Conformal Risk Control and Evidential Uncertainty Quantification for Detecting Diagnostic Inconsistency in High Stakes Multimodal Chest Radiograph Interpretation

Abstract

Medical vision language models deployed in high-stakes clinical diagnostic workflows must provide rigorous, calibrated uncertainty guarantees to alert attending radiologists when predictions are vulnerable to failure. A particularly dangerous vulnerability occurs when models express high overconfidence while producing contradictory diagnoses under semantically equivalent query rephrasings. Standard softmax probabilities fail to signal this fragility, creating severe hazards for autonomous clinical triage. To establish provable safety bounds, this paper introduces a distribution-free uncertainty quantification framework uniting Conformal Risk Control with evidential deep learning for medical vision language models. Our framework places subjective Dirichlet distributions over model logits to decompose predictive uncertainty into vacuity reflecting lack of visual evidence and dissonance reflecting internal prompt-induced contradiction. We then formulate a finite-sample conformal prediction algorithm that guarantees, with user-specified confidence 1-alpha, that the true diagnosis is contained within the returned prediction set, dynamically expanding to an abstain-and-defer signal whenever paraphrase instability is detected. Evaluated across 15,000 thoracic radiography cases spanning four hospital datasets, the proposed framework achieves exact finite-sample coverage at 95% and 99% confidence levels while identifying 88.7% of impending paraphrase-induced diagnostic flips prior to output generation. Furthermore, the conformal deferral policy maintains optimal triage efficiency by routing only genuinely ambiguous cases to expert radiologist review. This mathematical approach bridges empirical vision-language modeling and rigorous statistical risk guarantees, providing an indispensable safety layer for clinical AI deployments.