Paperdoi:10.1145/3842436.3843803
When not to diagnose: evaluating evidence-sufficiency recognition in ECG multimodal language models with ECGTrustBench (HumanSys '26)
HumanSys '26, pp. 83–88; issue date 2026-10-26
Licence: CC BY 4.0
TL;DR
ECGTrustBench frames evidence-sufficiency recognition as answerable-vs-insufficient QA over 3,300 image–question pairs. Corruptions (missing leads, calibration loss, photographic degradation) were unanimously approved by three licensed clinicians. Across ECG-specific, medical, general and proprietary MLLMs, ECG-specific models reach up to 77.3 % clean accuracy but abstain on at most 10 % of evidence-deficient cases, while proprietary models reach up to 82 % recognition. Models react more to visible degradation than to missing diagnostic evidence.
- Form factor: clinical (12-lead images, mobile capture)
Abstract
Mobile electrocardiogram (ECG) capture routes photographed or scanned 12-lead images to an AI model or a clinician, yet missing leads, calibration loss, or photographic degradation can remove diagnosis-critical evidence while the tracing remains visually readable. A condition-aware evidence-sufficiency check is therefore needed to flag unusable captures for recapture or clinician review before diagnosis. Existing ECG multimodal large language model (MLLM) evaluations cannot support this, as they assume every question is answerable. We introduce ECGTrustBench, which evaluates evidence-sufficiency recognition as answerable-versus-insufficient QA over 3,300 image-question pairs, with corruption criteria unanimously approved by three licensed clinicians. Across ECG-specific, medical-domain, general-domain, and proprietary MLLMs, we find a consistent gap between answering and abstaining: ECG-specific models reach up to 77.3% clean accuracy yet abstain on at most 10% of evidence-deficient cases, while proprietary models achieve recognition up to 82%. Overall, current models respond more to visible degradation than to missing diagnostic evidence, highlighting a critical gap for evidence-sufficiency gating.
Verbatim from Semantic Scholar (opens in a new tab) · Abstract via Semantic Scholar; © the authors/publisher
Results on ChatPPG leaderboards
All leaderboardsNo results from this paper on ChatPPG leaderboards yet. Browse the leaderboards.
Tasks & signals
Signals
Keywords
How it works
Cite this paper
APA
Huang, Q., Jiang, J., Syed, F. F., Riaz, A. H., Chen, Y., & Nie, J. (2026). When not to diagnose: evaluating evidence-sufficiency recognition in ECG multimodal language models with ECGTrustBench (HumanSys '26). HumanSys '26, pp. 83–88. https://doi.org/10.1145/3842436.3843803
BibTeX
@article{huang2026when,
title = {{When not to diagnose: evaluating evidence-sufficiency recognition in ECG multimodal language models with ECGTrustBench (HumanSys '26)}},
author = {Huang, Qijia and Jiang, Jingxia and Syed, Faisal F. and Riaz, Abdul Haseeb and Chen, Yiran and Nie, Jingping},
year = {2026},
journal = {HumanSys '26, pp. 83–88},
doi = {10.1145/3842436.3843803}
}Built from the hub record. Check it against the publisher before submitting.
Added to the ChatPPG research hub
