PaperarXiv:2603.14326

ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

Venue listed2026Added Oct 30, 2026

Authors: Jungwoo Oh, Hyunseung Chung, Junhee Lee, Min-Gyu Kim, Hangyul Yoon, Ki Seong Lee, Youngchae Lee, Muhan Yeo, Edward Choi (arXiv HTML v2)

NeurIPS 2026 Evaluations & Datasets Track (accepted per arXiv v2 comment); arXiv 2603.14326v2 (2026-10-28; v1 2026-03-15); arXiv licence CC BY 4.0

PDF arXiv2603.14326 Cite

Open code

Oh, Jungwoo et al., "ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation", arXiv:2603.14326, CC BY 4.0

TL;DR

Construction: an automated ECG analysis pipeline does wave detection and segmentation, then feature extraction, then rule-based diagnosis derivation. A sample is kept only when its source label agrees with the pipeline's diagnosis. The result is more than 6,400 samples across 17 core ECG diagnoses: 3,084 from PTB-XL and 3,359 from MIMIC-IV-ECG. Three board-certified internists validated 147 representative reasoning paths. Evaluation: multi-turn dialogue through Criterion Selection, then Finding Identification, then ECG Grounding (segment and measurement), then Diagnostic Decision.

Form factor: clinical-12-lead

Abstract

While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce ECG-Reasoning-Benchmark, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step- by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a systematic shortfall in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit consistently low success rates in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, highlighting a limitation in their visual grounding capabilities and motivating robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.

Verbatim from arXiv (opens in a new tab) · arXiv metadata (abstracts are released under CC0)

Results on ChatPPG leaderboards

All leaderboards

No results from this paper on ChatPPG leaderboards yet. Browse the leaderboards.

Tasks & signals

Signals

ECG

Keywords

MLLMBenchmarkClinical ReasoningGroundingNeurIPS 2026Datasets TrackMIT

Datasets used

How it works

Cite this paper

Cite this paper with the fields recorded here. Check it against the publisher before submitting.

Oh, J., Chung, H., Lee, J., Kim, M.-G., Yoon, H., Lee, K. S., Lee, Y., Yeo, M., & Choi, E. (2026). ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation. NeurIPS 2026 Evaluations & Datasets Track. https://arxiv.org/abs/2603.14326

Oh, Jungwoo, Hyunseung Chung, Junhee Lee, Min-Gyu Kim, Hangyul Yoon, Ki Seong Lee, Youngchae Lee, Muhan Yeo, and Edward Choi. 2026. "ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation." NeurIPS 2026 Evaluations & Datasets Track. https://arxiv.org/abs/2603.14326.

Oh, J., Chung, H., Lee, J., Kim, M.-G., Yoon, H., Lee, K. S., Lee, Y., Yeo, M. and Choi, E. (2026) 'ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation', NeurIPS 2026 Evaluations & Datasets Track. Available at: https://arxiv.org/abs/2603.14326

Oh, Jungwoo, et al. "ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation." NeurIPS 2026 Evaluations & Datasets Track, 2026. https://arxiv.org/abs/2603.14326.

J. Oh et al., "ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation," NeurIPS 2026 Evaluations & Datasets Track, 2026, [Online]. Available: https://arxiv.org/abs/2603.14326.

@inproceedings{oh2026ecgreasoningbenchmark,
  title         = {{ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation}},
  author        = {Oh, Jungwoo and Chung, Hyunseung and Lee, Junhee and Kim, Min-Gyu and Yoon, Hangyul and Lee, Ki Seong and Lee, Youngchae and Yeo, Muhan and Choi, Edward},
  year          = {2026},
  booktitle     = {NeurIPS 2026 Evaluations \& Datasets Track},
  eprint        = {2603.14326},
  archivePrefix = {arXiv}
}

Oct 30, 2026Added to the ChatPPG research hub