ChatPPG.
Paper

PRISM: Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis

Hamza Shafiq, Jialu Tang, Ting Dang, Jun Hu, Aaqib Saeed
arXiv preprint2026ECGEHR structuredText
arXiv Cite
PreprintFirst posted: Oct 9, 2026Added to atlas: Oct 12, 2026Source
Citations—Not available
GitHub stars—Not available
HF downloads—Not available

Activity counts measure attention, not model quality. Missing counts are unknown. Publication status follows the linked source; a listed venue alone does not confirm peer review.

PRISM is a multimodal language model for clinical prognosis that encodes numeric features as tokens, weights recent ECGs more heavily, and retrieves averaged embeddings of similar past patients. No individual retrieved record goes into the prompt. On the 1,443-task MDS-ED benchmark it reached AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission and 92.8% for mortality.

  • Result: On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality.
  • Result: With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED.
At a glance
Venue
arXiv preprint
Published
2026
DOI
Not reported
arXiv
2610.13510 (opens in a new tab)
PMID
Not reported
Licence
Not reported
Signals
ECGEHR structuredText
Form factor
Not reported
Subjects
Not reported
Code
Not reported
Added
Oct 12, 2026

Abstract

Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context. We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank. Retrieval operates in representation space: the model receives mean-pooled embeddings of similar training-cohort encounters, while no individual retrieved record enters the prompt. On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality. These results use a 96-hour pre-arrival window unavailable to reported baselines. With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED. A matched-input compact fusion model using the same five evidence streams and frozen encoders reaches 0.7652 AUROC, versus 0.9092 for PRISM. Replacing numerical EHR encoding with text serialization reduces AUROC by 3.66 points; removing pre-arrival history reduces it by 1.97 points; and zeroing retrieval reduces it by 0.99 points. Genuine retrieval outperforms random-neighbor retrieval by 0.97 macro-average points, although a paired task-level sign test does not establish a uniform advantage (p=0.067). On the external MC-MED cohort, supervised fine-tuning reaches 0.8611 AUROC, compared with 0.8227 for locally trained gradient boosting and 0.7841 for source-only PRISM. These results support structure-aware multimodal integration and supervised external adaptation.

Verbatim fromarXiv· arXiv metadata (abstracts are released under CC0)

PRISM: Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis: ChatPPG research summary card

How it works

How Method works: Setup → Method → ResultSetup: ECG (Electrocardiogram); EHR structured; Text (Language). Method: Method (Language-aligned). Result: Result (On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality.); Result (With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED.).SetupECG: ElectrocardiogramECGElectrocardiogramEHR structuredEHR structuredText: LanguageTextLanguageMethodMethodLanguage-alignedResultResult: On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality.ResultOn the 1,443-task MDS-ED benchmark…Result: With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED.ResultWith benchmark-window inputs only…

Cite

@article{prismretrievalaugmentedm2026,
  title = {{PRISM: Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis}},
  author = {Hamza Shafiq and Jialu Tang and Ting Dang and Jun Hu and Aaqib Saeed},
  year = {2026},
  journal = {arXiv preprint},
  eprint = {2610.13510}
}

History

  1. Added to the ChatPPG research hub

More on ChatPPG