PaperarXiv:2501.10885
CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attention
arXiv preprint
TL;DR
CEReBrO is a small EEG foundation model with per-channel patch tokens and alternating intra-channel and inter-channel attention. The authors pretrain three sizes, 3.58M, 39.95M, and 85.15M parameters, on more than 20,000 hours of public TUEG scalp EEG. The abstract's rounded range is 3.6M to 85M parameters. This paper does not evaluate PPG.
- arXiv:2501.10885v4; DOI 10.48550/arXiv.2501.10885
- University of Zurich and ETH Zürich Integrated Systems Laboratory
- Pretraining: TUEG, over 20,000 hours, over 10,000 subjects
- Reported sizes: 3.58M, 39.95M, 85.15M parameters
- Alternating attention: up to 2x speed and 6x less memory than standard self-attention (authors' claim)
- No PPG experiments in this paper; the ECG/PPG blood-pressure transfer is a separate preprint
Abstract
Electroencephalograph (EEG) is a crucial tool for studying brain activity. Recently, self-supervised learning methods leveraging large unlabeled datasets have emerged as a potential solution to the scarcity of widely available annotated EEG data. However, current methods suffer from at least one of the following limitations: i) sub-optimal EEG signal modeling, ii) model sizes in the hundreds of millions of trainable parameters, and iii) reliance on private datasets and/or inconsistent public benchmarks, hindering reproducibility. To address these challenges, we introduce a Compact Encoder for Representations of Brain Oscillations using alternating attention (CEReBrO), a new small EEG foundation model. Our tokenization scheme represents EEG signals at a per-channel patch granularity. We propose an alternating attention mechanism that jointly models intra-channel temporal dynamics and inter-channel spatial correlations, achieving 2x speed improvement with 6x less memory required compared to standard self-attention. We present several model sizes ranging from 3.6 million to 85 million parameters. Pre-trained on over 20,000 hours of publicly available scalp EEG recordings with diverse channel configurations, our models set new benchmarks in emotion detection and seizure detection tasks, with competitive performance in anomaly classification and gait prediction. This validates our models' effectiveness and efficiency.
Verbatim from arXiv (opens in a new tab) · arXiv metadata (abstracts are released under CC0)
Results on ChatPPG leaderboards
All leaderboardsNo results from this paper on ChatPPG leaderboards yet. Browse the leaderboards.
Tasks & signals
Signals
Keywords
Models
How it works
Cite this paper
APA
Dimofte, A., Bucagu, G. A., Ingolfsson, T. M., Wang, X., Cossettini, A., Benini, L., & Li, Y. (2025). CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attention. arXiv preprint. https://doi.org/10.48550/arXiv.2501.10885
BibTeX
@misc{dimofte2025cerebro,
title = {{CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attention}},
author = {Dimofte, Alexandru and Bucagu, Glenn Anta and Ingolfsson, Thorir Mar and Wang, Xiaying and Cossettini, Andrea and Benini, Luca and Li, Yawei},
year = {2025},
howpublished = {arXiv preprint},
doi = {10.48550/arXiv.2501.10885},
eprint = {2501.10885},
archivePrefix = {arXiv}
}Built from the hub record. Check it against the publisher before submitting.
Added to the ChatPPG research hub
