CM-TCC (Cross-Modal Temporal and Contextual Contrasting; ECG + ACC encoders, Emognition)

The growing use of wearable devices capturing multimodal signals, such as electrocardiogram (ECG) and accelerometer (ACC) data, offers opportunities for creating large, real-life datasets that can power self-supervised learning (SSL) models that can support a variety of applications. Using ECG and ACC wearables data, we train SSL models with both unimodal and cross-modal pretraining objectives and evaluate their performance on downstream tasks such as physical activity recognition and affect classification. In each of our downstream experiments, we identify dominant and secondary modalities and demonstrate that cross-modal pretraining can enhance downstream performance even when only a single modality is available. Additionally, we find that modalities can complement or contradict each other in certain cases, highlighting the importance of incorporating domain knowledge when designing multimodal systems. We publicly release encoders pretrained via Cross-Modal Temporal and Contextual Contrasting (CM-TCC) for ECG and ACC signals to support further research on the complementarity of cross-modal representation learning.

Verbatim fromOpenAlex· Abstract via OpenAlex (publisher-deposited); © the authors/publisher