ProtoMM: Shared Prototypes for a Multimodal Pulse-Motion Foundation Model

Wanting Mao, Maxwell A. Xu (co-first), Harish Haresamudram, Mithun Saha, Santosh Kumar, James M. Rehg

Summary

Multimodal SSL that anchors PPG and accelerometry embeddings to a shared prototype dictionary with swapped Multimodal Prototype Prediction (within- + between-modality), avoiding CLIP-style hard negatives that discard within-modality structure. Pretrains a Pulse-Motion FM on in-the-wild MOODS wrist PPG+ACC; frozen probes lead most tasks vs strong multimodal baselines.