IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text

Summary

IMU2CLIP (Meta AI / UC Berkeley) adapts the CLIP contrastive multimodal alignment framework to IMU motion sensors. The core idea: align wrist-worn IMU time series with (1) first-person egocentric video frames and (2) natural language narrations describing the activity, using the Ego4D dataset which provides all three modalities synchronized.