Paperdoi:10.1145/3432701

MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices

Venue listed2020Added Nov 12, 2026

  • DS
  • SH
  • VR

Authors: David Strömbäck, Sangxia Huang, Valentin Radu

Proc. ACM IMWUT 4(4), Article 168, December 2020

TL;DR

A two-stage logging pipeline on the MM-Fit dataset (21 sessions, >800 minutes, 10 exercises + non-exercise, 616 sets / 6,160 reps): CNN autoencoders + a joint segmentation/recognition classifier (non-exercise is an 11th class), then a peak-counting / autocorrelation rep counter on already-segmented sets. Devices include two TicWatch Pro watches at 100 Hz accel+gyro (and 1 Hz HR), an earbud, phones, and RGB-D pose. Start/end annotations are described as typically within 2–3 frames (60–100 ms).

  • Form factor: wrist

Abstract

Fitness tracking devices have risen in popularity in recent years, but limitations in terms of their accuracy and failure to track many common exercises presents a need for improved fitness tracking solutions. This work proposes a multimodal deep learning approach to leverage multiple data sources for robust and accurate activity segmentation, exercise recognition and repetition counting. For this, we introduce the MM-Fit dataset; a substantial collection of inertial sensor data from smartphones, smartwatches and earbuds worn by participants while performing full-body workouts, and time-synchronised multi-viewpoint RGB-D video, with 2D and 3D pose estimates. We establish a strong baseline for activity segmentation and exercise recognition on the MM-Fit dataset, and demonstrate the effectiveness of our CNN-based architecture at extracting modality-specific spatial temporal features from inertial sensor and skeleton sequence data. We compare the performance of unimodal and multimodal models for activity recognition across a number of sensing devices and modalities. Furthermore, we demonstrate the effectiveness of multimodal deep learning at learning cross-modal representations for activity recognition, which achieves 96% accuracy across all sensing modalities on unseen subjects in the MM-Fit dataset; 94% using data from the smartwatch only; 85% from the smartphone only; and 82% on data from the earbud device. We strengthen single-device performance by using the zeroing-out training strategy, which phases out the other sensing modalities. Finally, we implement and evaluate a strong repetition counting baseline on our MM-Fit dataset. Collectively, these tasks contribute to recognising, segmenting and timing exercise and non-exercise activities for automatic exercise logging.

Verbatim from the publisher's DOI record (opens in a new tab) · Publisher licence: https://www.acm.org/publications/policies/copyright_policy#Background

Results on ChatPPG leaderboards

All leaderboards

No results from this paper on ChatPPG leaderboards yet. Browse the leaderboards.

Tasks & signals

Tasks

Activity recognition

Signals

Keywords

HarExercise OnsetExercise RecognitionRep CountingMultimodalImwut 2020

How it works

How Method works: Setup → Method → TargetSetup: Signals (accel, gyro, hr); Form factor (wrist). Method: Method. Target: Activity recognition.SetupSignals: accel, gyro, hrxyzSignalsaccel, gyro, hrForm factor: wristForm factorwristMethodMethodTargetActivity recognitionActivity recognition

Cite this paper

Cite this paper with the fields recorded here. Check it against the publisher before submitting.

Strömbäck, D., Huang, S., & Radu, V. (2020). MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices. Proc. ACM IMWUT 4, , December 2020, 168. https://doi.org/10.1145/3432701
Strömbäck, David, Sangxia Huang, and Valentin Radu. 2020. "MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices." Proc. ACM IMWUT 4, , December 2020, 168. https://doi.org/10.1145/3432701.
Strömbäck, D., Huang, S. and Radu, V. (2020) 'MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices', Proc. ACM IMWUT 4, , December 2020, p. 168. doi: 10.1145/3432701.
Strömbäck, David, et al. "MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices." Proc. ACM IMWUT 4, , December 2020, 2020, p. 168. https://doi.org/10.1145/3432701.
D. Strömbäck, S. Huang, and V. Radu, "MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices," Proc. ACM IMWUT 4, , December 2020, p. 168, 2020, doi: 10.1145/3432701.
@inproceedings{stromback2020mmfit,
  title         = {{MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices}},
  author        = {Strömbäck, David and Huang, Sangxia and Radu, Valentin},
  year          = {2020},
  booktitle     = {Proc. ACM IMWUT 4, , December 2020},
  pages         = {168},
  doi           = {10.1145/3432701}
}

Added to the ChatPPG research hub