Paperdoi:10.1145/3432701
MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices
Proc. ACM IMWUT 4(4), Article 168, December 2020
TL;DR
A two-stage logging pipeline on the MM-Fit dataset (21 sessions, >800 minutes, 10 exercises + non-exercise, 616 sets / 6,160 reps): CNN autoencoders + a joint segmentation/recognition classifier (non-exercise is an 11th class), then a peak-counting / autocorrelation rep counter on already-segmented sets. Devices include two TicWatch Pro watches at 100 Hz accel+gyro (and 1 Hz HR), an earbud, phones, and RGB-D pose. Start/end annotations are described as typically within 2–3 frames (60–100 ms).
- Form factor: wrist
Abstract
Fitness tracking devices have risen in popularity in recent years, but limitations in terms of their accuracy and failure to track many common exercises presents a need for improved fitness tracking solutions. This work proposes a multimodal deep learning approach to leverage multiple data sources for robust and accurate activity segmentation, exercise recognition and repetition counting. For this, we introduce the MM-Fit dataset; a substantial collection of inertial sensor data from smartphones, smartwatches and earbuds worn by participants while performing full-body workouts, and time-synchronised multi-viewpoint RGB-D video, with 2D and 3D pose estimates. We establish a strong baseline for activity segmentation and exercise recognition on the MM-Fit dataset, and demonstrate the effectiveness of our CNN-based architecture at extracting modality-specific spatial temporal features from inertial sensor and skeleton sequence data. We compare the performance of unimodal and multimodal models for activity recognition across a number of sensing devices and modalities. Furthermore, we demonstrate the effectiveness of multimodal deep learning at learning cross-modal representations for activity recognition, which achieves 96% accuracy across all sensing modalities on unseen subjects in the MM-Fit dataset; 94% using data from the smartwatch only; 85% from the smartphone only; and 82% on data from the earbud device. We strengthen single-device performance by using the zeroing-out training strategy, which phases out the other sensing modalities. Finally, we implement and evaluate a strong repetition counting baseline on our MM-Fit dataset. Collectively, these tasks contribute to recognising, segmenting and timing exercise and non-exercise activities for automatic exercise logging.
Verbatim from the publisher's DOI record (opens in a new tab) · Publisher licence: https://www.acm.org/publications/policies/copyright_policy#Background
Results on ChatPPG leaderboards
All leaderboardsNo results from this paper on ChatPPG leaderboards yet. Browse the leaderboards.
Tasks & signals
Tasks
Signals
Keywords
How it works
Cite this paper
Strömbäck, D., Huang, S., & Radu, V. (2020). MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices. Proc. ACM IMWUT 4, , December 2020, 168. https://doi.org/10.1145/3432701
Strömbäck, David, Sangxia Huang, and Valentin Radu. 2020. "MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices." Proc. ACM IMWUT 4, , December 2020, 168. https://doi.org/10.1145/3432701.
Strömbäck, D., Huang, S. and Radu, V. (2020) 'MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices', Proc. ACM IMWUT 4, , December 2020, p. 168. doi: 10.1145/3432701.
Strömbäck, David, et al. "MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices." Proc. ACM IMWUT 4, , December 2020, 2020, p. 168. https://doi.org/10.1145/3432701.
D. Strömbäck, S. Huang, and V. Radu, "MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices," Proc. ACM IMWUT 4, , December 2020, p. 168, 2020, doi: 10.1145/3432701.
@inproceedings{stromback2020mmfit,
title = {{MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices}},
author = {Strömbäck, David and Huang, Sangxia and Radu, Valentin},
year = {2020},
booktitle = {Proc. ACM IMWUT 4, , December 2020},
pages = {168},
doi = {10.1145/3432701}
}Added to the ChatPPG research hub