Transformer Models Applied to PPG Analysis
Transformer models use multi-head self-attention mechanisms to model long-range dependencies in sequential data without recurrence, achieving state-of-the-art performance in PPG tasks including heart rate estimation, AF detection, and waveform reconstruction by attending to relevant temporal context across the entire signal window simultaneously.
The self-attention mechanism computes attention weights between all pairs of positions in the input sequence: Attention(Q,K,V) = softmax(QK^T / √d_k)V, where Q, K, V are query, key, and value matrices derived from linear projections of the input. Multi-head attention runs this operation in parallel across H attention heads with different projection matrices, capturing complementary aspects of temporal relationships (e.g., one head may attend to beat-to-beat periodicity, another to morphological similarity).
For PPG, positional encodings must capture the quasi-periodic cardiac rhythm structure. Standard sinusoidal positional encoding (as in the original Transformer) works for fixed-length windows. Learnable positional embeddings trained on large PPG corpora better capture cardiac-specific temporal patterns. Beat-aware positional encodings that index by beat number rather than sample number are emerging as more physiologically appropriate for rhythm analysis.
PPG-Transformer (Transformer adapted for wrist PPG) trained on 10 million segments from Apple Heart Study data demonstrated transfer learning superiority: fine-tuning on downstream tasks (AF detection, sleep staging, stress detection) with 100× fewer labeled examples achieved performance within 2–5% AUC of fully supervised CNNs/LSTMs. This sample efficiency advantage is transformers' key value proposition in medical AI where labeled data is scarce and expensive.
Frequently Asked Questions
Do transformers outperform LSTMs for all PPG tasks?
Not universally. Transformers dominate on large datasets and transfer learning. LSTMs with appropriate architecture can match transformer performance on small-to-medium datasets while requiring significantly less compute. For edge deployment, LSTMs remain more practical.
What is the computational cost of transformers for real-time PPG?
Standard transformers with sequence length N have O(N²) attention complexity. For 30-second PPG at 100 Hz (N=3000), this is computationally prohibitive for edge devices. Linear attention approximations (Linformer, Performer) reduce complexity to O(N).
How are transformers pre-trained on PPG data?
Common pre-training objectives include masked PPG segment reconstruction (analogous to BERT), future segment prediction, and contrastive learning between augmented views of the same recording. Self-supervised pre-training on large unlabeled wearable datasets has shown strong transfer to cardiovascular classification tasks.