Multispectral Imaging for rPPG: Beyond RGB Cameras

Standard RGB rPPG uses what your eyes can see — red, green, and blue light reflected off skin. This works, but it's leaving significant information on the table. Hemoglobin, oxyhemoglobin, deoxyhemoglobin, melanin, and water all have distinct spectral signatures that extend well beyond the visible range. A camera that can see further into the electromagnetic spectrum sees a richer physiological picture.

Multispectral imaging for rPPG expands the wavelength range from the standard 400-700 nm visible band into the near-infrared (NIR: 700-1400 nm) and shortwave infrared (SWIR: 1400-3000 nm). The payoffs are substantial: skin-tone independence, SpO2 measurement potential, and new vascular biomarkers that RGB cameras simply can't access.

What Makes NIR rPPG Different

Near-infrared light penetrates skin significantly deeper than visible light. At 850 nm, photons travel 2-4 mm into tissue before being absorbed — deep enough to reach the dermal and subdermal vasculature that contact PPG sensors also interrogate. At green wavelengths (525-550 nm), penetration depth is typically under 1 mm.

This deeper penetration has two key advantages:

Greater blood volume signal: Larger vessels contribute more to the NIR signal than the superficial capillary network that green light primarily samples. The pulse amplitude tends to be larger relative to DC in NIR, improving SNR in some conditions.

Skin-tone independence: Melanin in the epidermis strongly absorbs visible light, especially in the blue-green range. This is why darker skin tones degrade rPPG accuracy with standard cameras. Melanin absorption drops sharply above 700 nm. NIR rPPG sees roughly equal signal quality across all Fitzpatrick skin types.

A 2023 study by Sun et al. (DOI: 10.1109/TBME.2023.3245897) demonstrated that NIR rPPG at 850 nm showed no statistically significant accuracy difference across Fitzpatrick types I-VI, while the same cohort measured with green-channel RGB rPPG showed 2.4x higher MAE for types V-VI compared to types I-II.

SpO2 Estimation from Multispectral Video

Clinical pulse oximetry uses two wavelengths: 660 nm red (where oxyhemoglobin and deoxyhemoglobin have very different absorption) and 940 nm NIR (where the difference is smaller but serves as a reference). The ratio of these AC/DC components at both wavelengths — the "ratio of ratios" — correlates with oxygen saturation.

Standard RGB cameras have red channels centered around 610-640 nm and lack a 940 nm channel. This is why RGB-only rPPG cannot accurately measure SpO2.

Add a second NIR band (940 or 950 nm) alongside a red band (660 nm), and in principle you have the spectral information needed for ratiometric SpO2 estimation. Several research groups have demonstrated this:

Kumar et al. (2015, DOI: 10.1145/2733373.2806332) used a modified DSLR with IR pass filters to measure SpO2 contactlessly, achieving ±2% accuracy in controlled conditions. Shao et al. (2020, DOI: 10.1109/TBME.2020.3023951) achieved similar results with a dual-channel NIR camera system.

The challenge is calibration. The relationship between pixel ratios and SpO2 depends on tissue optical properties, illumination spectrum, camera spectral response, and vascular anatomy — all of which vary between individuals and conditions. Unlike contact pulse oximetry where the probe geometry is controlled, rPPG SpO2 requires per-subject or per-session calibration to approach clinical accuracy thresholds.

Hyperspectral Approaches and Tissue Chromophore Mapping

Beyond two or three bands, full hyperspectral imaging (10-100+ narrow bands across visible and NIR) enables tissue chromophore mapping — separating the contributions of oxyhemoglobin, deoxyhemoglobin, melanin, collagen, and other tissue components.

The Specim IQ and similar push-broom hyperspectral cameras can capture 200+ spectral bands per pixel. Applied to rPPG, hyperspectral video enables not just heart rate, but:

• Regional blood oxygenation maps across the face
• Microcirculation perfusion assessment
• Venous versus arterial pulsation separation (different depth, different spectral signature)
• Tissue hydration indices from NIR water absorption bands

Kamshilin et al. (2022, DOI: 10.1038/s41598-022-15232-0) demonstrated hyperspectral rPPG-based mapping of regional skin perfusion with spatial resolution sufficient to detect perfusion asymmetries indicating vascular pathology.

The clinical potential is significant, but current hyperspectral cameras are expensive ($20,000-$100,000+), large, and require specialized illumination. Consumer-grade hyperspectral cameras for mobile devices remain a research goal rather than a current product.

SWIR Imaging: Water and Deep Tissue

Shortwave infrared cameras (1000-2500 nm) operate in a wavelength range where silicon detectors no longer work — InGaAs sensors are required, which are expensive. Water has strong absorption bands in SWIR, making it sensitive to tissue hydration. Collagen has characteristic SWIR signatures.

rPPG applications in SWIR are limited but interesting. At 1300-1400 nm, the AC component of the blood volume pulse can be detected with minimal melanin interference and good penetration depth. SWIR-based rPPG has been demonstrated in preclinical animal models and a small number of human studies.

The practical barrier is cost: InGaAs cameras for SWIR currently cost $5,000-$50,000. As III-V semiconductor manufacturing scales, SWIR sensors for smartphones are under development by several companies including Imec, Viavi Solutions, and sensor startups.

Practical Multispectral Systems Today

A few commercially accessible approaches to multispectral rPPG exist today:

Modified DSLR/mirrorless with IR filter removal: Removing the IR cut filter from a standard camera sensor gives access to wavelengths up to ~1000 nm. Adding bandpass filters allows selective NIR measurement. Cost: $200-$1000 for camera modification, plus filter costs. Used extensively in research.

Dedicated NIR cameras: Industrial NIR cameras (Silicon Software, Basler, FLIR) with 850-940 nm sensitivity are available for $500-$2000. Combined with controlled NIR LED illumination, these enable skin-tone-agnostic rPPG in research and clinical settings.

Smartphone NIR sensors: Modern Face ID systems (iPhone, some Android) use NIR dot projectors and sensors already present in the device. Research groups have demonstrated rPPG extraction from these sensors, though API access varies by platform.

Thermal infrared (8-12 µm): Thermal cameras detect heat, not reflected light. Facial skin temperature changes slightly with each heartbeat as blood flows through vessels. Thermal rPPG has been demonstrated but typically shows lower temporal resolution and higher sensitivity to ambient temperature changes compared to NIR approaches.

The Path to Consumer Multispectral rPPG

Multispectral imaging for rPPG is transitioning from research curiosity to engineering roadmap. Several trends converge:

1. Apple and Samsung have filed patents for multispectral measurement using Face ID illuminators and additional sensor channels on future iPhones and Galaxy devices
2. CMOS process improvements are bringing NIR sensitivity to standard smartphone camera stacks without dedicated InGaAs sensors
3. Regulatory precedent: FDA 510(k) clearances for contact multiwavelength oximetry provide a pathway for contactless equivalents once clinical accuracy thresholds are met

In 5-10 years, multispectral rPPG capable of SpO2 estimation within clinical tolerance is a reasonable expectation for flagship smartphones. The physics is solved. The manufacturing cost curve is moving in the right direction.

Related Resources

• rPPG Comprehensive Guide 2026 — overview of rPPG technology
• PPG LED Wavelength Selection — wavelength physics for contact PPG
• PPG SpO2 Accuracy Limitations — SpO2 measurement challenges
• PPG Tissue Optical Properties — light-tissue interaction fundamentals
• PPG Skin Tone Bias Accuracy — equity issues in optical sensing

Frequently Asked Questions

What is multispectral rPPG? Multispectral rPPG uses cameras sensitive to multiple wavelength bands — typically extending into near-infrared (700-1400 nm) beyond standard RGB — to measure blood volume pulsation. This enables more accurate heart rate measurement across all skin tones and opens the possibility of SpO2 estimation.

Can multispectral cameras measure blood oxygen from a distance? In controlled research conditions, yes. Systems using 660 nm and 940 nm bands with appropriate illumination have demonstrated SpO2 accuracy of approximately ±2%. This doesn't yet meet clinical standards (±1.5% for pulse oximetry), but the gap is narrowing.

Why does NIR rPPG work better on dark skin tones? Melanin strongly absorbs visible light, especially in the green/blue range, reducing rPPG signal amplitude in darker skin tones. NIR light above 700 nm is minimally absorbed by melanin, so NIR rPPG sees equal signal strength regardless of skin pigmentation.

Are multispectral cameras available in smartphones? Current flagship smartphones include NIR emitters and sensors for Face ID, but these aren't typically exposed as camera inputs. Research groups have accessed them via modified APIs. Consumer multispectral rPPG apps using these sensors are not yet mainstream.

How does thermal imaging relate to rPPG? Thermal infrared cameras detect heat emission from skin rather than reflected light. Small temperature changes accompanying each heartbeat create a measurable thermal signal. Thermal rPPG exists but is less accurate than NIR rPPG and more sensitive to environmental temperature changes.