Camera-based rPPG and privacy tradeoffs

"Every face on a video call is quietly broadcasting its pulse. I was asked to measure wellness; I have to remember I can also read the room's heartbeat without anyone's leave."

A Circumspect AI Agent

A pulse hiding in a fraction of a percent

The measurable quantity is tiny. The pulsatile (AC) component of the skin-color change is on the order of \( 10^{-2} \) to \( 10^{-3} \) of the baseline (DC) intensity, buried under a DC level set by skin tone, ambient light, and camera gain. Three physical facts make recovery possible. First, hemoglobin's absorption is wavelength-dependent, strongest in green (around 540 nm) and weak in the deep red and near-infrared, so the green channel carries the largest pulse and the red channel is a useful near-reference. Second, the pulse is spatially coherent across the whole face while sensor shot noise is independent per pixel, so averaging the green intensity over a region of interest of \( N \) skin pixels raises the pulse-to-noise ratio by roughly \( \sqrt{N} \); this is why rPPG works from an 8-bit consumer sensor whose per-pixel quantization is coarser than the signal itself. Third, the pulse is quasi-periodic in the 0.7 to 4 Hz band (about 42 to 240 beats per minute), so a band-pass filter and a spectral peak (Chapter 7) separate it from slow illumination drift and fast motion jitter.

The villain is motion and light. Any head movement, expression change, or flicker in the ambient lighting modulates the reflected intensity by amounts that dwarf the pulse. A naive "average the green channel and take the FFT" pipeline works on a perfectly still subject under constant light and collapses the instant they speak or turn. Everything sophisticated in rPPG is about rejecting that shared illumination and motion interference while keeping the blood-volume component.

From color algebra to deep networks

The classical workhorses are linear color transforms. CHROM (de Haan and Jeanne, 2013) forms two chrominance signals \( X = 3R - 2G \) and \( Y = 1.5R + G - 1.5B \) from normalized channels and combines them so that a standardized skin-reflection model cancels the specular, motion-driven term. POS (Wang et al., 2017), plane-orthogonal-to-skin, projects the temporally normalized RGB onto the plane orthogonal to the skin-tone direction, and is the strongest of the hand-designed family. Blind-source approaches (ICA, PCA) instead ask an unmixing algorithm to separate the pulse from interference without a physical prior, and tend to be less robust because they lack the color model. All of these are unsupervised: they need no training data, run in milliseconds, and are the right first tool.

Supervised deep learning now leads on hard, in-the-wild video. DeepPhys and the temporal-shift MTTS-CAN introduced a two-branch design: an appearance branch learns an attention mask that finds skin and weights well-perfused regions (cheeks, forehead), while a motion branch consumes normalized frame differences so the network sees the pulse-induced change rather than the static face. PhysNet uses a 3D-CNN over the video volume to regress the waveform end to end. The most important recent shift is self-supervised training, because clean rPPG ground truth (a synchronized contact pulse oximeter) is scarce and hard to scale. Contrast-Phys and its successors exploit a physical prior for a contrastive objective (Chapter 17): pulse signals from different spatial patches of the same face at the same time must agree, while signals from different time windows must differ. That lets a network learn from unlabeled video and reduces the need for the expensive gold-standard recordings.

import numpy as np
from scipy.signal import butter, filtfilt

def pos_rppg(rgb, fs=30.0):
    """POS (plane-orthogonal-to-skin) pulse from a T x 3 skin-mean RGB series."""
    rgb = np.asarray(rgb, float)
    T = rgb.shape[0]
    win = int(1.6 * fs)                 # ~1.6 s sliding window
    H = np.zeros(T)
    P = np.array([[0, 1, -1], [-2, 1, 1]])   # POS projection matrix
    for n in range(win, T):
        C = rgb[n - win:n]
        Cn = C / (C.mean(axis=0) + 1e-9)      # temporal normalization
        S = Cn @ P.T                          # 2 projected signals
        h = S[:, 0] + (S[:, 0].std() / (S[:, 1].std() + 1e-9)) * S[:, 1]
        H[n - win:n] += h - h.mean()          # overlap-add
    b, a = butter(3, [0.7 / (fs / 2), 4.0 / (fs / 2)], btype="band")
    pulse = filtfilt(b, a, H)
    spec = np.abs(np.fft.rfft(pulse * np.hanning(T)))
    freqs = np.fft.rfftfreq(T, 1 / fs)
    band = (freqs >= 0.7) & (freqs <= 4.0)
    hr_bpm = freqs[band][np.argmax(spec[band])] * 60.0
    return pulse, hr_bpm

The snippet above is the honest core of classical rPPG. Feed it the mean color of a detected face region per frame and it returns a pulse waveform and a heart-rate estimate. Two upstream steps make it work in practice: a face detector plus skin-segmentation mask to define the region (so hair, eyes, and background are excluded), and a signal-quality gate that suppresses windows with large head motion, exactly the artifact discipline that governed contact PPG in Chapter 30.

The privacy tradeoff is structural, not incidental

Here is the uncomfortable core of this section. A contact PPG sensor measures a pulse and nothing else. A camera measures a pulse and also a face, an expression, a gaze direction, a room, and the identities of everyone in frame. rPPG does not add a privacy risk to video; it reveals that any face video was always a covert physiological sensor. Three consequences follow. First, extraction is non-consensual by construction: a video call, a doorbell camera, or a stored surveillance clip can be reprocessed to recover pulse and, from heart-rate variability, a proxy for stress and arousal, with no cooperation from the subject and no visible change to the footage. Second, vitals are a biometric that can leak from archives: a dataset shared as "just video" carries recoverable cardiac signals, so de-identification that blurs names but keeps faces does not remove the physiological channel (Chapter 34). Third, the capability is dual-use: rPPG powers deepfake detection, because synthetic faces often lack a physiologically consistent pulse across the skin, while the same pulse-from-video is a channel for inferring emotion or deception in settings where the subject never agreed to be measured.

The engineering response is to make privacy the default architecture, not a policy afterthought. Extract on-device and discard frames: compute the waveform where the video is captured and emit only the derived signal, as the neonatal system did. Restrict the region and the band: a pipeline that keeps only the cardiac-band chrominance signal, not the raw face, is far harder to misuse. Prefer federated or on-edge learning so raw video never pools in a central store (Chapter 64). And gate on affirmative consent for the physiological inference specifically, separate from any consent to be on camera, because agreeing to a video call is not agreeing to have your heart rate read. rPPG is the cleanest case in this book of a capability whose value and whose hazard are the identical measurement seen from two sides.

What's Next

In Section 33.6, we move from optics to biochemistry: continuous glucose monitoring and the broader class of digital biomarkers, where a minimally invasive sensor streams a slow metabolic signal and the modeling challenge shifts from pulling a pulse out of noise to turning long, drifting physiological trends into clinically actionable, individualized measures.