"You handed me nine months of wrist accelerometer data and four hundred labeled minutes. I have looked at every minute you gave me. I would like to look at the other three hundred eighty-nine thousand."
An Underspecified AI Agent
Prerequisites
This section assumes you understand supervised training of the neural encoders in Chapter 13 and Chapter 14, and that you have internalized the leakage-safe, subject-wise splitting discipline of Chapter 5. Basic probability and the notion of class imbalance from Chapter 4 are used lightly. Big-O reasoning and cross-validation are refreshed in Appendix B. You do not yet need any self-supervised machinery; that begins in the next section. This is the motivation section: it establishes precisely why the label is the bottleneck, so the rest of Chapter 17 has a problem worth solving.
The Big Picture
In vision and language the data and its labels arrive together: a photo already shows a cat, a sentence already carries meaning a person can read off directly. Sensing breaks that coupling. A stream of accelerometer counts, a raw ECG trace, a vibration spectrum, none of these announce what they mean. Collecting the raw signal is nearly free, because sensors run continuously and cost pennies, but attaching a trustworthy label to any window of it is expensive, slow, and sometimes impossible after the fact. The result is a permanent and lopsided ratio: oceans of unlabeled sensor data next to a thimble of labeled examples. This section quantifies that asymmetry, dissects the four distinct reasons labels go missing in physical sensing, and shows why the natural response is not to label harder but to learn from the raw signal itself. That pivot is the entire premise of self-supervised learning, and it is what the remaining sections of this chapter build.
The unlabeled ocean and the labeled drop
Start with the arithmetic, because it is the whole argument in one line. A single wearable sampling a six-axis inertial measurement unit at 50 Hz emits about 26 million samples per day per device. A fleet of ten thousand such devices emits over 250 billion samples a day, indefinitely, at essentially zero marginal cost. Now count the labels. Getting a reliable activity label for one second of that stream requires either a person watching synchronized video, a diary the user actually filled in, or a lab-grade reference rig, and a skilled annotator labels on the order of a few minutes of dense sensor data per working hour. The two rates are separated by many orders of magnitude, and the gap does not close with more money in any practical budget: the raw stream grows faster than any annotation workforce can ever catch up to. Unlabeled data is effectively infinite and free; labels are finite, slow, and dear. Every deployed sensing program lives inside that ratio whether it names it or not.
Contrast this with the domains that trained the field's intuitions. ImageNet labels were cheap because any literate person can name the object in a photo, so the work parallelizes across millions of crowd workers at seconds per image. Web text is self-labeling: the next word is its own supervision, harvested for free from documents that already exist. Sensor streams have neither property. A person cannot glance at a window of triaxial acceleration and say "brushing teeth" with any confidence, and the signal carries no built-in next-token target that corresponds to a task you care about. The cheap-label mechanisms that powered the deep-learning revolution simply do not fire here.
Key Insight
Label scarcity in sensing is not a temporary data-collection problem you can fund your way out of; it is structural. The signal is machine-native, not human-native, so a label almost always requires a second, more expensive measurement system (a reference instrument, synchronized video, or a clinical diagnosis) to exist at all. That means the cost of a label is set by the cost of the ground-truth apparatus, not by annotator wages, and no crowdsourcing platform can drive it toward zero. The abundance is on the unlabeled side, so the leverage must be too.
Why a sensor label is uniquely expensive to create
The core difficulty is that sensor signals are not self-interpretable. To label a window you need a source of truth outside the signal itself, and in physical sensing that source is usually an instrument. Sleep stages are labeled by polysomnography, a full night wired to EEG, EOG, EMG, and airflow channels scored by a trained technician, which is why a labeled night of wearable sleep data can cost hundreds of dollars and why large labeled sleep sets are rare, a theme picked up in Chapter 30. Energy expenditure is labeled against a metabolic cart or doubly-labeled water. Remaining useful life for a bearing is labeled only by running the machine until it actually fails, so each label is one destroyed asset and one run-to-failure experiment, as Chapter 36 details. In each case the label is a byproduct of a costly, sometimes destructive, measurement you cannot repeat cheaply and often cannot run retroactively on data you already collected.
Beyond raw cost, four distinct failure modes conspire to keep labels scarce, and it helps to name them separately because each demands a different response.
Rarity. The events that justify the whole system are usually the rarest. Falls, seizures, machine faults, arrhythmias, and cyber-physical intrusions occur in a vanishing fraction of operating time. If a fault appears in one window out of a hundred thousand, then a labeling budget spent on random windows buys almost no positive examples, and the labeled set is starved of exactly the class it exists to detect. Class imbalance here is not a nuisance measured in ratios of ten to one; it can be ten thousand to one.
Subjectivity. Many sensing labels are human judgments with real disagreement baked in. Two sleep technicians scoring the same night agree on roughly eight epochs in ten; two clinicians reading the same ambiguous ECG or two raters annotating "stress" from physiology disagree often enough that the label itself is noisy. When the ground truth is contested, more annotation does not converge to a clean target, it just accumulates variance.
Non-transfer. A label earned on one device, one body, or one machine frequently does not transfer to the next. A model trained on labeled data from one accelerometer placement, sampling rate, or user population degrades on another, because the signal statistics shift while the label semantics stay fixed. This is the distribution-shift problem of Chapter 66, and it means labels are not just scarce but perishable: each new device generation or deployment site can invalidate the labeled corpus you paid for.
Privacy and access. The richest ground truth is often the most sensitive. Labeling behavior needs video of people's homes, labeling health needs medical records, and both are gated by consent, regulation, and the biometric-privacy constraints of Chapter 34. The label may be unattainable not because it is hard to produce but because you are not allowed to.
In Practice: the activity recognizer with four labeled hours
A wearables team building the human-activity models of Chapter 26 collected inertial data from ten thousand consented users over three months, on the order of a hundred thousand device-days of raw signal. To label it they ran a small in-lab study: thirty volunteers performing scripted activities with a researcher annotating from synchronized video, yielding about four labeled hours of clean, balanced examples. That is the entire supervised budget against a mountain of field data, and it shows every failure mode at once. Rarity: near-falls and stair stumbles almost never appeared in the scripted protocol. Subjectivity: "walking" versus "shuffling" split the annotators. Non-transfer: the lab volunteers wore the device tightly on the dominant wrist, while field users wore it loose on either wrist, so the labeled distribution did not match the deployment one. Training a supervised network on four hours overfit the lab and collapsed in the field. The unlabeled hundred thousand device-days, meanwhile, sat untouched by any supervised loss, holding almost all of the information about how real people actually move. The strategic error was framing the problem as "collect more labels" when the leverage was in learning from the signal already in hand.
What scarcity costs, and where the leverage is
It is worth making the bottleneck concrete rather than rhetorical. Model the fleet as producing sensor-hours at a fixed rate while a fixed annotation workforce labels at its own, far slower rate, and ask what fraction of the stream is ever labeled. The listing below does that accounting and prints the coverage over a year for a plausible wearable deployment.
devices = 10_000 # fielded wearables
hours_per_day = 24 # each streams continuously
annotators = 20 # full-time labelers
label_min_per_hr = 5 # minutes of dense signal labeled per work hour
work_hr_per_day = 6 # productive annotation hours per day
produced_hr_day = devices * hours_per_day
labeled_hr_day = annotators * work_hr_per_day * (label_min_per_hr / 60)
coverage = labeled_hr_day / produced_hr_day
print(f"produced : {produced_hr_day:>12,.0f} sensor-hours/day")
print(f"labeled : {labeled_hr_day:>12,.1f} sensor-hours/day")
print(f"coverage : {coverage:>12.2e} (labeled fraction)")
print(f"to label 1 day of stream: {produced_hr_day/labeled_hr_day:,.0f} annotator-days")annotators upward and watch how little coverage buys.Run Listing 17.1.1 and the conclusion is not subtle: throwing annotators at the stream barely moves the coverage needle, because the raw signal outpaces any workforce by four to five orders of magnitude. The information is overwhelmingly on the unlabeled side, so any method that learns from labels alone is discarding almost everything it was given. That is the opening self-supervised learning walks through. Instead of asking a human for the target, you construct a target from the structure of the signal itself, so that the entire unlabeled ocean becomes training data. The oversampled, physically smooth, temporally redundant nature of sensor streams that made attention expensive in Chapter 16 is exactly what makes them rich in free supervisory signal here: neighboring samples predict each other, augmented views should agree, and masked spans can be reconstructed. Those are the levers the rest of this chapter pulls.
The Right Tool
To decide whether label scarcity is actually hurting you, measure a label-efficiency curve: model quality as a function of how many labels you use. You do not have to hand-roll the subsampling, the stratification, and the cross-validation loop. scikit-learn's learning_curve does the whole sweep, holding the leakage-safe splits fixed, in a few lines that replace roughly forty of manual bookkeeping:
import numpy as np
from sklearn.model_selection import learning_curve, GroupKFold
from sklearn.linear_model import LogisticRegression
# X: window features, y: labels, groups: subject id (no subject on both sides)
sizes, train_sc, val_sc = learning_curve(
LogisticRegression(max_iter=1000), X, y,
train_sizes=np.linspace(0.02, 1.0, 10),
cv=GroupKFold(5), groups=groups, scoring="f1_macro")
for n, v in zip(sizes, val_sc.mean(axis=1)):
print(f"{int(n):>6} labels -> macro-F1 {v:.3f}")GroupKFold keeps each subject on one side of the split (the discipline of Chapter 5), and the printed curve tells you how flat your accuracy stays as labels shrink. A curve that is already near its ceiling at a few percent of labels means supervised data is not your bottleneck; a steep one means self-supervised pretraining has room to pay off.Research Frontier
The clearest evidence that the unlabeled ocean is worth mining comes from scaling studies on sensor foundation models, the subject of Chapter 20. Google's Large Sensor Model work (Narayanswamy et al., 2024) pretrained on the order of forty million hours of unlabeled wearable data and reported that self-supervised representations improve smoothly with more unlabeled data and cut the labeled examples needed for downstream tasks by large factors, echoing the imputation and forecasting gains of masked and contrastive objectives. Apple's and academic wearable-pretraining efforts report the same shape: label-efficiency curves that reach supervised-baseline accuracy with a small fraction of the labels once a self-supervised encoder is in place. The live question the field is still settling is which sensing regimes benefit most, and how to certify that a pretrained representation transfers safely across the device and population shifts of Chapter 66 rather than merely fitting the pretraining fleet. Probing that transfer is exactly the evaluation work of Section 17.7.
Exercise
Take any labeled sensor dataset you have (a HAR set from Chapter 26 works well) and, using Listing 17.1.2, plot its label-efficiency curve with subject-wise GroupKFold. (1) Report the smallest labeled fraction that reaches within two macro-F1 points of the full-data model. (2) Now corrupt 20 percent of the training labels at random to simulate the subjectivity failure mode and re-plot; describe how the curve's ceiling and shape change. (3) Restrict the labeled pool to a single subject and evaluate on the rest to simulate the non-transfer failure mode; quantify the drop and explain which of the four scarcity mechanisms it isolates. (4) Using Listing 17.1.1, compute how many annotator-days your fully labeled set represents at 5 labeled minutes per work hour, and state whether more labeling or self-supervised pretraining is the better next investment.
Self-Check
1. ImageNet and web text both offered cheap supervision at scale. State the specific property each one had that let labels be cheap, and explain why raw sensor streams have neither.
2. Name the four failure modes that keep sensor labels scarce, and for each give a one-line example from a different domain (clinical, industrial, wearable, security).
3. A colleague proposes fixing label scarcity by hiring ten times more annotators. Using the logic of Listing 17.1.1, explain why this barely changes coverage, and describe what a self-supervised approach does differently with the same data.
What's Next
In Section 17.2, we turn the abundance into supervision. You will build the first self-supervised objective for sensor streams: contrastive learning, where physically meaningful augmentations (jitter, time warping, channel dropout, sensor-specific transforms) generate pairs of views that a network learns to pull together or push apart, extracting a useful representation from windows no human ever labeled.