Capstone: select an application and sensing stack; define measurement and task; build acquisition and preprocessing; train a classical baseline and a modern/foundation model; evaluate generalization, uncertainty, and robustness; deploy or simulate deployment; produce a final technical and responsible-use report

"A demo persuades a room. A protocol persuades a regulator, a night-shift technician, and the version of you that has to debug this at 3 a.m."

A Seasoned AI Agent

Choose the application, the sensing stack, and a task you can measure

A capstone lives or dies on the first paragraph of its own spec. Three choices lock in before any code: the application (who acts on the output and what changes when they do), the sensing stack (which transducers, at what rates, with what known failure modes, per Chapter 2), and the task written as a function with a metric attached. "Monitor the patient" is not a task. "Classify each 10-second PPG window as atrial-fibrillation or sinus rhythm, optimizing sensitivity at a fixed 1-per-24-hour false-alarm rate" is a task: it names the input window, the label space, the operating point, and the metric that the operating point implies. The discipline here is to write the measurement model down explicitly, \(y = h(s) + \varepsilon\), so the task connects a hidden state \(s\) you care about to the raw signal \(y\) you actually get, with a noise term \(\varepsilon\) whose structure you will exploit later.

Scope ruthlessly. A good capstone task is narrow enough to evaluate rigorously in the time you have and broad enough that a classical method and a foundation model genuinely disagree. If a hand-tuned threshold already hits 99% of achievable performance, you have chosen a problem that cannot teach the classical-versus-foundation tradeoff that is the point of the exercise.

Build the acquisition and preprocessing spine

The data spine is the part reviewers trust least and students rush most. Acquisition means capturing raw signal with the metadata that makes it splittable: subject or machine ID, session, device serial, timestamp, and site. Those columns are not bookkeeping; they are the only defense against the leakage that inflates every naive result. Preprocessing then does the unglamorous work: resample to a common rate and align clocks (Chapter 3), filter and denoise (Chapter 6), window the stream, and, critically, fit every transform on the training split only. A normalization statistic computed over the whole dataset has already told your model about the test set.

The single most common capstone failure is the grouped split done wrong. If the same subject, machine, or recording session appears in both train and test, the model can memorize identity instead of learning the task, and the reported score is pure leakage. Split by group, verify no group crosses the boundary, and prefer a temporally forward split (train on the past, test on the future) whenever the deployment will see time move in one direction, exactly the protocol argued in Chapter 5.

Race a classical baseline against a foundation model, on one protocol

The heart of the capstone is a fair fight. Build a classical baseline first, always: hand-engineered spectral and statistical features (Chapter 8) into a gradient-boosted tree or an SVM. It is fast, interpretable, hard to beat on small data, and it sets the number the fancy model must justify beating. Then bring the modern or foundation model: fine-tune or zero-shot a pretrained time-series or sensor model such as TimesFM, MOMENT, Chronos, or Google's LSM, or train a compact transformer or state-space model from Chapter 15 and Chapter 16.

The rule that makes the comparison honest: one protocol, one split, one metric, computed in one pass. Both models see identical windows, identical folds, and the identical held-out test set they never touched during selection. A win only counts if it is large enough to clear the confidence interval and consistent across folds. Report the classical number even when the foundation model wins; report it especially when it does not, because "the 20-line baseline matched the 200-million-parameter model" is one of the most useful results a capstone can produce.

import numpy as np
from sklearn.metrics import roc_auc_score

def paired_eval(clf_scores, fm_scores, y_true, n_boot=2000, seed=0):
    """Same test set, same labels: bootstrap the AUROC gap between two models."""
    rng = np.random.default_rng(seed)
    n = len(y_true)
    gaps = np.empty(n_boot)
    for b in range(n_boot):
        idx = rng.integers(0, n, n)                 # resample TEST windows, paired
        a = roc_auc_score(y_true[idx], clf_scores[idx])
        f = roc_auc_score(y_true[idx], fm_scores[idx])
        gaps[b] = f - a                             # foundation minus classical
    lo, hi = np.percentile(gaps, [2.5, 97.5])
    print(f"AUROC gap (FM - classical): {gaps.mean():+.3f}  95% CI [{lo:+.3f}, {hi:+.3f}]")
    print("verdict:", "FM wins" if lo > 0 else "no significant difference")
    return gaps

# clf_scores, fm_scores, y_true come from ONE forward pass on the SAME held-out split

The code above is the arbiter of the race: it turns "the transformer got 0.94 and the tree got 0.92" into a defensible claim about whether the two points differ at all once test-set sampling noise is accounted for.

Evaluate generalization, uncertainty, and robustness

A single test score is a claim about one distribution. A capstone must defend three separate properties. Generalization: does performance hold on data from an unseen subject, site, device, or time period? Report the grouped and forward-in-time results, and treat any distribution-shift or out-of-distribution degradation with the test-time-adaptation lens of Chapter 66. Uncertainty: is the model's confidence trustworthy? Produce a reliability diagram and an expected-calibration-error number, and wrap the predictions in a conformal set (Chapter 18) so that a stated 90% coverage is an empirically verified 90%. A well-calibrated 85%-accurate model is often more deployable than an overconfident 90%-accurate one.

Robustness: what happens under the sensor failures that Chapter 2 promised would occur? Perturb the test set deliberately: add band-limited noise, drop a channel, shift the sampling rate, inject a stuck-at value, and simulate the spoofing attacks of Chapter 68. A model whose accuracy falls off a cliff when one accelerometer axis saturates is not ready, no matter its clean-data score. Plot performance as a function of perturbation magnitude; the slope of that curve is your robustness result.

Deploy or simulate deployment, then write the report

A model that never leaves a notebook has not been evaluated on the thing that matters: latency, memory, power, and drift on the target. If you can deploy, do it to a real edge device and measure end-to-end inference time and energy, using the optimization toolkit of Chapter 59. If you cannot, simulate deployment faithfully: quantize the model, run it under the target's compute and memory ceiling, replay a live-ordered stream (never shuffled), and log the latency and confidence of every decision as if a technician were watching. Either way you produce an operational curve, not a single accuracy point.

The final deliverable is two documents fused into one. The technical report states the task, the data spine and its splits, the model race with its confidence intervals, and the three-property evaluation, each claim backed by an artifact a reader could regenerate. The responsible-use report (framed by Chapter 70) states the intended use and the out-of-scope uses, the subgroup performance breakdown (does it work as well across skin tones, machine types, or sites?), the privacy posture of the sensor data, the known failure modes, and the human-in-the-loop safeguard. A capstone without the second document is an experiment; with it, it is a system someone could responsibly ship.

What's Next

This is the last section of the last chapter, so what comes next is your own project and the shelf of references behind it. In Contents, the appendices supply the reusable scaffolding: the datasets and benchmarks catalogue, the evaluation-metrics reference, the toolchain guide, and the notation glossary that turn the capstone rubric above into a repeatable practice. The book ends where a working sensory-AI engineer begins: with a physical quantity, an honest protocol, and a report you would sign.