Vehicles and mobility (multimodal suites, cabin monitoring, fleet PdM, safety cases)

"The camera saw a truck, the radar saw a wall, the lidar saw a cloud of dust. My job was to decide which two were lying, in forty milliseconds, forever."

A Fused AI Agent

The outward suite: complementary physics, not redundant cameras

The reason a self-driving stack carries three or four sensor types is not belt-and-braces caution; it is that each modality fails on a physically different axis, and the failures are close to statistically independent. A camera gives dense semantic texture and reads signs and lane lines, but it degrades in darkness, glare, fog, and heavy rain, and it estimates depth poorly from a single frame. Radar measures range and, through the Doppler shift, radial velocity directly and works through fog and rain, but its angular resolution is coarse and it clutters on metal. Lidar gives precise geometry at range, but it is expensive, struggles with retroreflective and absorptive surfaces, and scatters on airborne dust and snow. The engineering value comes from the disjointness of these weakness sets: the conditions that blind a camera (fog) are exactly where radar shines, and the ambiguity that plagues radar (angle) is exactly where lidar and camera are strong.

That complementarity only pays off if the fusion is honest about time and uncertainty. Each sensor runs on its own clock and cadence (camera at 30 to 60 Hz, lidar at 10 to 20 Hz, radar at 15 to 20 Hz), so the first job is temporal alignment: every detection must be motion-compensated to a common timestamp using the vehicle's own ego-motion estimate from the inertial unit (Chapter 24). The dominant modern architecture lifts all modalities into a shared bird's-eye-view grid, where a camera pixel, a radar return, and a lidar point can be reasoned about in the same metric coordinates before a tracker fuses them over time. The tracker is where uncertainty becomes load-bearing: an object confirmed by two independent modalities gets a tight covariance and a high existence probability; an object seen by one sensor only stays "tentative" until corroborated or aged out.

The inward suite: cabin and driver monitoring

Regulation is now pulling a second sensing system into every new car. Driver monitoring systems (DMS) use an infrared driver-facing camera to estimate gaze direction, eyelid closure (the PERCLOS drowsiness metric), and head pose, so the vehicle can escalate a warning before a distracted or drowsy driver drifts. Occupant monitoring systems (OMS) add a wider cabin view and, increasingly, a 60 GHz in-cabin radar that detects the sub-millimeter chest motion of breathing. That radar solves a problem a camera cannot: detecting a sleeping child left in a footwell or under a blanket, where vital-sign presence detection (the contactless sensing of Chapter 33) prevents a hot-car fatality. The European New Car Assessment Programme has made both drowsiness/attention monitoring and child-presence detection scored requirements, which is why these sensors are appearing in mainstream vehicles rather than luxury trims.

The inward suite carries the book's privacy thread at maximum intensity. A driver-facing camera is a continuous biometric stream pointed at a person's face inside a private space. The defensible design keeps inference on the edge (Chapter 59), emits only derived states (attentive/drowsy, occupied/empty) rather than raw video, and retains nothing beyond the moment of decision unless a crash triggers an event buffer. Radar is genuinely privacy-preserving here: it senses presence and breathing without ever forming an identifiable image, which is a strong argument for choosing it over a camera when the only task is "is a living occupant present?"

Fleet predictive maintenance: from one car to ten thousand

A single vehicle is a machine; a fleet is a data asset. Commercial operators (trucking, delivery, transit, ride-hail) stream CAN-bus and telematics data (engine temperature, oil pressure, battery voltage and, for electric vehicles, cell-level state-of-health) off every vehicle to predict failures before they strand a driver or a shipment. This is predictive maintenance (the prognostics of Chapter 36) applied at fleet scale, and its defining challenge is heterogeneity: vehicles differ by model year, duty cycle, climate, and driver behavior, so a threshold tuned on one cohort produces false alarms on another. The fleet advantage is that with thousands of near-identical units, a vehicle can be scored against its own peers: a battery pack degrading faster than the fleet distribution for its mileage and climate is suspicious even while every absolute reading remains in spec.

The operational payoff is measured in avoided roadside failures and optimized parts inventory, not in model accuracy on a benchmark. That reframes evaluation: a prognostic that predicts a failure two weeks out is far more valuable than one that predicts it two hours out, even at lower nominal accuracy, because two weeks buys a scheduled shop visit instead of a highway breakdown. Managing the model across a churning, drifting fleet, retraining as new model years enter service and old ones retire, is the MLOps discipline of Chapter 69, and getting the train/test split right so a vehicle never appears in both is the leakage-safe evaluation of Chapter 5.

The snippet below scores each vehicle's health signal against its own peer cohort rather than a fixed threshold, the core move that makes fleet PdM robust to heterogeneity.

import numpy as np

def peer_relative_risk(value, cohort_values):
    """Robust z-score of one vehicle's signal vs its peer cohort.
    cohort = vehicles matched on model-year, mileage band, climate."""
    med = np.median(cohort_values)
    mad = np.median(np.abs(cohort_values - med)) + 1e-9
    return 0.6745 * (value - med) / mad   # modified z-score

# Example: one truck's coolant-temp trend vs 400 matched peers
peers = np.random.normal(90.0, 3.0, size=400)   # deg C, healthy cohort
score = peer_relative_risk(101.5, peers)
print(f"peer-relative risk = {score:.1f}")       # ~ 5 sigma: flag for inspection

The safety case: engineering an argument, not just a model

What makes automotive sensing categorically different from a fitness wearable is that shipping requires a safety case: a structured, auditable argument that the system is acceptably safe for its intended use. Two standards divide the work. ISO 26262 governs functional safety, faults arising when hardware or software breaks (a sensor dies, a bit flips). ISO 21448, Safety of the Intended Functionality (SOTIF), governs the harder automotive problem: the system has no fault at all, yet still behaves unsafely because the world presented a scenario its perception could not handle (a novel obstacle, a blinding sunset, an adversarial road marking). SOTIF is where sensor AI lives, because a perception model that is working exactly as trained can still be wrong on an out-of-distribution scene (Chapter 66).

A safety case turns the abstract callouts of this book into obligations you must discharge with evidence. Calibrated uncertainty stops being nice-to-have and becomes the mechanism by which the system knows to hand control back or slow down when confidence collapses (the conformal guarantees of Chapter 18). Robustness to sensor spoofing (a projected phantom pedestrian, a lidar-blinding laser) becomes a documented threat model with tested mitigations from Chapter 68. Every claim in the argument ("the system detects pedestrians at 80 m in daylight") must be backed by a leakage-safe test on data the model never saw, because a safety case built on a contaminated benchmark is not an argument, it is a liability.

What's Next

In Section 71.4, we step out of the vehicle and into the building, where the sensing problem inverts: instead of one mobile platform packed with sensors, we have a fixed space watched by many cheap, often non-camera sensors, and the playbook becomes occupancy inference, privacy-preserving ambient intelligence, and where to physically place a sensor so it hears what matters.