Security of physical-world AI

"I was trained to trust my sensors. Nobody mentioned that my sensors could be trained to lie to me."

A Recently Compromised AI Agent

This section assumes the sensor fault vocabulary and the physics of spoofing from Part XIII, specifically the functional-safety and sensor-spoofing mechanics of Chapter 68 and the industrial-control intrusion detection of Chapter 38. Those chapters teach how to detect a corrupted input; here we take the adversary's-eye view and ask, across the whole lifecycle, who can make the system wrong, and what we owe the people affected when they try.

Why physical-world AI has a bigger attack surface

A cloud classifier receives its inputs through an authenticated API. A sensing system receives its inputs through the open physical world, which no firewall covers. That single fact widens the attack surface along three axes. First, the signal channel is public: anyone within line of sight, radio range, or acoustic range of a sensor can inject energy into it. Second, the transduction is exploitable: an adversary who understands that a MEMS gyroscope has a mechanical resonance, or that a camera's rolling shutter samples rows in sequence, can craft a physical stimulus that the sensor faithfully converts into a malicious digital reading. Third, the model itself is an asset that ships to the edge, where it can be extracted, inverted, or fooled at leisure. Security for physical-world AI therefore has to reason about the confidentiality, integrity, and availability (the classic CIA triad) of three things at once: the raw signal, the model, and the training data that shaped it.

It helps to score exposure rather than treat every component as equally at risk. A crude but useful risk index for a component multiplies how reachable it is, how much an attacker gains, and how likely a successful attack goes unnoticed:

The term that most often dominates in sensing systems is the last one. A physically spoofed but in-range measurement, exactly the stuck-and-plausible failure of Chapter 68, has a low \(P_{\text{detect}}\) and so a high \(R\) even when the impact is moderate. Security effort should follow \(R\), which usually means investing in detection and cross-checks rather than only in perimeter hardening.

A threat taxonomy: four ways to attack a model

Adversarial machine learning organizes attacks by what the adversary manipulates and when. Four classes cover almost everything you will face in a sensing deployment.

• Evasion (test time). The model is fixed; the adversary perturbs the input at inference to force a wrong output. In the physical world this is a printed patch, an infrared LED array, an ultrasonic tone, or a crafted radar return. The perturbation need not be small or invisible, only effective and deniable.
• Poisoning (train time). The adversary corrupts the training data so the deployed model carries a hidden flaw. Sensor datasets are especially exposed because they are often crowd-collected, weakly labeled, or continuously refreshed by fleet telemetry, giving an attacker a supply line straight into your weights. This is the security face of the leakage-safe data discipline of Chapter 5.
• Backdoor (train time, triggered at test time). A special case of poisoning where the model behaves normally until it sees a chosen trigger (a specific vibration signature, a particular pixel pattern), then misbehaves on cue. Backdoors are dangerous precisely because they pass ordinary accuracy tests.
• Extraction and inversion (deploy time). By querying an exposed model, an adversary can steal it (model extraction) or reconstruct sensitive training records (model inversion, membership inference). For biometric and health models this is a privacy breach, tying security directly to the biometric-privacy duties of Section 70.2 and the defenses of Chapter 64.

These map cleanly onto the CIA triad: evasion and poisoning attack integrity, jamming attacks availability, extraction and inversion attack confidentiality. Naming the class tells you where in the lifecycle the control belongs, at the sensor, in the data pipeline, or around the served model.

The model and data supply chain

Most real compromises do not come from a clever gradient; they come from an unsigned artifact. A modern sensor model is assembled from a pretrained backbone (often downloaded from a public hub), third-party datasets, an open-source training stack, quantization and compilation tools, and an over-the-air update channel to the fleet. Every hop is a place to inject a backdoor or a trojaned dependency. Treating the model like any other software artifact, with a bill of materials, cryptographic signing, and verified provenance, closes the cheapest attacks. The principle is simple: a device should refuse to run a model whose origin it cannot verify, exactly as a browser refuses an untrusted certificate. The snippet below shows the minimum viable check an edge device should perform before loading a model, verifying an HMAC signature over the exact bytes it is about to execute.

import hmac, hashlib

def verify_model(model_bytes: bytes, signature_hex: str, key: bytes) -> bool:
    """Refuse to load a model artifact whose signature does not match.
    key is a per-fleet secret provisioned at manufacture; signature_hex
    is shipped alongside the artifact by the trusted build server."""
    expected = hmac.new(key, model_bytes, hashlib.sha256).hexdigest()
    # constant-time compare avoids a timing side channel on the tag
    return hmac.compare_digest(expected, signature_hex)

# On device, before handing weights to the runtime:
blob = open("obstacle_net.q8.bin", "rb").read()
if not verify_model(blob, sig_from_manifest, FLEET_KEY):
    raise SystemExit("untrusted model artifact: refusing to load")
load_and_run(blob)

Code 70.4.1 is deliberately minimal to show the shape of the control; a fleet in production wants asymmetric signatures, a rotating key hierarchy, and a transparency log, none of which you should hand-roll.

Defense in depth as a governance duty

No single control stops all four attack classes, so security for physical-world AI is layered, and the layering is a responsibility you should be able to show an auditor. A workable stack has four tiers. At the signal tier, add redundancy and physical authentication: cross-check independent modalities so a spoof must fool all of them at once (the missing-modality and fusion logic of earlier parts is also your anti-spoofing logic), and where possible use sensors with built-in challenge-response or physical-layer fingerprints. At the data tier, enforce provenance, quarantine fleet-collected data, and run poisoning and backdoor audits before any retrain touches production weights. At the model tier, sign artifacts, rate-limit and monitor query patterns to blunt extraction, and consider adversarial training where the threat justifies its accuracy cost. At the operations tier, keep an incident-response plan, an audit log, and the ability to roll back a model fleet-wide in minutes, because the responsible question after a compromise is not only "is it fixed" but "who was harmed and how do we tell them," which is the accountability thread of Section 70.7. Agentic sensing systems that can take actions in the world (Chapter 22) raise the stakes of every tier, because a compromised perception now drives a compromised action.

What's Next

In Section 70.5, we turn from defending the pipeline to documenting it: dataset governance through datasheets and dataset cards, the artifacts that make provenance, consent, and known limitations legible to the people who reuse your data, and that turn the poisoning and privacy defenses of this section into a paper trail an auditor can actually follow.