"You want ten thousand labeled bearing failures. The factory will happily supply them: one every eighteen months, undocumented, at three in the morning. Or I can synthesize them before lunch. Which deadline do you prefer?"
A Pragmatic AI Agent
Prerequisites
This section leans hard on the measurement-model view of a sensor from Chapter 2: a sensor is a known map from a physical quantity to a digital reading, with its own gain, bandwidth, and noise. It also uses the sampling, quantization, and timing vocabulary of Chapter 3, and the noise and distribution primer from Chapter 4. Predictive models of how sensor state evolves in time were built in Section 55.1; here we add the sensor's measurement layer on top to emit synthetic readings. No physics engine or graphics background is assumed; the few ideas we need are defined as they appear.
The Big Picture
Every data-hungry model in this book quietly assumes that labeled sensor data exists in the quantity and variety you need. For real deployments that assumption is usually false. The interesting events, a bearing spalling, a patient's arrhythmia, a pedestrian stepping off a curb in fog, are rare, dangerous, or expensive to provoke, and the ground-truth label is often the very thing you cannot measure. Simulation flips the problem around. Instead of waiting for the world to produce a labeled example, you write down the physics, run it forward, and pass the result through a model of your sensor to get a synthetic reading whose label you already know because you set it. This section is about doing that with discipline: what a synthetic sensor pipeline is, the layered structure that separates world physics from the sensor's own quirks, and where the practice pays off versus where it quietly lies to you. The reality gap and how to close it belong to Section 55.3; here we build the generator.
Why synthesize sensor data at all
The what is straightforward: synthetic sensor data is a stream of readings produced by a computational model rather than by a physical transducer, together with the labels that model knows by construction. The why is four recurring pains that real data cannot cheaply cure. First, rarity: safety-critical failures are, by design, uncommon, so the positive class you most need to detect is the class you have almost no examples of. Second, labeling: many targets (true joint angle, exact fault severity, the instant a crack initiated) are unobservable in the field even when the raw signal is abundant, whereas a simulator hands you the latent state for free. Third, cost and safety: you cannot crash a hundred cars, induce a hundred cardiac events, or spall a hundred production bearings to build a training set. Fourth, coverage: real logs sample the operating conditions that happened to occur, not the corner cases (a heatwave, a sensor half-occluded by mud, a resonance at an rpm you never ran) that break models in deployment. A simulator lets you dial each of these knobs deliberately, which is exactly what the fault-injection goal of this chapter's lab requires.
The counterweight, stated up front so it colors everything below, is that synthetic data is only as trustworthy as the model that produced it. A generator encodes your beliefs about the physics and the sensor; anything you did not model, the real world will still deliver. That is why synthetic data is most defensible for augmentation, pretraining, and stress-testing, and most dangerous when it silently becomes your only evidence. We return to this discipline in the validation of Section 55.7.
The simulation stack: from world state to a reading
The how is cleaner if you refuse to treat the simulator as one opaque black box and instead separate it into three composable layers. Keeping them distinct is the single most useful habit in this section, because it lets you reuse a physics model across many sensors and swap a sensor model without touching the physics.
Layer 1, the world / physics. A generative model of the latent state \(\mathbf{x}(t)\): rigid-body dynamics in a physics engine, a hydraulic or thermal model of a plant, an electrophysiological model of the heart, a procedural traffic scene. This is the layer Section 55.1 built as a dynamics model; its output is ground truth, noise-free and fully labeled.
Layer 2, the measurement model. The map \(h(\cdot)\) from world state to the ideal quantity your sensor responds to, exactly the observation function of Chapter 2. An accelerometer sees the second derivative of position projected onto its axis and scaled by its sensitivity; a camera sees a projection of the scene; a lidar sees ranges along rays. This layer is where a single simulated world can feed many virtual sensors at once.
Layer 3, the corruption / acquisition model. The transducer is not ideal. This layer applies the imperfections catalogued in Chapters 2 and 3: additive and quantization noise, limited bandwidth, gain and bias errors, saturation, dropouts, and the finite sample rate. A generic scalar sensor reduces to
\[ y_k = Q\!\Big( g \cdot h\big(\mathbf{x}(t_k)\big) + b + n_k \Big), \qquad n_k \sim \mathcal{N}(0,\sigma^2), \]where \(g\) is sensitivity, \(b\) a bias, \(Q(\cdot)\) the quantizer, and \(t_k\) the (possibly jittered) sample times. Richer sensors replace the scalar \(h\) and the Gaussian \(n_k\) with a rendering equation and structured artifacts, but the three-layer decomposition survives. The lesson is that realism lives mostly in Layer 3: a physically perfect world plus a naive sensor model produces suspiciously clean data that a model trained on it will never recognize in the field.
Key Insight: fidelity that matters is fidelity your model can perceive
It is tempting to chase maximal physical realism everywhere. The productive question is narrower: which discrepancies between synthetic and real data can the downstream model actually exploit or be fooled by? A vibration classifier keyed on envelope-spectrum peaks does not care whether your rendered lighting is photoreal, but it collapses if you forget to model the structural resonance that shapes those peaks. A depth network cares intensely about sensor noise statistics and edge artifacts and not at all about the color of the walls. So budget fidelity where the model looks: match the signal statistics in the feature space the model consumes, not the raw pixels or volts in the abstract. This is why the same simulator can be gloriously over-engineered for one task and dangerously naive for another.
A worked generator: a bearing vibration stream with an injected fault
Concreteness helps. A rolling-element bearing with a defect on its outer race produces, on each pass of a ball over the defect, a sharp impact that rings the sensor-plus-structure resonance and decays. The impacts recur at the ball-pass frequency (BPFO), and the whole train is scaled by the accelerometer sensitivity and buried in broadband noise. That is a Layer-1 impulse train, a Layer-2 resonance-and-sensitivity map, and a Layer-3 noise-and-quantize step, exactly the three layers above. Listing 55.1 synthesizes one second of it with a labeled fault severity.
import numpy as np
fs, T = 20_000, 1.0 # 20 kHz accelerometer, 1 second
t = np.arange(0, T, 1/fs)
# Layer 1: world physics = periodic impacts at the ball-pass freq (BPFO)
bpfo, severity = 104.0, 0.8 # Hz; severity in [0,1] is our LABEL
impact_times = np.arange(0, T, 1/bpfo)
excitation = np.zeros_like(t)
for tk in impact_times:
excitation[int(tk*fs)] = severity # unit impulses, scaled by defect size
# Layer 2: measurement model = ring the structural resonance, apply sensitivity
f_res, zeta = 3200.0, 0.02 # resonance 3.2 kHz, light damping
decay = np.exp(-zeta * 2*np.pi*f_res * (t % (1/bpfo)))
ring = decay * np.sin(2*np.pi*f_res * (t % (1/bpfo)))
signal = np.convolve(excitation, ring[:int(fs/bpfo)], mode="same")
sensitivity = 100.0 # mV per g
signal = sensitivity * signal
# Layer 3: acquisition = broadband noise + 12-bit quantization
signal += np.random.normal(0, 3.0, size=t.shape) # sensor + electronic noise
lsb = (signal.max() - signal.min()) / 2**12 # 12-bit ADC step
reading = np.round(signal / lsb) * lsb # quantized digital reading
print(reading.shape, "samples; label severity =", severity)
severity is a free, exact label that no real accelerometer could ever hand you. Sweeping bpfo, severity, f_res, and the noise level generates a labeled fleet of faults on demand.Listing 55.1 is deliberately small so the three layers are visible, but it already does the thing that matters: it emits a realistic envelope spectrum whose peaks sit at BPFO and its harmonics, with an exact severity label attached. Feed a few thousand such sequences, sweeping fault type, speed, and noise, to a detector and you have the transferable training set the lab asks for. Notice what stayed in your hands: the resonance frequency, the sensitivity, and the noise variance are all your assumptions, and a real bearing on a real gearbox may resonate elsewhere, which is precisely the gap Section 55.3 teaches you to randomize over rather than guess.
Practical Example: seeding a wind-turbine fleet monitor before the first failure
A wind-farm operator commissions a new turbine model and wants gearbox fault detection working from day one, but a fresh fleet has, by definition, no failures yet. The team builds a generator in the shape of Listing 55.1, extended with the real gearbox kinematics (tooth counts, shaft speeds) so the fault frequencies are correct, and with SCADA-driven rpm so impacts smear realistically as the rotor speed varies in wind. They synthesize tens of thousands of labeled sequences across outer-race, inner-race, and tooth-crack faults at graded severities, pretrain a condition-monitoring model on them, then fine-tune on the trickle of real healthy data as it accumulates. The synthetic set carries the class the real data lacks (severe faults) and the exact labels the field cannot supply. When the first real spall finally appears eleven months in, the detector flags it early, and the team's job in Chapter 37 becomes confirming the synthetic priors held rather than starting from zero. The prognostics horizon this buys is the subject of Chapter 36.
Right Tool: don't hand-roll a full sensor simulator
Listing 55.1 is a teaching-sized generator for one scalar sensor. Full-fidelity rigs are another order of effort, and mature simulators already own the hard parts. CARLA emits synchronized camera, lidar, radar, IMU, and GNSS streams from a driving scene with pixel- and point-level ground truth; NVIDIA Isaac Sim does the same for robot arms and mobile bases with contact-rich physics and configurable sensor noise; gazebo covers general robotics. Reproducing even one of CARLA's calibrated lidar sensors, with ray casting, intensity, dropout, and motion distortion, is well over a thousand lines and a research effort in its own right; the simulator reduces it to a sensor descriptor and a callback. Use these for the world-and-rendering layers and reserve your own code for the sensor-specific corruption model the simulator does not capture. The neural-field and Gaussian-splatting reconstructions of Chapter 51 are an increasingly popular data-driven alternative to hand-built graphics for the world layer.
Physics-based versus learned generators, and how to choose
Not every synthetic pipeline is a physics engine. Two families exist and they trade off cleanly. Physics-based (mechanistic) generators, like Listing 55.1 and CARLA, encode known equations. They are interpretable, extrapolate to conditions never observed (that is their whole point), give exact labels, but demand that you actually know the physics and pay in engineering effort and compute. Learned (data-driven) generators, GANs, diffusion models, and the world models of Chapter 53, fit a distribution from real examples and then sample new ones. They capture subtle real-world texture the physics misses, but they cannot conjure a mode absent from their training data (a fault they never saw), their labels are only as good as the conditioning, and they can hallucinate physically impossible signals. The practical choice tracks the failure you are curing: reach for mechanistic simulation when you need coverage of rare or unseen regimes and trustworthy labels, and for learned generators when you have real data of the right kind and want more of the same, only more diverse. The hybrid that gets the best of both, a physics core with a learned residual, is exactly the subject of Section 55.5.
Research Frontier: sensor-realistic generative simulation
The current frontier is fusing physics and generative learning so the output is both controllable and photo/signal-realistic. On the driving side, NVIDIA's neural reconstruction and DRIVE Sim line and academic systems such as UniSim and MARS rebuild real logged scenes as editable neural scenes, then re-render new camera and lidar sensor data along counterfactual trajectories, closed-loop and sensor-realistic, so you can ask "what if the pedestrian had crossed here" and get labeled multi-sensor data. In wearables and health, generative models synthesize labeled IMU and ECG segments to balance rare classes. The two hard open problems are label fidelity under generative sampling (a diffusion model can produce a beautiful signal whose attached label is subtly wrong) and closing the reality gap without over-fitting to the simulator, which pushes directly into the domain-randomization and sim-to-real methods of Section 55.3 and the validation protocols of Section 55.7.
Exercise
Extend Listing 55.1 into a small labeled dataset generator. (1) Add a second fault type by placing impacts at an inner-race frequency (BPFI) and amplitude-modulating them at shaft speed, so the two fault classes are physically distinguishable. (2) Randomize resonance frequency, sensitivity, noise variance, and rpm across a few thousand one-second clips, saving each clip with its (fault_type, severity) label. (3) Train any classifier from Chapter 23 or Chapter 37 on it, then test on the real CWRU bearing dataset. Report the accuracy drop from synthetic-test to real-test, and argue which of your three layers is most responsible for the gap. Keep synthetic and real strictly separated in the split, per the leakage discipline of Chapter 5.
Self-Check
1. Name the three layers of the simulation stack and say which one produces the label and which one produces most of the realism. 2. Give one target that a simulator can label exactly but a real sensor cannot measure at all, and explain why that asymmetry is the core value of synthetic data. 3. When would you choose a learned generative model over a physics-based simulator, and what failure mode of the learned model would make you switch back?
What's Next
In Section 55.3, we confront the gap that Listing 55.1 quietly assumed away: the real bearing resonates at a frequency you guessed wrong, in a noise field you did not model. Rather than trying to match reality exactly, domain randomization deliberately scrambles the simulator's parameters so the trained model treats reality as just one more randomized sample, and the sim-to-real toolkit turns a generator like this one into a policy that survives contact with the physical world.