Noise sources and signal-to-noise ratio

"Every reading I return is the truth plus an apology I never asked to write."

A Self-Aware AI Agent

This section assumes you are comfortable with the resolution, sensitivity, and dynamic-range vocabulary from Section 2.2, and that you have met random variables, variance, and expectation. If those feel shaky, the Probability, Estimation, and Uncertainty Primer (Chapter 4) is the companion that makes the statistics here rigorous. We stay at the physical and definitional level: what the noise is, why it is there, and how to measure it.

Where the noise comes from

Noise is any component of the sensor output that does not correspond to the observable you intend to measure. It is useful to sort noise by its physical origin, because origin dictates behavior: how it scales with signal, temperature, and bandwidth, and therefore how you can fight it.

Thermal (Johnson-Nyquist) noise is the voltage jitter produced by charge carriers rattling around in any resistive element at non-zero temperature. Its power is \(P = 4 k_B T R \, \Delta f\), so it grows with temperature \(T\), resistance \(R\), and measurement bandwidth \(\Delta f\). It is white (flat across frequency) and unavoidable above absolute zero, which is why cooled detectors and narrow bandwidths are the two classic weapons against it.

Shot noise arises because signals are quantized into discrete carriers: electrons across a junction, photons hitting a pixel. When you count \(N\) discrete events, the count fluctuates with standard deviation \(\sqrt{N}\) (Poisson statistics). This is why a photodiode in bright light has more absolute noise but better relative noise than in dim light, and why low-light imaging is fundamentally hard.

Flicker (1/f) noise has a power spectral density that rises at low frequencies. It dominates slow measurements and DC-coupled sensors, and it is the reason a temperature probe left untouched still wanders over minutes. Quantization noise is the rounding error the analog-to-digital converter injects when it snaps a continuous voltage to the nearest code; we treat it in depth alongside sampling in Signals, Sampling, Time, and Synchronization (Chapter 3). Layered on top are environmental and interference terms: 50/60 Hz mains pickup, motion artifact in a wrist sensor, electromagnetic coupling from a nearby motor.

Defining signal-to-noise ratio

SNR compresses "how clean is this measurement" into one number: the ratio of signal power to noise power,

Because power is amplitude squared, an SNR expressed from amplitudes uses \(20 \log_{10}(A_\text{signal}/A_\text{noise})\). The decibel scale is logarithmic, so every 10 dB is a factor of ten in power, every 3 dB roughly a doubling. As a feel for the numbers: a clean lab-grade measurement might sit at 60 to 80 dB, a wearable photoplethysmogram on a moving wrist can drop below 6 dB during exercise, and detection near 0 dB means signal and noise carry equal power (a coin flip without further processing).

Two cousins deserve names. The noise floor is the output you see with no input signal present, and it sets the smallest change you can resolve. The minimum detectable signal is the input that produces an output equal to the noise floor, conventionally where SNR reaches 1. A sensor's usable dynamic range from Section 2.2 is bounded below by this floor and above by saturation, a ceiling we treat in Section 2.4.

Measuring and improving SNR in practice

The single most reliable lever is coherent averaging. If a measurement is repeated \(N\) times with independent, zero-mean white noise, the signal adds coherently while the noise adds in quadrature, so amplitude SNR improves by \(\sqrt{N}\): a 100-fold repeat buys 10x amplitude SNR, or 20 dB. This is the workhorse behind lock-in amplifiers, averaged evoked potentials, and multi-frame image stacking. It only works when the noise is white and the signal is stationary across the repeats, which is exactly the assumption that colored noise and drift violate. Other levers include narrowing bandwidth to the band the signal actually occupies (thermal noise power scales with \(\Delta f\)), matched filtering when the signal shape is known, and cooling the front end.

To estimate SNR from real data you separate a clean-signal estimate from a residual and compare their variances. The snippet below simulates a sine buried in white noise, recovers SNR, and demonstrates the \(\sqrt{N}\) averaging gain.

import numpy as np

rng = np.random.default_rng(0)
fs, f0, T = 1000, 5.0, 2.0           # sample rate, tone freq, duration
t = np.arange(0, T, 1/fs)
clean = np.sin(2*np.pi*f0*t)         # the observable we want
noise_sigma = 1.5                    # white noise standard deviation

def snr_db(sig, noisy):
    resid = noisy - sig
    return 10*np.log10(np.var(sig) / np.var(resid))

single = clean + rng.normal(0, noise_sigma, t.size)
print(f"single-shot SNR:  {snr_db(clean, single):5.1f} dB")

N = 64                               # coherent averaging over N repeats
stack = np.mean([clean + rng.normal(0, noise_sigma, t.size)
                 for _ in range(N)], axis=0)
print(f"averaged (N={N}): {snr_db(clean, stack):5.1f} dB")
print(f"predicted gain:   {10*np.log10(N):5.1f} dB")

Why SNR is the ceiling for AI

Here is the connection this whole chapter is building toward. A learned model cannot recover information the measurement never contained. Once signal and noise overlap in the feature the model relies on, no architecture, however large, can separate them with certainty; it can only report a calibrated probability. SNR therefore sets an information-theoretic ceiling on accuracy, and the honest job of a model near that ceiling is to quantify its own uncertainty rather than hallucinate confidence. That is why sensor-facing systems pair predictions with uncertainty, a thread that runs from the estimation foundations in Chapter 4 through calibration and conformal methods later in the book. A practical corollary: reporting the SNR of your training data is as important as reporting the model, because a benchmark collected at 40 dB tells you nothing about deployment at 5 dB.

What's Next

In Section 2.4, we turn from the random part of the error to its structured, repeatable part: bias, drift, hysteresis, and saturation. These are the errors that do not average away and that a naive noise model will silently absorb into the wrong place, and understanding them completes the measurement-error picture before we assemble the full transfer function.