"I reported the temperature as 21.4 degrees with the confidence of a stone tablet. Nobody asked me the one question that mattered: 21.4, plus or minus what?"
A Recalibrated AI Agent
Prerequisites
This section assumes you can read the additive measurement model \(x = h(s) + \eta\) from Chapter 2 and that you know a sensor stream is a sequence of samples \(x[n]\) produced by sampling a continuous signal, as developed in Chapter 3. You need first-year calculus (integrals and sums) and a little linear algebra for the covariance matrix at the end. No prior probability course is required; the distributions and moments you need are built here and collected in Appendix A. Everything downstream in this chapter, estimators, Bayesian priors, and the uncertainty split, rests on the language introduced in this one section.
Why a single number is never the whole measurement
A sensor never hands you a value; it hands you a value drawn from a distribution. Point the same accelerometer at the same still table and read it a thousand times and you get a thousand slightly different numbers, jittering around the truth. That jitter is not a defect to be apologized for. It is information: its shape tells you the noise floor, its width tells you how much to trust any single reading, and its tails tell you how often the sensor will lie badly. This section gives you the vocabulary to describe that jitter precisely, so that every later chapter can answer "how sure are we?" instead of just "what did we get?" A model that consumes raw values and ignores their distribution is discarding half of what the hardware told it.
We treat each sample as a random variable: a rule that assigns a number to the outcome of a random experiment, here the experiment being "let the transducer report once." A random variable \(X\) is described completely by its distribution, and for a real-valued sensor reading that distribution is captured by a probability density function \(p_X(x)\), where \(p_X(x)\,dx\) is the probability that a reading lands in the sliver \([x, x+dx]\). The density integrates to one, \(\int_{-\infty}^{\infty} p_X(x)\,dx = 1\), which simply says the sensor always returns something.
From one sample to a random variable: what, why, and when
What changes when we call a reading a random variable is that we stop asking "what is the value?" and start asking "what values are plausible, and with what weight?" Why this matters for sensory AI is that the physical noise sources of Chapter 2, thermal agitation, quantization, shot noise, are themselves random processes, so the only faithful description of a reading is probabilistic. How we work with it is through two summaries that we can actually compute from data: the density (or its cumulative form \(F_X(x) = P(X \le x)\)) when we want the full picture, and a handful of moments when we want a compact fingerprint. When you can get away with just the moments rather than the full density is the recurring judgment call of this chapter, and the Gaussian case, where two moments are the whole density, is why the Gaussian shows up everywhere.
The distributions that actually appear in sensor data
A small cast of distributions covers most of what a sensor stream throws at you, and each earns its place from a physical mechanism, not mathematical convenience.
The Gaussian (normal) distribution, \(p(x) = \tfrac{1}{\sqrt{2\pi}\sigma}\exp\!\big(-\tfrac{(x-\mu)^2}{2\sigma^2}\big)\), dominates because of the central limit theorem: when many small independent disturbances add up, as they do in an analog front end, their sum tends toward a Gaussian regardless of the individual shapes. That is why thermal noise on a voltage rail is modeled as Gaussian, and why the Kalman filter of Chapter 9 can be optimal and cheap at the same time. The uniform distribution describes quantization error: an analog-to-digital converter with step \(q\) rounds the truth, and the leftover error is spread flat across \([-q/2, q/2]\), giving the famous quantization-noise variance \(q^2/12\). The Poisson distribution counts discrete arrivals, photons hitting a pixel, ticks on a Geiger counter, so a photon-limited camera or lidar return is Poisson, and its signal-dependent noise (variance equal to the mean) is why dark scenes look grainier. The Rayleigh distribution appears whenever you take the magnitude of two independent Gaussian components, which is exactly the envelope of a radar or RF signal; it is the reason Chapter 44 reasons about detection thresholds probabilistically. Finally, heavy-tailed distributions (Student-t, log-normal) model the rare large outliers that a pure Gaussian badly underestimates, such as a motion spike from a footstep on a wearable.
Two moments are a promise, not a description
Reporting a mean and a standard deviation is equivalent to asserting the data is Gaussian, because those two numbers pin down a Gaussian exactly and pin down nothing else exactly. When your sensor noise is genuinely Gaussian, that promise is free and everything downstream simplifies. When it is not, for example a bimodal reading from a sensor that flips between two states, or a heavy-tailed stream punctuated by spikes, the mean-and-variance summary quietly discards the very structure that matters for detection and safety. The discipline is to check the higher moments and the histogram before you commit to a two-number summary, not after a false alarm forces you to.
Moments: the compact fingerprint of a distribution
Moments are expectations of powers of the variable, and the first four carry almost all the intuition you need. The mean \(\mu = \mathbb{E}[X] = \int x\,p(x)\,dx\) is the center of mass, the value a long average converges to; for a sensor it is the true signal plus any constant bias. The variance \(\sigma^2 = \mathbb{E}[(X-\mu)^2]\) is the spread, and its square root, the standard deviation \(\sigma\), lives in the same units as the reading, which is why we quote "plus or minus \(\sigma\)." The skewness \(\gamma_1 = \mathbb{E}[(X-\mu)^3]/\sigma^3\) measures asymmetry: a positive skew means occasional large-positive excursions, the signature of a rectified or count-based signal. The kurtosis \(\gamma_2 = \mathbb{E}[(X-\mu)^4]/\sigma^4\) measures tail heaviness; a Gaussian sits at exactly 3, and anything meaningfully above it warns you that rare large events happen more often than a Gaussian would predict, which is a direct input to how you set alarm thresholds and to the outlier-robust filters of Chapter 6.
import numpy as np
from scipy import stats
# Simulated 100 Hz accelerometer: gravity bias + Gaussian noise + rare spikes
rng = np.random.default_rng(0)
n = 20_000
clean = 9.81 * np.ones(n) # true value (m/s^2)
noise = rng.normal(0.0, 0.05, n) # thermal jitter, sigma = 0.05
spikes = rng.binomial(1, 0.002, n) * rng.normal(0, 1.0, n) # rare footstep taps
x = clean + noise + spikes
print(f"mean = {x.mean():.4f}") # ~ 9.81, recovers the true value
print(f"std = {x.std():.4f}") # inflated above 0.05 by the spikes
print(f"skewness = {stats.skew(x):+.3f}") # near 0: spikes are symmetric
print(f"kurtosis = {stats.kurtosis(x):+.3f}") # >> 0 (excess): heavy tails, not Gaussian
The code above makes the earlier point concrete. Both a clean Gaussian stream and the spiked stream can report the same mean and a similar standard deviation, yet only the kurtosis exposes that one of them will occasionally throw a value that a Gaussian safety margin never budgeted for.
Right tool: distribution fitting in three lines
Coding a maximum-likelihood fit and moment estimator by hand is a page of numerics per distribution. SciPy collapses it: params = stats.t.fit(x) fits a heavy-tailed Student-t and returns the degrees of freedom, location, and scale in one call, and stats.probplot(x, dist="norm") produces the Q-Q plot that visually confirms non-normality. That is roughly 40 lines of hand-rolled optimization and quantile math reduced to 2, with the library handling numerical stability, edge cases, and a catalog of 100-plus distributions. Keep the from-scratch moment code above for understanding; reach for scipy.stats in production.
Signals are sequences of random variables: joints, stationarity, and correlation
A single sample is a random variable; a signal \(x[0], x[1], \dots\) is a whole family of them, a random process. What ties the samples together is the joint distribution, and the practical shortcut for describing it is the autocovariance \(C[k] = \mathbb{E}[(x[n]-\mu)(x[n+k]-\mu)]\), which measures how strongly a reading predicts one \(k\) steps later. Real sensor noise is rarely "white" (uncorrelated across time); a drifting gyroscope bias, for instance, produces samples that stay correlated for seconds. We usually assume wide-sense stationarity, meaning the mean and autocovariance do not change over time, because it lets us estimate those statistics by averaging along the stream instead of needing many parallel sensors. When two channels are involved, an accelerometer's three axes, the pairwise second moments assemble into a covariance matrix \(\Sigma\), whose off-diagonal entries reveal cross-axis coupling and whose structure feeds directly into the multivariate Gaussian that Chapter 9 propagates through time.
A wearable that learned to distrust its own tail
A fall-detection algorithm on a wrist wearable was tuned assuming Gaussian accelerometer noise, so it flagged any reading more than four standard deviations from the resting mean. In the lab it worked. In the field it fired dozens of false alarms a day, because ordinary activities, clapping, setting down a mug, a firm handshake, produced sharp taps that a Gaussian model rated as one-in-a-million events. Plotting the field histogram exposed excess kurtosis near 12, far from the Gaussian's 3. The fix was not a bigger model; it was a better distribution. The team refit the resting-state noise as a Student-t, whose heavy tails absorbed the everyday taps, and reserved the alarm for the genuinely extreme, sustained excursions that a real fall produces. False alarms dropped by an order of magnitude. The moral: the fourth moment was the whole story, and the two-moment summary had been hiding it.
Exercise: quantization noise from first principles
An ADC quantizes to steps of size \(q\), so the rounding error is uniform on \([-q/2, q/2]\). (a) Show from the definition of variance that the error has mean 0 and variance \(q^2/12\). (b) A 12-bit converter spans a 5 V range. Compute \(q\) and the resulting noise standard deviation in millivolts. (c) Your Gaussian sensor noise already has \(\sigma = 3\) mV. Using the fact that independent noise variances add, decide whether the quantization step is negligible or a real contributor, and state the rule of thumb this gives for choosing ADC resolution relative to the analog noise floor.
Self-check
- Why does reporting only a mean and standard deviation amount to an implicit claim that the data is Gaussian, and when is that claim safe?
- A pixel's photon count follows a Poisson distribution with mean \(\lambda\). What is its variance, and why does this make dark regions of an image noisier than bright ones in a relative sense?
- Two accelerometer axes have a large off-diagonal entry in their covariance matrix. What physical or mounting fact might explain it, and why can you not treat the axes as independent?
What's Next
In Section 4.2, we turn from describing distributions to inferring them from data. The moment estimates you computed above were guesses at unknown truths, and every guess has a bias and a variance of its own. We will formalize what makes an estimator good, expose the bias-variance tradeoff that governs every fit in this book, and derive maximum likelihood, the principled engine that turns a stream of samples into the best parameters for the distribution behind them.