Bayesian inference and priors

"You gave me three noisy readings and asked what I believe. I already believed something before the first one arrived; the readings only told me how much to change my mind."

A Well-Calibrated AI Agent

This section assumes you are comfortable with the random variables, densities, and expectations from Section 4.1 and with the likelihood and maximum-likelihood estimation (MLE) developed in Section 4.2. Where 4.2 asked "which parameter best explains the data," this section asks the fuller question: "given the data and what I already knew, what is my full belief about the parameter, uncertainty included." We keep the book's notation: a hidden quantity of interest is \(\theta\) (a sensor bias, a state, a heart rate), the observed measurements are \(x\), and the measurement model that links them was introduced as \(x = h(\theta) + \eta\) in Chapter 2.

Bayes' theorem as a belief-update rule

Bayes' theorem is one line of algebra that follows directly from the definition of conditional probability:

Read it right to left. The prior \(p(\theta)\) is what you believe about \(\theta\) before this measurement. The likelihood \(p(x \mid \theta)\), the same object you maximized in 4.2, scores how well each candidate \(\theta\) explains the data \(x\). Their product, renormalized by the evidence \(p(x)=\int p(x\mid\theta)p(\theta)\,d\theta\), is the posterior \(p(\theta\mid x)\): your updated belief. The proportionality on the right is what you actually compute with, because the evidence is just the constant that makes the posterior integrate to one.

What makes this more than a formula is that the posterior is a full distribution, not a point. It carries both a best guess and a calibrated spread, which is why Bayesian inference is the natural home for the aleatoric-versus-epistemic split that Section 4.4 makes central: the likelihood width encodes irreducible sensor noise, while the posterior width over \(\theta\) encodes how much the data has (or has not) pinned down the unknown.

Priors: what to encode, why, and how much they cost

A prior encodes structure you know before seeing this particular data: physical bounds (a probability lies in \([0,1]\), a distance is non-negative), physics (a vehicle's acceleration is bounded, so its state cannot jump), and history (last year's calibration, the population distribution of resting heart rates). The practical question is always how strong to make it. A prior's strength is measured in pseudo-observations: a Gaussian prior with variance \(\tau^2\) on a quantity measured with per-sample noise \(\sigma^2\) is worth about \(\sigma^2/\tau^2\) real measurements. Make it too weak and you throw away hard-won knowledge, forcing the sensor to relearn physics from every cold start. Make it too strong and the prior overrides evidence the data was trying to deliver, so a genuinely fast heart rate gets dragged back toward 60.

The mechanically convenient case is a conjugate prior: a prior whose functional form is preserved by the update, so the posterior is the same family with updated parameters and no integral to evaluate. Two pairs cover most of what a sensing engineer needs. For a Gaussian measurement with known noise variance, a Gaussian prior on the mean yields a Gaussian posterior. For a binary event counted over trials (a fall detector firing, a beat classified as ectopic), a Beta prior on the success probability plus Binomial counts yields a Beta posterior. The Gaussian-Gaussian case is precisely the update rule that, run recursively over time, becomes the Kalman filter of Chapter 9.

The Gaussian-Gaussian posterior mean is worth memorizing, because it shows exactly how prior and data trade off. With prior \(\theta\sim\mathcal{N}(\mu_0,\tau^2)\) and \(n\) readings of mean \(\bar{x}\) at noise \(\sigma^2\):

The posterior mean is a precision-weighted average of the prior mean and the data mean, and precisions (inverse variances) simply add. As \(n\) grows, the data term dominates and the posterior converges to the MLE, which is why a well-chosen prior helps most exactly when data is scarce and fades gracefully when data is plentiful.

The following snippet implements the conjugate update directly, so you can watch the posterior tighten as readings arrive.

import numpy as np

def gaussian_update(mu0, tau2, readings, sigma2):
    """Sequential Gaussian-Gaussian posterior for a scalar mean."""
    mu, prec = mu0, 1.0 / tau2          # track mean and precision (1/variance)
    for x in readings:
        prec_new = prec + 1.0 / sigma2  # precisions add
        mu = (prec * mu + (1.0 / sigma2) * x) / prec_new
        prec = prec_new
    return mu, 1.0 / prec               # posterior mean, posterior variance

mu, var = gaussian_update(125.0, 20.0**2, [132, 119, 141], 12.0**2)
print(f"posterior mean {mu:.1f} bpm, std {np.sqrt(var):.1f} bpm")
# posterior mean 130.1 bpm, std 6.5 bpm

import pymc as pm

with pm.Model() as m:
    theta = pm.Normal("theta", mu=125.0, sigma=20.0)     # the prior
    pm.Normal("obs", mu=theta, sigma=12.0,               # the likelihood
              observed=[132, 119, 141])
    idata = pm.sample(1000, progressbar=False)           # posterior samples

Reporting a belief: MAP, posterior mean, and credible intervals

A posterior is a distribution, but a device often has to emit one number. Three summaries dominate. The maximum a posteriori (MAP) estimate is the posterior mode, \(\arg\max_\theta p(\theta\mid x)\); it is MLE with the prior's regularizing pull, and for a Gaussian posterior it coincides with the mean. The posterior mean minimizes expected squared error and is the standard choice when the cost of being wrong grows with the square of the error. A credible interval, for example the central region containing 95 percent of posterior mass, is the honest uncertainty report, and unlike a frequentist confidence interval it means exactly what a newcomer expects: given the data and prior, there is a 95 percent probability the parameter lies inside it.

MAP also exposes the bridge to everyday deep learning. Maximizing \(\log p(x\mid\theta) + \log p(\theta)\) is a data-fit term plus a penalty term, and a zero-mean Gaussian prior makes that penalty a sum of squares. That is L2 weight regularization, and a Laplace (double-exponential) prior is L1. Every time a model is trained with weight decay, a prior is being asserted, whether or not anyone says so. This is the same identity the calibration and uncertainty methods of Chapter 18 build on.

When to reach for Bayes, and when not to

Go Bayesian when data is scarce relative to the number of unknowns, when you genuinely hold prior knowledge worth encoding (physics, calibration history, population statistics), and above all when you must report calibrated uncertainty rather than a bare point estimate, because a device that acts on its own confidence needs that confidence to be honest. The cost is real: you must choose a prior and, outside conjugate cases, pay for approximate inference. When data is abundant and a point estimate suffices, MLE from Section 4.2 is cheaper and the prior washes out anyway. The recursive form of the Gaussian update seen here is the engine of the entire state-estimation part of this book, and the multi-sensor generalization, where each sensor contributes a likelihood and priors propagate between them, is the subject of Chapter 49 on probabilistic fusion.

What's Next

In Section 4.4, we split the uncertainty this section quantified into its two irreducible kinds: aleatoric uncertainty, the noise the world will never let you remove, and epistemic uncertainty, the ignorance that more data can. The posterior width you just learned to compute is where that distinction becomes operational, and it is the thread that runs through every calibration, fusion, and safety chapter that follows.