Tuning process and measurement noise

"Every Kalman filter I have ever met was correct. Most of them were also lying, because someone guessed Q and R over lunch and never checked."

A Skeptical AI Agent

Recall the notation remap flagged back in Chapter 1: throughout this chapter the state is \(\mathbf{x}\) and the measurement is \(\mathbf{z}\), following sixty years of control-engineering tradition. The prediction step propagates the state through the dynamics \(\mathbf{x}_k = \mathbf{F}\mathbf{x}_{k-1} + \mathbf{w}_k\) with process noise \(\mathbf{w}_k \sim \mathcal{N}(\mathbf{0}, \mathbf{Q})\), and the measurement arrives as \(\mathbf{z}_k = \mathbf{H}\mathbf{x}_k + \mathbf{v}_k\) with measurement noise \(\mathbf{v}_k \sim \mathcal{N}(\mathbf{0}, \mathbf{R})\). You need only comfort with covariance and the Gaussian; if that feels shaky, Chapter 4 is the place to firm it up.

What Q and R actually represent

What they are. \(\mathbf{R}\) is the covariance of the sensor's noise: how much each raw measurement scatters around the true value it is reporting. \(\mathbf{Q}\) is the covariance of the process noise: how much the real state wanders away from what your dynamics model predicts in one time step. They answer two different questions. \(\mathbf{R}\) asks "how badly does my sensor lie?" and \(\mathbf{Q}\) asks "how wrong is my model of the world?"

Why they are not symmetric in difficulty. \(\mathbf{R}\) is a property of hardware you can measure on a bench. \(\mathbf{Q}\) is a property of a model you invented, so it is a fudge factor by construction. A constant-velocity tracker applied to a pedestrian who accelerates, turns, and stops has a dynamics model that is wrong in a way no datasheet describes. \(\mathbf{Q}\) is where you confess that wrongness. Setting \(\mathbf{Q}\) too small tells the filter "my model is nearly perfect," so it trusts its prediction, ignores fresh measurements, and lags every real maneuver. Setting it too large tells the filter "my model is worthless," so it chases each noisy sample and the estimate rattles. Neither failure raises an exception; the filter runs happily while being quietly useless.

The gain feels only the ratio

To see why the ratio is what matters, look at the scalar case. With prior variance \(P^-\), measurement variance \(R\), and \(H = 1\), the Kalman gain is

When \(P^- \gg R\) the gain approaches 1: the filter throws away its prediction and adopts the measurement. When \(P^- \ll R\) the gain approaches 0: the filter ignores the measurement and coasts on its model. And \(P^-\) grows by exactly \(Q\) at every predict step (\(P^- = P + Q\) in the scalar constant-position case), so \(Q\) is the throttle that keeps the filter listening to its sensor. A tiny \(Q\) lets \(P^-\) collapse toward zero, the gain follows, and the filter goes deaf. This is the single most common Kalman pathology in the field, and it never announces itself. The estimate simply drifts away from a slow-moving truth while the reported covariance shrinks with false confidence.

Set R from the sensor, Q from the motion

How to set \(\mathbf{R}\). Measure it. Hold the sensor still (or against a known reference), collect a few thousand samples, and compute the sample covariance of the residual from the known value. For many sensors the datasheet already gives you a noise density or an RMS figure you can square into a variance; grounding \(\mathbf{R}\) in the physics of the transducer is exactly the material of Chapter 2. Two cautions. First, datasheet noise is often quoted under ideal conditions; real \(\mathbf{R}\) inflates with temperature, motion, and aging, so measure it in your deployment regime. Second, \(\mathbf{R}\) is a covariance, not a variance: if two measurement channels share a common error source, the off-diagonal terms are real and dropping them costs you accuracy.

How to set \(\mathbf{Q}\). There is no bench for \(\mathbf{Q}\), so anchor it to physics instead of taste. For a constant-velocity model driven by unknown acceleration, the standard construction treats acceleration as white noise with intensity \(\sigma_a^2\) and integrates it into the state, giving the continuous-white-noise-acceleration \(\mathbf{Q}\)

where \(\Delta t\) is the sample interval and \(\sigma_a\) is the biggest acceleration you expect the target to sneak in between updates. Now \(\mathbf{Q}\) has one interpretable knob, \(\sigma_a\), that you can reason about physically: a tracked pedestrian might peak near \(1\ \text{m/s}^2\), an aggressive drone near \(20\). This turns a matrix of mystery numbers into a single question with a real-world answer. Chapter 23 revisits exactly this construction for inertial motion models.

Let the innovations grade your tuning

The elegant part of Kalman tuning is that the filter checks its own homework. At each step it predicts a measurement and the actual one arrives; the difference is the innovation \(\boldsymbol{\nu}_k = \mathbf{z}_k - \mathbf{H}\hat{\mathbf{x}}_k^-\), and the filter also predicts that innovation's covariance \(\mathbf{S}_k = \mathbf{H}\mathbf{P}_k^-\mathbf{H}^\top + \mathbf{R}\). If \(\mathbf{Q}\) and \(\mathbf{R}\) are right, the innovations are white (uncorrelated in time) and their squared, normalized magnitude, the normalized innovation squared (NIS) \(\boldsymbol{\nu}_k^\top \mathbf{S}_k^{-1} \boldsymbol{\nu}_k\), follows a chi-square distribution with degrees of freedom equal to the measurement dimension. Averaged over a run it should sit near that dimension. If the average NIS runs high, the filter is overconfident: \(\mathbf{Q}\) or \(\mathbf{R}\) is too small. If it runs low, the filter is underconfident and you are leaving accuracy on the table. This is a rigorous, quantitative consistency test, not a look-at-the-plot heuristic, and it is the same residual-whitening idea that underlies the diagnostics in Section 9.7.

import numpy as np
from scipy.stats import chi2

def average_nis(innovations, S_list):
    """Mean normalized innovation squared over a run.
    innovations: list of (m,) arrays; S_list: list of (m,m) innovation covariances."""
    nis = [float(nu @ np.linalg.solve(S, nu)) for nu, S in zip(innovations, S_list)]
    return np.mean(nis)

# One-dimensional position measurements (m = 1): NIS should average ~1.
m, N = 1, 500
mean_nis = average_nis(innovations, S_list)      # from your filter run
lo, hi = chi2.ppf([0.025, 0.975], df=m*N) / N     # 95% consistency band
verdict = ("OVERCONFIDENT: raise Q or R" if mean_nis > hi else
           "UNDERCONFIDENT: lower Q or R" if mean_nis < lo else
           "CONSISTENT")
print(f"mean NIS = {mean_nis:.3f}  band = [{lo:.3f}, {hi:.3f}]  -> {verdict}")

The companion test, when you have ground truth (in simulation, or from a motion-capture reference rig), is the normalized estimation error squared (NEES), which normalizes the true estimation error by the filter's reported state covariance \(\mathbf{P}_k\). NEES catches an overconfident \(\mathbf{P}\) that NIS can miss, so use NEES offline during development and NIS online where truth is unavailable. Both are consumers of the same principle: a well-tuned filter's own uncertainty claims must match the errors it actually commits, and calibration of that kind is a theme this book returns to in Chapter 18.

from filterpy.kalman import KalmanFilter
from filterpy.stats import NEES
import numpy as np

kf = KalmanFilter(dim_x=2, dim_z=1)   # set F, H, Q, R, x, P as usual
errs, ps = [], []
for z, x_true in zip(measurements, truth):
    kf.predict(); kf.update(z)
    errs.append(x_true - kf.x); ps.append(kf.P)
    # kf.y is the innovation, kf.S its covariance -> NIS = kf.y.T @ inv(kf.S) @ kf.y
print("mean NEES:", np.mean(NEES(errs, ps)))     # compare to dim_x = 2

When to automate: adaptive and data-driven tuning

Manual NIS sweeps are the right tool when the noise is roughly stationary and you can tune once offline. When it is not, three escalating options exist. Adaptive filtering adjusts \(\mathbf{Q}\) or \(\mathbf{R}\) online from a sliding window of innovations, useful when a sensor's noise changes with regime (GNSS in a tunnel, a camera in glare). Autocovariance least squares (ALS) is an offline method that estimates \(\mathbf{Q}\) and \(\mathbf{R}\) jointly from the autocorrelation of the innovation sequence on logged data, giving a principled fit rather than a hand sweep. Maximum-likelihood tuning treats \(\mathbf{Q}\) and \(\mathbf{R}\) as parameters and maximizes the filter's log-likelihood over a batch, the most rigorous route when you have representative logs. Reach for automation only after the manual NIS test has told you your model structure is sound; an adaptive scheme wrapped around a misspecified dynamics model will chase the wrong target faster. The nonlinear filters of Chapter 10 inherit every one of these tuning concerns, with the added twist that \(\mathbf{Q}\) must also absorb linearization error.

What's Next

In Section 9.5, we move from tuning to the failure that no amount of tuning can rescue: when the sensor geometry simply does not constrain part of the state. We formalize observability, show how an unobservable mode makes a covariance quietly blow up, and connect a diverging filter back to the consistency diagnostics you just learned, so you can tell a tuning problem from a structural one before it drives off the curb.