Failure modes and diagnostics

"A Kalman filter never tells you it is wrong. It tells you, with great confidence, exactly the wrong thing. The innovation sequence is the only witness that will testify against it."

A Skeptical AI Agent

This section assumes you are comfortable with the linear Kalman recursion and its covariance bookkeeping from Sections 9.3 and the noise-tuning intuition from Section 9.4. The consistency tests below are chi-square arguments, so the Gaussian-and-quadratic-form machinery from Chapter 4 is the prerequisite you will lean on hardest.

The innovation sequence: your filter's built-in witness

At every measurement update the filter computes the innovation (measurement residual): the gap between what it predicted the sensor would read and what the sensor actually reported.

$$\nu_k = z_k - H_k \hat{x}_{k|k-1}, \qquad S_k = H_k P_{k|k-1} H_k^\top + R_k$$

Here \(\nu_k\) is the innovation and \(S_k\) is its predicted covariance. The theory makes a sharp, testable claim: if the model, the noise covariances, and the arithmetic are all correct, the innovation sequence is zero-mean, white, and has exactly covariance \(S_k\). That is not a soft heuristic; it is a structural consequence of the Kalman gain being optimal. A well-tuned filter has extracted all the predictable structure from the data, so what is left over must be unpredictable (white) noise. Any departure from whiteness means predictable signal is still sitting in the residuals, which means the filter is leaving information on the table or, worse, has the wrong model.

This is what makes \(\nu_k\) the single most valuable diagnostic you own: it is computed anyway, on every step, with no ground truth required. You do not need to know the true state to check that your residuals look like the noise they are supposed to be. Three cheap checks fall out immediately. A running mean of \(\nu_k\) that drifts away from zero signals a bias, often an uncalibrated sensor offset or an unmodeled acceleration. An autocorrelation at nonzero lag signals that the dynamics model is too simple to explain the motion. And a sample variance that persistently exceeds \(S_k\) signals that your covariances are optimistic.

Consistency tests: NIS and NES made quantitative

To turn "looks white" into a threshold an alarm can fire on, we normalize the innovation by its own predicted covariance. The Normalized Innovation Squared (NIS) is

$$\epsilon_{\nu,k} = \nu_k^\top S_k^{-1} \nu_k.$$

Under the correct model, \(\epsilon_{\nu,k}\) is a chi-square random variable with \(m\) degrees of freedom, where \(m\) is the measurement dimension. That gives you a calibrated confidence interval with no tuning: for a scalar sensor, \(\epsilon_{\nu,k}\) should land inside \([0.001, 5.0]\) about 95% of the time, and its time average should sit near \(m\). If NIS runs consistently high, the filter is overconfident: it believes its predictions more than the data warrant, so \(R\) or \(Q\) is too small. If NIS runs consistently low, the filter is underconfident and wasting information. NIS needs no ground truth, so it is the test you run in production.

When you do have ground truth, in simulation, on a motion-capture rig, or from a survey-grade reference, use the Normalized Estimation Error Squared (NEES), \(\bar{\epsilon}_{x,k} = \tilde{x}_k^\top P_{k|k}^{-1} \tilde{x}_k\) with \(\tilde{x}_k = x_k - \hat{x}_{k|k}\). NEES is chi-square with \(n\) degrees of freedom (\(n\) the state dimension) and directly answers the question NIS cannot: is the reported covariance \(P\) an honest description of the actual error? A filter can pass NIS and still fail NEES if the innovations look fine but the state uncertainty is miscalibrated. Run NEES on your bench before deployment; run NIS forever after. Both feed the same downstream story as the calibration diagnostics you will meet in Chapter 18: a number is only useful if its stated uncertainty is trustworthy.

import numpy as np
from scipy.stats import chi2

def nis_monitor(innovations, S_list, alpha=0.05):
    """Flag steps where NIS leaves the chi-square consistency band."""
    m = innovations[0].shape[0]              # measurement dimension
    lo, hi = chi2.ppf(alpha/2, m), chi2.ppf(1 - alpha/2, m)
    nis = np.array([nu @ np.linalg.solve(S, nu)
                    for nu, S in zip(innovations, S_list)])
    flags = (nis < lo) | (nis > hi)
    print(f"mean NIS = {nis.mean():.2f} (expected {m}); "
          f"{100*flags.mean():.1f}% of steps outside [{lo:.2f}, {hi:.2f}]")
    return nis, flags                        # sustained high mean -> overconfident filter

The nis_monitor above is deliberately stateless so you can drop it into a replay harness. In a live system you would keep a sliding window of NIS values and alarm on the windowed average, which smooths out the single-sample chi-square noise while still catching a sustained drift within a few seconds.

The failure taxonomy: divergence, collapse, and numerical breakdown

Three failure families cover almost everything you will see in the field, and each has a signature in the diagnostics above.

Divergence is when the estimate walks away from the truth while the covariance stays small. It comes from model mismatch: a constant-velocity filter tracking a turning target, or an unmodeled bias. The signature is unmistakable in the residuals, NIS climbs and stays high, and the innovation mean drifts, even as \(P\) reports serene confidence. This is the failure covered structurally in Section 9.5; here the point is that NIS detects it. Covariance collapse is the overconfidence endgame: after many consistent measurements \(P\) shrinks so far that the gain \(K_k \to 0\) and the filter stops listening to its sensors. New measurements, even correct ones, no longer move the estimate, so a later disturbance goes untracked. The classic cure is a floor on \(P\) or a fading-memory factor that gently inflates the prediction covariance so the filter never fully stops learning. Numerical breakdown is arithmetic, not statistics: the standard covariance update \(P_{k|k} = (I - K_k H_k) P_{k|k-1}\) is not guaranteed to stay symmetric or positive definite in finite precision, and once \(P\) loses positive-definiteness the whole recursion produces nonsense (negative variances, NaNs).

Two robustness upgrades handle the last two families almost for free. The Joseph-form update, \(P_{k|k} = (I - K_k H_k)P_{k|k-1}(I - K_k H_k)^\top + K_k R_k K_k^\top\), is algebraically identical to the short form but preserves symmetry and positive-definiteness because it is a sum of two positive-semidefinite terms. For ill-conditioned problems, propagate a factor of \(P\) instead of \(P\) itself (square-root or UD filtering), which doubles the effective numerical precision. Neither changes the estimate when everything is well conditioned; both keep the filter alive when it is not.

From diagnostics to fixes: gating and the robustness playbook

The same NIS quantity that flags model mismatch also defends the filter against bad measurements. Outliers, an optical multipath return, a GNSS reflection off a building, a mis-associated radar detection, arrive as innovations with huge NIS. Gating exploits this: before applying an update, test whether \(\nu_k^\top S_k^{-1}\nu_k\) exceeds a chi-square threshold (say the 99% quantile), and if it does, reject or down-weight the measurement rather than let it yank the state. This Mahalanobis gate is the workhorse of data association in multi-target tracking and the entry point to the robust and adaptive filters of Chapter 10. A word of caution: gating and divergence look alike through the same lens. If nearly every measurement gets gated out, the problem is not a run of outliers; it is that your filter has diverged and is rejecting correct data because its prediction is wrong. A gate that rejects more than a few percent of measurements is itself an alarm.

Where does this leave the diagnostic loop? Monitor NIS in production; run NEES on the bench; gate on NIS to survive outliers; reach for Joseph form and square-root propagation to survive the arithmetic; add fading memory to survive collapse. Persistent inconsistency that none of these fix is the honest signal that your model is wrong, which is the doorway to the nonlinear and particle methods ahead. The residual-based reasoning here is the same logic that underpins classical change and anomaly detection in Chapter 12 and the spoofing defenses in Chapter 68: a residual that stops looking like noise is always trying to tell you something.

What's Next

In Chapter 10, we confront the failure this section could only diagnose: a model that is wrong because the world is nonlinear. When persistent NIS inconsistency traces back to curved dynamics rather than mistuned noise, the fix is not a bigger \(Q\) but a better filter. The Extended and Unscented Kalman filters linearize the problem, particle filters abandon the Gaussian assumption entirely, and the diagnostics you just learned become the yardstick for choosing among them.