Consistency checks and fault detection

"A filter that never doubts itself is not confident. It is broken, and it will take you down with it."

A Self-Auditing AI Agent

The innovation is the filter's confession

The one quantity you can check at every timestep is the innovation \(\nu_k = z_k - \hat{z}_k\), the gap between the measurement that arrived and the measurement the filter predicted. The what: the innovation is the only place where the model meets reality before it is absorbed, so it carries every discrepancy the filter has not yet explained away. The why it is checkable is that a correctly modeled Gaussian filter makes a hard, testable promise about it. The innovation should be zero-mean, and its spread should match the innovation covariance \(S_k = H_k P_k^- H_k^\top + R_k\), which the filter computes anyway during the update: \(H_k P_k^- H_k^\top\) is the predicted-state uncertainty seen through the measurement, and \(R_k\) is the sensor noise. If the real innovations are consistently larger than \(S_k\) allows, the filter is overconfident (its covariance is a lie, the dangerous direction); consistently smaller and it is needlessly conservative.

To make "larger than \(S_k\) allows" precise, collapse the vector innovation to a scalar with the Mahalanobis form from Section 49.1, giving the Normalized Innovation Squared (NIS):

The how of the test is a theorem: if the filter is consistent and the true noise is Gaussian, \(\epsilon_k\) follows a chi-square distribution with \(m\) degrees of freedom, where \(m\) is the measurement dimension. That distribution has a known mean (\(m\)) and known tail quantiles, so a single measurement gives you a calibrated yes/no: does this reading fall inside the ellipsoid my covariance drew for it?

Gating: reject the outlier before it poisons the estimate

The first job of the NIS statistic is defensive. Before folding a measurement into the state, test it: if \(\epsilon_k\) exceeds a threshold \(\gamma\) drawn from the chi-square inverse CDF at a chosen confidence (say the 99th percentile), the reading is gated out and the update is skipped for that sensor this step. This is the same validation gate that guards data association in Section 49.4, used here for fault rejection rather than track assignment. The why gating matters: a single wild measurement (a multipath GPS jump, a lidar return off a mirror, a wireless outlier) enters the Kalman update multiplied by the gain, and because the filter believes it, the bad value drags the mean and, worse, shrinks the covariance around the wrong place. One ungated outlier can corrupt an estimate for many subsequent steps. The gate is a chi-square bouncer at the door.

Gating on a single sample, though, is a coin flip against slow faults. A sensor that drifts by a few millimeters per second produces innovations that each sit comfortably inside the gate while their sum marches steadily off. The fix is to accumulate. Averaging NIS over a sliding window of \(N\) steps gives a statistic distributed as chi-square with \(mN\) degrees of freedom (scaled), which tightens the band around the mean and exposes a persistent bias that any single sample would hide. For a step change of unknown onset, the sequential tools of Chapter 12, CUSUM and the generalized likelihood ratio, run directly on the innovation sequence and detect the change far faster than a fixed-window average, at the price of a tuned false-alarm rate.

import numpy as np
from scipy.stats import chi2

def nis(nu, S):
    """Normalized Innovation Squared for one measurement update."""
    return float(nu.T @ np.linalg.solve(S, nu))

m = 2                                    # measurement dimension (e.g. x, y)
gate = chi2.ppf(0.99, df=m)              # single-step 99% gate ~ 9.21

rng = np.random.default_rng(0)
S = np.array([[0.20, 0.05], [0.05, 0.30]])   # innovation covariance from the filter
L = np.linalg.cholesky(S)

# Healthy stream: innovations really are N(0, S)
healthy = [nis(L @ rng.standard_normal(m), S) for _ in range(500)]

# Faulted stream: sensor develops a slow bias after step 250
faulted, bias = [], np.zeros(m)
for k in range(500):
    if k > 250:
        bias += np.array([0.004, 0.0])       # creeping drift, invisible per-sample
    faulted.append(nis(L @ rng.standard_normal(m) + bias, S))

print(f"healthy mean NIS  {np.mean(healthy):.2f}  (expect ~{m})")
print(f"healthy gate viol {np.mean(np.array(healthy) > gate)*100:.1f}%  (expect ~1%)")
print(f"faulted mean NIS  {np.mean(faulted):.2f}")
# A 20-step moving average crosses the fault long before any single sample trips the gate
mov = np.convolve(faulted, np.ones(20)/20, mode="valid")
alarm = np.argmax(mov > chi2.ppf(0.99, df=m*20)/20) + 20
print(f"windowed alarm at step {alarm}  (true onset 250)")

As Listing 49.7 shows, the healthy filter certifies its own honesty (mean NIS at \(m\), violations near the design rate), while the same statistic, accumulated, surfaces a creeping bias that no individual reading betrays. That gap between per-sample and windowed detection is exactly the tradeoff you tune for your latency and false-alarm budget.

From detection to isolation: which sensor lied?

Detecting that something is wrong is half the job; a safety case needs to know which source failed so the estimator can drop it and keep running. This is fault detection and isolation (FDI), and Bayesian fusion offers two complementary handles. The first is analytical redundancy: run a bank of filters, each excluding one sensor, and compare their innovation histories. If dropping sensor \(j\) restores every other channel's NIS to its healthy band, sensor \(j\) is the culprit. This is the residual-structuring idea behind the receiver autonomous integrity monitoring (RAIM) used in aviation GNSS, generalized to arbitrary sensor sets. The second handle is hardware redundancy read through the same statistic: with three or more sources measuring one quantity, a per-channel innovation cross-check performs a probabilistic majority vote, outvoting the deviant even when you cannot say a priori which one it is.

The when to reach for which: analytical redundancy is cheap in sensors but costly in computation and modeling (you must trust every excluded-filter model), so it fits mass-market systems that cannot afford triplicated hardware. Hardware redundancy is expensive in parts but robust and simple to certify, so it dominates aerospace and medical devices where a missed fault is catastrophic. Most serious stacks blend the two, and both ultimately rest on the same claim this section has built: a residual that violates its own predicted covariance is evidence, and the pattern of which residuals violate is the diagnosis.

from filterpy.kalman import KalmanFilter
from filterpy.stats import mahalanobis
import scipy.stats as st

kf = KalmanFilter(dim_x=4, dim_z=2)      # set F, H, Q, R, P as usual
kf.predict()
d2 = mahalanobis(z, kf.H @ kf.x, kf.S) ** 2      # == NIS, S computed by the filter
if d2 <= st.chi2.ppf(0.99, df=2):
    kf.update(z)                          # accept: consistent measurement
# else: gate out this reading, coast on the prediction

What's Next

In Chapter 50, we leave the Gaussian world where consistency has a closed-form chi-square test and enter deep multimodal fusion, where a network blends camera, lidar, and audio through learned encoders. The auditing question sharpens rather than disappears: when a modality drops out or turns adversarial, how does a neural fusion model notice, and how does it degrade gracefully instead of hallucinating confidence? The consistency instinct you built here, comparing predicted uncertainty against realized error, becomes the design principle for missing-modality robustness.