Confidence-weighted combination

"I asked three sensors the same question, and instead of taking a vote I asked each how sure it was. The nervous one I let whisper; the confident one I let speak."

A Judiciously Weighting AI Agent

This section assumes you are comfortable with variance and the Gaussian from the probability and estimation primer of Chapter 4, and that you know what it means for a predicted probability to be calibrated, which is developed in full in Chapter 18. We deliberately stay at the level of combining scalar estimates and class scores. The full covariance machinery, where confidence becomes a matrix and cross-sensor correlations must be tracked, is the subject of Chapter 49; here confidence is a single number per sensor.

Inverse-variance weighting: the one formula

Suppose two sensors each report an unbiased estimate of the same true value \(x\): sensor \(i\) returns \(x_i\) with variance \(\sigma_i^2\). What single number best combines them? If the errors are independent and zero-mean, the minimum-variance unbiased estimate is the precision-weighted average, where precision is the reciprocal of variance:

The weight is precision, not variance, and the intuition is exactly right: a sensor you trust (small \(\sigma_i^2\)) gets a large weight, a sensor you distrust gets a small one, and a useless sensor with \(\sigma_i^2 \to \infty\) gets weight zero and quietly drops out. That last property is the graceful-degradation mechanism behind missing-modality fusion in Section 48.4: a dropped sensor is just one whose confidence went to zero. The reward for weighting correctly is a fused variance that is smaller than either input:

Precisions add. Two sensors of equal variance \(\sigma^2\) fuse to variance \(\sigma^2/2\), the familiar factor-of-two from averaging. But two unequal sensors do better than either alone only because the weighting respected their variances; a plain mean of a good and a bad sensor lands between them and can be worse than the good one by itself.

This same estimator is the measurement-update heart of the Kalman filter in Chapter 9: the Kalman gain is precisely the fraction of the correction assigned to the measurement rather than the prediction, computed from their relative variances. If you understand inverse-variance fusion, you already understand the scalar Kalman update.

The same idea for classifiers: softmax and log-opinion pools

Continuous estimates weight by precision. Classifiers weight by confidence too, but the arithmetic moves into log space. Given per-class probability vectors \(p^{(i)}\) from several models or sensors, two pooling rules dominate. The linear opinion pool averages the probabilities, \(\bar{p} = \sum_i \alpha_i\, p^{(i)}\); it is forgiving and keeps a class alive if any sensor supports it. The logarithmic opinion pool averages the log-probabilities and renormalizes, which multiplies the distributions and behaves like a soft AND: a class survives only if every weighted sensor grants it non-trivial mass. The weights \(\alpha_i\) are again per-sensor confidences, and setting them from a validation set is the classifier analogue of estimating \(\sigma_i^2\).

The subtlety unique to classifiers is where the confidence hides. A softmax already emits a number that looks like confidence, the top-class probability, but a raw neural network softmax is systematically overconfident. Temperature scaling divides the logits by a scalar \(T>1\) learned on held-out data, which relaxes the distribution toward its true reliability without changing which class wins. In fusion this is not cosmetic: an uncalibrated, spiky softmax hands one sensor an enormous log-pool weight it did not earn, and the pool follows it off a cliff. Calibrate first, then pool.

Static reliability versus per-sample confidence

There are two clocks on which a weight can move. A static reliability weight is fixed per sensor: the microphone is generally cleaner than the contact sensor, so it always gets more say. You estimate it once, offline, from each channel's error on a validation set, and it never changes at run time. It is cheap, robust, and the right default when a sensor's quality is stable.

A per-sample confidence moves with the current input: the same camera is trustworthy in daylight and near-useless in a tunnel, and its weight should crater the instant it enters the dark. This is strictly more powerful and strictly more dangerous, because now the confidence itself is an estimate that can be wrong at exactly the wrong moment. Sources of a live per-sample confidence include a signal-quality index (SNR, saturation flags, contact impedance), a model's own predicted variance from a heteroscedastic head, or an out-of-distribution score. When those live signals feed neural fusion, the weighting becomes a learned gate or attention over modalities, the mechanism developed in depth for deep multimodal fusion in Chapter 50. The gate is confidence-weighted combination with the weights produced by a small learned network rather than a variance formula.

When confidence weighting betrays you

The failure mode worth memorizing is the confidently wrong sensor. Inverse-variance weighting assumes each sensor is unbiased and that its reported variance is its true variance. Break either assumption and the formula amplifies the damage. A stuck sensor that reports a constant value with a tiny (and false) variance receives an enormous weight and drags the fused estimate to its stuck value; the honest sensors are outvoted by a broken one shouting. This is why serious fusion pairs weighting with a consistency or innovation check that detects a sensor whose reading is statistically incompatible with the others and zeroes its weight, a fault-detection loop covered in Chapter 49. The second betrayal is correlation: the additive-precision result assumes independent errors. Two cameras behind the same fogged windshield share a failure, so counting them as two independent votes overstates confidence exactly when both are wrong together.

import numpy as np
x   = np.array([72.0, 68.0, 75.0])      # three heart-rate estimates (bpm)
var = np.array([4.0, 25.0, 100.0])      # each sensor's error variance
w   = 1.0 / var
fused      = np.average(x, weights=w)                 # precision-weighted mean
fused_var  = 1.0 / w.sum()                            # fused variance
print(f"{fused:.2f} bpm  +/- {np.sqrt(fused_var):.2f}")

The short program below makes the payoff and the betrayal concrete in one run: it fuses honest sensors, then re-runs with one sensor lying about its variance so you can watch the fused estimate get captured.

import numpy as np

def fuse(x, var):
    w = 1.0 / np.asarray(var)
    return np.average(x, weights=w), 1.0 / w.sum()

true = 70.0
x = np.array([70.5, 69.2, 71.1])          # three roughly-unbiased readings
honest = np.array([4.0, 9.0, 16.0])
print("honest :", np.round(fuse(x, honest), 3))     # near 70, tight variance

# sensor 3 is stuck at 55 but claims near-zero variance
x_bad   = np.array([70.5, 69.2, 55.0])
lying   = np.array([4.0, 9.0, 0.05])
print("lying  :", np.round(fuse(x_bad, lying), 3))  # captured, dragged to ~55

What's Next

In Section 48.6, we turn from building fused estimators to judging them: how to evaluate a fused system so that its headline accuracy is not an artifact of a lucky sensor, how to measure whether fusion actually beats the best single modality, and how to keep the evaluation leakage-safe when several correlated streams share a subject. The confidence weights you learned to trust here become one of the things that evaluation must probe, because a system that reports honest uncertainty is worth more than one that is merely accurate on average.