Section 4.5: Information, entropy, and mutual information

"A measurement is worth taking only if it surprises me. I priced the surprise, in bits, before I bothered to look."
A Parsimonious AI Agent

The Big Picture

Every sensing decision is really a question about surprise. How uncertain am I right now? How much of that uncertainty would this next reading remove? Which of three candidate sensors would remove the most? Information theory answers all three with a single currency: the bit. Entropy measures how much doubt a distribution holds. Cross-entropy and relative entropy (KL divergence) measure how badly one distribution stands in for another, which is exactly the loss most classifiers minimize. Mutual information measures how much knowing one variable tells you about another, which is exactly what a measurement is supposed to do. This section builds those three quantities from one definition and shows why they are the right yardstick for sensor selection, feature ranking, and deciding what to measure next.

This section assumes the random-variable and expectation language of Section 4.1 and builds directly on the aleatoric-versus-epistemic split of Section 4.4: entropy is how we will put a number on "how much doubt," and mutual information is how we will price the reduction. If you want the physical reason a sensor channel has finite capacity in the first place, the noise floors derived in Chapter 2 are where the bits run out.

Entropy: doubt measured in bits

Start with a single surprising event. If an outcome has probability $p$, its self-information (or surprisal) is $-\log_2 p$ bits. A certain event ($p=1$) carries zero bits: no surprise, no news. A one-in-a-million event carries about 20 bits. The logarithm is the whole trick, because it makes information from independent events add. Shannon entropy is just the expected surprisal of a distribution:

$$H(X) = -\sum_{x} p(x)\,\log_2 p(x) = \mathbb{E}\big[-\log_2 p(X)\big].$$

Entropy is maximal when the distribution is uniform (every outcome equally likely, maximal doubt) and zero when one outcome is certain. A fair coin has $H = 1$ bit; a fair die has $\log_2 6 \approx 2.585$ bits. For continuous signals the sum becomes an integral and we speak of differential entropy; a Gaussian of variance $\sigma^2$ has differential entropy $\tfrac{1}{2}\log_2(2\pi e\,\sigma^2)$, which grows with the log of the noise width. That single fact quietly connects entropy to the variance-based uncertainty you already met: wider noise, more bits of doubt. Choosing the base of the logarithm just changes units. Base 2 gives bits; the natural log gives nats. Nothing conceptual changes, so pick whichever your library defaults to and stay consistent.

Relative entropy and cross-entropy: the cost of a wrong model

Sensing systems almost never know the true distribution; they carry an approximation $q$ of the real $p$. The relative entropy, or Kullback-Leibler divergence, measures the penalty for that mismatch:

$$D_{\mathrm{KL}}(p \,\|\, q) = \sum_x p(x)\,\log_2 \frac{p(x)}{q(x)} \;\ge\; 0,$$

with equality only when $q = p$. It is the average number of extra bits you waste encoding data drawn from $p$ while believing it came from $q$. Two cautions earn their keep in practice. First, it is not symmetric: $D_{\mathrm{KL}}(p\|q) \ne D_{\mathrm{KL}}(q\|p)$, so it is a divergence, not a distance. Second, it blows up wherever $q(x)=0$ but $p(x)>0$, which is the mathematical form of the epistemic disaster from Section 4.4: a model that assigns zero probability to a regime it never saw pays an infinite bill when that regime arrives. The closely related cross-entropy $H(p,q) = H(p) + D_{\mathrm{KL}}(p\|q)$ is the actual loss function behind almost every classifier you will train in Parts IV and V; minimizing cross-entropy is minimizing the divergence between your predicted label distribution and the truth.

Key Insight

Mutual information is the reduction in entropy about one variable from observing another: $I(X;Y) = H(X) - H(X \mid Y)$. Read it as a before-and-after ledger. $H(X)$ is your doubt about the world state before the measurement; $H(X \mid Y)$ is your expected doubt after seeing sensor reading $Y$; the difference is what the measurement bought you, in bits. It is symmetric, $I(X;Y) = I(Y;X)$, and it is exactly $D_{\mathrm{KL}}\!\big(p(x,y)\,\|\,p(x)p(y)\big)$, the divergence between the joint and the product of marginals. So mutual information is zero if and only if $X$ and $Y$ are independent, meaning a sensor that carries no mutual information with the quantity you care about is, formally, useless for estimating it.

Mutual information: the value of a measurement

Mutual information is the quantity a sensing engineer actually wants, because it prices a measurement in advance. Written through the joint distribution,

$$I(X;Y) = \sum_{x,y} p(x,y)\,\log_2 \frac{p(x,y)}{p(x)\,p(y)}.$$

It is the workhorse behind three recurring jobs. Feature selection: rank candidate features by their mutual information with the label and keep the informative ones, the information-theoretic backbone of Chapter 8. Sensor and channel selection: prefer the modality whose reading shares the most bits with the hidden state, and drop redundant sensors whose information is already covered, a principle that reappears in the fusion chapters of Part X. Representation learning: modern self-supervised objectives such as InfoNCE are, at heart, tractable lower bounds on mutual information between different views of a signal, as Chapter 17 develops. One caveat carries through all three: mutual information counts statistical dependence, not causation and not usefulness for a specific downstream cost. A feature can be richly informative and still awkward to act on. Treat it as a strong prior on value, not a verdict.

Practical Example: The Robot That Priced Its Next Glance

A warehouse robot must decide whether a shelf bin is empty, half-full, or full before it plans a grasp. It carries three sensors it can poll: a cheap downward proximity sensor, a wrist camera it must rotate to aim (costing a second of motion), and a load cell in the gripper that requires an exploratory touch. Rather than firing all three, its planner estimates the mutual information $I(\text{bin state};\text{reading})$ for each from a small calibration log. The proximity sensor turns out to share only about 0.2 bits with the three-way state (it confuses half-full and full), the camera shares roughly 1.4 bits, and the touch shares 0.9 bits but is slow. The robot picks the camera first because it buys the most doubt-reduction per glance, and only falls back to touch when the camera's posterior entropy stays high (a glare-blinded frame). Before adopting this ledger the team had hand-tuned a fixed sensor order that wasted motion on the near-useless proximity ping. Pricing each measurement in bits turned a brittle heuristic into a rule that adapts per bin.

The code below computes entropy and mutual information from a small joint sample, the exact calculation the robot's planner runs over its calibration log. It uses base-2 logs so the answers read directly in bits.

import numpy as np

def entropy(counts):
    p = counts / counts.sum()
    p = p[p > 0]                       # 0*log0 = 0
    return -np.sum(p * np.log2(p))     # bits

def mutual_information(joint):
    joint = joint / joint.sum()
    px = joint.sum(axis=1, keepdims=True)
    py = joint.sum(axis=0, keepdims=True)
    nz = joint > 0
    return np.sum(joint[nz] * np.log2(joint[nz] / (px @ py)[nz]))

# rows = hidden bin state (empty/half/full), cols = camera reading (low/mid/high)
joint = np.array([[40,  5,  1],
                  [ 6, 30,  8],
                  [ 1,  7, 42]], dtype=float)

print("H(state)      =", round(entropy(joint.sum(axis=1)), 3), "bits")
print("I(state;cam)  =", round(mutual_information(joint), 3), "bits")
# H(state) ~ 1.58 bits of prior doubt; the camera removes ~0.9 of them.

Estimating entropy and mutual information from a joint count table. H(state) is the robot's prior doubt about the bin (near the 1.585-bit maximum of a three-way choice); I(state;cam) is how many of those bits one camera reading is expected to remove. Ranking candidate sensors by this second number is the whole of information-driven sensor selection.

Right Tool: Skip the Estimator Plumbing

The hand-rolled functions above are fine for discrete tables, but real sensor readings are continuous, and estimating mutual information from continuous samples needs binning or nearest-neighbor tricks that are easy to get subtly wrong. scipy.stats.entropy gives entropy and KL divergence in one call, and scikit-learn's mutual_info_classif and mutual_info_regression estimate feature-label mutual information directly from raw continuous data using a validated k-nearest-neighbor estimator. That collapses a careful 60-to-80-line binning-and-bias-correction implementation into a single line, roughly a 95 percent reduction, while handling the continuous-variable bias that trips up naive histograms.

Information gain and deciding what to measure next

The payoff of this section is active sensing: choosing the next measurement that maximizes expected information gain. If your current belief about a state $X$ has entropy $H(X)$, and a candidate action $a$ would yield an observation $Y_a$, the expected reduction in doubt is the mutual information $I(X; Y_a)$ under your current belief. Greedily picking $\arg\max_a I(X; Y_a)$ is the information-theoretic core of Bayesian experimental design, active learning (label the points that shrink model entropy the most), and next-best-view planning in robotics. It is also the honest, quantitative version of the epistemic-uncertainty story from Section 4.4: high epistemic uncertainty means high entropy that data can remove, and mutual information tells you which datum removes the most. When you reach the Bayesian filters of Chapter 9 and the fusion rules of Part X, this same ledger decides which sensor to trust and when a reading is redundant. Bits, once you can count them, become a budget you can spend wisely.

Exercise

(a) A binary sensor fires with the correct label 80 percent of the time and is wrong 20 percent, independent of a uniform prior over two equally likely states. Compute $H(X)$, $H(X \mid Y)$, and $I(X;Y)$ in bits, and interpret the gap. (b) You are offered a second sensor whose reading is a perfect copy of the first. What is the mutual information of the pair with the state, and what does that tell you about paying for redundant sensors? (c) Using the code in this section, modify the joint table so the camera becomes useless (zero mutual information) without changing the marginal over states, and verify numerically.

Self-Check

Why is KL divergence not a valid distance metric, and what practical failure occurs when a model assigns zero probability to an event that actually happens?
State mutual information as an entropy difference and as a KL divergence between joint and product-of-marginals, and explain why both must give the same number.
Your feature has high mutual information with the label but adds nothing to a model that already uses a correlated feature. Which information-theoretic quantity captures that redundancy, and why is raw mutual information alone the wrong ranking?

What's Next

In Section 4.6, we turn the doubt-in-bits view into decisions with consequences: hypothesis testing and detection theory. Where mutual information asks how much a measurement could tell you, detection theory asks where to put the threshold once the reading arrives, trading false alarms against missed detections. The likelihood ratio you will meet there is the same log-probability machinery you just used to price surprise, now aimed at saying yes or no.