Deep ensembles, MC-dropout, Bayesian and evidential DL

"A single network will happily tell you the pressure is 3.2 bar with total conviction, even when it has never seen a valve like this one. The cure is to ask several networks and notice when they start to argue."

An Appropriately Doubtful AI Agent

This section assumes the variance decomposition and Bayesian posterior notation of Chapter 4, and it inherits the calibration vocabulary of Section 18.2: an uncertainty estimate is only useful once it is calibrated, so everything here is measured on a held-out set, never on training windows. The through-line is the law of total variance, which lets us split any distribution over predictions into the two components we care about:

Here \(\theta\) is the network weights, \(\mu_\theta(x)\) and \(\sigma^2_\theta(x)\) are a heteroscedastic head's predicted mean and variance, and the outer expectation and variance run over a distribution of plausible weights. Every method below is a different recipe for that weight distribution.

Deep ensembles: the strong, embarrassingly parallel baseline

What. A deep ensemble (Lakshminarayanan et al., NeurIPS 2017) is \(M\) copies of the same architecture, each trained from an independent random initialization on the same data, their predictions averaged at inference. Nothing more exotic. The random seeds send stochastic gradient descent into different basins of the loss surface, so the members disagree most where the data did not pin them down.

Why it works. The disagreement is the epistemic term. On an in-distribution window every member has seen similar data and they concur, so \(\operatorname{Var}_\theta[\mu_\theta(x)]\) is small. On a novel window the members extrapolate differently and the spread explodes. Despite being theoretically the least "Bayesian" of the four, deep ensembles remain the most reliable in practice: across the large-scale shift benchmarks they consistently give the best calibration under distribution shift, which is precisely the regime a deployed sensor model lives in. For regression, pair each member with a heteroscedastic head that predicts \(\mu\) and \(\sigma^2\) via the Gaussian negative-log-likelihood; the ensemble then supplies both terms of the variance decomposition in one shot.

When. Choose ensembles when accuracy and honest uncertainty matter more than training cost and you can afford \(M\) times the parameters (typically \(M = 5\) captures most of the benefit). The cost is real but embarrassingly parallel: members train independently, and at inference on the edge you can distill them or use the memory-sharing tricks below.

Monte Carlo dropout: one network, many masks

What. MC-dropout (Gal & Ghahramani, ICML 2016) keeps dropout active at inference and runs \(T\) stochastic forward passes, each with a different random mask, then treats the \(T\) outputs as samples from an approximate predictive distribution. The mean is your prediction; the spread is your epistemic estimate.

Why it is (approximately) Bayesian. Gal and Ghahramani showed that training a network with dropout and \(L_2\) weight decay is equivalent to variational inference with a specific Bernoulli approximate posterior over the weights. Each dropout mask is therefore a cheap sample \(\theta_t \sim q(\theta)\), and averaging over masks approximates the predictive integral \(\int p(y\mid x,\theta)\,q(\theta)\,d\theta\). The appeal is that you get uncertainty from a model you probably already trained, with no architectural change and one hyperparameter (the dropout rate).

When, and the caveats. Use MC-dropout when compute or memory forbids an ensemble and the network already contains dropout layers. Know its limits: the uncertainty quality is highly sensitive to the dropout rate and where the layers sit, it tends to underestimate epistemic uncertainty far from the data compared to an ensemble, and it does nothing if your architecture uses batch normalization instead of dropout. For sequence models covered in Chapter 14, apply the same mask across timesteps (variational RNN dropout) rather than resampling per step, or the recurrent state's uncertainty will be nonsense.

import torch

def mc_dropout_predict(model, x, T=30):
    """Heteroscedastic net returning (mu, log_var). Keep dropout ON."""
    model.train()                        # forces dropout active; freeze BN separately if present
    mus, var_a = [], []
    with torch.no_grad():
        for _ in range(T):
            mu, log_var = model(x)       # (B, D) each
            mus.append(mu)
            var_a.append(log_var.exp())
    mus = torch.stack(mus)               # (T, B, D)
    var_a = torch.stack(var_a)           # (T, B, D)

    pred_mean = mus.mean(0)              # predictive mean
    aleatoric = var_a.mean(0)            # E_theta[ sigma^2 ]  -> irreducible noise
    epistemic = mus.var(0)              # Var_theta[ mu ]     -> model ignorance
    return pred_mean, aleatoric, epistemic

The snippet above makes the practical point concrete: whether the samples come from dropout masks or from ensemble members, the downstream bookkeeping is identical, and only the epistemic term should react to novelty.

Bayesian deep learning: posteriors without full retraining

What. Bayesian deep learning aims for the real object, a posterior \(p(\theta \mid \mathcal{D})\) over weights, rather than a point estimate. Full Hamiltonian Monte Carlo over millions of weights is intractable, so practice uses scalable approximations. Three matter for sensor work. Mean-field variational inference (Bayes by Backprop) replaces each weight with a learned Gaussian, doubling the parameter count and adding training noise. SWAG (Maddox et al., NeurIPS 2019) fits a Gaussian to the trajectory of SGD iterates in the final epochs, turning ordinary training into a cheap posterior for free. Laplace approximation (revived at scale by Daxberger et al., NeurIPS 2021, "Laplace Redux") fits a Gaussian around an already-trained maximum-a-posteriori solution using the loss curvature, so it is fully post-hoc: train normally, then add uncertainty in a few lines.

Why and when. Reach for these when you want a principled posterior and its downstream benefits (marginal likelihood for model selection, coherent propagation into a fusion stage as in Chapter 49) without paying for \(M\) full trainings. The Laplace approximation is the pragmatic sweet spot for deployed models: it leaves your trained weights and accuracy untouched, needs no retraining, and only the last layer is often enough to get well-calibrated uncertainty at a fraction of an ensemble's cost.

Evidential deep learning: uncertainty in a single forward pass

What. Evidential deep learning removes the need for sampling entirely by having the network output the parameters of a distribution over the predictive distribution. For classification (Sensoy et al., NeurIPS 2018) the model emits the concentration parameters of a Dirichlet, whose total \(S=\sum_k \alpha_k\) acts as accumulated evidence: high total means confidence, low total means "I have seen little like this." For regression (Deep Evidential Regression, Amini et al., NeurIPS 2020) the model emits the four parameters of a Normal-Inverse-Gamma prior, from which aleatoric and epistemic terms read off in closed form. One forward pass, both uncertainties, no ensemble, no dropout loop, which is decisive on the microcontrollers of Chapter 61 where a 30-pass MC loop or five networks simply will not fit.

When, with eyes open. Evidential methods are the cheapest route to per-sample uncertainty at inference, so they are attractive for edge and streaming deployment. But their epistemic signal is a learned regressor, not a sample-based estimate, and it can be poorly calibrated or arbitrary far from the training data. Treat evidential uncertainty as a fast first-line screen and validate it hard on true out-of-distribution data before trusting it in a safety loop; where a guarantee is required, the distribution-free machinery of Section 18.4 is the safer floor.

Choosing among the four

The choice is a budget decision. If you can afford the compute and want the best calibration under shift, use a deep ensemble; it is the default that research keeps failing to beat. If you cannot, and the model already has dropout, MC-dropout is the cheapest add-on, accepting that it under-reports far-from-data uncertainty. If you have a trained model you must not disturb, a post-hoc Laplace approximation adds a calibrated posterior with no retraining. If you are on a microcontroller where even a second forward pass is a luxury, evidential methods give single-pass uncertainty, validated carefully. Whatever you pick, the epistemic estimate is your trigger for the abstention and fallback logic of Section 18.6 and for any functional-safety argument in Chapter 68.

What's Next

In Section 18.4, we stop approximating posteriors and start demanding guarantees. Conformal prediction turns any of these point predictors or uncertainty scores into prediction sets with a finite-sample coverage guarantee that holds regardless of the model, provided the data are exchangeable, and we confront what happens to that guarantee when a drifting sensor stream breaks exchangeability.