"A network that has never heard of the second law of thermodynamics will happily heat a room by cooling it, and report low loss the whole way down."
A Thermodynamically Chastened AI Agent
Prerequisites
This section builds on the measurement model \(x = h(s) + \eta\) and the governing physics of a sensed quantity from Chapter 2, and on the estimation-and-uncertainty vocabulary (priors, ill-posedness, regularization) from Chapter 4. It assumes comfort with automatic differentiation and gradient training at the level of Chapter 13. The inverse-problem framing here is the optimization-based cousin of the recursive state estimation in Chapter 11.
The Big Picture
A digital twin is only as good as the physics it obeys and the parameters it is told. Physics-informed neural networks (PINNs) attack both problems with one idea: make the governing differential equation part of the loss function, so the network is penalized not just for missing your sparse sensor readings but for violating the conservation law, diffusion equation, or wave equation that the world actually follows. This turns a network into a mesh-free field that interpolates where you have no sensors and, in the inverse mode that matters most for sensing, reads out the hidden physical parameters that best explain the readings you do have: a thermal diffusivity, a leak location, a corrosion rate. The prize is a twin that stays physical between sensors and self-calibrates from data.
Baking the equation into the loss
A PINN is an ordinary neural network \(u_\theta(x,t)\) that maps space-time coordinates to a physical field, temperature, pressure, displacement, trained with a loss that has two jobs. The first is the familiar data term: wherever a sensor sat, the field must match its reading. The second is the physics residual term, and it is what makes the method work. If the field must satisfy a partial differential equation written abstractly as \(\mathcal{N}[u] = 0\) (for the 1D heat equation, \(\mathcal{N}[u] = u_t - \alpha\,u_{xx}\)), then we sample a cloud of collocation points throughout the domain, evaluate the network there, and use automatic differentiation to compute the exact derivatives \(u_t\) and \(u_{xx}\) of the network output with respect to its inputs. The residual at each point is a number that should be zero, and its mean square becomes a loss. The total objective is
\[ \mathcal{L}(\theta) = \underbrace{\frac{1}{N_d}\sum_i \lVert u_\theta(x_i,t_i) - x_i^{\text{obs}}\rVert^2}_{\text{fit the sensors}} \; + \; \lambda\,\underbrace{\frac{1}{N_c}\sum_j \lVert \mathcal{N}[u_\theta](x_j,t_j)\rVert^2}_{\text{obey the physics}} \; + \; \mathcal{L}_{\text{bc/ic}}. \]The why is that the physics term needs no labels. Collocation points are free: you invent them by sampling coordinates, and the equation itself tells you the target (zero). So a PINN can be trained on a handful of real sensor readings plus millions of unlabeled points where it must merely be self-consistent with the law. That is how it fills the gaps between sensors with something better than a spline: an interpolant that a differential equation vouches for. Because derivatives come from autodiff rather than a finite-difference grid, the method is mesh-free, which is a real advantage on irregular domains and in high dimensions where meshing is painful.
Key Insight
The physics residual is a self-supervised, label-free regularizer with a physical guarantee. Ordinary regularizers (weight decay, smoothness penalties) express a vague preference for simple functions. The PINN residual expresses the exact statement "this field conserves energy" or "this field diffuses at rate \(\alpha\)." That is why a PINN can generalize from absurdly little data: the equation removes most of the function space before the sensors ever speak. The cost is that this constraint is soft, enforced by a penalty weight \(\lambda\), not a hard guarantee, and choosing \(\lambda\) badly lets the network satisfy one term by wrecking the other.
The inverse problem: reading physics out of sensors
For sensing, the forward use (solve the PDE, predict the field) is the less interesting half. The half that earns its place in a digital-twin chapter is the inverse problem: the equation is known in form but one or more of its coefficients, sources, or boundary conditions are unknown, and you want to recover them from sparse, noisy sensor data. This is exactly the situation of a physical asset whose material properties drift with age, damage, or fouling. Classically these problems are ill-posed in the sense of Chapter 4: many parameter values fit the data almost equally well, so you need a prior or a regularizer to pick one.
PINNs turn the inverse problem into a startlingly small code change. You simply promote the unknown coefficient, say the diffusivity \(\alpha\), from a fixed constant into a learnable parameter and let the same optimizer that fits the network weights also fit \(\alpha\). The data term pins the field to the sensors; the physics residual now says "and whatever field you learn must satisfy the diffusion equation for some \(\alpha\)," and gradient descent finds the \(\alpha\) that reconciles the two. The sensors constrain the field, the field constrains the equation, and the equation hands you the hidden parameter. Nothing about the loss changes except which tensors carry gradients. Listing 55.1 shows the whole inverse solver for a 1D heat rod in a few dozen lines.
import torch, torch.nn as nn
# Field network u(x,t); the UNKNOWN we want from sensors is the diffusivity alpha.
net = nn.Sequential(nn.Linear(2, 32), nn.Tanh(),
nn.Linear(32, 32), nn.Tanh(), nn.Linear(32, 1))
log_alpha = nn.Parameter(torch.tensor(0.0)) # learn alpha = exp(log_alpha) > 0
opt = torch.optim.Adam([*net.parameters(), log_alpha], lr=1e-3)
# Sparse noisy sensor readings: (x, t) -> measured temperature (few probes, few times).
xt_obs = torch.rand(40, 2); u_obs = true_field(xt_obs) + 0.02 * torch.randn(40, 1)
def residual(xt): # PDE residual u_t - alpha * u_xx
xt = xt.clone().requires_grad_(True)
u = net(xt)
g = torch.autograd.grad(u, xt, torch.ones_like(u), create_graph=True)[0]
u_t, u_x = g[:, 1:2], g[:, 0:1]
u_xx = torch.autograd.grad(u_x, xt, torch.ones_like(u_x), create_graph=True)[0][:, 0:1]
return u_t - torch.exp(log_alpha) * u_xx
for step in range(20_000):
xt_col = torch.rand(1000, 2) # free, unlabeled collocation points
loss = ((net(xt_obs) - u_obs) ** 2).mean() + (residual(xt_col) ** 2).mean()
opt.zero_grad(); loss.backward(); opt.step()
print("recovered diffusivity:", torch.exp(log_alpha).item())log_alpha is registered as a learnable parameter alongside the network weights, so one Adam optimizer recovers the hidden thermal diffusivity from 40 noisy probe readings while the residual term forces the learned field to obey the heat equation. Making the unknown a nn.Parameter is the entire difference between the forward and inverse problem.As Listing 55.1 shows, the inverse problem is not a separate algorithm; it is the forward loss with one extra trainable scalar. Estimating a whole spatial field of unknowns (a variable-conductivity map, a distributed source) works the same way: replace the scalar with a second small network \(\alpha_\phi(x)\) and let its parameters ride along in the optimizer.
In Practice: locating a leak in a district heating pipe
A municipal heating operator instruments a buried pipe with a dozen temperature probes spaced hundreds of meters apart. A slow leak somewhere between two probes bleeds heat, but the sparse sensors cannot see where. The operator sets up a PINN whose field is the pipe-wall temperature governed by an advection-diffusion equation, feeds it the twelve probe streams as the data term, and adds a learnable spatial sink term \(q_\phi(x)\) as the unknown. Training drives \(q_\phi\) to concentrate exactly where the physics can only be reconciled with the readings by removing heat, and the recovered sink peaks at the leak, tens of meters resolved from probes kilometers apart. The same machinery underlies condition monitoring in Chapter 36: the recovered parameter (here a heat sink, elsewhere a corrosion rate or a stiffness loss) is itself the health indicator you want to trend and alarm on.
When PINNs win, and when they fight back
PINNs are not a free lunch, and knowing their failure modes is the difference between a working twin and a plausible-looking one. They shine when the governing equation is trusted, sensor data is sparse, and you want either a continuous field between probes or a physical parameter that no sensor measures directly. They struggle on stiff or multiscale problems, sharp fronts, turbulence, high-frequency waves, where the residual landscape is brutal and vanilla training stalls. The soft penalty weight \(\lambda\) is a genuine nuisance: too small and the network ignores the physics, too large and it satisfies the equation with a trivial field that fits no sensor. Training pathologies are well documented; the residual and data gradients can pull at wildly different magnitudes, which is why practical PINNs lean on adaptive loss weighting, curriculum schedules, and second-order optimizers such as L-BFGS for the final polish.
It also pays to place PINNs among their neighbors. Unlike the recursive filters of Chapter 11, a PINN solves the whole space-time window at once as a batch optimization, which is powerful offline but not a streaming estimator; Section 55.6 couples a twin to an EKF precisely to get the online update PINNs lack. And unlike the neural fields of Chapter 51, which fit a coordinate network to observations alone, a PINN adds the differential-equation residual that makes its interpolation obey physical law rather than merely look smooth.
Research Frontier
The field is moving fast beyond vanilla PINNs. Neural operators, the Fourier Neural Operator (FNO) and DeepONet, learn the solution map of an entire PDE family in one training run, so at inference they solve a new boundary condition or coefficient field in a single forward pass instead of re-optimizing, which is what a real-time twin needs. Adaptive-weighting schemes and the "conservative" and separable PINN variants tackle the stiff-loss pathologies, and Bayesian PINNs attach calibrated uncertainty (Chapter 18) to the recovered parameters so an inverse estimate arrives with error bars rather than a bare point value. The open problem for sensing remains scaling honest, uncertainty-aware inverse solutions to three-dimensional, multiphysics assets in real time.
The Right Tool
Listing 55.1 hand-rolls the autodiff residual, collocation sampling, and loss weighting: on a real multiphysics problem that grows into hundreds of lines of derivative bookkeeping and weight scheduling that are easy to get subtly wrong. Libraries such as DeepXDE, NVIDIA Modulus, and PINA package geometry sampling, boundary-condition handling, adaptive loss weights, and the L-BFGS polish behind a declarative API. The same inverse heat problem drops to roughly ten lines:
import deepxde as dde
alpha = dde.Variable(1.0) # the unknown to recover
def pde(x, u):
u_t = dde.grad.jacobian(u, x, i=0, j=1)
u_xx = dde.grad.hessian(u, x, i=0, j=0)
return u_t - alpha * u_xx
data = dde.data.TimePDE(geom_time, pde, [observe_u], num_domain=1000, anchors=xt_obs)
model = dde.Model(data, dde.nn.FNN([2] + [32]*3 + [1], "tanh", "Glorot normal"))
model.compile("adam", lr=1e-3, external_trainable_variables=alpha)
model.train(iterations=20000, callbacks=[dde.callbacks.VariableValue(alpha, period=1000)])DeepXDE: declaring alpha as a dde.Variable and passing it to external_trainable_variables replaces the manual parameter registration, autodiff residual, and training loop of Listing 55.1, cutting roughly forty lines to about ten while adding built-in collocation sampling and callback logging of the recovered value.As Listing 55.2 shows, the library removes the plumbing, not the judgment. Choosing the right equation, trusting it, and deciding whether the recovered parameter is identifiable from your sensor layout remain firmly yours.
Exercise
Take Listing 55.1 and reduce the number of sensor readings from 40 down to 4, then to 1, retraining each time and recording the recovered diffusivity and its error. At what point does the estimate destabilize? Now double the observation noise and repeat. Write one paragraph explaining, in the ill-posedness language of Chapter 4, why the physics residual lets you recover a parameter from as little as a single well-placed probe, and what sensor placement would make the parameter unidentifiable no matter how many readings you add.
Self-Check
1. A PINN trains on 30 sensor readings but 500,000 collocation points. Where do the labels for the collocation points come from, and why are they free?
2. What is the single code change that converts a forward PINN (predict the field for known \(\alpha\)) into an inverse PINN (recover unknown \(\alpha\) from data)?
3. Your inverse PINN drives the physics residual to nearly zero but ignores the sensor readings entirely. Which knob is most likely misset, and in which direction?
What's Next
In Section 55.5, we widen the lens from "physics as a loss term" to full hybrid mechanistic-plus-data-driven models, where a trusted simulator handles the part of the system we understand and a learned residual model soaks up the friction, wear, and unmodeled dynamics it cannot, giving twins that are both physically grounded and empirically honest.