Section 72.3: Neural fields, differentiable sensors, and inverse sensing

"Forward models predict measurements. Sensing is the harder job of running them backward and admitting the answer is not unique."
An Invertible AI Agent

Prerequisites

This section builds directly on the neural-field machinery of Chapter 51 (radiance fields, signed distance fields, positional encoding, the volume rendering integral). It assumes the measurement-model view of a sensor from Chapter 2, the estimation and uncertainty vocabulary of Chapter 4, and automatic differentiation from Appendix B. Where this section goes past Chapter 51 is the frontier claim: that the sensor itself can be made differentiable, and that perception is an inverse problem solved by running that differentiable sensor backward. Chapter 51 teaches the representation; this section teaches the research program around it.

The Big Picture

Nearly every method in this book runs the pipeline one way: measurements come in, a model reads features out. Inverse sensing turns the pipeline around. Write down a differentiable forward model $g_\phi$ that predicts what a sensor would record given a scene $z$ and sensor parameters $\phi$, then recover the scene by optimizing $z$ (and often $\phi$) until the predicted measurements match the real ones. A neural field is the ideal unknown $z$: a continuous, coordinate-queried function with no fixed resolution, differentiable end to end. The forward model is the physics of the sensor: ray marching for a camera, time-of-flight for lidar, the radar range equation, the wave equation for sound. When both halves are differentiable, gradient descent on measurement error reconstructs geometry, appearance, material, and even the sensor's own calibration at once. This unifies photogrammetry, tomography, SLAM, and calibration under a single objective, and it is where a large slice of frontier physical-AI research now lives.

The inverse-sensing objective

What is inverse sensing. A forward sensor model is a map $y = g_\phi(z) + \varepsilon$, where $z$ is the latent state of the world (a scene, a field, a set of physical parameters), $\phi$ are sensor parameters (poses, intrinsics, gains, timing), and $\varepsilon$ is noise. The forward problem, "given the world, predict the reading," is well posed and cheap. Sensing is the inverse problem, "given the readings, recover the world," and it is generically ill posed: many worlds explain the same measurements, so the answer needs a prior. The classical estimator is a regularized least-squares fit,

$$\hat{z},\hat{\phi} \;=\; \arg\min_{z,\phi}\; \sum_{i} \big\lVert y_i - g_\phi(z)_i \big\rVert^2_{\Sigma^{-1}} \;+\; \lambda\,\mathcal{R}(z),$$

with $\Sigma$ the measurement covariance and $\mathcal{R}$ a prior (smoothness, sparsity, the eikonal constraint of an SDF). Why neural fields changed this: for decades $z$ was a voxel grid or a mesh, and optimizing it meant hand-deriving adjoints of the forward model. Represent $z$ as a neural field $f_\theta$ and make $g_\phi$ a differentiable renderer, and the whole objective becomes one autograd graph. How you solve it is then the loop you already know: forward-simulate the measurement, take the residual, backpropagate to $\theta$ and $\phi$, step. When to reach for it: whenever you trust a forward physics model and want to fuse many partial, noisy, differently-posed measurements into one consistent scene, especially when the sensor is exotic and no pretrained network exists for it.

Key Insight

The gradient that trains the field and the gradient that calibrates the sensor are computed by the same backward pass. Because $y=g_\phi(f_\theta(\cdot))$ is one differentiable composition, $\partial \mathcal{L}/\partial\theta$ reconstructs the scene while $\partial \mathcal{L}/\partial\phi$ refines poses, intrinsics, timing offsets, and even lens distortion. This is why "bundle-adjusting" neural fields can start from rough or wrong poses and still converge: calibration is not a separate preprocessing step, it is a free byproduct of the same optimization. The practical corollary is that a bad forward model, not a bad network, is the usual cause of failure. If your reconstruction is blurry or warped, audit the sensor physics before you touch the architecture.

Differentiable sensors: one framework, many modalities

Inverse sensing is a frontier program, not a single trick, because the forward model $g_\phi$ is swappable: change the physics and the same neural field learns from a different instrument. The volume rendering integral of Chapter 51 is only the camera case. The research move is to write differentiable forward models for the rest of the sensor suite in Chapter 40 onward.

Lidar and depth. The forward model returns expected range and drop probability along each beam by integrating density to the first hard surface; supervising a field with raw returns yields NeRF-LOAM-style differentiable lidar mapping that jointly refines the trajectory (this is neural SLAM, extending Chapter 52).
Radar. The forward model is the range-Doppler response and the radar equation; a differentiable renderer that predicts a returned power cube lets a field absorb the multipath and specularity that make raw radar (Chapter 44) so hard to read.
Acoustic and ultrasound. The wave equation is differentiable, so neural-acoustic-field work reconstructs room impulse responses and material absorption from a handful of microphone recordings.
Tomographic and transmission sensors. CT, MRI, and electrical-impedance sensing are the original inverse problems; a coordinate field regularizes the reconstruction where classical filtered back-projection would streak.

Why this matters for sensory AI specifically: it gives you a principled way to fuse modalities that live in incompatible coordinate systems. A camera, a lidar, and a radar disagree pixel-by-pixel but agree about the one scene that produced all three. Optimizing a single field against all three differentiable forward models is fusion by shared cause, a complement to the feature-level fusion of Chapter 50.

Practical Example: waking a mine-inspection robot in the dark

An autonomous inspection rover crawls a decommissioned mine drift where there is no light and GPS is dead. It carries a spinning lidar, a solid-state radar, and a contact microphone on its manipulator. No pretrained model exists for this sensor mix, and the walls are wet, dusty, and specular, exactly the conditions that defeat off-the-shelf lidar odometry. The team represents the drift as one signed distance field and attaches three differentiable forward models: a lidar that predicts beam ranges, a radar that predicts the range-Doppler cube, and a contact-acoustic model that predicts the tap response of struck rock. A single optimization minimizes the sum of the three residuals plus an eikonal prior. The lidar pins down clean geometry; the radar fills the dust-occluded gaps; the acoustic taps resolve whether a flat wall is solid rock or a thin shotcrete shell hiding a void. The recovered field is watertight enough to plan a collision-free path and to flag the void. Crucially, the rover's trajectory came out of the same solve: the poses were parameters $\phi$, refined by the very same gradients, so no separate SLAM front end was needed.

Task-driven and self-calibrating acquisition

Making the sensor differentiable unlocks a second, more radical move: optimize the sensor, not just the scene. If $\phi$ includes controllable acquisition parameters (where to point next, which exposure, which waveform, which coded aperture), you can differentiate the downstream task loss with respect to $\phi$ and choose measurements that are maximally informative. What this is: end-to-end co-design of the sensor and the estimator. Why it works: the derivative $\partial \mathcal{L}_\text{task}/\partial\phi$ tells you which physical setting most reduces task error, answering the active-sensing question of "where should I look next" with gradients instead of heuristics. This connects to the uncertainty machinery of Chapter 18: the next-best measurement is the one whose expected posterior variance reduction is largest. How it appears: differentiable optics that learn a lens phase mask jointly with the reconstruction network, coded-illumination microscopy that learns its LED pattern, and next-best-view planners. When to invest: when acquisition is expensive or slow (a robot arm repositioning a probe, a clinical scan minute) so that fewer, better measurements pay off.

Research Frontier

The live problems as of 2026. Speed: per-scene optimization is slow, so hybrids that swap the pure MLP for explicit primitives (3D Gaussian Splatting, with differentiable forward splatting for lidar and event cameras) or hash-grid encodings (Instant-NGP) now dominate when latency matters. Priors: single-scene fits overfit thin views; feed-forward models such as pixelNeRF and LRM amortize the inverse solve across scenes so a new scene needs seconds, not minutes. Dynamics: deformable and 4D fields, and event-based differentiable rendering that exploits the microsecond timing of neuromorphic cameras (Chapter 46), are unsolved at scale. Uncertainty: inverse problems are non-unique, yet most reconstructions return a point estimate with no calibrated error bar, a gap revisited in Section 72.6. The reference implementations are nerfstudio, gsplat, and SDFStudio.

Building the loop in code

The whole idea compresses to a short optimization loop. The listing below is a differentiable range sensor: a signed distance field is queried along each beam, a soft rendering turns the field into an expected range, and one loss on range residuals produces gradients that flow to both the field weights and the sensor pose. It is deliberately minimal, but it is the exact skeleton that neural-SLAM and inverse-sensing systems scale up.

import torch, torch.nn as nn

class SDFField(nn.Module):                    # z: the unknown scene, f_theta
    def __init__(self, w=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(3, w), nn.Softplus(),
                                 nn.Linear(w, w), nn.Softplus(),
                                 nn.Linear(w, 1))
    def forward(self, x):                      # x:(...,3) -> signed distance (...,)
        return self.net(x).squeeze(-1)

def differentiable_range(field, origin, dirs, tn=0.1, tf=8.0, N=128, beta=40.0):
    """Forward sensor model g_phi: predict expected range per beam from the SDF."""
    t = torch.linspace(tn, tf, N)              # samples along every beam
    pts = origin + t[:, None, None] * dirs     # (N, B, 3), B beams
    s = field(pts)                             # signed distance at each sample
    occ = torch.sigmoid(-beta * s)             # soft inside-surface indicator
    alpha = occ.clamp(1e-6, 1 - 1e-6)          # hit probability per segment
    T = torch.cumprod(1 - alpha + 1e-9, dim=0) # transmittance up to each sample
    w = T * alpha                              # weight = first-hit probability
    w = w / (w.sum(0, keepdim=True) + 1e-9)    # normalize into a range pmf
    return (w * t[:, None]).sum(0)             # expected range per beam

field = SDFField()
pose  = nn.Parameter(torch.zeros(3))           # phi: sensor position, also learned
opt   = torch.optim.Adam(list(field.parameters()) + [pose], lr=1e-3)

for step in range(2000):
    dirs = sample_beam_directions()            # unit vectors, (B,3)
    pred = differentiable_range(field, pose, dirs)
    loss = ((pred - measured_ranges(dirs)) ** 2).mean()   # inverse-sensing residual
    opt.zero_grad(); loss.backward(); opt.step()          # updates BOTH z and phi

A minimal differentiable range sensor. The same loss.backward() reconstructs the scene field and refines the sensor pose, because prediction and calibration share one autograd graph. Real systems replace the toy MLP with a hash-grid encoding and add an eikonal regularizer, but the forward-model-plus-residual loop is unchanged.

Library Shortcut

The listing above is the teaching skeleton. In production you would not hand-write the sampler, the encoding, the pose graph, and the mesh extraction. nerfstudio plus gsplat reduce a full inverse-sensing pipeline (camera and depth forward models, hash-grid fields, pose optimization, exporting a mesh) from roughly 800 to 1000 lines of custom CUDA and Python to about 20 lines of config plus a ns-train command. The library handles ray generation, mixed-precision sampling, appearance embeddings, camera-pose refinement, and marching-cubes export. Reach for the from-scratch loop only to understand the gradients or to bolt on a sensor whose forward model no library ships yet, which is precisely the frontier case.

Exercise

Take the code above and make the sensor pose wrong: initialize pose at an offset of 0.5 m from the true origin used to generate measured_ranges. (1) Train with the pose frozen (remove it from the optimizer) and inspect the reconstructed surface. (2) Train with the pose learnable, as written. Compare the final range residual and the recovered pose error. Then add an eikonal penalty $\lambda(\lVert\nabla f_\theta\rVert-1)^2$ on random query points and report how it changes surface quality. Which of the three (pose freezing, pose learning, eikonal prior) mattered most, and why does that ranking depend on how many beams you have?

Self-Check

Inverse sensing is ill posed: many scenes explain the same measurements. Name the two ingredients in the objective that make the solution unique, and say which one the eikonal loss supplies.
Why can a neural field "bundle-adjust" its way out of bad initial poses, when a classical voxel reconstruction with fixed poses cannot?
You have a camera, a lidar, and a radar of the same scene. Explain how a single neural field fuses all three without ever aligning them pixel-to-pixel, and what property each modality contributes to the shared field.

What's Next

In Section 72.4, we step from reconstructing the world as it is to predicting the world as it will be: world models for physical AI that learn the forward dynamics of a scene and let an agent roll them out to imagine, plan, and act, closing the loop from the differentiable perception of this section to differentiable control.