Neural radiance fields and signed distance fields

"A scene is not a pile of points. It is a function of space you have not yet bothered to learn."

A Continuous AI Agent

The radiance field and its rendering integral

A NeRF is a function \(F_\theta:(\mathbf{x}, \mathbf{d}) \mapsto (\mathbf{c}, \sigma)\). It takes a 3D position \(\mathbf{x}\in\mathbb{R}^3\) and a unit viewing direction \(\mathbf{d}\), and returns an emitted RGB color \(\mathbf{c}\) and a volume density \(\sigma \ge 0\). Density depends only on position (geometry does not change with where you look), while color depends on both, so that a wet road or a glossy panel can look different from different angles. The what is deliberately minimal: \(F_\theta\) is a small MLP, often eight layers wide, with no convolutions and no explicit 3D data structure.

The why it can be trained from ordinary photographs is the rendering model. To predict the color of one pixel, you cast the camera ray \(\mathbf{r}(t)=\mathbf{o}+t\mathbf{d}\) into the scene, sample points along it, and composite their colors weighted by how much light reaches the camera from each. This is the volume rendering integral:

$$C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt,\qquad T(t)=\exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right).$$

The transmittance \(T(t)\) is the probability a ray travels from the camera to \(t\) without being absorbed; \(\sigma\) governs how fast that probability decays. A point contributes to the pixel only if it is dense and nothing dense sits in front of it. The how of training is then pure supervision by re-projection: render the predicted pixel color \(\hat C(\mathbf{r})\), compare it to the true captured pixel \(C_\text{gt}(\mathbf{r})\), and descend the photometric loss \(\mathcal{L}=\sum_{\mathbf{r}} \lVert \hat C(\mathbf{r}) - C_\text{gt}(\mathbf{r})\rVert^2\). No 3D labels are needed; the geometry emerges because reproducing many overlapping views is only possible if \(\sigma\) concentrates on the true surfaces. The when: NeRF shines when you have dozens of well-posed images of a mostly static scene and want photorealistic novel views, and it struggles when poses are unknown, the scene moves, or you have only a handful of frames.

Signed distance fields: geometry as a level set

Density is a soft, fog-like quantity: it is superb for rendering but it never tells you exactly where the surface is, only where light tends to stop. When the deliverable is geometry, a lidar-quality mesh, a collision surface for a robot, a metric measurement, an SDF is the better field. An SDF is a function \(s_\theta:\mathbb{R}^3\to\mathbb{R}\) returning the signed distance to the nearest surface: negative inside the object, positive outside, and exactly zero on it. The surface is the implicit set \(\{\mathbf{x}: s_\theta(\mathbf{x})=0\}\). The why to prefer this: a true distance function has a defined normal everywhere (\(\mathbf{n}=\nabla s_\theta\)), extracts a watertight mesh by marching cubes on the zero crossing, and supports exact ray-surface intersection by sphere tracing, stepping along a ray in increments of the current distance value, which cannot overshoot the surface.

For \(s_\theta\) to be a genuine distance rather than an arbitrary field with the right sign, it must satisfy the eikonal equation \(\lVert\nabla s_\theta(\mathbf{x})\rVert=1\) almost everywhere: moving one meter through space should change the distance-to-surface by one meter. This is imposed during training as a soft penalty, the eikonal loss \(\mathcal{L}_\text{eik}=\mathbb{E}_\mathbf{x}\big(\lVert\nabla s_\theta(\mathbf{x})\rVert-1\big)^2\), evaluated on random points using the network's own autodiff gradient. The how of supervision depends on your sensor. A lidar or depth camera gives near-direct supervision: points on the beam return are zero-distance surface samples, and free space between sensor and return is known to be empty (positive distance). RGB-only supervision is subtler; methods such as NeuS and VolSDF convert the SDF into a density profile so that the same volume rendering integral above can train geometry and appearance jointly, marrying NeRF's easy photometric supervision to the SDF's clean surface.

import torch, torch.nn as nn

def positional_encoding(x, L=6):                 # x: (..., 3) coordinates
    freqs = 2.0 ** torch.arange(L) * torch.pi    # [pi, 2pi, 4pi, ...]
    xb = x[..., None] * freqs                     # (..., 3, L)
    enc = torch.cat([xb.sin(), xb.cos()], dim=-1) # (..., 3, 2L)
    return torch.cat([x, enc.flatten(-2)], dim=-1)

class TinyField(nn.Module):                       # (x,y,z) -> (rgb, density)
    def __init__(self, L=6, w=128):
        super().__init__()
        d_in = 3 + 3 * 2 * L
        self.net = nn.Sequential(nn.Linear(d_in, w), nn.ReLU(),
                                 nn.Linear(w, w), nn.ReLU(),
                                 nn.Linear(w, 4))        # 3 color + 1 density
    def forward(self, x):
        h = self.net(positional_encoding(x))
        return torch.sigmoid(h[..., :3]), torch.relu(h[..., 3])

def render_ray(field, o, d, tn=2.0, tf=6.0, N=64):
    t = torch.linspace(tn, tf, N)                 # sample points along the ray
    pts = o + t[:, None] * d                      # (N, 3)
    rgb, sigma = field(pts)                        # (N,3), (N,)
    delta = (t[1:] - t[:-1])                       # segment lengths
    delta = torch.cat([delta, delta[-1:]])
    alpha = 1.0 - torch.exp(-sigma * delta)        # opacity per segment
    T = torch.cumprod(torch.cat([torch.ones(1), 1 - alpha[:-1]]), 0)  # transmittance
    w = T * alpha                                  # compositing weights
    return (w[:, None] * rgb).sum(0)               # rendered pixel color

Listing 51.1 makes the central seam visible: the network is trivial, and every bit of physics lives in the rendering step that turns field values along a ray into one pixel. Swap the color-plus-density head for a distance head and reuse the same ray machinery and you have moved from a radiance field to a rendered SDF.

From slow MLPs to real hash grids

The original NeRF paid for its elegance with time: a single scene took a day on a GPU because every one of millions of ray samples ran a full MLP forward and backward pass. The decisive engineering shift, Instant-NGP (Muller et al., 2022), keeps the field idea but moves most of the capacity out of the MLP and into a multiresolution hash grid of trainable feature vectors. A query point looks up and interpolates feature vectors from several grid resolutions, concatenates them, and passes the result through a tiny two-layer MLP. The why this is faster: the grid does the memorizing, so the network can be small, and the lookups are cheap; training drops from hours to seconds. The how it stays compact: fine levels index a fixed-size table through a hash, accepting rare collisions in exchange for bounded memory. This is the representation most production neural-field and splatting systems build on, and it is why neural fields moved from a research curiosity to something you can fit on a robot's logged drive over lunch. The same acceleration underlies the near-real-time reconstruction pressures taken up later in this chapter and the mapping backbones of Chapter 51's neighbor, Chapter 52 on SLAM.

# reconstruct a scene from a folder of images, then export a mesh
# (shell, one line each)
# ns-process-data images --data ./substation_frames --output-dir ./proc
# ns-train nerfacto --data ./proc
# ns-export poisson --load-config outputs/.../config.yml --output-dir ./mesh

What's Next

In Section 51.2, we trade the queried network for an explicit, rasterizable representation: 3D Gaussian splatting, which scatters the scene across millions of differentiable Gaussians that render in real time, keeping the photometric supervision of NeRF while dropping the per-sample MLP cost that made these fields slow.