3D representations (voxels, points, meshes, SDFs)

"The world does not come with a data structure attached. I have to choose one, and every choice throws away something I might later miss."

A Reluctant AI Agent

This section assumes you can already produce a point cloud from a depth map and place it in a metric coordinate frame, both covered in Sections 40.2 and 40.3. It also leans on the probability vocabulary of occupancy and log-odds from Chapter 4. No prior graphics background is required; we build each representation from its query.

Points: the raw sample, kept sparse

A point cloud is simply a set \(P = \{\mathbf{x}_i \in \mathbb{R}^3\}\), optionally carrying color, intensity, or a normal per point. It is what almost every depth sensor emits natively, so it is the representation with zero conversion loss: nothing has been quantized, smoothed, or interpolated. That fidelity is its strength. Its weaknesses follow from what it lacks. A point cloud has no connectivity (you cannot ask "what is adjacent to this point" without building a spatial index), no notion of free space (the gaps between points are ambiguous: unseen, occluded, or genuinely empty?), and no fixed size, which is awkward for tensor-shaped neural networks.

Points shine when the surface is thin relative to the scene and you want to defer any commitment about topology. Lidar perception stacks (Chapter 42) operate directly on points precisely because voxelizing a 100 m lidar sweep at centimeter resolution would be almost entirely empty. Memory scales with surface area sampled, not scene volume, which is the decisive advantage outdoors.

Voxels: the volumetric grid

A voxel grid partitions space into a regular 3D lattice of cells of edge length \(v\), and stores a value per cell: a binary occupancy flag, an occupancy probability, a semantic label, or a learned feature vector. The appeal is that voxels turn 3D into an image-like tensor, so the entire machinery of convolution, pooling, and 3D U-Nets transfers directly. Free space is represented explicitly and for free: an empty cell is a stored zero, which is exactly what a robot's motion planner or an AR occlusion test wants to query.

The cost is cubic. Memory and compute scale as \((L/v)^3\) for a scene of extent \(L\), so halving the voxel size multiplies storage eightfold. A dense \(10\,\text{m}\) room at \(1\,\text{cm}\) resolution is \(10^9\) cells, most of them empty. Two fixes dominate practice. Sparse voxels store only non-empty cells in a hash map, which is how sparse-convolution backbones stay tractable. Octrees refine resolution adaptively, subdividing only where surface detail lives, so a flat wall costs one large node while a textured object costs many small ones.

Meshes: surfaces with connectivity

A triangle mesh is a set of vertices plus a set of faces that connect them, so it encodes the surface and its topology explicitly. This is the representation graphics pipelines, physics engines, and CAD tools expect, because connectivity is what lets you compute a consistent surface normal, integrate area, ray-trace efficiently, or test watertightness. A mesh is compact for large smooth regions: a planar floor is two triangles regardless of physical size, whereas the same floor costs thousands of voxels or points.

Meshes are less convenient as a neural substrate. Their irregular, variable connectivity does not tensorize cleanly, and topology changes (a hole opening, two objects merging) are awkward to represent as a smooth deformation, which complicates learning. In a sensing stack meshes usually appear as an output product: you reconstruct one for visualization, collision geometry, or export, rather than running inference on it directly. Recovering a mesh from points or a volume is the job of reconstruction algorithms such as Poisson surface reconstruction or the classic Marching Cubes, which extracts a triangle surface from a scalar field, the field we turn to next.

SDFs: the implicit surface as a field

A signed distance function assigns to every point \(\mathbf{x}\) the distance to the nearest surface, signed negative inside the object and positive outside:

The surface is the level set where \(f(\mathbf{x}) = 0\). This implicit view has three properties that make it the backbone of modern 3D reconstruction. First, the surface is defined everywhere at arbitrary resolution: you sample the field as finely as you like without pre-committing to a grid. Second, the gradient \(\nabla f\) is the outward surface normal at the zero-crossing, so geometry and orientation come from the same object. Third, distances compose, which makes collision and nearest-obstacle queries trivial.

In practice an SDF is stored two ways. A truncated SDF (TSDF) keeps distance values in a voxel grid but only near the surface (truncated beyond a band), and is what depth-fusion systems like KinectFusion integrate frame by frame: it averages many noisy depth maps into one clean field and is the standard bridge from a stream of depth images to a mesh. A neural SDF represents \(f\) as the weights of a small network, trading memory for a learned, continuous, differentiable field; this is the door into neural fields and their Gaussian-splatting cousins in Chapter 51.

The workhorse conversion in that story, TSDF volume to triangle mesh, is a few lines with a modern geometry library, and getting it right by hand (correct triangle winding, watertight seams, per-vertex normals) is exactly the kind of detail a library has already solved.

import numpy as np
import open3d as o3d

# A stream of RGB-D frames fused into a Truncated SDF volume,
# then extracted as a mesh. Voxel size sets resolution; the
# truncation band (sdf_trunc) sets how far from the surface
# distances are stored.
volume = o3d.pipelines.integration.ScalableTSDFVolume(
    voxel_length=2.0 / 1000.0,   # 2 mm voxels
    sdf_trunc=8.0 / 1000.0,      # 8 mm truncation band
    color_type=o3d.pipelines.integration.TSDFVolumeColorType.RGB8,
)

for color, depth, cam_intrinsic, cam_pose in rgbd_stream:      # your calibrated frames
    rgbd = o3d.geometry.RGBDImage.create_from_color_and_depth(
        color, depth, depth_scale=1000.0, convert_rgb_to_intensity=False)
    volume.integrate(rgbd, cam_intrinsic, np.linalg.inv(cam_pose))

mesh = volume.extract_triangle_mesh()   # Marching Cubes on the zero level set
mesh.compute_vertex_normals()           # normals from the SDF gradient
o3d.io.write_triangle_mesh("part.ply", mesh)

Choosing, and converting

Read the choice off your dominant query. Need to run a detector over a wide outdoor sweep? Stay on points so memory tracks surface, not volume. Need explicit free space for planning or occupancy learning, on an image-like tensor? Use voxels, sparse or octree once resolution bites. Need to render, simulate collisions, measure area, or export to CAD? Produce a mesh. Need a continuous, denoised, differentiable surface with cheap distance and normal queries, or you are fusing many noisy frames? Use an SDF/TSDF. Conversions are routine and mostly one-directional in loss: points to voxels (bin and count), TSDF to mesh (Marching Cubes), mesh to points (surface sampling), points to SDF (unsigned distance, then sign via ray-casting or learned occupancy). Every conversion discards something, so convert as late as possible and keep the representation your accuracy-critical stage needs closest to the raw sensor.

What's Next

In Section 40.6, we stop treating these geometries as exact and confront depth uncertainty head-on: where the error in a depth measurement comes from, how it grows with range and baseline, and how to carry a per-point or per-voxel confidence through the representations we just built so that downstream fusion and planning know how much to trust each surface.