"I used to store the world as a black-box function you had to query one ray at a time. Now I store it as a million little glowing ellipsoids I can throw at the screen. My latency thanks me."
A Recently Rasterized AI Agent
The Big Picture
Neural SLAM proved a moving camera could carve a dense, photorealistic map out of nothing but images. It just did so at a crawl, because every pixel of every query cost a march of network evaluations along a ray. Gaussian-splatting SLAM keeps the payoff, a map you can re-render from any viewpoint, and throws out the bottleneck: it represents the scene as an explicit cloud of 3D Gaussians and renders them by rasterization instead of ray marching. The result is the first family of dense photometric SLAM systems that track, map, and render at interactive rates on a single GPU, and increasingly on an embedded board bolted to a robot.
Prerequisites
This section assumes you know what a 3D Gaussian splat is (mean, anisotropic covariance, opacity, view-dependent color) and how differentiable rasterization composites them; that groundwork is laid in Chapter 51. It also builds directly on the ray-marched neural SLAM of Section 52.3, and leans on camera geometry and RGB-D depth from Chapter 40. Pose optimization on the \(SE(3)\) manifold is used here in the same spirit as the smoothing back-ends of Chapter 11.
Why the map representation changed
Neural SLAM stores the scene implicitly, as weights of a network that answers "what color and density is this point?" one query at a time. That is expressive but slow: rendering a single pixel means sampling dozens of points along a ray and evaluating an MLP at each. Gaussian splatting inverts the primitive. The map becomes an explicit set of anisotropic 3D Gaussians, each with a mean \(\mu_i\), a covariance \(\Sigma_i\) (factored into a scale and a rotation so it stays positive-definite), an opacity \(o_i\), and a color. Rendering projects each Gaussian to a 2D "splat" on the image plane and alpha-composites the sorted splats front to back:
\[ C(\mathbf{u}) = \sum_{i} c_i\, \alpha_i \prod_{jKey Insight
Classical SLAM (Section 52.2) tracks by matching sparse features; neural SLAM tracks by ray-marching an implicit field. Gaussian-splatting SLAM tracks by differentiable rendering: freeze the map, render the current pose's predicted image, compare it to the real frame pixel by pixel, and slide the pose down the photometric gradient. The map is the measurement model. There is no feature detector, no descriptor, no data-association step to get wrong.
Tracking and mapping as one photometric optimization
Every system in this family alternates two optimizations over the same explicit map. Tracking estimates the new camera pose \(T \in SE(3)\) by holding the Gaussians fixed and minimizing the residual between the rendered image \(\hat{I}(T)\) and the observed frame \(I\), typically a photometric term plus a depth or silhouette term when RGB-D is available:
\[ T^\star = \arg\min_{T \in SE(3)} \; \lambda_p \, \lVert \hat{I}(T) - I \rVert_1 \; + \; \lambda_d \, \lVert \hat{D}(T) - D \rVert_1 . \]The pose is updated on the Lie algebra \(\mathfrak{se}(3)\) so the estimate stays a valid rigid transform. Mapping then flips which variables are frozen: it holds recent keyframe poses (mostly) fixed and optimizes the Gaussians against a window of keyframes, growing new Gaussians where the render is starved of geometry (densification) and deleting near-transparent or oversized ones (pruning). MonoGS derives the pose Jacobians analytically so tracking needs no autograd graph over the renderer, which is a large part of why it hits interactive rates. The heart of the projection, the step that turns a 3D ellipsoid into the 2D footprint you optimize against, is short enough to write out.
import numpy as np
def project_covariance(mu_cam, cov3d, fx, fy):
"""Project a 3D Gaussian covariance into the 2D image plane (EWA splatting).
mu_cam: Gaussian mean in the CAMERA frame (x, y, z), z > 0.
cov3d: 3x3 world/camera-aligned covariance (already rotated into cam frame).
Returns the 2x2 image-space covariance of the splat, in pixels^2."""
x, y, z = mu_cam
# Jacobian of the perspective projection u = fx*x/z, v = fy*y/z at mu_cam.
J = np.array([[fx / z, 0.0, -fx * x / z**2],
[0.0, fy / z, -fy * y / z**2]])
cov2d = J @ cov3d @ J.T # 2x2 footprint on the image plane
cov2d += np.eye(2) * 0.3 # low-pass dilation, keeps sub-pixel splats visible
return cov2d
mu = np.array([0.10, -0.05, 2.0]) # 2 m in front of the camera
cov3d = np.diag([0.02**2, 0.02**2, 0.05**2]) # a 2 cm x 2 cm x 5 cm ellipsoid
print(np.round(project_covariance(mu, cov3d, fx=525.0, fy=525.0), 3))
J is exactly the term that couples camera pose to splat shape: differentiating the rendered image through this map is what yields the analytic pose gradient MonoGS tracks with. A distant Gaussian (large z) projects to a tiny footprint, which is why far geometry contributes weak pose constraints, the same depth-precision falloff you met with stereo and RGB-D in Chapter 40.The code above prints the pixel-space footprint of one splat; scale that to a million Gaussians rasterized and sorted per frame and you have the render that both localizes the camera and gets sculpted by the map optimizer. Because the whole thing is a rasterizer, not a ray marcher, the forward and backward passes are tile-parallel on the GPU, and that is the entire speed story.
Right Tool: don't hand-roll the rasterizer
The differentiable tile rasterizer, the alpha-compositing forward pass and its analytic backward pass, is roughly 2,000 lines of hand-tuned CUDA. The gsplat library (from the Nerfstudio project) exposes it as two calls, gsplat.project_gaussians(...) and gsplat.rasterize_gaussians(...), turning the excerpt above plus its full backward pass into about 15 lines of Python. You inherit fused CUDA kernels, correct gradients, and multi-GPU support for free; you write only the SLAM logic (keyframing, densification, pose optimization) on top. Build the projection once by hand to understand it, as we just did, then never ship your own kernel.
The three systems: MonoGS, SplaTAM, Photo-SLAM
The three canonical 2024 systems stake out different corners of the input-and-speed design space. MonoGS (Matsuki et al., CVPR 2024, originally titled "Gaussian Splatting SLAM") is the purest expression of the idea: a monocular system that does direct photometric tracking straight against the Gaussian map with analytic pose Jacobians, adding geometric regularizers (isotropy and depth-order priors) to fight the scale ambiguity a single camera cannot resolve. It also runs with RGB-D or stereo when you have them. SplaTAM (Keetha et al., CVPR 2024) commits to RGB-D and leans on a clever trick: it renders a silhouette (an accumulated-opacity mask) alongside color and depth, so tracking only trusts pixels the map is actually confident about, and mapping only densifies where the silhouette says geometry is missing. That silhouette-guided masking makes its dense tracking notably robust. Photo-SLAM (Huang et al., CVPR 2024) is the pragmatist: it keeps a classical ORB feature front-end for pose tracking and bundle adjustment, and attaches a Gaussian "hyper-primitive" map purely for photorealistic rendering. That split lets it run monocular, stereo, or RGB-D, and, crucially, hit real time on an embedded Jetson-class board, which the pure-photometric systems struggle to do.
Field note: a warehouse robot that maps as it patrols
A logistics startup mounts a single global-shutter RGB camera on an autonomous floor robot that patrols aisles overnight. They want two things from one pass: an accurate trajectory for navigation, and a photorealistic 3D map an operator can later fly through to spot a fallen box or a mislabeled pallet. Classical ORB-SLAM3 gave them the trajectory but only a sparse point cloud no human could read. Swapping in a Photo-SLAM-style pipeline, ORB tracking for the pose, a Gaussian map for appearance, they keep the metric trajectory (stereo baseline fixes the monocular scale) and gain a map the operator scrubs through like a video game level. The Gaussian map streams to a workstation; the robot itself runs the ORB front-end on a Jetson Orin, exactly the embedded-deployment envelope Chapter 59 is about. The photometric map is the deliverable; the trajectory is the by-product.
What breaks, and where the frontier is
Three failure modes recur. Scale drift haunts every monocular variant: a single camera recovers geometry only up to an unknown global scale, so MonoGS needs geometric priors and any metric use needs a depth sensor, a stereo baseline, or the learned depth priors of Chapter 41. Catastrophic forgetting is subtler: because mapping optimizes against a sliding window of recent keyframes, Gaussians for a room you left ten minutes ago can be nudged out of shape by gradients that never see that room again, so systems keep anchor keyframes and freeze old Gaussians to preserve them. The deepest gap is loop closure. Most early Gaussian-splatting SLAM systems are locally consistent but have no global back-end, so when a long trajectory returns to its start, the accumulated pose error shows up as a ghosted double-wall in the map. Fixing it means deforming a rigid explicit map after a loop is detected, which is harder than re-linking a factor graph.
Research Frontier
The 2024 trio, MonoGS, SplaTAM, and Photo-SLAM, established real-time dense photometric SLAM on 3D Gaussians. The live frontier is global consistency: Gaussian-SLAM (Yugay et al.) pushed dense RGB-D accuracy, and LoopSplat (2024) added loop closure by registering Gaussian submaps directly and rigidly deforming the map on closure, the piece the originals lacked. Parallel work fuses splatting with lidar and inertial data for outdoor, metric-scale mapping, which is precisely the Gaussian-LIC system you meet next in Section 52.5. Open problems: reflective and transparent surfaces that violate the opacity model, dynamic scenes (Section 52.6), and shrinking the memory of a million-Gaussian map to fit a phone.
Exercise
Using the project_covariance function above, place one Gaussian at depth \(z = 1\ \mathrm{m}\) and an identical one at \(z = 5\ \mathrm{m}\), both with the same 3D covariance. Compare the areas of their 2D footprints (the determinant of cov2d scales with squared footprint area). Then argue, in two or three sentences, why a Gaussian-splatting tracker gets weak pose constraints from distant geometry and strong ones from nearby geometry, and connect that to why these systems drift faster in large open outdoor scenes than in cluttered rooms.
Self-Check
- Why can Gaussian-splatting SLAM track a camera without ever detecting or matching a feature, and what plays the role of the measurement model instead?
- SplaTAM renders a silhouette alongside color and depth. What two distinct jobs does that silhouette do, one in tracking and one in mapping?
- Photo-SLAM keeps a classical ORB front-end rather than tracking purely photometrically. Name one capability that choice buys it that pure-photometric MonoGS lacks.
What's Next
In Section 52.5, we take the Gaussian map outdoors and metric. Pure-camera splatting SLAM fights scale drift and dark, textureless surfaces; bolting on a lidar for direct range and an IMU for high-rate motion turns the same photorealistic representation into a robust, drift-bounded mapper. We work through Gaussian-LIC and the tight-coupling that lets lidar geometry, inertial preintegration, and photometric splatting sharpen one map together.