Dynamic-scene and semantic mapping

"For years I assumed the world held still while I mapped it. Then a bus drove through my map, and I spent three keyframes politely pretending the wall behind it had moved."

A Chastened AI Agent

Why moving objects break SLAM, and the three ways to cope

The core estimator behind classical SLAM minimizes reprojection error under a static-world assumption: a 3D landmark projects to where the camera geometry says it should, and any discrepancy is measurement noise. A moving object silently breaks that model. Its features are geometrically consistent frame to frame (they really are the same corner of the same bus), so a naive front-end happily matches them, then the back-end tries to explain their motion by moving the camera instead. A single bus filling a third of the frame can drag the trajectory meters off course and bake a translucent bus-shaped scar into the map.

Three strategies handle this, in rising order of ambition. The first is reject: detect dynamic regions and simply exclude their features from tracking and mapping, treating them as outliers. The second is reconstruct: keep the static background clean but also model each moving object as its own rigidly moving submap, so you end up with the scene plus per-object trajectories. The third is deform: allow genuinely non-rigid geometry (a person bending, a cloth folding) and fit a time-varying representation, the regime dynamic Gaussian splatting now targets. Most deployed systems live in the first tier, the richest research lives in the third.

Detecting what moves: geometry, semantics, or both

How do you know a pixel is moving? Two independent signals exist, and the strongest systems fuse them. Geometric cues need no labels: after estimating an initial camera motion, a feature that violates multi-view epipolar geometry, or whose measured depth disagrees with the reprojected static prediction, is probably on a moving body. Formally, for a matched point with a candidate motion, the epipolar residual \(r = \mathbf{x}_2^\top \mathbf{F}\, \mathbf{x}_1\) should be near zero for static structure; large \(|r|\) flags a dynamic outlier, which classical robust back-ends already down-weight. The weakness is that geometric cues fail exactly when they matter most: an object moving along the epipolar line (a car driving straight ahead of you) produces a small residual and hides.

Semantic cues cover that blind spot. A segmentation network labels every pixel, and a prior list of movable classes (person, vehicle, animal) marks those regions as suspect regardless of whether they happen to be moving this instant. This is the design of the widely cited DynaSLAM (Bescos et al., 2018), which layers Mask R-CNN masking plus multi-view geometry on top of ORB-SLAM2 and then inpaints the background occluded by removed objects, and of DS-SLAM, which pairs SegNet with a moving-consistency check. The catch is the mirror image of the geometric failure: semantics over-reject. A parked car is labeled "vehicle" and discarded even though it is a perfectly good, rock-solid landmark. The fix is the fusion: use the class prior to decide what could move, then use the geometric consistency check to decide what is moving, and only reject the intersection.

import numpy as np

# Per-feature dynamic gate: fuse a semantic movable-class prior with a
# geometric epipolar-consistency test. A feature is trusted as a static
# landmark only if it is NOT flagged by both signals.
def static_feature_mask(epi_residual, is_movable_class,
                        resid_thresh=1.5e-3):
    """epi_residual: |x2^T F x1| per matched feature (array).
       is_movable_class: bool array, True if pixel's class can move.
       Returns bool mask of features to KEEP for pose estimation."""
    moving_geo = np.abs(epi_residual) > resid_thresh   # violates epipolar geom
    suspect = is_movable_class & moving_geo            # movable AND actually moving
    return ~suspect                                    # keep everything else

res  = np.array([2e-4, 5e-3, 1e-4, 8e-3, 3e-4])
movable = np.array([False, True, True, True, False])   # class prior
keep = static_feature_mask(res, movable)
print(keep)   # a parked car (movable but low residual) is KEPT

The code above is the heart of every "reject" system in one function. Everything else, the segmentation network, the fundamental-matrix estimate, the keyframe bookkeeping, feeds these two arrays. In practice the movable-class prior comes from a real-time panoptic model and the residual from the same robust estimator you already run for outlier rejection, so the marginal cost of dynamic handling is mostly the segmentation forward pass.

From labels to object-level and open-vocabulary maps

Rejecting dynamics keeps the map clean; semantic mapping makes the map mean something. The simplest form fuses per-pixel labels into the 3D representation: as each frame's segmentation is back-projected, every voxel, surfel, or Gaussian accumulates a running distribution over class labels, converging to a dense semantic map. SemanticFusion did this over a surfel map by Bayesian label fusion across frames; Kimera builds a real-time metric-semantic mesh alongside its VIO and pose-graph back-end. The next step up is object-level SLAM, where the map's units are whole objects rather than pixels. QuadricSLAM represents objects as dual quadrics (ellipsoids) that are jointly optimized with camera poses in the factor graph, and Fusion++ maintains a per-object TSDF. An object-centric map is compact, supports data association by object identity across large loops, and lets a robot reason about "the chair" as one movable entity, closing the loop back to dynamic handling.

The 2023 shift is open-vocabulary mapping. Instead of committing to a fixed label set at training time, each 3D primitive stores a high-dimensional vision-language feature (a CLIP-style embedding, the machinery of Chapter 21). ConceptFusion fuses pixel-aligned foundation-model features into a 3D map so you can later query it with an arbitrary text or image prompt; LERF and the newer language-embedded Gaussian splats bake the same idea into radiance-field and splatting maps. A query becomes a dot product: embed the phrase "the red backpack," compare it against every primitive's stored feature, and the map lights up where the similarity is high, no retraining, no closed label list. This is the representation an embodied agent needs to follow a natural-language instruction into a space it has mapped but never been told the vocabulary of.

The moving frontier: dynamic reconstruction and consistency

The hardest regime is reconstructing geometry that genuinely deforms, not just rejecting it. Here the map itself becomes time-dependent: a set of Gaussians (or a neural field) whose positions and shapes are functions of \(t\), fit so that rendering at each timestamp matches that frame. This is where SLAM meets 4D reconstruction, and it inherits every ambiguity of monocular dynamic vision, motion and depth trade off against each other, so the community leans hard on learned depth and motion priors to constrain the fit.

What's Next

In Section 52.7, we ask the uncomfortable question this whole chapter has been deferring: how do you actually measure whether a SLAM system is good? We work through trajectory metrics (absolute and relative pose error), map-quality and rendering metrics, the standard benchmarks (TUM RGB-D, EuRoC, KITTI, Replica), and the leakage traps that make an impressive number on paper evaporate in the field.