ROS 2 perception integration

"The algorithm was correct. The transform was ten milliseconds stale, the QoS was mismatched, and the depth cloud arrived on a topic nobody subscribed to. My beautiful fusion node saw nothing at all."

A Freshly Humbled Integration AI Agent

This section assumes the timing and synchronization foundations of Chapter 3 and the real-time budgeting of Section 57.6. Here we make those concrete inside one concrete middleware rather than in the abstract.

The graph: nodes, topics, and DDS

A ROS 2 system is a set of nodes (independent units of computation) that communicate over named topics using a publish-subscribe model. A camera driver node publishes on /camera/image_raw; a detector node subscribes to it and publishes bounding boxes on /detections; a tracker subscribes to those. No node holds a reference to another. This decoupling is what lets you swap a classical detector for a neural one, or run the detector on a separate compute board, by changing a topic name rather than rewriting the caller. Underneath, ROS 2 delegates transport to DDS (the Data Distribution Service, an industrial pub-sub standard), which handles discovery, serialization, and delivery with no central broker. That decentralization matters for robots: there is no ROS master to become a single point of failure, and two nodes on the same machine can bypass the network stack entirely.

Perception messages are standardized so that any node can consume any sensor. The sensor_msgs package defines Image, PointCloud2, LaserScan, Imu, and CameraInfo among others; every one carries a std_msgs/Header with a timestamp and a frame_id. Those two fields, when the measurement was taken and in which coordinate frame, are the entire basis for fusing heterogeneous sensors, and forgetting to populate them correctly is the single most common integration bug.

tf2: the coordinate-frame spine

A robot has many frames: base_link at its center, lidar_link where the lidar sits, camera_optical_frame, an odom frame that drifts smoothly, a map frame that is globally consistent. A lidar point is meaningless until you know it is expressed in lidar_link and can transform it into base_link to reason about obstacles near the body. tf2 is the ROS 2 subsystem that maintains this tree of time-stamped rigid transforms and answers queries of the form "give me the transform from frame A to frame B at time \(t\)." Because transforms are time-stamped, tf2 interpolates: if a wheel moved between two published odom updates, tf2 returns the pose at the exact instant your camera frame was captured. This is what makes proprioception-exteroception fusion geometrically honest. The composed transform from a sensor frame to the map frame is just a chain of homogeneous matrices,

and tf2 walks that chain for you, raising an exception if any link is missing or too old. The static links (sensor mounts) are published once; the dynamic links (odom to base) stream from the estimator of Section 57.4.

Time synchronization and quality of service

Fusion nodes usually need several streams that belong to the same instant: an RGB image, its depth map, and the IMU state when both were captured. The streams arrive on separate topics at different rates with jitter, so you cannot simply grab the latest of each. ROS 2 provides message_filters with an approximate-time synchronizer that buffers recent messages per topic and emits a tuple only when it finds one message from each whose stamps fall inside a tolerance window. This is the practical realization of the alignment problem from Chapter 3, packaged as a callback.

Equally important is QoS (quality of service), the per-topic contract that governs delivery. A Reliable publisher retransmits until every subscriber acknowledges; BestEffort fires and forgets. High-rate sensor data (lidar, images) is almost always BestEffort with a shallow KEEP_LAST history, because a robot moving at speed wants the freshest scan, not a guaranteed-delivered stale one. A mismatched QoS, a Reliable subscriber on a BestEffort topic, causes the connection to silently never form: the subscriber simply receives nothing, with no error. This is the classic "my node sees no data" failure, and it is a delivery contract problem, not a code problem.

import rclpy
from rclpy.node import Node
from message_filters import Subscriber, ApproximateTimeSynchronizer
from sensor_msgs.msg import Image, PointCloud2
from rclpy.qos import QoSProfile, ReliabilityPolicy, HistoryPolicy

SENSOR_QOS = QoSProfile(reliability=ReliabilityPolicy.BEST_EFFORT,
                        history=HistoryPolicy.KEEP_LAST, depth=5)

class FusionNode(Node):
    def __init__(self):
        super().__init__("rgbd_fusion")
        img = Subscriber(self, Image, "/camera/image_raw", qos_profile=SENSOR_QOS)
        cloud = Subscriber(self, PointCloud2, "/lidar/points", qos_profile=SENSOR_QOS)
        # pair messages whose stamps agree within ~20 ms
        self.sync = ApproximateTimeSynchronizer([img, cloud], queue_size=10, slop=0.02)
        self.sync.registerCallback(self.on_pair)

    def on_pair(self, image, cloud):
        dt = abs((image.header.stamp.sec + image.header.stamp.nanosec * 1e-9) -
                 (cloud.header.stamp.sec + cloud.header.stamp.nanosec * 1e-9))
        self.get_logger().info(f"aligned pair, |dt|={dt*1e3:.1f} ms")
        # ... transform cloud into camera frame via tf2, run fusion ...

def main():
    rclpy.init(); rclpy.spin(FusionNode()); rclpy.shutdown()

The code above is referenced in the lab below. Note how little of it is perception: the substance is delivery contracts and stamp alignment, exactly the concerns that never appear in a paper but decide whether the paper's method survives contact with a robot.

Composition, lifecycle, and Nav2

Passing a multi-megabyte point cloud between processes means serializing and copying it, which at 20 Hz wastes real bandwidth. ROS 2 answers with components: nodes compiled as shared libraries and loaded into one process, where intra-process communication passes a pointer instead of a copy. A perception pipeline (driver, filter, detector) composed into a single container can cut end-to-end latency dramatically, which connects directly to the budgets of Section 57.6. ROS 2 also offers managed lifecycle nodes with explicit unconfigured, inactive, and active states, so a sensor driver can be brought up, checked, and only then allowed to publish, which is the clean hook for the failure detection and recovery of Section 57.5. Above all this sits Nav2, the navigation stack that consumes the fused, transformed perception to build costmaps and plan paths, tying this chapter's perception to the spatial reasoning of Chapter 52.

When ROS 2 is the right choice

ROS 2 earns its complexity when you have many heterogeneous sensors, multiple compute units, and a need to reuse community drivers and stacks like Nav2. For a single-sensor embedded classifier on a microcontroller (Chapter 59), it is overkill; a bare inference loop is simpler and lighter. The decision hinges on graph complexity, not on prestige. And a running caution on evaluation: a perception node that passes in a simulator can still fail on hardware purely from QoS, stamping, or frame errors, so the leakage-safe discipline of validating on real recorded rosbags, not just simulation, applies to integration exactly as it applies to models.

What's Next

With perception wired into a running ROS 2 graph, the robot can see, transform, and fuse the world in real time. In Chapter 58, we close the loop from perception to action: vision-language-action models that map raw sensor streams and natural-language goals directly to robot commands, turning the perceptual substrate of this chapter into embodied behavior.