Visuo-tactile fusion

"The camera told me the mug was full. My fingers told me it was empty. One of them was looking at a reflection, and it was not the fingers."

A Skeptical AI Agent

Why the two modalities are complementary, not redundant

The case for touch begins where vision ends. Consider a robot picking a coffee mug. Vision, covered as a sensing modality throughout Parts IX and X, localizes the mug, proposes a grasp, and guides the hand to it. But once the fingers close, the fingertips and the mug occlude one another in every camera view, and the questions that decide success become physical rather than geometric: is the grip about to slip, how much force is on the handle, is the wall thin porcelain that will crack. None of these are visible. Touch answers all of them directly, because a contact-rich sensor measures the deformation field at the interface (Section 56.4). The two senses are therefore complementary in the strict sense of Chapter 48: each supplies information the other structurally cannot, so fusing them expands what is observable rather than merely averaging down noise.

Formally, let the manipulation state be \(s\) (object pose, contact configuration, grasp stability). Vision induces a likelihood \(p(v \mid s)\) that is sharp over global pose but nearly flat over contact forces; touch induces \(p(\tau \mid s)\) that is sharp over local contact but silent about anything beyond the fingertip. Their product \(p(s \mid v, \tau) \propto p(v \mid s)\, p(\tau \mid s)\, p(s)\) is sharp in both subspaces at once, which is the entire point. The interesting twist is that touch is not merely noisier where vision is strong; it is undefined there. Before contact, \(\tau\) carries no information about grasp force because there is no contact to measure. Touch switches on.

Registering touch into the visual frame

To fuse the streams you must place them in a common frame, exactly as Chapter 48 demands for any spatial fusion. A tactile reading is expressed in the sensor's own frame, on the fingertip. Vision lives in the camera frame. The bridge is the robot's kinematics: forward kinematics maps the fingertip frame to the base frame through the measured joint angles, and a one-time hand-eye calibration \(T_{\text{cam}\leftarrow\text{base}}\) maps the base frame to the camera. Composing them projects the contact patch to a pixel region, so a GelSight imprint can be overlaid on the exact spot of the object the camera sees. This registration rests on the measurement-model discipline of Chapter 2: every frame transform carries calibration error, and a few millimeters of hand-eye error will misplace the contact patch by more than a fingertip is wide.

Two practical consequences follow. First, the fusion is only as good as the calibration, so contact-rich work budgets real effort for hand-eye calibration and joint-encoder accuracy. Second, when registration is expensive or unreliable, many systems skip explicit geometric alignment and instead let a learned model align the modalities in a shared latent space, which is the dominant modern approach and the subject of the next subsection.

Fusion architectures: from cross-attention to contrastive pretraining

Because touch and vision meet most usefully as learned features, visuo-tactile fusion is overwhelmingly middle fusion in the taxonomy of Chapter 48: a vision encoder and a tactile encoder each produce an embedding, and the two are combined before a shared head. Three combination patterns dominate. Concatenation with a gate is the simplest and handles the intermittent channel gracefully, because a learned gate can zero out the tactile embedding before contact and open it after. Cross-attention lets the tactile tokens query the visual tokens (or vice versa), so a slip signal at the fingertip can attend to the visual region it belongs to; this is the same mechanism that powers the multimodal models of Chapter 21. Contrastive pretraining aligns the two encoders without labels by pulling paired vision and touch crops together in embedding space and pushing unpaired ones apart, the recipe of Chapter 17; the tactile foundation models of Section 56.5 (UniTouch and its kin) are built this way and give a strong initialization for downstream fusion.

Listing 56.6.1 implements the gated variant for grasp-success prediction. The gate is driven by a scalar contact flag, so the network is told, per sample, whether touch is live. This single mechanism turns the intermittent-modality headache into a first-class input.

import torch, torch.nn as nn

class VisuoTactileGrasp(nn.Module):
    """Predict grasp success from a vision embedding and a tactile embedding,
    with a learned gate that respects whether contact has occurred."""
    def __init__(self, d_vis, d_tac, d=64):
        super().__init__()
        self.vis = nn.Linear(d_vis, d)
        self.tac = nn.Linear(d_tac, d)
        self.gate = nn.Sequential(nn.Linear(1, d), nn.Sigmoid())  # driven by contact flag
        self.head = nn.Sequential(nn.Linear(2 * d, 64), nn.ReLU(), nn.Linear(64, 1))

    def forward(self, e_vis, e_tac, in_contact):      # in_contact: (B,1) in {0.,1.}
        v = self.vis(e_vis)
        t = self.tac(e_tac) * self.gate(in_contact)   # touch is masked before contact
        return self.head(torch.cat([v, t], dim=-1)).squeeze(-1)  # grasp-success logit

As Listing 56.6.1 shows, the architecture is a few lines; the design content is the gate and the contrastively pretrained encoders that feed it. This mirrors Chapter 50: robustness to an absent modality comes from telling the model, not from hoping it infers absence from zeros.

from torchmultimodal.modules.layers.transformer import TransformerCrossAttentionLayer
xattn = TransformerCrossAttentionLayer(d_model=64, n_head=4)
# tactile tokens query visual tokens; key_padding_mask hides touch before contact
fused = xattn(hidden_states=e_tac, encoder_hidden_states=e_vis,
              encoder_key_padding_mask=no_contact_mask)

Evaluating fusion rigorously, and when it actually helps

Fusion is not free, and it does not always win. The discipline is to measure the vision-only baseline first and demand that fusion beat it on held-out contact conditions, not merely on aggregate accuracy where vision already suffices. Paired visuo-tactile datasets make this possible: Touch-and-Go (Yang and colleagues, 2022) provides human-collected vision and GelSight touches on real objects, and ObjectFolder (Gao and colleagues) supplies simulated multisensory objects with vision, touch, and audio. The book's Lab 56 uses exactly this setup: train a tactile classifier, then fuse it with vision on Touch-and-Go and confirm the lift is concentrated where vision is weak (transparent, reflective, or texture-ambiguous surfaces). The recurring failure mode is a fusion model that quietly ignores touch, learning to route everything through the vision branch because that branch minimizes training loss fastest; the gate value and per-modality ablations are your instruments for catching it.

What's Next

Fusion tells you that vision and touch agree on a contact state, but not yet what that contact is doing moment to moment. In Section 56.7, we zoom into the tactile stream itself to infer slip onset, contact geometry, and material properties: the fast, local judgments that let a hand tighten its grip a few milliseconds before an object escapes, and that the fusion models of this section rely on as their tactile signal.