Why touch is the missing modality

"Vision tells me the cup is on the table. Touch tells me it is slipping out of my hand. Only one of those is an emergency."

A Grasping AI Agent

What touch measures that nothing else can

The first argument for touch is informational: it is the only sensor that observes the mechanical state of contact directly. When two surfaces meet, the physics at the interface is a small set of quantities, and none of them radiate outward for a camera to catch. There is the normal force pressing the surfaces together, the shear (tangential) force resisting sliding, the slip state (are the surfaces locked or moving relative to each other), the local compliance or hardness revealed by how the contact patch deforms under load, the microtexture felt as high-frequency vibration when a fingertip is dragged, and the thermal signature of the material (metal feels cold because it conducts heat away from the skin, wood does not). The what is this contact wrench and its evolution; the why it matters is that manipulation is controlled by exactly these variables, not by object identity. You do not lift a glass by knowing it is a glass; you lift it by keeping the tangential load inside the friction cone.

Formally, Coulomb's law says a static contact holds only while the tangential force stays under a friction-limited fraction of the normal force:

The instant \(\lVert \mathbf{f}_t \rVert\) crosses \(\mu f_n\), the object slips. Here is the crux: \(\mathbf{f}_t\), \(f_n\), and \(\mu\) are all interface quantities. A camera can watch an object begin to fall after it has already left the friction cone, but it cannot measure the margin \(\mu f_n - \lVert \mathbf{f}_t \rVert\) that predicts the fall. Touch can, and it can do so milliseconds before any visible motion. That predictive margin is the single clearest example of information that is present in the tactile channel and absent, in principle, from every distal one.

Why vision goes blind precisely at contact

The second argument is geometric, and it is the reason touch cannot simply be inferred from better cameras. At the moment that matters most, the moment of contact, the manipulator occludes the contact. When a gripper closes on a connector, the fingers hide the connector; when a hand grasps a mug, the palm and fingers cover the very surface patch bearing the load. This is the "last inch" problem of manipulation: perception is strong while the object is in free space and collapses in the final centimeter, exactly when the task succeeds or fails. The why is unavoidable geometry, not sensor quality: an eye-in-hand camera good enough to see under its own fingers would have to see through them.

Touch inverts this profile. It carries no signal at a distance (an untouched skin patch reports nothing) and maximum signal at contact, which is the complement of vision's curve. That complementarity is why touch belongs in the fusion stack of Chapter 50 rather than as a vision replacement: the two sensors are informative in disjoint regimes of the same task. It is also why touch is the natural partner of proprioception, the robot's sense of its own joint angles and end-effector forces developed in Chapter 57: proprioception tells the robot where its hand is, touch tells it what the hand has found.

Why it has been "missing": data, sensors, and representation

If touch is so informative, why is it absent from the models that already read ECG, radar, and lidar? The honest answer is not physics but infrastructure, and it is a three-part deficit. First, data: there is no ImageNet or LibriSpeech for touch. Tactile datasets are small, robot-specific, and expensive to collect, because every sample requires a physical contact rather than a scrape of the web. Second, sensor heterogeneity: a camera is a camera, but "a tactile sensor" spans camera-based gels, capacitive arrays, piezoresistive skins, and barometric taxels (all catalogued in Section 56.3), each with different resolution, modality count, and output format, so a model trained on one rarely transfers to another. Third, representation: the field has not agreed on what a tactile "frame" even is, an image, a force vector field, a point set, or a time series, which is the subject of Section 56.4.

These are the same three deficits that self-supervised learning solved for other modalities in Chapter 17, which is exactly why tactile foundation models are the current turning point rather than a distant hope. The small code below makes the friction-cone argument concrete and shows why the signal that resolves a grasp is a temporal one that only a high-rate contact sensor sees.

import numpy as np

# A held object: constant grip (normal force), rising pull (tangential force)
# as the arm accelerates. mu is the surface friction coefficient.
mu, f_n = 0.6, 5.0                      # newtons of grip
t  = np.linspace(0, 1.0, 200)          # 200 Hz tactile stream, 1 second
f_t = 6.0 * t                           # tangential load grows as arm lifts
margin = mu * f_n - f_t                 # >0 = held, <=0 = slipping

slip_idx = np.argmax(margin <= 0)       # first sample the object slips
print(f"friction limit = {mu*f_n:.1f} N")
print(f"slip begins at t = {t[slip_idx]*1000:.0f} ms, "
      f"pull = {f_t[slip_idx]:.2f} N")
# A 30 Hz camera samples every ~33 ms and only sees motion AFTER slip.
# The 200 Hz tactile margin crosses zero first, giving ~tens of ms warning.

As Listing 56.1 shows, the decisive variable is a force margin sampled fast, not an image. That is the shape of the whole modality: local, mechanical, high-bandwidth, and invisible from afar.

from datasets import load_dataset
# streaming avoids downloading the whole multi-GB tactile corpus
touch = load_dataset("touch_and_go", split="train", streaming=True)
sample = next(iter(touch))          # {'tactile': image, 'vision': image, 'material': label}

What's Next

In Section 56.2, we meet the sensors that actually capture these contact signals, starting with the camera-based family (GelSight, DIGIT, and the omnidirectional DIGIT 360) that turns a lump of soft gel and a tiny camera into a dense, high-resolution image of touch, converting the mechanical quantities argued for here into a stream a neural network can read.