Tactile representations and features

"A finger does not report megapascals. It reports where the world pushed back, how hard, and which way it is starting to slide. My job is to write that down in numbers a network can love."

A Recently Fingertipped AI Agent

This section assumes you can turn a signal into features (the general toolkit of Chapter 8), read a spectrum (Chapter 7), and reason about learned embeddings for sensor streams (Chapter 13). What is new here is the physics of contact, which constrains every feature: touch is local, it carries a direction (normal versus shear), and its signature spans two very different time scales.

The three shapes of a tactile signal

Before choosing features you must name the data structure, because it fixes the architecture exactly as the modality axes did in Chapter 1. Tactile sensors emit one of three shapes.

The tactile image. A camera-based sensor gives a dense 2D field: raw RGB, or, after photometric stereo, a per-pixel surface-normal map \(\mathbf{n}(u,v)\) that integrates to a height map \(z(u,v)\) of the imprinted geometry. Printed markers add a second field, a displacement vector \(\mathbf{d}(u,v)\) per marker tracking how the gel sheared. This is the richest tactile shape, spatial resolution in the tens of microns, and it is processed like any image with 2D convolutions or a vision transformer.

The taxel grid over time. A capacitive or piezoresistive array is a low-resolution pressure movie: a tensor of shape \((T, H, W)\) with \(H, W\) in the range of 4 to 32. Treat it as a single-channel video. The spatial dimension is coarse but the temporal dimension is fast, which is where slip and vibration hide.

The irregular graph. Electronic skin wrapped over a curved robot link is not a grid at all. Taxels sit at scattered 3D positions with a known neighbor topology, so the natural object is a graph whose nodes carry pressure and whose edges carry geodesic distance. This is exactly the setting for the graph neural networks of Chapter 54, and it is why "flatten to a vector" quietly destroys e-skin data: it throws away the geometry that says which taxels are physically adjacent.

Handcrafted features: geometry, force, and texture

Before reaching for a network, a handful of physics-grounded features solve a surprising share of tactile tasks, and they stay interpretable, which matters for the safety cases of Chapter 68. Group them by the physical question they answer.

Contact geometry. Threshold the pressure or height field to get a binary contact mask, then read its image moments. The zeroth moment is contact area \(A\); the first moments give the pressure centroid \((\bar u, \bar v)\), the point about which contact torque balances; the second central moments give the covariance whose eigenvectors are the contact's principal axes and whose eigenvalue ratio is its elongation. An edge presses as a thin high-eccentricity blob, a flat face as a broad round one. These few numbers already separate most everyday contact primitives.

Force and its direction. Summing pressure over the mask estimates total normal force \(F_z\). Direction comes from the marker field: the mean displacement \(\overline{\mathbf{d}}\) is proportional to net shear \((F_x, F_y)\), while the divergence of the displacement field reports normal load and its curl reports applied torque about the contact normal. The ratio of shear magnitude to normal force, compared against an estimated friction coefficient \(\mu\), is the classic incipient-slip signal that Section 56.7 builds on.

Texture and micro-vibration. Slide across a surface and fine structure excites vibrations. The features here are spectral, not spatial: band-power, spectral centroid, and roughness estimates from the high-frequency channel, the tactile analogue of the acoustic features in Chapter 7. Corduroy and glass differ far more in their vibration spectrum than in any static pressure map.

The code below computes the geometry block from a single pressure frame. It is the seed of a real tactile feature extractor and is referenced again in the exercise.

import numpy as np

def contact_features(P, thresh=0.05):
    """Geometry features from one HxW tactile pressure frame P (>=0)."""
    mask = P > thresh * P.max()
    if not mask.any():
        return dict(area=0, cx=np.nan, cy=np.nan, force=0.0, elong=np.nan)

    ys, xs = np.nonzero(mask)
    w = P[ys, xs]                       # pressure as pixel weight
    force = float(w.sum())              # total normal force (uncalibrated)
    cx = float((w * xs).sum() / w.sum())  # pressure centroid, x
    cy = float((w * ys).sum() / w.sum())  # pressure centroid, y

    # weighted second central moments -> principal-axis elongation
    dx, dy = xs - cx, ys - cy
    cov = np.array([[(w*dx*dx).sum(), (w*dx*dy).sum()],
                    [(w*dx*dy).sum(), (w*dy*dy).sum()]]) / w.sum()
    lam = np.linalg.eigvalsh(cov)       # ascending eigenvalues
    elong = float(np.sqrt(lam[1] / max(lam[0], 1e-9)))
    return dict(area=int(mask.sum()), cx=cx, cy=cy, force=force, elong=elong)

# a synthetic "edge" contact: a thin diagonal ridge
P = np.zeros((16, 16))
for i in range(16):
    P[i, min(15, i)] = 1.0
print(contact_features(P))

Learned tactile representations

Handcrafted features are transparent but brittle at the edges: they assume you already know which physical quantity matters. When the task is "does this grasp feel stable" across thousands of unseen objects, the mapping from raw gel image to answer is too tangled to hand-design, and you learn the representation instead. Three patterns dominate.

Treat the tactile image as an image. A ResNet or vision-transformer backbone consumes the raw or normal-map frame and emits an embedding, following the neural-representation playbook of Chapter 13. Because a tactile frame really is an image, ImageNet-pretrained weights transfer as a useful initialization, an accident of camera-based tactile hardware that array sensors do not enjoy.

Encode the sequence. For slip and manipulation, a single frame is not enough; a temporal convolution or transformer over a window produces a spatio-temporal embedding that carries the dynamic band the geometry features drop.

Pretrain without labels. Tactile labels are expensive (you cannot crowdsource the feel of an object), so self-supervised objectives, contrastive learning across augmented touches or across the visual and tactile views of one contact, learn a general encoder from unlabeled interaction logs. This is the self-supervised recipe of Chapter 17, and it is the direct on-ramp to the tactile foundation models of Section 56.5.

Making representations sensor-invariant

A tactile feature is worthless if it means something different on Tuesday. Three effects break naive representations, and handling them is the unglamorous core of a deployable pipeline.

Zero drift and offset. Piezoresistive and capacitive taxels drift with temperature and viscoelastic relaxation, so the "no contact" baseline wanders. Always subtract a running no-load reference before extracting features; report pressure relative to baseline, never as a raw reading. This is the tactile instance of the leakage-safe hygiene stressed in Chapter 5: if calibration state leaks across the train and test split, your accuracy is a fiction.

Per-sensor gain. No two gel fingers or e-skin patches are identical; illumination, membrane thickness, and taxel sensitivity all vary. Normalize each sensor to a common scale (per-taxel min-max or z-scoring against a calibration press) so a model trained on finger A transfers to finger B. Cross-sensor generalization is the headline weakness of hand-tuned tactile systems and the main thing the foundation models above buy back.

Illumination and marker bias (camera sensors). Convert to a photometric-stereo normal map early. The normal map depends on geometry, not on the LED that happens to be brightest, so it is far more portable across sensor units than raw RGB.

from digit_interface import Digit          # or gelsight sdk
d = Digit("D20001"); d.connect()
frame = d.get_frame()                       # calibrated, baseline-subtracted
# gelsight_sdk: nn reconstruction to gradients -> height
gx, gy = reconstructor.get_gradients(frame)
height  = poisson_reconstruct(gx, gy)       # one call

What's Next

In Section 56.5, we take the "reuse a learned representation" thread to its conclusion: tactile foundation models. Sparsh, UniTouch, and T3 replace the per-sensor feature engineering of this section with a single pretrained encoder that transfers across sensor bodies and tasks, and we will see what they can and cannot do, and how they connect to the visuo-tactile fusion of Section 56.6.