Slip, contact, and material inference

"I gripped the wine glass with the confidence of a model that had never felt anything shift. The floor felt it first."

A Chastened AI Agent

Contact detection and localization: the gating event

Everything begins with knowing that skin has touched something. Contact detection is a change-detection problem in the exact sense of Chapter 12: the sensor sits in a quiet baseline, and contact is the abrupt onset that must be caught with low latency and few false positives. For a camera-based sensor the signal is the deviation of the elastomer image from its no-load reference; for a taxel array it is a pressure crossing a threshold. The trap is that the threshold is not universal. A gel that has drifted with temperature, or an e-skin whose baseline creeps with viscoelastic relaxation (a failure mode flagged in Section 56.3), will fire spuriously if the threshold is fixed. The robust practice is a per-taxel adaptive baseline plus a spatial-coherence test: real contact activates a contiguous patch, whereas sensor noise flickers independently across taxels.

Localization refines "something touched" into "an object touched here, over this area, with this normal force." From the deformation field you recover the contact patch centroid \(\mathbf{c}\), its area \(A\), and, by integrating the estimated pressure, the normal force \(F_n = \int_A p(\mathbf{x})\, d\mathbf{x}\). These three quantities are the currency of contact-rich control: the centroid steers a regrasp, the area distinguishes an edge from a face, and \(F_n\) closes the force loop. Because they are derived from a physical measurement model, their error budget inherits directly from the sensor-physics discipline of Chapter 2.

Slip detection: catching failure before it happens

Slip is the crown jewel because of its timing. Coulomb's law says a grasp holds as long as the tangential force stays inside the friction cone, \(F_t \le \mu F_n\), where \(\mu\) is the friction coefficient. The naive strategy, measure \(F_t\) and \(F_n\) and watch the ratio, fails in practice because \(\mu\) is unknown and varies with material, moisture, and speed. The decisive insight is that a compliant fingertip warns you first. Before the whole contact patch slides (gross slip), the outer annulus of the contact begins to micro-slide while the center still sticks. This incipient slip is a partial-slip regime, and it is directly observable: on a marker-based sensor like GelSight or GelSlim, the tracked dots at the patch periphery start moving while the central dots stay pinned, so the displacement field develops a tell-tale divergence and non-uniform shear.

Two complementary detectors dominate. The geometric one watches the marker displacement field \(\mathbf{u}(\mathbf{x})\) and flags incipient slip when the fraction of the contact area that is sliding, estimated from where local shear exceeds the stick limit, crosses a threshold. The spectral one exploits the fact that micro-slips excite high-frequency vibrations in the elastomer or in a piezoelectric skin: sliding across texture produces a broadband buzz in roughly the 100 Hz to 1 kHz band that is absent during a stable stick. This is the time-frequency reasoning of Chapter 7 applied to skin: the onset of spectral energy is the onset of slip, and it precedes the geometric collapse of the grasp. Fusing both is standard, and learned detectors (often a small temporal-convolutional or recurrent model from Chapter 14) simply learn this joint signature from labeled slip episodes.

Listing 56.7.1 shows a minimal incipient-slip detector from a marker displacement field. It measures how non-uniform the shear is across the contact: a rigid stick moves all markers together (low dispersion), while partial slip stretches the field (high dispersion at the rim). The scalar it returns is the kind of feature a downstream controller thresholds or feeds to a learned model.

import numpy as np

def incipient_slip_score(markers_ref, markers_now, contact_mask):
    """Estimate an incipient-slip score from tracked marker displacements.
    markers_ref/now: (N, 2) pixel positions at reference and current frame.
    contact_mask:    (N,) bool, markers inside the contact patch.
    Returns a scalar in [0, inf): ~0 for a rigid stick, growing as the
    contact-patch periphery begins to micro-slide (partial slip)."""
    u = markers_now - markers_ref                 # per-marker displacement
    u_in = u[contact_mask]
    if len(u_in) < 4:
        return 0.0
    mean_u = u_in.mean(axis=0)                     # bulk (rigid) shift
    residual = u_in - mean_u                       # non-uniform part
    dispersion = np.linalg.norm(residual, axis=1)  # per-marker slide magnitude
    # rim markers sliding while the center sticks -> high 90th percentile
    return float(np.percentile(dispersion, 90))

Material inference: hardness, texture, and thermal cues

A grasp that respects the object also has to know what the object is made of, and skin can infer this from three quasi-orthogonal cues. Hardness (compliance) comes from the indentation response: press with a known normal force and watch how the contact area grows. A soft object spreads under load (large \(dA/dF_n\)); a rigid one barely deforms. This is a static, single-touch measurement. Texture and roughness come from motion: sliding the fingertip across a surface at velocity \(v\) over a spatial period \(\lambda\) excites a vibration at frequency \(f \approx v/\lambda\), so a coarse weave and a fine silk produce different spectral fingerprints, exactly the duality of Chapter 7. Because \(f\) scales with exploration speed, the classifier must be trained with speed as a nuisance variable or fed velocity as a covariate, or it will call the same fabric different names at different speeds. Thermal cues separate materials by heat draw: a heated skin pressed to metal loses temperature fast (high thermal effusivity) while wood or foam barely cools it, so a simple cooling curve separates metal from plastic that looks identical to a camera.

Across all three inferences, evaluation is where projects quietly fail. If train and test episodes share the same physical object instances, a model memorizes the object rather than learning slip physics or material properties, and the leakage-safe splitting discipline of Chapter 5 is not optional. Split by object identity and by exploration trajectory, never by random frame, or a slip detector will post a stellar number in the lab and drop the first unseen mug.

What's Next

In Chapter 57, touch stops being a standalone sense and joins the robot's body. We turn to proprioception and exteroception: encoders, force/torque sensors, and IMUs that tell the machine where its own limbs are, fused with the outward-looking depth and lidar streams that tell it where the world is. Slip and contact become one input to a full perception stack that must localize, detect failures, and act within a real-time budget.