"They gave me a camera, pointed it at a lump of jelly, and called it a fingertip. I have never felt more, or seen less."
A Recently Fingertipped AI Agent
Why turn a fingertip into a camera
Section 56.1 argued that touch is the modality machine perception keeps skipping. This section covers the trick that made high-resolution touch cheap and practical: stop trying to wire thousands of tiny pressure cells and instead watch a soft skin deform with an ordinary camera. A camera-based (optical) tactile sensor presses a transparent elastomer against the world, lights it from inside, and films the geometry that the contact carves into the gel. One CMOS sensor already made in the billions replaces a bespoke taxel array, and every mature tool from computer vision, convolutional networks, optical flow, photometric stereo, transfers to the touch signal unchanged. This is the design behind GelSight, DIGIT, and DIGIT 360, and it is why a robot fingertip can now resolve a fingerprint ridge, a bar-code embossing, or the exact instant an object begins to slip.
This section assumes you are comfortable with the measurement model \(x = h(s) + \eta\) from Chapter 2 and with treating an image as a tensor a convolutional network can consume, developed in Chapter 13. The recurring theme here is that an optical tactile sensor is a small, deliberately blinded camera whose "scene" is the deformation of its own skin, so the hidden state \(s\) we want (contact geometry, force, slip) has to be inferred from pixels by inverting a well-controlled optical setup.
The retrographic principle: sensing shape by lighting it from within
The core idea, introduced by GelSight, is retrographic sensing. A slab of soft, transparent silicone is coated on its outer face with an opaque, matte, reflective membrane. When an object presses in, the membrane conforms to the object's microgeometry, but because the coating is opaque the camera never sees the object itself. It sees only the painted skin. That is the point. The membrane's reflectance is fixed and known, so its apparent brightness depends only on its local surface orientation relative to the internal lights, not on the color, transparency, or texture of whatever is being touched. A glass edge, a black rubber seal, and a printed letter all produce the same clean geometric readout.
Illumination is what makes the orientation legible. Colored LEDs are placed around the gel at grazing angles, typically red, green, and blue from three different directions. Each channel of the resulting RGB frame is, in effect, one image of the same surface under one light. A facet tilted toward the red light reddens; one tilted toward the blue light blues. This is exactly the setup of photometric stereo, a shape-from-shading technique detailed among the depth methods of Chapter 40. Under a Lambertian membrane the measured intensity in channel \(i\) is
$$ I_i = \rho\,\mathbf{n} \cdot \mathbf{l}_i, \qquad i \in \{R, G, B\}, $$where \(\rho\) is the fixed albedo, \(\mathbf{n}\) is the unit surface normal at that pixel, and \(\mathbf{l}_i\) is the known direction to light \(i\). Three lights give three equations, enough to solve for the two free angles of \(\mathbf{n}\) per pixel. In practice the map from \((R,G,B)\) to the surface gradient \((\partial z/\partial x, \partial z/\partial y)\) is not solved analytically but calibrated once by pressing a ball of known radius against the gel and recording which color triple corresponds to which slope. That lookup table absorbs every non-ideality of the real optics.
The signal is geometry, not pressure
A capacitive taxel array (Section 56.3) measures force per cell. An optical tactile sensor primarily measures shape: the height map of the contact patch at camera resolution, often hundreds of thousands of effective points from a single frame. Force and slip are recovered as second-order quantities, from how much the gel bulges and from how printed marker dots drift. This is why one megapixel CMOS sensor can out-resolve a thousand-taxel array by three orders of magnitude, and why the hard problems become optical (calibration, gel wear, illumination drift) rather than electrical (wiring, crosstalk).
From frame to geometry: gradients, integration, and marker fields
Once the calibrated lookup turns each pixel's color into a surface gradient, recovering the height map \(z(x,y)\) is an integration problem. The gradient field is generally not exactly conservative because of noise, so we solve for the height whose gradients best match the measurement in a least-squares sense. The classic Frankot-Chellappa method does this in one Fourier-domain step, and it is short enough to run in a few lines of NumPy. The snippet below synthesizes the height a gel would take pressing on a small bump, throws away the height to keep only gradients (mimicking what the sensor actually measures), and reconstructs the height back.
import numpy as np
# The surface the gel takes when pressing on a small bump (ground-truth height).
H = W = 128
yy, xx = np.mgrid[0:H, 0:W]
r2 = (xx - W / 2) ** 2 + (yy - H / 2) ** 2
truth = 6.0 * np.exp(-r2 / (2 * 15.0 ** 2)) # height in gel units
# A GelSight frame yields surface *gradients* via the RGB->normal calibration,
# not height. Emulate that measurement:
gx, gy = np.gradient(truth)
# Frankot-Chellappa: integrate gradients into a height map in Fourier space.
fx = np.fft.fftfreq(W).reshape(1, W)
fy = np.fft.fftfreq(H).reshape(H, 1)
denom = (2 * np.pi * fx) ** 2 + (2 * np.pi * fy) ** 2
denom[0, 0] = 1.0 # avoid divide-by-zero at DC
Gx, Gy = np.fft.fft2(gx), np.fft.fft2(gy)
num = -1j * (2 * np.pi * fx) * Gx - 1j * (2 * np.pi * fy) * Gy
z = np.real(np.fft.ifft2(num / denom))
z -= z.min()
print(f"peak height truth={truth.max():.2f} recovered={z.max():.2f}")
The second measurement channel is shear. Many optical tactile skins print a regular grid of black dots on the membrane. When the contact drags or twists, the dots move, and their displacement field is a direct read on tangential force and incipient slip. Tracking the dots is ordinary sparse optical flow: a diverging dot field means the object is being squeezed, a uniform drift means it is sliding, and a rotating field means torque. Slip, contact, and material inference built on these fields are the subject of Section 56.7; here the point is only that both geometry and shear fall out of standard vision operators applied to the gel image.
Robotics: seating a USB connector by feel
A tabletop manipulator at a small-electronics assembly cell has to insert a USB-A plug into a recessed port, a task where a millimeter of misalignment jams the connector and vision is occluded by the gripper the instant contact begins. Fitting DIGIT sensors on the two fingers turned the problem into an image problem. As the plug's leading edge touches the port lip, the gel renders a sharp height ridge whose orientation reveals the misalignment angle; the marker field shears in the direction the plug is being pushed off-axis. A small convolutional policy reads the two 320-by-240 gel streams and nudges the wrist until the ridge centers and the shear relaxes, then commits to the push. The cell reached reliable one-try insertion on connectors whose tolerances defeated a vision-only pipeline, because at the moment of contact the fingertip, not the overhead camera, held the information.
The sensor family: GelSight, DIGIT, and DIGIT 360
GelSight is the progenitor and the high-fidelity end of the family. Its emphasis is geometric resolution fine enough to read a \$20 bill's intaglio or measure surface roughness in microns, which made it a laboratory instrument for metrology and material study before it was a robot finger. GelSight units tend to be larger and are the reference against which resolution is quoted.
DIGIT, released open-source by Meta AI, re-engineered the same retrographic principle for manipulation at scale: compact enough to mount on a robot fingertip, built from injection-moldable parts, and cheap enough to instrument a whole hand. It trades some of GelSight's peak resolution for ruggedness, a snap-in replaceable gel (the wear part), and published hardware so labs can build identical sensors and share data. That standardization is what let tactile datasets and the foundation models of Section 56.5 accrete around a common sensor.
DIGIT 360 is the multimodal successor. Instead of a flat gel it uses a hemispherical fingertip with a wide-field lens, so contact anywhere on a rounded fingertip is imaged, closer to the omnidirectional sensitivity of a human pad. It adds on-sensor compute and channels beyond geometry, including contact-borne sound and other transduction, moving a single fingertip toward reporting geometry, force, vibration, and acoustic cues together. It reframes the fingertip as a small multimodal sensor hub rather than a lone camera.
Research Frontier
DIGIT 360 (Lambeta et al., Meta AI, 2024, "Digitizing Touch with an Artificial Multimodal Fingertip") pushes optical touch from a single geometry image toward a dense multimodal fingertip: a hyperfovea lens covering the whole hemispherical pad, over eight million effective sensing elements, on-device inference, and additional modalities such as contact audio and heat. The open frontier is representation and fusion: how to encode a fingertip that emits geometry, shear, vibration, and sound at once into a token stream that a policy can act on, and how to pretrain across the many incompatible optical-tactile designs so one model transfers between GelSight, DIGIT, and DIGIT 360. Those cross-sensor tactile foundation models (Sparsh, UniTouch, T3) are the topic of Section 56.5.
Tradeoffs and when to reach for optical touch
Optical tactile sensing buys resolution and cheap manufacturing at a real cost. The gel is a consumable: it abrades, tears, and clouds, and its response drifts as it ages, exactly the non-stationarity flagged in Chapter 1, so recalibration is a maintenance task, not a one-off. The sensor is bulky compared to a thin e-skin patch because it needs a few millimeters of gel plus focal distance for the camera, which limits where it fits on a dexterous hand. Latency and power are those of running a camera and a small network per fingertip. And because collecting real tactile data means physically pressing thousands of objects, the field leans hard on simulators: TACTO and Taxim render synthetic GelSight and DIGIT images from mesh contacts, part of the synthetic-data and digital-twin toolkit in Chapter 55, so policies can be trained in sim and transferred. Reach for an optical sensor when you need fine contact geometry and slip on a fingertip-scale contact and can tolerate the bulk; reach for the capacitive and piezoresistive arrays of Section 56.3 when you need thin, large-area, low-power coverage of a whole limb.
Let the sensor SDK handle capture and calibration
Streaming and configuring a DIGIT by hand means opening the USB video device, matching the right resolution and frame rate, driving the RGB LED intensities, and demosaicing, easily 150 to 200 lines of OpenCV and device plumbing that differ per operating system. The official digit-interface package collapses that to a handful:
from digit_interface import Digit
d = Digit("D20001") # connect by serial
d.connect()
d.set_intensity(Digit.LIGHTING_MAX)
d.set_resolution(Digit.STREAMS["QVGA"]) # 320x240 @ 60 fps
frame = d.get_frame() # calibrated BGR gel image, ready for your model
digit-interface. The library owns device enumeration, LED control, resolution and frame-rate negotiation, and color conversion, so downstream code sees a plain image tensor.The output is the same BGR frame you would feed the gradient-integration code above, or a convolutional network, so the SDK removes the hardware boilerplate without changing the perception pipeline.
Exercise: geometry versus force ambiguity
Take the reconstruction snippet above and press two different "objects" into the simulated gel: a tall narrow spike and a broad shallow dome scaled so both reach the same peak height. (1) Modify truth to produce each, run the Frankot-Chellappa recovery, and confirm the height maps are recovered correctly. (2) Now argue why the height map alone cannot tell you the normal force in newtons behind each contact, and describe two extra measurements a real GelSight-style sensor provides (hint: gel bulk deformation and marker displacement) that would break the ambiguity. (3) State which of the two ambiguity-breakers would still work if the object were perfectly rigid and stationary.
Self-check
- Why does an opaque, coated membrane make an optical tactile reading independent of the touched object's color and transparency, and what physical quantity does the camera actually measure through it?
- Three colored LEDs plus one RGB camera implement photometric stereo in a single frame. Write the per-channel intensity equation and explain why three lights are enough to recover a surface normal.
- Name one advantage and one hard limitation of DIGIT relative to a thin capacitive e-skin patch, and state a task where each design is the right choice.
What's Next
In Section 56.3, we cross to the other major tactile family: capacitive, piezoresistive, and electronic-skin arrays that measure force directly across large, thin, flexible surfaces. Where optical sensors trade bulk for resolution, these trade resolution for coverage and conformability, and comparing the two makes the design space of machine touch concrete.