"Tell me eleven joint angles and I will draw you a hand. Tell me six accelerations and I will guess the other forty."
A Kinematically Constrained AI Agent
Prerequisites
This section builds directly on the inertial measurement of Chapter 23 (what an accelerometer and gyroscope actually report) and on the orientation-estimation and drift material of Chapter 24, since every body-worn pose system lives or dies by how it fights integration drift. The estimation-under-uncertainty framing comes from Chapter 4, and the recurrent sequence models we use to lift sparse sensors to full pose are developed in Chapter 14. Unlike Section 27.2, which classifies discrete inertial gestures, this section estimates a continuous articulated configuration.
The Big Picture
A gesture is a label; a pose is a full articulated configuration: every joint angle of the hand, or the orientation of every segment of the body, recovered continuously in real time. That is a far richer and far harder target. A human hand has roughly 20 to 27 degrees of freedom; a full body skeleton has more than 50. Camera-based motion capture solves this from the outside, but wearables must solve it from the inside, with a handful of sensors strapped to a moving, deforming, sweating body. This section is about that inversion: how flex, stretch, magnetic, and inertial sensors report a low-dimensional shadow of a high-dimensional pose, and how a kinematic model plus a learned prior turns six noisy signals into a plausible forty-joint skeleton. Get this right and you drive avatars, surgical training, rehabilitation, and dexterous teleoperation from a garment instead of a camera rig.
What "pose" means, and why it is underdetermined
Represent a hand or body as a kinematic tree: a root segment and a chain of rigid links connected by joints, each joint contributing one or more rotational degrees of freedom. The pose is the vector of joint angles \(\boldsymbol{\theta}\); forward kinematics maps \(\boldsymbol{\theta}\) to the 3D position of every landmark \(\mathbf{p}_i = \mathrm{FK}_i(\boldsymbol{\theta})\) by composing per-joint rotations down the chain. Pose sensing is the inverse: recover \(\boldsymbol{\theta}\) from measurements. The central difficulty is that wearable sensors are almost always fewer than the degrees of freedom. A five-sensor bend glove gives one number per finger; the hand has three flexion joints per finger plus abduction. Six body-worn IMUs must explain a skeleton with dozens of joints. The map from measurements to pose is therefore underdetermined: many poses fit the data equally well, so recovery is impossible without a prior over how bodies actually move. That prior, whether a hand-tuned coupling ratio or a learned distribution, is the real product of this section.
Instrumenting the hand: bend, stretch, and magnetic gloves
Data gloves attach a sensor across each joint so its output tracks joint angle directly. Three transduction families dominate. Resistive bend sensors (the classic CyberGlove approach) are strips whose resistance changes as they flex; two per finger capture the two main flexion joints, and the transfer is near-linear over the usable range but drifts with temperature and glove fit. Capacitive stretch sensors use a soft elastomer dielectric whose capacitance rises as it stretches over a knuckle; they are washable, conform to the skin, and are the basis of modern soft sensing gloves, at the cost of hysteresis and a slow viscoelastic relaxation you must model or filter out (the denoising tools of Chapter 6 apply directly). Magnetic and inertial gloves place a tiny IMU or a magnetic tracker (Polhemus-style) on each fingertip and phalanx, giving orientation rather than a single bend scalar, which resolves abduction and the subtle joints a single bend strip misses, but multiplies wiring, power, and calibration burden.
Whatever the transducer, the glove reports a per-sensor scalar or orientation that must be turned into anatomically valid joint angles. This needs a per-user calibration: the same finger flex produces different raw values on different hands and different glove donnings, so systems ask the user to make a few canonical poses (flat hand, fist, spread) and fit an affine or piecewise map from raw signal to angle. Skipping calibration is the single most common reason a glove that demoed well feels wrong on a new user, a cross-user generalization problem that Section 27.5 attacks head-on for neuromotor signals.
Key Insight
A wearable never measures pose; it measures a projection of pose plus drift plus body-fit nuisance. The information you are missing is supplied by the kinematic model and a motion prior, not by the sensor. This is why the strongest hand and body trackers are not "better sensors" but better priors: they exploit the fact that human joints are coupled (you cannot bend the last knuckle without the middle one moving), that limbs have fixed lengths, and that real motion occupies a tiny, smooth manifold of the combinatorially huge joint-angle space. The sensor picks the point on that manifold; the prior supplies the manifold.
Full-body inertial mocap and the sparse-IMU problem
For the whole body, the dominant wearable approach straps an IMU to each major segment. A dense suit (Xsens MVN uses 17 IMUs; Perception Neuron is similar) covers pelvis, spine, upper and lower limbs, and hands. Each IMU gives a segment orientation via the fusion of Chapter 24; a skeletal model with known bone lengths then chains those orientations into a full pose and, by integrating the root, an approximate trajectory in space. The two enemies are familiar from dead reckoning: orientation drift (gyro bias integrated over time, corrected by accelerometer gravity and, where available, magnetometer heading) and positional drift of the un-anchored root, which is why inertial suits use foot-contact detection as a zero-velocity update to pin the feet and bound accumulated error.
Seventeen IMUs is accurate but intrusive; nobody wears a full suit to answer email. The research frontier is the sparse regime: recover a full body pose from only five or six IMUs (head, wrists, lower legs, pelvis), or even from the sensors already in a phone, watch, and earbuds. This is severely underdetermined, and it is exactly where learned priors earn their place.
import numpy as np
def finger_fk(theta, L=(0.045, 0.028, 0.020)):
"""Planar forward kinematics of one finger: 3 flexion joints -> fingertip (x, y).
theta: (mcp, pip, dip) joint angles in radians. L: phalanx lengths (m)."""
x = y = 0.0
a = 0.0 # cumulative angle along the chain
for ang, l in zip(theta, L):
a += ang # each joint rotates relative to the previous segment
x += l * np.cos(a)
y += l * np.sin(a)
return np.array([x, y])
def angles_from_bend(bend, coupling=0.66):
"""One bend scalar per finger -> 3 joint angles using an anatomical coupling prior.
A single sensor underdetermines 3 DOF; the prior distributes flexion across joints."""
mcp = bend # sensor tracks the base knuckle
pip = coupling * bend # coupled: distal joints follow, scaled
dip = coupling * coupling * bend
return np.array([mcp, pip, dip])
bend = np.deg2rad(60) # sensor reads a 60-degree base flexion
theta = angles_from_bend(bend)
print("joint angles (deg):", np.round(np.rad2deg(theta), 1))
print("fingertip (mm): ", np.round(finger_fk(theta) * 1000, 1))
coupling constant is the entire "model" that resolves the two missing degrees of freedom; a learned prior replaces this hand-picked ratio with a data-driven distribution. Referenced in the underdetermination and learned-prior discussions.The code above makes the underdetermination concrete: one sensor, three unknowns, closed only by the coupling constant. Replace that constant with a network trained on real hand motion and you have the modern approach.
Learned priors: lifting sparse sensors to full pose
The state of the art casts sparse-IMU pose estimation as sequence-to-sequence regression onto a parametric body model. The near-universal target is SMPL (Loper et al., 2015), which represents any body as a low-dimensional pose-and-shape vector that decodes to a full mesh, so the network predicts a compact code instead of dozens of raw angles. DIP (Deep Inertial Poser, Huang et al., 2018) first showed a bidirectional RNN mapping six IMUs to full SMPL pose in real time. TransPose (Yi et al., 2021) added a multi-stage design that also estimates global translation from the same six sensors, and PIP and Transformer Inertial Poser (2022) folded in a physics prior and attention so the recovered motion respects balance and foot contact rather than floating or sliding. IMUPoser (Mollyn et al., 2023) pushed the idea to the sensors people already carry, a phone, a watch, and earbuds, accepting lower fidelity for zero extra hardware.
Research Frontier
The live problem is physically plausible pose from ever-fewer sensors. TransPose and PIP set the modern bar for six-IMU real-time full-body capture, and physics-aware variants now suppress the foot-skating and ground-penetration artifacts that pure regression produces. Two directions are hot: (1) fusing sparse IMUs with a sparse visual or electromagnetic anchor to bound global drift, and (2) self-supervised pretraining on large unlabeled inertial corpora, following the wearable-foundation-model program of Chapter 20, so a single backbone transfers across users and sensor layouts without per-person retraining. The open weakness everywhere is heading drift from unreliable indoor magnetometers, which still forces periodic re-anchoring.
In Practice: a stroke-rehab glove that had to trust the prior
A neurorehabilitation group built a soft capacitive-stretch glove to track home exercises for stroke survivors relearning hand control. The clinical requirement was a faithful joint-angle trace for the therapist, not a game score. Two realities broke the naive readout. First, hemiparetic hands exhibit abnormal coupling: patients often cannot flex one finger without the neighbors flexing too, so a healthy-hand coupling prior systematically mis-estimated exactly the impairment the clinician needed to see. Second, the elastomer's viscoelastic creep meant a held fist slowly drifted in the raw signal even as the hand stayed still. The team fixed both by separating the two roles the prior was silently playing: they kept a strong kinematic prior for bone lengths and joint limits (which are patient-independent) but learned the coupling per patient from a short calibration session, and they modeled the creep as a slow baseline they subtracted, much as the capacitive front end of Section 27.1 tracks and removes drift. The result recovered the pathological coupling as signal instead of erasing it as noise, the exact failure mode a one-size-fits-all prior invites in clinical use.
Right Tool: don't hand-build the kinematic tree
Composing per-joint rotations, enforcing bone lengths, and decoding to a mesh is hundreds of lines of quaternion bookkeeping that is easy to get subtly wrong (a flipped axis desyncs the whole chain). A parametric-body library does it for you: with the official SMPL/SMPL-X body_models package you pass a pose-and-shape tensor and get posed joints and a full mesh back in about five lines, replacing a multi-hundred-line forward-kinematics and skinning implementation. Feed it the pose your network predicts and it handles skinning, joint regression, and shape blending; hand-roll the tree only for teaching, exactly as the toy code above does.
Fusion, drift, and when a wearable is the right choice
Choose a bend or stretch glove when you need hand joint angles in a garment, tolerate per-user calibration, and want washable comfort over per-finger orientation. Choose a per-segment IMU glove or suit when you need full 3D orientation and abduction and can pay the wiring and calibration cost. Choose the sparse learned approach when unobtrusiveness beats millimeter accuracy, for consumer avatars, fitness, and always-on interaction. In every case the wearable's blind spot is global position: IMUs are dead-reckoning devices whose absolute translation drifts without an external anchor, which is why serious systems fuse them with the localization sources of Chapter 25, or with a camera, and why the sensor-fusion machinery of Chapter 48 underlies every production body tracker. Pose feeds forward too: the joint-angle stream this section produces is the input to the activity and behavior recognition of Chapter 26 and to the dexterous teleoperation of robots in Chapter 57.
Exercise
You have six body-worn IMUs (head, both wrists, both lower legs, pelvis) streaming orientation at 60 Hz, and you want a full SMPL pose. (1) Explain, using the degree-of-freedom count, why a per-frame least-squares fit of joint angles to the six orientations is ill-posed, and name the two priors that make it solvable. (2) The recovered figure "skates" (feet slide along the floor) during walking. Which physical constraint is missing, and how would a zero-velocity foot-contact update or a physics term remove the artifact? (3) Design a leakage-safe evaluation: given a dataset of 30 subjects, how do you split train and test so a reported joint-angle error reflects true cross-user generalization rather than memorized body shapes (recall the discipline of Chapter 5)?
Self-Check
1. What does forward kinematics compute, and why is pose sensing the harder inverse of it?
2. A single bend sensor per finger cannot recover all three flexion angles. What supplies the missing information, and what goes wrong when that prior is copied from healthy hands onto an impaired one?
3. Why do inertial pose suits detect foot contact, and which kind of drift does that detection specifically bound?
What's Next
In Section 27.4, we move from measuring the moving hand to measuring the intent to move it, reading the surface electromyographic signals of the forearm muscles before the fingers act. That shifts the sensing target upstream of kinematics entirely, and it is the basis of the Meta neuromotor wristband and the emg2qwerty typing benchmark.