"Give me a clean label and I will learn anything. Give me a taxonomy and I will learn that the world was negotiable all along."
A Well-Supervised AI Agent
Prerequisites
This chapter assumes the inertial signal model of Chapter 23 (what an accelerometer and gyroscope actually measure) and the leakage-safe dataset discipline of Chapter 5. Basic probability and the notion of a classifier's output distribution come from the primer in Chapter 4. The feature and window concepts used in the code example are developed in Chapter 8.
The Big Picture
Before a single window of accelerometer data reaches a model, someone decided what counts as an "activity." That decision, the label set and how its members relate, quietly determines what your classifier can ever be right or wrong about. Is "walking upstairs" a separate class from "walking," or a refinement of it? Is "commuting" an activity, or a story we tell about a sequence of sitting, standing, and walking? Human activity recognition (HAR) lives or dies on these choices long before architecture matters. This section designs the label space itself: the taxonomy that names activities and the hierarchy that says which ones are kin. Get it wrong and you will spend the rest of the chapter fighting a problem you created in a spreadsheet.
What a taxonomy has to name
An activity taxonomy is a controlled vocabulary of the behaviors a system must recognize, with definitions precise enough that two annotators watching the same clip assign the same label. It is hard because human motion spans radically different time scales, and a flat list papers over that structure. Practitioners converge on a few natural strata. Postures (sitting, standing, lying) are near-static configurations. Locomotion or ambulation (walking, running, climbing stairs, cycling) is quasi-periodic whole-body motion. Transitions (sit-to-stand, stand-to-walk) are brief, non-stationary bridges that earn their own treatment in Section 26.3. Gestures and manipulations (opening a drawer, drinking) are short, limb-local actions. And activities of daily living or ADLs (cooking, commuting, exercising) are composite, goal-defined episodes assembled from all of the above.
The trap is mixing strata in one flat label set. If your classes are {walking, running, sitting, cooking}, you have asked the model to decide, from one window, between a locomotion mode, a posture, and a minutes-long episode that itself contains walking and sitting. Those decisions do not compete at the same level, and a softmax that treats them as mutually exclusive is being asked an incoherent question. First fix the level of abstraction you are labeling; then, if you need several, represent them explicitly rather than flattening them.
Granularity, the NULL class, and multi-label reality
Granularity is the first real design lever. Coarse labels (moving vs still) are cheap to annotate but carry little information; fine labels (walking-upstairs-carrying-object) are informative but data-hungry and inconsistently annotated because the boundaries blur. The right granularity follows the downstream decision, not what is technically distinguishable. A fall-detection wearable needs the coarse {upright, on-the-ground} distinction at high reliability; a gait clinic wants stride-level refinement. Do not annotate finer than your application will consume, because every extra class multiplies labeling cost and the leakage-safe evaluation burden of Chapter 5.
Two facts about real deployments break the tidy single-label picture. First, the NULL or "other" class: in continuous recording, most time is spent doing nothing your taxonomy names. The Opportunity dataset, a kitchen-ADL benchmark, is dominated by its NULL class, and a model that ignores it will hallucinate activities in the gaps. NULL is not a nuisance to discard; it is the majority prior that keeps precision honest. Second, activities co-occur. Walking while talking on the phone is two simultaneous truths, so context-rich benchmarks such as ExtraSensory (in-the-wild smartphone and smartwatch data) label each moment with a set of tags. When labels co-occur, the problem is multi-label classification with independent per-tag sigmoids, not single-label softmax, and confusing the two silently caps your accuracy.
Key Insight
The label set is a modeling assumption, not a given. "Single-label, mutually exclusive, flat" is the default a naive pipeline inherits, yet real activity data is hierarchical (walking-upstairs is-a walking), multi-label (walking and phoning), and NULL-dominated (mostly unlabeled time). Each mismatch shows up not as an error message but as a stubborn accuracy ceiling you cannot explain. Choose the label structure to match the phenomenon before you choose the loss.
Label hierarchies: trees, losses, and metrics
Once you accept that activities have kinship, you encode it as a hierarchy: a tree (or, for messier ontologies, a directed acyclic graph) whose leaves are the fine classes and whose internal nodes are coarser groupings. The UCI HAR smartphone benchmark, for instance, has six leaves under two parents, {walking, walking-upstairs, walking-downstairs} under dynamic and {sitting, standing, lying} under static. PAMAP2, with its eighteen protocol activities, groups likewise into locomotion, posture, and household strata. This structure changes three things.
Prediction. Hierarchical classification either trains one classifier per parent node and routes a sample down the tree, or trains a flat leaf classifier and reconciles its output with the hierarchy afterward. The per-node approach lets each decision use features suited to its level and degrades gracefully: an uncertain model can stop at "dynamic" rather than guess between three stair-like classes.
Loss. A flat cross-entropy treats confusing walking with running as exactly as costly as confusing walking with lying, which is rarely true. A hierarchy-aware loss weights each mistake by tree distance, so near-misses are penalized less: \[ \mathcal{L}_{\text{hier}} = -\sum_{c} w_{y,c}\, \log p_c, \qquad w_{y,c} = 1 + \lambda\, d_{\mathcal{T}}(y, c), \] where \(d_{\mathcal{T}}(y,c)\) is the edge count between true class \(y\) and class \(c\), and \(\lambda\) tunes how much the geometry matters.
Evaluation. Flat accuracy hides the difference between a model that confuses siblings and one that confuses strangers. Hierarchical precision and recall fix this by crediting shared ancestors. Writing \(\mathrm{An}(\cdot)\) for a label's ancestors (including itself), the hierarchical F-measure is built from \[ P_h = \frac{|\mathrm{An}(\hat y)\cap \mathrm{An}(y)|}{|\mathrm{An}(\hat y)|}, \qquad R_h = \frac{|\mathrm{An}(\hat y)\cap \mathrm{An}(y)|}{|\mathrm{An}(y)|}. \] Predicting "walking-downstairs" when the truth is "walking-upstairs" scores well above zero because both share the dynamic and walking ancestors, matching the intuition that the model nearly understood the motion. This connects to the leakage-safe, subject-independent protocols of Chapter 65: report hierarchical metrics on held-out subjects, never held-out windows.
In Practice: the clinical wearable that scored 96% and helped no one
A rehabilitation group deployed a wrist accelerometer to track post-stroke patients' daily mobility, using a flat six-class set copied from a smartphone benchmark. Offline the model hit 96% window accuracy, so it shipped. Clinicians promptly distrusted it: it confidently labeled slow, shuffling assisted walking as "standing," because the benchmark's "walking" implicitly meant healthy-gait walking and had no node for pathological locomotion. The fix was taxonomic, not architectural. The team added a walking parent with independent and assisted children, re-annotated with a physiotherapist writing the definitions, and switched to hierarchical recall on held-out patients. Window accuracy fell to 91%, but the metric now credited the clinically meaningful distinction, and the tool became usable. The lesson recurs across HAR: an impressive flat number on the wrong label space is worse than a modest number on the right one.
Ontologies, drift, and where labels come from
Beyond a fixed benchmark, the taxonomy is a living artifact. New activities appear (a user takes up rowing), definitions drift (what "exercise" means to a fitness app changes with a feature launch), and different sources use incompatible vocabularies. Mature systems therefore treat the label set as an ontology: named classes with written definitions, explicit parent links, cross-dataset synonym mappings, and a versioning policy. That is what lets you merge UCI HAR, PAMAP2, and your own logs into one corpus without colliding two senses of "cycling." It is also the substrate on which the personalization of Section 26.4 and the continual learning of Section 26.5 operate: you cannot add a class coherently with no principled place to hang it. Design the ontology so adding a leaf is a local edit, not a global relabel.
import numpy as np
# A small activity hierarchy as a parent map (child -> parent); root = None.
PARENT = {
"root": None,
"static": "root", "dynamic": "root",
"sitting": "static", "standing": "static", "lying": "static",
"walking": "dynamic", "upstairs": "walking", "downstairs": "walking",
}
def ancestors(label):
"""Set of a label and all its ancestors up to the root."""
out, node = set(), label
while node is not None:
out.add(node)
node = PARENT[node]
return out
def hierarchical_prf(y_true, y_pred):
"""Hierarchical precision, recall, F1 over ancestor sets."""
inter = tp = tpred = ttrue = 0
for yt, yp in zip(y_true, y_pred):
at, ap = ancestors(yt), ancestors(yp)
shared = at & ap
inter += len(shared); tpred += len(ap); ttrue += len(at)
P = inter / tpred
R = inter / ttrue
F = 2 * P * R / (P + R)
return P, R, F
truth = ["upstairs", "sitting", "walking"]
pred = ["downstairs", "standing", "walking"] # 2 near-misses, 1 exact
print("hierarchical P/R/F1 = %.3f %.3f %.3f" % hierarchical_prf(truth, pred))
# Flat accuracy would score the first two as 0; the hierarchy gives partial credit.
Right Tool: hierarchical classification without the plumbing
The parent map, per-node routing, and consistent hierarchical scoring above are exactly what hiclass packages. Wrapping any scikit-learn base estimator in LocalClassifierPerParentNode(RandomForestClassifier()) and calling .fit(X, Y_paths) gives you tree-consistent prediction and built-in hierarchical precision/recall, replacing roughly 120 lines of routing logic, ancestor bookkeeping, and metric code with about 5. Reach for it once your taxonomy has more than two levels; hand-rolling the routing is where subtle label-leakage and inconsistent-prediction bugs breed.
Exercise
Take the six UCI HAR classes and design a three-level hierarchy for a fall-prevention wearable whose only safety-critical decision is "upright vs on-the-ground." (1) Draw the tree and state, in one sentence each, the annotation definition of every node. (2) Argue which two leaves you could merge with no cost to the safety decision, and which single new leaf you would add. (3) Give one mistake the hierarchical F1 would forgive that flat accuracy would not, and explain why forgiving it is correct for this application.
Self-Check
1. Why is a flat softmax over {walking, sitting, cooking} an incoherent question, and what label structure fixes it?
2. In a continuously recorded HAR stream, why is discarding the NULL class a mistake even though NULL is "not an activity"?
3. Two models both score 90% flat accuracy; one confuses only sibling classes, the other confuses distant ones. Which hierarchical metric separates them, and which model would you ship?
What's Next
In Section 26.2, we turn from the label space to the physical one: where on the body a sensor sits and which modality you choose. A taxonomy that distinguishes stair-climbing from walking is only realizable if the sensor placement can actually see the difference, so the label design of this section and the hardware choices of the next are two halves of one decision.