Feature selection

"I generated 794 features per window, kept the 30 with the highest correlation to the label, and only later noticed I had ranked them on the same data I then tested on. My paper reported 96 percent. Reality reported 71."

A Chastened AI Agent

Why fewer features, precisely

What feature selection buys you is easiest to see through the geometry of the curse of dimensionality. With \(d\) features and \(n\) windows, a linear model has \(d\) parameters to fit from \(n\) examples; once \(d\) approaches or exceeds \(n\) the model can fit noise perfectly and generalizes badly. Why this bites sensor data especially hard is that windows are cheap to over-count and true examples are scarce: a study with 20 subjects and thousands of overlapping windows has an effective sample size closer to 20, because windows from one person are near-duplicates. So the honest \(n\) is small while \(d\) balloons. How selection helps is threefold. It improves generalization by reducing the parameter count relative to the effective \(n\); it slashes inference cost, which matters when a Chapter 59 edge target must compute every kept feature in real time; and it produces a model whose inputs a domain expert can name and sanity-check. When to prefer selection over the transforms of Section 8.6 is when interpretability or per-feature compute cost is a hard requirement, since PCA components are dense combinations of all original features and cost just as much to evaluate.

Filter methods: rank before you model

Filter methods score each feature against the label without training the downstream model, then keep the top ranked. They are fast, model-agnostic, and the natural first pass on a 5000-feature table. The simplest is a variance threshold: drop features that barely move across windows, since a near-constant feature carries no information. Next come univariate association scores. For a regression target, Pearson or Spearman correlation \(|\rho|\) ranks linear or monotone dependence; for classification, an ANOVA F-test or, better, mutual information

which unlike correlation captures nonlinear and non-monotone dependence, exactly the kind a threshold-crossing feature or a multi-modal spectral band produces. The catch with pure univariate ranking is redundancy: the top 30 features by mutual information are often 30 slightly different flavors of "how much energy is in this window," all strongly correlated with each other. You pay to compute 30 features and get the information content of about three.

Wrapper and embedded methods: let the model vote

Wrapper methods treat the downstream model as a black box and search over feature subsets, scoring each subset by cross-validated performance. Recursive feature elimination (RFE) is the common form: train the model, drop the least important feature, refit, and repeat until the target count remains. Wrappers find subsets tuned to the specific model and can capture interactions a filter misses, but they are expensive (many model fits) and prone to overfitting the selection to the validation folds unless nested carefully. Embedded methods get selection for free during a single training run. An L1-penalized (Lasso) linear or logistic model adds \(\lambda \sum_j |w_j|\) to the loss, which drives many coefficients exactly to zero, so the surviving nonzero features are the selection; the strength \(\lambda\) trades sparsity against fit. Tree ensembles (random forests, gradient boosting) expose an importance per feature, and permutation importance, shuffling one feature and measuring the accuracy drop, gives a more honest ranking than the built-in impurity importance, which is biased toward high-cardinality features. Embedded methods are usually the best accuracy-per-effort choice; filters are the best first-pass thinning; wrappers are worth the compute only when a small final subset must be squeezed for maximum accuracy. These importances also feed the interpretability and root-cause tooling of Chapter 67.

The leakage trap that inflates half of published results

Feature selection uses the label, so it is part of the model, and it must be fit only on training data. The single most common sensor-ML mistake is to rank or filter features on the whole dataset and only afterward split into train and test. This lets the test set vote on which features look good, and the reported accuracy is optimistic, often by 10 to 25 points, exactly the gap in this section's epigraph. The correct pattern is to wrap selection and model in one pipeline and cross-validate the whole thing, so selection is refit from scratch on each training fold and never sees its fold's test data. Sensor data adds a second requirement on top: the splits must be grouped by subject, device, or session (Chapter 5), because otherwise a feature that encodes subject identity will be selected as if it were predictive, and your grouped test set will expose it. Rigorous benchmarking of the whole selection-plus-model pipeline is the subject of Chapter 65.

The code below builds selection into a scikit-learn pipeline and evaluates it with grouped cross-validation, so the numbers it prints are honest by construction.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_selection import VarianceThreshold, SelectKBest, mutual_info_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, GroupKFold

# X: (n_windows, n_features), y: labels, groups: subject id per window
pipe = Pipeline([
    ("drop_dead", VarianceThreshold(threshold=1e-8)),   # remove near-constant features
    ("rank",      SelectKBest(mutual_info_classif, k=20)),  # keep 20 by mutual information
    ("clf",       LogisticRegression(penalty="l1", C=0.5, solver="liblinear")),
])

# selection is REFIT inside every training fold; folds never share a subject
cv = GroupKFold(n_splits=5)
scores = cross_val_score(pipe, X, y, groups=groups, cv=cv, scoring="f1_macro")
print(f"grouped CV macro-F1: {scores.mean():.3f} +/- {scores.std():.3f}")

Swap SelectKBest outside the pipeline and rank on all of X, and the printed F1 typically jumps, but that jump is the leak, not skill. Keeping the two selectors and the classifier in one estimator is what makes the difference between a number you can ship and a number you have to retract.

Stability: does the selection survive a different sample

A subset that changes wildly when you resample the data is a warning sign, not a model. Stability selection addresses this by running the selector on many bootstrap resamples (grouped by subject) and keeping only features chosen in, say, at least 70 percent of runs. The payoff is a subset that reflects reproducible structure rather than one lucky split, which matters enormously when the feature list becomes a specification: the 18 features the maintenance team ships must be computable on every deployed controller, and a list that reshuffles each retraining is unmaintainable. Report selection frequency alongside accuracy; a feature kept 95 percent of the time is trustworthy, one kept 55 percent is a coin flip dressed as a signal.

What's Next

In Section 8.6, we turn from keeping a subset of the original features to transforming them into new axes: PCA and ICA for linear compression and source separation, and the manifold methods UMAP and t-SNE that unfold nonlinear structure for visualization and embedding. Where selection preserves named, auditable features, these methods trade interpretability for a denser, lower-dimensional representation, and we will see exactly when that trade is worth making for sensor streams.