PCA, ICA, and manifold methods (UMAP, t-SNE) for sensors

"Give me two hundred correlated channels and I will hand you back the three that were doing all the talking."

A Parsimonious AI Agent

This section assumes the covariance and eigen-decomposition background from Chapter 4, and it inherits the windowed feature tables built across the rest of this chapter. Every method here is unsupervised: it never sees your labels, which is both its strength (it works before you have labels) and its risk (the axis of greatest variance is not always the axis that matters).

PCA: rotate to the axes of maximum variance

PCA answers one question: if I must describe each feature vector with only \(k\) numbers instead of \(d\), which \(k\) linear directions preserve the most variance? Center the data, form the covariance matrix \(\mathbf{C} = \tfrac{1}{N}\mathbf{X}^\top\mathbf{X}\), and take its eigenvectors. The eigenvector with the largest eigenvalue is the first principal component, the direction along which the data spreads most; the second is the orthogonal direction of next-most spread, and so on. Projecting onto the top \(k\) eigenvectors gives the best possible \(k\)-dimensional linear approximation in a least-squares sense:

Why it works for sensors: physically coupled channels are highly correlated, so a handful of components typically recover 90 to 99 percent of the variance of a table with hundreds of columns. That buys you faster models, less overfitting, and a natural denoising step, because the smallest components are usually where broadband sensor noise hides. When to use it: as a first-line compressor before a classifier, as a whitening stage in front of ICA, and as a sanity check on redundancy. The one non-negotiable preprocessing step is standardization. PCA chases variance, so a feature measured in raw accelerometer counts (variance in the thousands) will swamp a feature measured as a dimensionless ratio unless you first scale every column to unit variance.

ICA: unmix statistically independent sources

PCA finds uncorrelated directions; ICA finds independent ones, a strictly stronger and often more physically meaningful goal. The setting is the classic cocktail party: several sensors each record a linear mixture \(\mathbf{x} = \mathbf{A}\mathbf{s}\) of several underlying sources \(\mathbf{s}\), and you want to recover an unmixing matrix \(\mathbf{W}\approx\mathbf{A}^{-1}\) so that \(\hat{\mathbf{s}} = \mathbf{W}\mathbf{x}\) gives back the sources. The trick that makes it possible is non-Gaussianity: by the central limit theorem a mixture of independent sources is more Gaussian than any single source, so algorithms like FastICA search for the projection directions that are maximally non-Gaussian (maximizing kurtosis or negentropy). Order and sign of the recovered sources are arbitrary, which you must account for downstream.

Where ICA earns its keep in sensing is artifact removal and source separation. In electroencephalography, eye blinks and cardiac interference are independent of brain activity and mix linearly across the scalp electrodes, so ICA can isolate them into their own components that you zero out before reconstructing a clean signal, a workhorse technique for the brain-computer interfaces of Chapter 31. It also separates overlapping motor-unit trains in high-density electromyography and untangles crosstalk in multi-antenna radio arrays. Standard practice is to whiten with PCA first (which also lets you drop the noise dimensions), then run ICA on the whitened, dimension-reduced data.

import numpy as np
from sklearn.decomposition import PCA, FastICA

rng = np.random.default_rng(0)
t = np.linspace(0, 8, 4000)
s1 = np.sin(2 * np.pi * 1.0 * t)            # a slow oscillation
s2 = np.sign(np.sin(2 * np.pi * 3.0 * t))   # a square wave
s3 = rng.uniform(-1, 1, t.size)             # noise-like source
S = np.c_[s1, s2, s3]
A = np.array([[1.0, 0.8, 0.4],              # unknown mixing matrix
              [0.6, 1.0, 0.7],
              [0.9, 0.5, 1.0]])
X = S @ A.T                                 # 3 "sensors", each a mixture

X = (X - X.mean(0)) / X.std(0)              # standardize before either method
pcs = PCA(n_components=3).fit_transform(X)  # uncorrelated, variance-ranked
src = FastICA(n_components=3, random_state=0).fit_transform(X)  # independent

def corr_to_truth(est):
    return np.abs(np.corrcoef(est.T, S.T)[:3, 3:]).max(1).round(2)
print("PCA best match to each true source:", corr_to_truth(pcs))
print("ICA best match to each true source:", corr_to_truth(src))

The code above makes the distinction concrete: PCA hands back components that are uncorrelated yet still mixtures of the true sources, whereas ICA recovers the individual square wave, sinusoid, and noise source, because independence is a stronger constraint than mere decorrelation.

Manifold methods: t-SNE and UMAP for seeing structure

PCA is linear: it can only rotate and project. Real sensor data often lies on a curved, low-dimensional manifold embedded in high-dimensional feature space (think of the smooth loop a gait cycle traces), and a linear projection smears that structure. Manifold methods embed the data into two or three dimensions while preserving local neighborhoods. t-SNE (t-distributed stochastic neighbor embedding) models pairwise similarities as probabilities and arranges points so near neighbors stay near; its perplexity parameter sets the effective neighborhood size. UMAP (uniform manifold approximation and projection) builds a fuzzy nearest-neighbor graph and optimizes a low-dimensional layout that matches it; it is faster, scales to millions of points, better preserves global structure, and (unlike classic t-SNE) can transform new points with a fitted model.

When to use them: almost exclusively for visualization and diagnosis, not as features for a downstream model. A UMAP plot of your windowed features colored by class tells you at a glance whether the classes are separable at all, whether two labels are hopelessly overlapping, and whether one subject or device forms its own island (a red flag for the leakage discipline of Chapter 5). What they are not: a metric space. Distances between clusters, cluster sizes, and densities in a t-SNE or UMAP plot are largely artifacts of the algorithm and the hyperparameters. Read them as "these points are neighbors," never as "this gap is twice that gap."

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import umap
emb = make_pipeline(StandardScaler(), PCA(n_components=30),
                    umap.UMAP(n_neighbors=15, min_dist=0.1)).fit_transform(X)

Using them honestly for sensors

Three cautions keep these methods from betraying you. First, fit on train only. A PCA basis, an ICA unmixing matrix, and a UMAP model are all fitted objects with learned parameters; fitting any of them on the full dataset before splitting leaks test information and inflates your reported accuracy, the exact trap Chapter 5 warns against. Fit on the training fold, then transform validation and test. Second, match the tool to the goal: PCA to compress and denoise, LDA when you have labels and want separation, ICA to recover physical sources, manifold methods to look. Third, resist using t-SNE or UMAP coordinates as model inputs; they are non-deterministic across runs, unstable to hyperparameters, and distort the metric, so a classifier trained on them is standing on sand. Compress with PCA if you need learned low-dimensional features; visualize with UMAP.

What's Next

In Section 8.7, we step back and ask the honest question that hangs over this entire chapter: when do these carefully handcrafted, dimensionality-reduced features still beat an end-to-end deep network, and when is it time to let a model learn its own representation from the raw signal?