Section 37.3: First-shot and domain-generalized anomalous sound detection (DCASE)

"You have never heard this machine before, you have never heard it break, and you must decide, on its very first recording, whether it is dying. Welcome to first-shot detection."
An Acoustically Nervous AI Agent

The Big Picture

Section 37.2 gave you the machinery of acoustic and vibration anomaly detection: turn a microphone stream into a log-mel spectrogram, learn what "normal" sounds like, and flag departures. That works beautifully when you can collect a fat library of normal recordings from the exact pump you will monitor, in the exact room, at the exact operating point. Factories do not grant that luxury. A new machine arrives, a line is re-tooled, the same valve model is installed on three continents with different mains frequency and background noise, and you are asked to have a detector running on day one with only a handful of normal clips and zero examples of failure. The DCASE (Detection and Classification of Acoustic Scenes and Events) challenge turned this exact pain into a public, ruthlessly-scored benchmark. Its machine-condition task has walked the field from "one model per machine, tuned lovingly" to domain generalization (survive a shift you were warned about but shown no labels for) and finally to first-shot detection (perform on a machine type you have never seen, with no per-type tuning at all). This section explains what those two words mean precisely, why the winning systems are discriminative rather than reconstructive, and how the scoring rules stop you from fooling yourself.

This section assumes you have the spectrogram and time-frequency vocabulary of Chapter 7 and the unsupervised acoustic detectors of Section 37.2. The embedding trick at its heart is self-supervised representation learning from Chapter 17, and the domain-shift framing connects directly to the out-of-distribution and test-time-adaptation ideas of Chapter 66.

From closed-set to first-shot: how the DCASE task evolved

DCASE Task 2 is unsupervised anomalous sound detection (ASD): training data contains only normal machine sounds, anomalies appear only at test time, and you must output a real-valued anomaly score per 10-second clip. The task defines the frontier of the field, so tracking its versions is the fastest way to understand what "hard" currently means.

2020 (closed-set). Six machine types (fan, pump, valve, slider, and two toy sets from MIMII and ToyADMOS). You trained a dedicated model per machine type and even per machine ID. Autoencoders dominated: reconstruct the normal spectrogram, score the reconstruction error.
2021 (domain shift introduced). Training data came from a source domain (say, a specific speed and load) but test data mixed in a target domain (a different speed, a noisier room). A few target-domain normal clips were provided, so teams could adapt.
2022 (domain generalization). The rules hardened: you were given plenty of source-domain data and only a handful (often ten) of target normals, and were forbidden from knowing at test time which domain a clip came from. One model had to handle both. MIMII-DG and ToyADMOS2 were the datasets.
2023 onward (first-shot). The training and evaluation machine types no longer overlap, and no per-machine-type hyperparameter tuning is allowed. Your system meets a completely new machine type (a gearbox, a bandsaw) at scoring time and must run out of the box. This is the closest public analogue to a real factory deployment.

The arc matters because each step deletes an assumption that hand-tuned condition monitoring silently relies on: same machine, same operating point, same room, engineer available to tune. First-shot deletes all of them at once.

Domain generalization, not adaptation

The distinction that trips people up is generalization versus adaptation. In domain adaptation you get labeled or at least abundant target data and are allowed to fit to it. In domain generalization you must produce one fixed detector that works on target conditions you were shown only faintly (a few normal clips) or not at all, without any test-time retraining keyed to the target. Formally, let a clip's log-mel feature be $\mathbf{x}$ and let $d \in \{\text{source}, \text{target}\}$ be its hidden domain. The training distribution is dominated by source, $P_{\text{train}}(\mathbf{x}) \approx P(\mathbf{x} \mid d=\text{source})$, yet you are scored on both. A detector that memorizes the source acoustic background will fire on every target clip and destroy its false-positive rate. The goal is an anomaly score $a(\mathbf{x})$ whose normal-versus-anomalous boundary is stable across $d$, even though the marginal $P(\mathbf{x})$ shifts underneath it.

Key Insight

The reason plain autoencoders lost to discriminative models on this benchmark is a domain-shift failure, not a capacity failure. A reconstruction detector scores anomaly as reconstruction error $\lVert \mathbf{x} - \hat{\mathbf{x}} \rVert$, which is large whenever the input is unfamiliar for any reason, including a benign change of room or speed. Under domain shift, "unfamiliar" and "anomalous" stop being the same thing, so reconstruction error conflates a healthy target-domain pump with a broken source-domain one. The systems that win instead learn an embedding where the axis that separates machine states is decorrelated from the axis that separates domains, and they get that decorrelation almost for free from a metadata-classification pretext task.

The discriminative recipe: classify the metadata, then measure distance

The dominant paradigm since 2021 inverts the autoencoder logic. Instead of modeling normal sound directly, you train a network to classify auxiliary metadata that ships with every normal clip: the machine type, the section (an anonymized grouping of individual units), and attribute labels such as speed, load, or microphone position. This is a self-supervised pretext in the sense of Chapter 17: no anomaly labels are used, only free metadata. To classify these fine-grained conditions the encoder is forced to attend to the precise spectral signature of correct operation. At inference you discard the classifier head and keep the embedding $\mathbf{z} = f_\theta(\mathbf{x})$. Normal clips cluster tightly per section; an anomalous clip lands far from every normal cluster. Scoring is then a simple distance in embedding space, and this is where domain generalization is won: you fit a separate reference set (or Gaussian) for the source cluster and for the few target normals, and score against the nearest one.

Two distance rules dominate. The Mahalanobis score uses each section's normal mean $\boldsymbol{\mu}_c$ and shared covariance $\boldsymbol{\Sigma}$:

$$a(\mathbf{x}) = \min_c\, (\mathbf{z}-\boldsymbol{\mu}_c)^\top \boldsymbol{\Sigma}^{-1} (\mathbf{z}-\boldsymbol{\mu}_c).$$

The k-nearest-neighbor score is the mean distance to the $k$ closest normal embeddings, pooling source and the handful of target normals into one reference bank so that both domains define "normal." kNN is the more robust default under first-shot conditions precisely because it makes no Gaussian assumption about a machine type it has never studied.

import numpy as np
from sklearn.neighbors import NearestNeighbors

# z_train: (N, D) embeddings of NORMAL clips from a trained metadata classifier
#          (source + the few provided target normals, pooled)
# z_test:  (M, D) embeddings of clips to score
def first_shot_scores(z_train, z_test, k=4):
    z_train = z_train / (np.linalg.norm(z_train, axis=1, keepdims=True) + 1e-9)
    z_test  = z_test  / (np.linalg.norm(z_test,  axis=1, keepdims=True) + 1e-9)
    nn = NearestNeighbors(n_neighbors=k, metric="euclidean").fit(z_train)
    dist, _ = nn.kneighbors(z_test)          # distance to k nearest normals
    return dist.mean(axis=1)                  # higher = more anomalous

# A machine-type-agnostic threshold is set on a validation set of normals only,
# e.g. the 90th percentile of scores, honoring the low-false-alarm target.

A first-shot kNN anomaly scorer over pretrained embeddings. The encoder that produces z_train was trained only to classify machine section and attributes (never to see anomalies), and the same scoring code runs unchanged on a machine type it was never tuned for. Pooling target normals into the reference bank is what buys domain generalization.

The scorer above is deliberately machine-type-agnostic: the same four lines run on a valve, a gearbox, or a bandsaw, which is exactly the first-shot constraint. All the machine-specific knowledge lives in the learned embedding, not in tuned thresholds or per-type architecture.

Library Shortcut

Building the full DCASE pipeline by hand (log-mel front end, a metadata-classification backbone with ArcFace-style margin loss, embedding extraction, per-section Mahalanobis and kNN scoring, and the partial-AUC harness) is roughly 500 to 700 lines. The official DCASE Task 2 baseline repository ships all of it, and general anomaly toolkits such as anomalib reduce the modeling half to a handful of config lines. A PyTorch Lightning wrapper around a pretrained audio backbone plus the kNN scorer above turns the whole detector into about 30 lines. Write the pieces from scratch once to understand the metadata trick; reach for the baseline repo when you actually need a leaderboard number, because its scoring code is the one graders trust.

Practical Example: A Valve Line Re-Tooled Overnight

A packaging plant monitors 40 pneumatic valves by microphone. The team trained an autoencoder detector on three months of normal source-domain audio and it held a clean AUC near 0.95. Then maintenance swapped the compressor, raising line pressure and shifting every valve's hiss upward in frequency. Overnight the autoencoder's false-alarm rate exploded: reconstruction error rose on every valve because the whole soundscape was now "unfamiliar," and operators started ignoring the alerts. Rebuilt as a metadata-classification embedding with a pooled kNN reference bank (source clips plus ten normal clips recorded after the swap), the detector treated the pressure change as a benign domain shift and kept its target-domain AUC above 0.9, while still catching a genuinely stuck valve two shifts later. The lesson generalized to the next machine type the plant added, a labeler drive, with no retuning: that is first-shot behavior in production.

Scoring without self-deception: partial AUC and the harmonic mean

DCASE does not report plain accuracy, and neither should you. Anomalies are rare and a maintenance team will abandon any detector that cries wolf, so the challenge scores the partial AUC (pAUC): the area under the ROC curve restricted to the low false-positive region, typically the leftmost $p=0.1$ of the false-positive axis. A detector can only earn pAUC by ranking true anomalies above normals at operating thresholds a plant would actually tolerate. Results are reported separately for source-domain AUC, target-domain AUC, and pAUC, then aggregated across all machine types with the harmonic mean, which punishes any single machine type you fail on far harder than an arithmetic mean would. You cannot buy a good score by acing fans and ignoring valves.

Research Frontier

The 2023 to 2025 DCASE Task 2 leaders pair a pretrained audio backbone (an AudioSet-trained model such as BEATs or a self-supervised spectrogram transformer) with the metadata-classification fine-tuning above, then score with a smoothed kNN or a per-section flow model. Open problems are squarely research-frontier: how to weight the few target normals against abundant source data without overfitting to ten clips; how to make the embedding invariant to domain while staying sensitive to faults, since the two are entangled; and whether general-purpose audio foundation models can deliver true zero-shot detection on an unheard machine type with no normals at all, which would push "first-shot" toward "zero-shot." Calibrating the anomaly score into a trustworthy probability, using the conformal tools of Chapter 18, remains largely unsolved for these non-exchangeable, domain-shifted streams.

Exercise

Take the MIMII-DG development set (or the DCASE baseline's synthetic embeddings). (a) Train a small classifier to predict machine section from log-mel features, discard the head, and score test clips with the kNN function above; report source-AUC, target-AUC, and pAUC at $p=0.1$ per machine type, then their harmonic mean. (b) Replace kNN with a per-section Mahalanobis score and compare, especially on the target domain where the Gaussian assumption is shakiest. (c) Now remove the ten target normals from the reference bank and re-score. Quantify how much target-AUC collapses, and explain in one sentence why this experiment is the difference between domain adaptation and domain generalization.

Self-Check

In one sentence each, distinguish domain adaptation, domain generalization, and first-shot detection as the DCASE task defines them.
Why does a reconstruction-error autoencoder degrade under domain shift while a metadata-classification embedding does not, even though neither ever sees an anomaly during training?
The challenge scores partial AUC at $p=0.1$ and aggregates with a harmonic mean. What specific failure mode of a naive detector does each of those two choices catch?

What's Next

In Section 37.4, we open up the two score functions that underlie most detectors in this chapter: reconstruction error and forecasting residuals. You will see when a machine that predicts its own next window beats one that merely rebuilds the current one, why both need the honest, low-false-alarm evaluation you just met, and how the embedding-distance idea from this section fits into that broader family.