Validation of synthetic data and twins

"My synthetic data passed every test I wrote. That is exactly the problem: I graded my own homework, in my own handwriting, with an answer key I invented."

A Self-Aware AI Agent

Two different objects, two different questions

The word "validation" hides two tasks that need different tools, and conflating them is the most common mistake in this space. The first object is a synthetic dataset: a static pile of generated readings and labels. The question is does training or testing on this data tell me the truth about real performance? The second object is a digital twin: a live, coupled model that ingests real measurements and produces state estimates and predictions, as built in Section 55.6. The question there is does this twin still faithfully track the physical asset, right now, and will its predictions hold? Data validation is a one-time offline audit; twin validation is a continuous online commitment. A generator can pass its audit and yet, wired into a twin whose parameters have drifted, mislead you completely. Keep the two ledgers separate.

Underneath both sits a single honest principle worth stating plainly: a model can only be validated against evidence it did not itself produce. Synthetic data that validates synthetic data is circular. The gold standard for both objects is a quantity of real, task-relevant, leakage-free measurements held in reserve, exactly the reserve that Chapter 5 taught you never to spend early.

Validating synthetic data: fidelity, utility, diversity

Three properties must each be checked, because a generator can ace one while failing the others. Fidelity asks whether synthetic samples are individually plausible and collectively match the real distribution. Utility asks whether a model trained on synthetic data actually works on real data. Diversity (coverage) asks whether the generator spans the real modes rather than reproducing a few memorized ones.

The sharpest single fidelity probe is the classifier two-sample test (C2ST). Pool real and synthetic samples, label them by origin, and train a discriminator to tell them apart under proper cross-validation. If the best classifier lands at chance (an AUC or accuracy near \(0.5\)), the two distributions are, to that classifier's eyes, indistinguishable, which is exactly the honest goal. If it reaches \(0.95\), the classifier just told you which features betray your synthetic data, and its most important inputs are a to-do list for the generator. This subsumes the intuition behind a GAN discriminator and behind the Frechet-distance family (FID and its time-series analogues), which compare Gaussian summaries of a feature embedding; the kernel two-sample test (MMD) is its non-parametric cousin. The complementary utility metric is Train-on-Synthetic, Test-on-Real (TSTR): fit the downstream model purely on generated data, evaluate on a held-out real set, and compare against Train-on-Real (TRTR). A small TSTR-to-TRTR gap is the operational license to synthesize. Diversity needs its own check because both C2ST and TSTR can be fooled by a generator that copies a slice of the training set; precision-and-recall-for-distributions and nearest-neighbour distance to the real training points catch that mode collapse and memorization.

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

def c2st_auc(real, synth, seed=0):
    """Classifier two-sample test: AUC ~0.5 means real and synthetic
    are indistinguishable (good). AUC near 1.0 exposes the tells."""
    X = np.vstack([real, synth])
    y = np.r_[np.ones(len(real)), np.zeros(len(synth))]      # origin label
    clf = GradientBoostingClassifier(random_state=seed)
    return cross_val_score(clf, X, y, cv=5, scoring="roc_auc").mean()

def tstr_gap(train_synth, train_real, test_real, fit_eval):
    """Utility: performance drop from Train-on-Real to Train-on-Synthetic,
    both scored on the SAME held-out real test set. fit_eval(Xtr)->score."""
    return fit_eval(train_real, test_real) - fit_eval(train_synth, test_real)

rng = np.random.default_rng(0)
real  = rng.normal(0.0, 1.0, size=(2000, 8))
good  = rng.normal(0.0, 1.0, size=(2000, 8))          # matched generator
naive = rng.normal(0.2, 0.7, size=(2000, 8))          # wrong mean and scale
print(f"C2ST AUC good  = {c2st_auc(real, good):.3f}   (want ~0.5)")
print(f"C2ST AUC naive = {c2st_auc(real, naive):.3f}   (tells exposed)")

Listing 55.7 makes the discipline concrete: fidelity and utility are measured, never assumed, and both verdicts are cast on real data the generator never touched. Report the C2ST AUC and the TSTR gap together; a generator that passes fidelity but fails utility is over-fit to the wrong features, and one that passes utility but fails fidelity is getting lucky on one task and will not transfer to the next.

Validating a running twin: residuals, calibration, credibility

A twin is validated online and continuously, because the asset it mirrors ages, wears, and drifts. Three layers of evidence matter. First, innovation residuals: the twin already produces a prediction of each incoming measurement, so the residual \(r_k = y_k - \hat{y}_{k|k-1}\) is free. If the twin is faithful and its noise model correct, residuals are zero-mean and white with the covariance the filter claims, so the normalized innovation squared \( \epsilon_k = r_k^{\top} S_k^{-1} r_k \) follows a chi-squared law with known degrees of freedom. Persistent bias, growing magnitude, or autocorrelation in the residuals is the twin telling you it has diverged from the plant, and a CUSUM or generalized-likelihood-ratio test from Chapter 12 turns that signal into a dated alarm. Second, predictive calibration: a twin that forecasts must own its uncertainty, so its prediction intervals must achieve their nominal coverage on real outcomes, checked with the reliability diagrams and conformal wrappers of Chapter 18. A twin that is accurate on the mean but overconfident in its intervals is unsafe to act on. Third, formal credibility: safety-critical twins are certified against structured verification-and-validation frameworks (ASME V&V 10 for computational solid mechanics, V&V 40 for medical-device models) that force you to declare the intended use, the validation metric, and the acceptance threshold before looking at the data, closing the door on grading your own homework.

The failure modes validation exists to catch

Naming the specific ways synthetic pipelines lie makes the tests above feel less abstract. Covariate gaps: the generator never sampled the operating regime that breaks the model in the field, so it is a coverage failure that OOD detection from Chapter 66 should catch at inference time even when validation missed it. Label leakage through the simulator: because the generator knows the latent label, an artefact correlated with it (a rendering seed, a fixed noise phase) can leak the answer, inflating synthetic accuracy to numbers real data will never reproduce. Memorization and mode collapse: a learned generator that echoes its training set passes fidelity and utility while offering no new information and, worse, leaking private records. Sim-to-real optimism: reporting only the synthetic-test score, which is exactly the "wins-only on the wrong test set" trap. The countermeasure to all four is the same discipline: pre-register the metric and threshold, always cast the final verdict on real held-out data, and treat any suspiciously large synthetic-to-real gap as a bug to be diagnosed rather than a rounding error.

What's Next

In Chapter 56, we leave the world of simulated and mirrored signals and touch the physical world directly, building the tactile sensors and electronic skin that give robots a sense of contact, pressure, and slip, and asking how a modality with almost no public data and a punishing sim-to-real gap validates the very synthetic pipelines this chapter just taught you to trust.