"They told me I was zero-shot. Then I found my test set living inside my training data, wearing a fake mustache."
A Disillusioned AI Agent
The big picture
Sections 19.3 through 19.5 handed you a shelf of pretrained forecasters and a promise: point them at a new sensor stream and read off state-of-the-art numbers with no fitting. This section asks the harder question that decides whether any of those numbers mean anything. When a model has been pretrained on tens of billions of time points scraped from every public archive on the internet, the phrase "zero-shot on this dataset" is only trustworthy if you can prove the dataset was not already inside the pretraining corpus. It usually cannot be proven. That is the evaluation crisis: the same forces that make a time-series foundation model (TSFM) powerful, a giant heterogeneous corpus like LOTSA, also quietly contaminate the very benchmarks (Monash, GIFT-Eval) we use to judge it. This section teaches you to read TSFM claims like a skeptic and to run a benchmark that a skeptic would believe.
This is a research-frontier section, so it assumes you have met the model families of Section 19.3 and the zero-shot tasks of Section 19.4. It leans on two ideas taught earlier in the book: leakage-safe data handling from Chapter 5 (Sensor Data Engineering and Leakage-Safe Datasets), and proper scoring of probabilistic forecasts from Chapter 18 (Uncertainty, Calibration, and Conformal Prediction). If either is unfamiliar, skim them first; everything here builds on them.
Three ways a foundation-model benchmark lies
Leakage is any path by which information about the test target reaches the model before it is scored. For task-specific models the paths are familiar and local. For foundation models a new, global path opens up, and it is easy to miss because it was created months earlier by someone else during pretraining. It helps to name three distinct failure modes rather than lumping them together.
Corpus leakage is the headline problem. A TSFM is pretrained on a corpus assembled from public archives; a benchmark is assembled from public archives; the overlap is often nonzero and frequently large. If the electricity or traffic series you are "zero-shot" evaluating was one of the series scraped into pretraining, the model has literally seen the answer, and its low error measures memorization, not generalization. Windowing leakage is the classic mistake wearing new clothes: normalizing, imputing, or selecting a forecast horizon using statistics computed over a window that includes the future you are trying to predict. Selection leakage is the meta-level version: choosing which model, context length, or checkpoint to report after peeking at test scores across many benchmarks, so the reported configuration is overfit to the leaderboard rather than to any real deployment.
Key insight
For a task-specific model, "did the test data leak into training?" is a question you can answer, because you built the training set. For a foundation model you did not build the training set, so the honest default is to assume corpus leakage until the pretraining data manifest proves otherwise. A zero-shot number without a leakage audit is not a measurement; it is a rumor. This inverts the usual burden of proof: the vendor claiming zero-shot generalization owes you the corpus manifest, not the other way around.
The corpora and the benchmarks: LOTSA, Monash, GIFT-Eval
Three named artifacts anchor this whole discussion, and the relationship between them is the crux of the crisis. The Monash Time Series Forecasting Archive (Godahewa et al., NeurIPS 2021) is the older reference: roughly thirty curated univariate collections spanning energy, traffic, tourism, weather, and web traffic, packaged with fixed horizons so results are comparable across papers. LOTSA (Large-scale Open Time Series Archive, released with Moirai by Woo et al., 2024) is a pretraining corpus, not a benchmark: on the order of \(2.7 \times 10^{11}\) observations pulled from many domains, explicitly built to feed a foundation model. GIFT-Eval (Aksu et al., Salesforce, 2024) is a purpose-built evaluation suite covering many datasets, ten domains, and multiple frequencies and horizons, designed to be a fair scoreboard for TSFMs.
The problem is that these overlap. Monash collections were widely reused, so parts of Monash flowed into pretraining corpora including LOTSA-style archives, which means a model pretrained on the archive and then evaluated on Monash is grading its own homework. The GIFT-Eval authors took this seriously and published a train/test-split discipline plus a "GIFT-Eval Pretrain" partition kept separate from the evaluation partition, precisely so that a model can be pretrained on one and cleanly tested on the other. That separation is the single most important design idea in modern TSFM benchmarking, and it is the one earlier archives lacked.
In practice: a grid operator vets a "zero-shot" load forecaster
A regional utility evaluates a vendor's TSFM for day-ahead substation load forecasting. The pitch deck shows the model beating the utility's tuned gradient-boosting baseline by 18% MASE "zero-shot, no fitting." The utility's data-science lead does one thing before believing it: she asks for the pretraining corpus manifest. The vendor's corpus included the ELECTRICITY and the ISO-NE public load sets, and two of the utility's own historical feeders had been contributed to a public archive years earlier. On those two feeders the model's error was implausibly low; on three private feeders never published anywhere, the 18% edge shrank to 4% and vanished into the noise band once seasonal-naive was scored correctly. The demo was not fraud; it was corpus leakage plus selection leakage, and a single manifest request exposed it. The utility deployed the model, but budgeted for the 4% it could actually defend, not the 18% on the slide.
Non-stationarity: why a random split leaks the future
Even with a clean corpus, sensor time series break a load-bearing assumption of ordinary machine learning: they are non-stationary. The statistics that generate the signal (mean, variance, seasonality, the effect of covariates) drift over time as machines wear, seasons turn, firmware updates, and user behavior shifts. This is the same distribution shift that Chapter 66 (Distribution Shift, OOD, and Test-Time Adaptation) treats as a first-class deployment problem. For evaluation it has one blunt consequence: a random train/test split is invalid. Randomly holding out timestamps places future points into training and past points into test, so the model interpolates across a gap it will never have at deployment, and every metric flatters it.
The correct protocol is temporal. Split at a cutoff time \(t_c\): train on \(t \le t_c\), test on \(t > t_c\), never the reverse. Better still is rolling-origin evaluation (backtesting): slide the cutoff forward through several origins and average, so the score reflects performance across many recent regimes rather than one lucky window. Formally, over origins \(t_1 < t_2 < \dots < t_K\), the backtest error is
$$\mathrm{BT} = \frac{1}{K}\sum_{k=1}^{K} \mathcal{L}\big(y_{t_k+1:t_k+h},\ \hat{y}_{t_k+1:t_k+h}\big),$$where \(h\) is the forecast horizon and \(\mathcal{L}\) a proper scoring rule. Each origin uses only data up to \(t_k\) to forecast the next \(h\) steps, so no evaluation window ever peeks past its own cutoff. This is exactly the leakage-safe backtesting philosophy of Chapter 65 (Evaluation Protocols and Leakage-Safe Benchmarking), specialized to forecasting.
Scoring fairly: the seasonal-naive floor and the right metric
A benchmark number is meaningless without a floor to compare against, and the floor for forecasting is the seasonal naive predictor: repeat the value from one season ago. The Mean Absolute Scaled Error (MASE) bakes this in by dividing forecast error by the in-sample error of seasonal naive:
$$\mathrm{MASE} = \frac{\frac{1}{h}\sum_{i=1}^{h}\lvert y_i - \hat{y}_i\rvert}{\frac{1}{n-m}\sum_{t=m+1}^{n}\lvert y_t - y_{t-m}\rvert},$$with seasonal period \(m\). A MASE below 1 means you beat seasonal naive; a MASE at or above 1 means an expensive foundation model lost to a one-line heuristic, which happens more often than leaderboards advertise. For probabilistic forecasts the analogous proper score is the Continuous Ranked Probability Score (CRPS) from Chapter 18; report it alongside MASE so a confidently-wrong model cannot hide behind a good point estimate. One last trap is aggregation: averaging raw errors across datasets lets a single high-variance series dominate, so GIFT-Eval and the M-competitions report a normalized, rank-based or geometric aggregation instead of a raw arithmetic mean. The listing below runs a rolling-origin backtest and computes MASE against the seasonal-naive floor so you can see the whole discipline in one place.
import numpy as np
def seasonal_naive(history, h, m):
# repeat the last full season forward to horizon h
return history[-m:][np.arange(h) % m]
def mase(y_true, y_pred, history, m):
num = np.mean(np.abs(y_true - y_pred))
denom = np.mean(np.abs(history[m:] - history[:-m])) # in-sample seasonal naive
return num / denom
def rolling_origin_backtest(series, forecaster, origins, h, m):
scores = []
for t_c in origins: # each origin sees only the past
history = series[:t_c]
y_true = series[t_c:t_c + h]
y_pred = forecaster(history, h, m)
scores.append(mase(y_true, y_pred, history, m))
return np.mean(scores) # backtest MASE across origins
rng = np.random.default_rng(0)
t = np.arange(600)
series = 10 + 3*np.sin(2*np.pi*t/24) + 0.01*t + rng.normal(0, 0.5, size=t.size) # daily + drift
origins = [400, 450, 500, 550] # four rolling cutoffs, all late in the series
bt = rolling_origin_backtest(series, seasonal_naive, origins, h=24, m=24)
print(f"seasonal-naive backtest MASE: {bt:.3f}")
seasonal_naive for a TSFM's predict gives a number a skeptic will accept; a value at or above 1.0 means the foundation model failed to beat the one-line baseline.Run Listing 19.6.1 and the seasonal-naive predictor scores a MASE near 1.0 by construction, because it is the denominator. The point is the scaffold: replace the forecaster argument with any model from Section 19.3 and the same code yields a directly comparable, leakage-safe score. Anything that cannot beat this scaffold's 1.0 on your data does not belong in your pipeline, no matter how many billions of parameters it carries.
Right tool: don't hand-roll the backtest harness
Listing 19.6.1 is about 20 lines and still omits multi-series batching, missing-value handling, quantile losses, and the normalized aggregation GIFT-Eval expects. The GIFT-Eval harness and GluonTS give you all of it: a rolling-origin Evaluator, MASE, CRPS (as weighted quantile loss), and rank aggregation across the full suite in a few lines.
from gluonts.evaluation import make_evaluation_predictions, Evaluator
forecast_it, ts_it = make_evaluation_predictions(dataset=test_data, predictor=predictor)
agg, per_series = Evaluator()(ts_it, forecast_it) # MASE, CRPS, coverage, all at once
That is a reduction from tens of lines to two, and more importantly the library fixes the split boundary correctly every time, which is exactly where hand-rolled harnesses leak.
Research frontier
As of 2025, the GIFT-Eval leaderboard (Salesforce) is the reference scoreboard, with TimesFM (Das et al., Google), Moirai and Moirai-MoE (Woo et al.), and Chronos / Chronos-Bolt (Ansari et al., Amazon) trading the top ranks depending on frequency and horizon. The live frontier is the leakage audit itself: because pretraining corpora such as LOTSA are only partially documented, groups now publish overlap analyses between pretraining sets and evaluation suites, and GIFT-Eval ships a dedicated non-overlapping "Pretrain" partition so authors can demonstrate a clean separation. Expect the next generation of benchmarks to require a machine-readable corpus manifest as a submission condition, turning "prove there was no leakage" from a courtesy into an entry rule.
Exercise
You are handed a TSFM that reports MASE 0.72 on a Monash energy collection, zero-shot. (a) List the three leakage modes from this section and, for each, name one concrete check you would run to rule it out. (b) The vendor cannot share the pretraining manifest. Design a held-out test using only your own private sensor data that would still let you estimate the true generalization gap. (c) Your private-data MASE comes out at 0.95. Write two sentences a decision-maker can act on, distinguishing what you measured from what the vendor claimed.
Self-check
- Explain why a random train/test split is invalid for a non-stationary sensor stream, and what rolling-origin evaluation does instead.
- A model reports MASE 1.4 on your benchmark. What does that single number tell you about whether to deploy it, and why is the seasonal-naive denominator the reason it is interpretable?
- Distinguish corpus leakage from selection leakage. Which one is unique to foundation models, and which one predates them entirely?
What's Next
In Section 19.7, we turn the crisis into a decision rule: once you can measure a foundation model without self-deception, when does its zero-shot convenience actually justify its cost, and when does a small task-specific model, trained on your own leakage-safe split, quietly win on accuracy, latency, and trust?