"I gave up trying to repaint every leaf the wind would move. I only predict that the branch will still be a branch."
An Abstemious AI Agent
Prerequisites
This section builds directly on the self-supervised and contrastive pretraining machinery of Chapter 17, especially the collapse problem and the trick of an exponential-moving-average target network. The backbone is the vision transformer and patch tokenization of Chapter 15. It contrasts throughout with the reconstruction-based, pixel-space world models of Section 53.2, so read that first if the phrase "recurrent latent state" is unfamiliar.
The Big Picture
A joint-embedding predictive architecture (JEPA) is a world model that makes its predictions in representation space rather than in signal space. The recurrent world models of the previous section, and every generative model of the next one, learn by reconstructing pixels or samples: given part of the world, predict the missing part in full detail. JEPA refuses that. It encodes the observed context and the masked target through the same network, and trains a predictor to map the context embedding to the embedding of the target, never to its raw pixels. The payoff is that the model is not forced to model the parts of the sensor stream that are inherently unpredictable (the exact texture of foliage, the precise speckle of a lidar return, the microsecond jitter of a noisy accelerometer). It spends capacity only on structure that a downstream perception or control task can actually use. Yann LeCun's group has pushed this from I-JEPA on still images to V-JEPA 2 on internet-scale video and, critically for this Part, to action-conditioned prediction that plans robot motions in latent space.
Why predict in latent space at all
The what is a change of target. A generative predictor minimizes a loss in observation space, \(\lVert x_{\text{target}} - \hat{x} \rVert\); a JEPA minimizes a loss in embedding space, \(\lVert s_{\theta}(x_{\text{target}}) - g(s_{\theta}(x_{\text{context}})) \rVert\), where \(s_\theta\) is a shared encoder and \(g\) is a predictor conditioned on where the target sits. The why is the signal-to-noise structure of real sensor data. Most of the bits in a camera frame or a raw waveform are high-frequency detail that no model can predict from context and that no perception task needs; a pixel reconstruction loss forces the network to burn representational capacity chasing that irreducible noise, which is why masked-autoencoder features often need heavy fine-tuning to become linearly useful. By moving the target into a learned embedding, JEPA lets the encoder discard unpredictable detail before the loss is computed. The prediction problem becomes "will there be an edge here, roughly this oriented" instead of "what are the exact 768 sub-pixel values," and the resulting features are strong off-the-shelf, without labels.
This is the same energy-based, non-generative philosophy as the contrastive methods of Chapter 17, but with a decisive difference. Contrastive learning needs carefully mined negative pairs and hand-built augmentations to avoid collapse; JEPA needs neither. It is a predictive rather than invariance objective: it does not ask two views to be identical, it asks one view to predict another's representation given their spatial or temporal relationship, which preserves information about position and motion that invariance objectives throw away.
Key Insight: asymmetry is what stops collapse without negatives
A predictor that maps context to target embedding has a trivial cheat: make the encoder output a constant, so every prediction is perfect and every embedding is identical. JEPA blocks this the same way BYOL and DINO do, through architectural asymmetry rather than negative samples. The target encoder is not trained by gradient descent; its weights are an exponential moving average (EMA) of the context encoder, and gradients are stopped from flowing into it. The context encoder must therefore chase a slowly moving, non-degenerate target it cannot directly control, and a collapsed constant is no longer a fixed point of the dynamics. The predictor is deliberately narrow so it cannot memorize the target. No contrastive negatives, no clustering, no reconstruction: the EMA plus stop-gradient plus a bottlenecked predictor is the entire anti-collapse mechanism.
I-JEPA: masked prediction on a single image
The how is cleanest in the image case. I-JEPA (Assran and colleagues, 2023) tokenizes an image into patches with a vision transformer, then samples one context block and several target blocks of patches. The context encoder sees only the context patches. The target encoder (the EMA copy) encodes the full image and the target patch embeddings are read off from it. A predictor, given the context embedding plus positional tokens marking where each target block lives, must produce the target encoder's embeddings for those blocks. The design choices that matter are non-obvious and were found empirically: the target blocks should be reasonably large (semantic, not a single patch), the context should be spatially distributed rather than a tight crop, and masking should be informed so the task is neither trivial nor impossible. Because everything happens on embeddings, there is no pixel decoder at all; I-JEPA trains a ViT-Huge to strong linear-probe accuracy far faster than pixel-reconstruction masked autoencoders, precisely because it never pays the decoder tax.
import torch, torch.nn.functional as F, copy
def jepa_step(ctx_encoder, tgt_encoder, predictor, imgs, ctx_mask, tgt_mask,
opt, ema=0.996):
# target embeddings: full image through the EMA encoder, no gradient
with torch.no_grad():
z_tgt = tgt_encoder(imgs) # (B, N_patches, D)
z_tgt = z_tgt[tgt_mask].detach() # keep only target patches
z_ctx = ctx_encoder(imgs, keep=ctx_mask) # context patches only
z_hat = predictor(z_ctx, target_pos=tgt_mask) # predict IN embedding space
loss = F.smooth_l1_loss(z_hat, z_tgt) # no pixels, no negatives
opt.zero_grad(); loss.backward(); opt.step()
# EMA update of the target encoder (stop-gradient side)
for p_t, p_c in zip(tgt_encoder.parameters(), ctx_encoder.parameters()):
p_t.data.mul_(ema).add_(p_c.data, alpha=1 - ema)
return loss.item()
torch.no_grad() with .detach() (stop-gradient), its weights follow the context encoder only through the EMA loop at the bottom, and the loss compares embeddings, never pixels. Swap the image tokenizer for a spatiotemporal one and this identical skeleton becomes V-JEPA.Listing 53.3 makes the whole family concrete: the loss line never touches imgs again after encoding, which is the mechanical statement of "predict in representation space." Everything specific to images lives in the tokenizer and the masking, so the same loop generalizes to video and, in principle, to any patchable sensor stream.
V-JEPA 2: from video features to an action-conditioned world model
Extending the idea to video (V-JEPA, then V-JEPA 2 in 2025) means masking spatiotemporal tubes: the context is a subset of patches across space and time, and the model predicts the embeddings of masked tubes elsewhere in the clip. Trained self-supervised on over a million hours of internet video, V-JEPA 2 learns features that transfer to action recognition and, more importantly for sensing, encode intuitive physics: probed on the "surprise" benchmarks where an object should not pass through a wall, its prediction error spikes on physically impossible events, which is exactly the anticipation signal Section 53.6 will exploit. V-JEPA 2 then adds a second stage, V-JEPA 2-AC (action-conditioned), post-trained on a modest set of unlabeled robot interaction videos. Here the predictor is conditioned on the robot's action, so it forecasts the next latent state given a candidate action. That turns a passive video model into a controllable world model: to reach a goal image, the robot rolls out candidate action sequences in latent space and picks the one whose predicted embedding lands nearest the goal embedding, the latent planning we develop in Section 53.5. It enables zero-shot pick-and-place on a robot the base model never saw, planning entirely in learned representation space.
Practical Example: a warehouse arm that plans without a simulator
Picture a bin-picking arm at a fulfillment center that must move a bottle from a cluttered tote to a shelf. The classical route is a hand-built physics simulator plus a reward-tuned policy, brittle whenever the tote lighting or the object set changes. An action-conditioned JEPA controller instead takes a single goal photograph of the bottle on the shelf and encodes it. At each control step it images the current scene, encodes it, and imagines several short action sequences by rolling the action-conditioned predictor forward in embedding space, a few tens of milliseconds each because no pixels are rendered. It executes the first action of the sequence whose predicted latent is closest to the goal latent, then re-plans. Because the world model was pretrained on generic human and robot video and only lightly adapted, the same controller handles a new bottle shape it never trained on: it does not need the pixels to be right, only the embedding of "bottle now on shelf" to match. That robustness to unmodeled visual detail is the direct dividend of not being a pixel predictor.
Right Tool: load the pretrained encoder, do not pretrain it
Pretraining a V-JEPA 2 backbone is a million-GPU-hour undertaking; reproducing the masking, EMA schedule, multi-block target sampling, and distributed training from scratch is several thousand lines and a data pipeline you do not want to own. The released checkpoints reduce that to a handful of lines: AutoModel.from_pretrained("facebook/vjepa2-vitl-fpc64-256") plus the matching video processor gives you a frozen spatiotemporal encoder, and an attentive-probe head turns it into a task model in under 50 lines. The library owns the collapse-prevention bookkeeping (EMA target, stop-gradient, target-block sampling) that is easy to get silently wrong and whose failure looks like a loss that cheerfully drops to zero while the features become useless.
Research Frontier: JEPA as the non-generative bet on world models
The state of the art here is an explicit wager against generation. I-JEPA (2023) and V-JEPA / V-JEPA 2 (Bardes, Assran and colleagues, 2024 to 2025) argue that reconstructing every pixel, the path taken by Sora-style video generators and the interactive models of Section 53.4, wastes capacity on detail irrelevant to reasoning and control. V-JEPA 2-AC's zero-shot robot manipulation, driven purely by latent-space planning, is the current evidence for the bet. Open questions that matter for sensing: how to fold multiple non-visual modalities (IMU, tactile, audio) into a shared JEPA embedding without a dominant modality collapsing the others, how to calibrate the prediction-error "surprise" signal so it is a trustworthy anomaly detector rather than a confident artifact, and whether the EMA target can be replaced by a principled objective. For embodied agents, the through-line runs to the vision-language-action models of Chapter 58.
When JEPA is the right world model, and when it is not
The when follows from what the objective throws away. Reach for a JEPA when you need strong, label-free representations of a high-bandwidth sensor stream and your downstream task is perception, anticipation, or latent-space control, where the exact reconstructed signal is irrelevant. It shines when unpredictable detail dominates the raw data, which is nearly always true for cameras, video, and dense depth. It is the wrong tool when you actually need the pixels or samples back: if the deliverable is a rendered future frame, a synthesized waveform, or an interactive playable environment, a generative model (Section 53.4) is what you want, because a JEPA has deliberately discarded the information needed to reconstruct. It is also weaker when the signal is low-dimensional and already clean, an inertial-only state estimator or a small tabular telemetry vector, where the reconstruction tax the method is designed to avoid was never large. In short: JEPA buys transferable understanding by giving up the ability to regenerate the input, and that trade is excellent for rich sensors and control, poor when the output itself must be a signal.
Exercise
Take a pretrained I-JEPA or V-JEPA 2 encoder and probe its predictive-surprise signal. (1) Freeze the encoder and, on a small held-out video set, compute the predictor's embedding-space error for normal clips versus clips you edit to violate physics (an object that teleports, or a stitched frame where a ball vanishes). Report whether error separates the two. (2) Ablate the anti-collapse mechanism: retrain the toy loop in Listing 53.3 with the EMA disabled (target encoder shares weights and receives gradients) and show the loss collapsing to zero while a linear probe on the features degrades to chance. (3) Contrast the linear-probe accuracy of the JEPA features against a masked-autoencoder baseline trained for the same wall-clock time, and connect the gap to the decoder-tax argument in the text.
Self-Check
1. State the single change to the loss function that distinguishes a JEPA from the reconstruction-based world model of Section 53.2, and name one class of sensor detail this change lets the model ignore. 2. JEPA uses no contrastive negatives; name the three architectural ingredients that prevent representational collapse instead. 3. What does the "AC" stage add to V-JEPA 2, and why is that the step that turns a passive video model into something a robot can plan with?
What's Next
In Section 53.4, we cross to the other side of the bet: generative interactive world models such as Genie, which do reconstruct the observation and, in exchange, let an agent act inside an imagined, playable environment learned from unlabeled video. Holding the JEPA trade-off in mind, discarding pixels for understanding, sharpens exactly what generation buys back and what it costs.