English

When Does Depth Survive Composition? Compute--Quality Regimes in Latent World Models

Machine Learning 2026-07-11 v1 Artificial Intelligence

Abstract

Adaptive-compute world models -- early-exit or mixture-of-depths predictors that spend variable depth per step -- assume depth buys better predictions and can be routed adaptively. In autoregressive rollouts, the first assumption requires depth's per-step precision to survive composition. We test this with a pre-registered instrument, the shallow penalty ρ=err(shallowest-exit rollout)/err(full-depth rollout)\rho=\mathrm{err}(\text{shallowest-exit rollout})/\mathrm{err}(\text{full-depth rollout}), across nine DeepMind Control tasks under matched single-step (K=1K=1) and multi-step (K=4K=4) training, three seeds each. We find three regimes: on 6/9 tasks depth helps rollouts (intrinsic, ρ\rho up to 4.7×4.7\times), on 2/9 the shallow exits beat the full stack (inversion, ρ\rho down to 0.85×0.85\times), and one is flat. The robust inversion (cheetah) is not a property of the dynamics but is created by training: an ablation supervising early exits only at the first rollout step erases it (ρ:0.871.18\rho: 0.87\to1.18, n=8n=8, Δ=+0.31\Delta=+0.31), while an intrinsic-tradeoff task is unaffected -- a double dissociation we call the routability catch-22, since the supervision that makes exits routable is what trains them to out-roll the full stack. The regime is partly predictable a priori: observation/action dimensionality and one-step model error correlate with ρ\rho at Spearman0.75|\text{Spearman}|\approx0.75 (n=9n=9). Inside a CEM planner, ρ\rho's sign predicts whether planning benefits from depth, most sharply on the inversion task, where shallow planning beats deep. Finally, three cautions: a task's regime depends on the metric space, the rollout horizon, and the encoder. All thresholds and gates were fixed before the compute campaign, including a pre-registered negative for the hypothesis that motivated the study.

Cite

@article{arxiv.2607.10203,
  title  = {When Does Depth Survive Composition? Compute--Quality Regimes in Latent World Models},
  author = {Achyuthan Sivasankar},
  journal= {arXiv preprint arXiv:2607.10203},
  year   = {2026}
}

Comments

15 pages, 9 figures, 2 tables