Immediate Derivatives Suffice for Online Recurrent Adaptation
Abstract
For three decades online recurrent learning has been assumed to require propagating a Jacobian tensor through the network's dynamics at per step. We show it doesn't. Dropping the propagation entirely (, memory) matches full RTRL within CI on held-out BCI cross-session drift (TOST equivalent within pp at , Adam, float64), and across vanilla-RNN synthetic cells (sine and Lorenz under Adam and SGD) and LSTM/sine under Adam. A decomposition explains why. On BCI, concentrates in a single direction (top-1 singular fraction 0.62-0.74 across four optimizers, vs 0.333 for ), and the four-optimizer full-RTRL-vs- recovery gap tracks each optimizer's per-layer update-magnitude ratio monotonically. A stationary (no-drift) control collapses both concentrations to ~0.6: the drift-specific signal is the differential, not 's absolute rank-1 structure. The signature and the behavioral gap both collapse on LSTM, consistent with a mechanism specific to additive linear recurrence. On synthetic sine, is redundant with , which predicts the synthetic null. Full RTRL's one robust advantage is LARS (+17 to +27 pp), but +LARS also fails to adapt independently; the gap is an optimizermethod interaction, not a method-quality claim. We characterize the regime: +Adam+float64 is robust; SGD, Adafactor, and float32 have specific fragilities documented in the paper. On the evaluated cells, the memory saving at ( vs ) comes with no measured recovery cost.
Cite
@article{arxiv.2603.28750,
title = {Immediate Derivatives Suffice for Online Recurrent Adaptation},
author = {Aur Shalev Merin},
journal= {arXiv preprint arXiv:2603.28750},
year = {2026}
}
Comments
25 pages, 4 figures, 19 tables. Submitted to NeurIPS 2026