Linear Recurrent Neural Networks as Time-Delay Embeddings

Sequence models, and particularly Linear Recurrent Neural Networks (LRNNs) of the form $\mathbf{h}_{k+1} = \mathbf{W} \mathbf{h}_{k} + \mathbf{y}_k + \mathbf{b}$, are widely applicable in time-series analysis for dynamical systems, yet, as black-box algorithms, much is unknown about why they perform well. In this work, we leverage Takens' embedding theorem, which provides conditions under which partially observed time series organized into delay-coordinate vectors can faithfully represent the original system's dynamics, as a theoretical framework for explaining how and why sequence models preserve and reconstruct dynamical systems. For LRNNs, concatenating output states into delay-coordinate vectors gives rise to a ``delay" matrix $\mathbb{M}_{n,m}\in \mathbb{C}^{(nm) \times (n+1)m}$: a block matrix consisting of identity matrices $\mathbf{I} \in \mathbb{R}^{m \times m}$ repeated $n$ times along the main diagonal and weight matrices $\mathbf{W} \in \mathbb{C}^{m \times m}$ featured $n$ times along the super-diagonal. $\mathbb{M}_{n,m}$ relates the delay-coordinates of the input time series to those of the LRNN output states, and, for $\mathbb{M}_{n,m}$ to be an embedding, it must be full row-rank. We provide explicit conditions for $\mathbb{M}_{n,m}$ to be full row-rank and prove the condition number of $\mathbb{M}_{n,m}$ and determinant of $\mathbb{M}_{n,m} \mathbb{M}_{n,m}^*$--measures of embedding stability--are bounded independent of $n$, at least for certain ranges of $\mathbf{W}$'s singular values: namely, when $\sigma_{\max}(\mathbf{W}) \le 1$. This result explains why the spectrum of $\mathbf{W}$ for trained LRNNs tends to converge to within the unit circle.