In deep learning, the recently introduced state space models utilize HiPPO (High-order Polynomial Projection Operators) memory units to approximate continuous-time trajectories of input functions using ordinary differential equations (ODEs), and these techniques have shown empirical success in capturing long-range dependencies in long input sequences. However, the mathematical foundations of these ODEs, particularly the singular HiPPO-LegS (Legendre Scaled) ODE, and their corresponding numerical discretizations remain unsettled. In this work, we fill this gap by establishing that HiPPO-LegS ODE is well-posed despite its singularity, albeit without the freedom of arbitrary initial conditions. Further, we establish convergence of the associated numerical discretization schemes for Riemann integrable input functions.
Cite
@article{arxiv.2412.08595,
title = {Numerical Analysis of HiPPO-LegS ODE for Deep State Space Models},
author = {Jaesung R. Park and Jaewook J. Suh and Youngjoon Hong and Ernest K. Ryu},
journal= {arXiv preprint arXiv:2412.08595},
year = {2025}
}