Related papers: On Lyapunov Exponents for RNNs: Understanding Info…
Reactivity, contractivity, and Lyapunov exponents are powerful tools for studying the stability properties of dynamical systems and have been extensively investigated in the literature for decades. In this paper, we review and extend the…
Recurrent neural networks (RNNs) have yielded promising results for both recognizing objects in challenging conditions and modeling aspects of primate vision. However, the representational dynamics of recurrent computations remain poorly…
Recurrent neural networks (RNNs) are a class of neural networks used in sequential tasks. However, in general, RNNs have a large number of parameters and involve enormous computational costs by repeating the recurrent structures in many…
The Linear Representation Hypothesis (LRH) states that neural networks learn to encode concepts as directions in activation space, and a strong version of the LRH states that models learn only such encodings. In this paper, we present a…
The performance of graph neural networks (GNNs) is susceptible to discrepancies between training and testing sample distributions. Prior studies have attempted to mitigating the impact of distribution shift by reconstructing node features…
We describe recurrent neural networks (RNNs), which have attracted great attention on sequential tasks, such as handwriting recognition, speech recognition and image to text. However, compared to general feedforward neural networks, RNNs…
We use finite-time Lyapunov exponent (FTLE) distributions to probe transition mechanisms in high-dimensional reservoir maps trained on low-dimensional chaotic dynamics across multiple regimes. While trained reservoirs accurately predict…
Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). It shows that RNNs can efficiently represent bounded hierarchical structures that are…
Recurrent neural networks (RNNs) are nonlinear dynamical models commonly used in the machine learning and dynamical systems literature to represent complex dynamical or sequential relationships between variables. More recently, as deep…
Feed-forward neural networks (FNNs) work as standard building blocks in applying artificial intelligence (AI) to the physical world. They allow learning the dynamics of unknown physical systems (e.g., biological and chemical) {to predict…
Modelling real world systems involving humans such as biological processes for disease treatment or human behavior for robotic rehabilitation is a challenging problem because labeled training data is sparse and expensive, while high…
The spectrum of Lyapunov exponents is powerful tool for the analysis of the complex system dynamics. In the general framework of nonlinear dynamical systems a number of the numerical technics have been developed to obtain the spectrum of…
To be effective in sequential data processing, Recurrent Neural Networks (RNNs) are required to keep track of past events by creating memories. While the relation between memories and the network's hidden state dynamics was established over…
Deep neural networks (DNNs) are powerful black-box function approximators which have been shown to yield improved performance compared to traditional neural network (NN) architectures. However, black-box algorithms do not incorporate known…
The computation of the entire Lyapunov spectrum for extended dynamical systems is a very time consuming task. If the system is in a chaotic spatio-temporal regime it is possible to approximately reconstruct the Lyapunov spectrum from the…
Recurrent Neural Networks (RNNs) offer fast inference on long sequences but are hard to optimize and slow to train. Deep state-space models (SSMs) have recently been shown to perform remarkably well on long sequence modeling tasks, and have…
Models based on deep convolutional networks have dominated recent image interpretation tasks; we investigate whether models which are also recurrent, or "temporally deep", are effective for tasks involving sequences, visual and otherwise.…
Recurrent Neural Networks (RNNs) are widely used for online regression due to their ability to generalize nonlinear temporal dependencies. As an RNN model, Long-Short-Term-Memory Networks (LSTMs) are commonly preferred in practice, as these…
Recommender systems objectives can be broadly characterized as modeling user preferences over short-or long-term time horizon. A large body of previous research studied long-term recommendation through dimensionality reduction techniques…
Lyapunov exponents (LEs) are key indicators of chaos in dynamical systems. In general relativity the classical definition of LE meets difficulty because it is not coordinate invariant and spacetime coordinates lose their physical meaning as…