Related papers: Reservoir Transformers
Transformers have achieved great success across a wide range of applications, yet the theoretical foundations underlying their success remain largely unexplored. To demystify the strong capacities of transformers applied to versatile…
A machine-learning approach called "reservoir computing" has been used successfully for short-term prediction and attractor reconstruction of chaotic dynamical systems from time series data. We present a theoretical framework that describes…
After showing the efficiency of feedforward networks to estimate control in high dimension in the global optimization of some storages problems, we develop a modification of an algorithm based on some dynamic programming principle. We show…
Reservoir computing is a popular approach to design recurrent neural networks, due to its training simplicity and approximation performance. The recurrent part of these networks is not trained (e.g., via gradient descent), making them…
The Reservoir Computing (RC) paradigm utilizes a dynamical system, i.e., a reservoir, and a linear classifier, i.e., a read-out layer, to process data from sequential classification tasks. In this paper the usage of Cellular Automata (CA)…
Reservoir computing provides an alternative to recurrent neural networks by overcoming the common problems of backpropagation through time and by training only a simple readout layer. The emerging field of quantum computing offers a new…
Reservoir computing is an information processing technique, derived from the theory of neural networks, which is easy to implement in hardware. Several reservoir computer hardware implementations have been realized recently with performance…
Dissipation induced by interactions with an external environment typically hinders the performance of quantum computation, but in some cases can be turned out as a useful resource. We show the potential enhancement induced by dissipation in…
Reservoir Computing is an Unconventional Computation model to perform computation on various different substrates, such as recurrent neural networks or physical materials. The method takes a 'black-box' approach, training only the outputs…
In Chaos, a minor divergence between two initial conditions exhibits exponential amplification over time, leading to far-away outcomes, known as the butterfly effect. Thus, the distant future is full of uncertainty and hard to forecast. We…
Multilingual Machine Translation promises to improve translation quality between non-English languages. This is advantageous for several reasons, namely lower latency (no need to translate twice), and reduced error cascades (e.g., avoiding…
Reservoir Computing offers a great computational framework where a physical system can directly be used as computational substrate. Typically a "reservoir" is comprised of a large number of dynamical systems, and is consequently…
Stacking non-linear layers allows deep neural networks to model complicated functions, and including residual connections in Transformer layers is beneficial for convergence and performance. However, residual connections may make the model…
Large transformer-based language models have been shown to be very effective in many classification tasks. However, their computational complexity prevents their use in applications requiring the classification of a large set of candidates.…
Reservoir computing (RC) can efficiently process time-series data by transferring the input signal to randomly connected recurrent neural networks (RNNs), which are referred to as a reservoir. The high-dimensional representation of…
Transformers have demonstrated exceptional performance across a wide range of domains. While their ability to perform reinforcement learning in-context has been established both theoretically and empirically, their behavior in…
We investigate the computational potential and limitations of a passive linear optical reservoir with a photodetector at the optical-to-electrical interface as the sole source of nonlinearity. In contrast to conventional nonlinear…
Reservoir computing is a bio-inspired computing paradigm for processing time-dependent signals. Its hardware implementations have received much attention because of their simplicity and remarkable performance on a series of benchmark tasks.…
Embedding layers in transformer-based NLP models typically account for the largest share of model parameters, scaling with vocabulary size but not yielding performance gains proportional to scale. We propose an alternative approach in which…
Following the traditional paradigm of convolutional neural networks (CNNs), modern CNNs manage to keep pace with more recent, for example transformer-based, models by not only increasing model depth and width but also the kernel size. This…