Related papers: Coinductive guide to inductive transformer heads

Transformers have reshaped machine learning by utilizing attention mechanisms to capture complex patterns in large datasets, leading to significant improvements in performance. This success has contributed to the belief that "bigger means…

Modern distributed networks, notably transformers, acquire a remarkable ability (termed `in-context learning') to adapt their computation to input statistics, such that a fixed network can be applied to data from a broad range of systems.…

Training Transformers on algorithmic tasks frequently demonstrates an intriguing abrupt learning phenomenon: an extended performance plateau followed by a sudden, sharp improvement. This work investigates the underlying mechanisms for such…

Adaptive inference is a promising technique to improve the computational efficiency of deep models at test time. In contrast to static models which use the same computation graph for all instances, adaptive networks can dynamically adjust…

We introduce a family of periods of mixed Tate motives called dissection polylogarithms, that are indexed by combinatorial objects called dissection diagrams. The motivic coproduct on the former is encoded by a combinatorial Hopf algebra…

We introduce a natural Hopf algebra structure on the space of noncommutative symmetric functions which was recently studied as a vector space by Rosas and Sagan. The bases for this algebra are indexed by set partitions. We show that there…

In a recent series of communications we have shown that the reordering problem of bosons leads to certain combinatorial structures. These structures may be associated with a certain graphical description. In this paper, we show that there…

The $q$--deformation $U_q (h_4)$ of the harmonic oscillator algebra is defined and proved to be a Ribbon Hopf algebra.Associated with this Hopf algebra we define an infinite dimensional braid group representation on the Hilbert space of the…

This paper presents a mathematical interpretation of self-attention by connecting it to distributional semantics principles. We show that self-attention emerges from projecting corpus-level co-occurrence statistics into sequence context.…

Transformer-based models have been achieving state-of-the-art results in several fields of Natural Language Processing. However, its direct application to speech tasks is not trivial. The nature of this sequences carries problems such as…

The controlled-NOT gate of qubit quantum circuits is shown to be described in terms of a Hopf algebra. Accordingly, any qubit quantum circuit can be expressed as the Hopf algebraic computations and unitary transformations on one qubit.

Modern data acquisition routinely produce massive amounts of event sequence data in various domains, such as social media, healthcare, and financial markets. These data often exhibit complicated short-term and long-term temporal…

Inductive learning aims to construct general models from specific examples, guided by biases that influence hypothesis selection and determine generalization capacity. In this work, we focus on characterizing the relational inductive biases…

There is a recent growing interest in applying Deep Learning techniques to tabular data, in order to replicate the success of other Artificial Intelligence areas in this structured domain. Specifically interesting is the case in which…

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long…

The transformer neural network has significantly out-shined all other neural network architectures as the engine behind large language models. We provide a theoretical analysis of the expressivity of the transformer architecture through the…

The purpose of this paper is two fold: we study the behaviour of the forgetful functor from S-modules to graded vector spaces in the context of algebras over an operad and derive from this theory the construction of combinatorial Hopf…

Pairwise dot product-based attention allows Transformers to exchange information between tokens in an input-dependent way, and is key to their success across diverse applications in language and vision. However, a typical Transformer model…

Transformers can generate predictions in two approaches: 1. auto-regressively by conditioning each sequence element on the previous ones, or 2. directly produce an output sequences in parallel. While research has mostly explored upon this…

We use the Hopf algebra structure of the time-ordered algebra of field operators to generate all connected weighted Feynman graphs in a recursive and efficient manner. The algebraic representation of the graphs is such that they can be…