Related papers: Multi-Headed Transformer Architectures as Time-dep…
Pretrained language models based on the transformer architecture have shown great success in NLP. Textual training data often comes from the web and is thus tagged with time-specific information, but most language models ignore this…
This article deals with the consensus problem involving agents with time-varying singularities in the dynamics or communication in undirected graph networks. Existing results provide control laws which guarantee asymptotic consensus. These…
Turbulence remains one of the last unresolved problems of classical physics and a major bottleneck to accurate flow prediction in climate, aerospace, and energy systems. Industrial simulations therefore rely on averaged representations of…
Large Language Models (LLMs) based on autoregressive, decoder-only Transformers generate text one token at a time, where a token represents a discrete unit of text. As each newly produced token is appended to the partial output sequence,…
In this study, we propose a design methodology of distributed controllers for multi-agent systems on a class of directed interaction networks by extending the gradient-flow method. Although the gradient-flow method is a common design tool…
Over the past two decades, complex network theory provided the ideal framework for investigating the intimate relationships between the topological properties characterizing the wiring of connections among a system's unitary components and…
The self-attention mechanism, while foundational to modern Transformer architectures, suffers from a critical inefficiency: it frequently allocates substantial attention to redundant or noisy context. Differential Attention addressed this…
A known limitation of time-dependent mean-field approaches is a lack of quantum tunneling for collective motions such as in sub-barrier fusion reactions. As a first step toward a solution, a time-dependent model is considered using a…
Transformers have revolutionized various domains of artificial intelligence due to their unique ability to model long-range dependencies in data. However, they lack in nuanced, context-dependent modulation of features and information flow.…
Multi-head attention is a driving force behind state-of-the-art transformers, which achieve remarkable performance across a variety of natural language processing (NLP) and computer vision tasks. It has been observed that for many…
In this work we propose an energy functional along the lines of Modern Hopfield Networks (MNH), the stationary points of which correspond to the attention due to Vaswani et al. [12], thus unifying both frameworks. The minima of this…
Separated flow transition is a very popular phenomenon in gas turbines, especially low-pressure turbines (LPT). Low-fidelity simulations are often used for gas turbine design. However, they are unable to predict separated flow transition…
Sequence modeling has produced diverse architectures -- from classical recurrent neural networks to modern Transformers and state space models (SSMs) -- yet a unified theoretical understanding of expressivity and trainability trade-offs…
Traffic forecasting problem remains a challenging task in the intelligent transportation system due to its spatio-temporal complexity. Although temporal dependency has been well studied and discussed, spatial dependency is relatively less…
Understanding strongly correlated systems is essential for advancing quantum chemistry and materials science, yet conventional methods like Density Functional Theory (DFT) often fail to capture their complex electronic behavior. To address…
The dot product attention mechanism, originally designed for natural language processing tasks, is a cornerstone of modern Transformers. It adeptly captures semantic relationships between word pairs in sentences by computing a similarity…
Graph transformers are the state-of-the-art for learning from graph-structured data and are empirically known to avoid several pitfalls of message-passing architectures. However, there is limited theoretical analysis on why these models…
Understanding how transient dynamics unfold in response to localized inputs is central to predicting and controlling signal propagation in network systems, including neural processing, epidemic intervention, and power-grid resilience.…
Interacting particle systems provide a fundamental framework for modeling collective behavior in biological, social, and physical systems. In many applications, stochastic perturbations are essential for capturing environmental variability…
We study distribution-on-distribution regression problems in which a response distribution depends on multiple distributional predictors. Such settings arise naturally in applications where the outcome distribution is driven by several…