Related papers: PIDformer: Transformer Meets Control Theory

Transformers are increasingly adopted for modeling and forecasting time-series, yet their internal mechanisms remain poorly understood from a dynamical systems perspective. In contrast to classical autoregressive and state-space models,…

Deep reinforcement learning (DRL) has seen several successful applications to process control. Common methods rely on a deep neural network structure to model the controller or process. With increasingly complicated control structures, the…

Transformers encode structure in sequences via an expanding contextual history. However, their purely feedforward architecture fundamentally limits dynamic state tracking. State tracking -- the iterative updating of latent variables…

The popular single-input single-output control structures and classic design methods (e.g., root locus analysis) for the power control of grid-forming converters have limitations in applying to different line characteristics and providing…

In deep learning theory, the covariance matrix of the representations serves as a proxy to examine the network's trainability. Motivated by the success of Transformers, we study the covariance matrix of a modified Softmax-based attention…

Despite the effectiveness of deep neural networks in numerous natural language processing applications, recent findings have exposed the vulnerability of these language models when minor perturbations are introduced. While appearing…

In this paper, we develop a rigorous optimal control-theoretic approach to Transformer training that respects key structural constraints such as (i) realized-input-independence during execution, (ii) the ensemble control nature of the…

Since proportional-integral-derivative (PID) controllers absolutely dominate the control engineering, numbers of different control structures and theories have been developed to enhance the efficiency of PID controllers. Thus, it is…

Enabling image generation models to be spatially controlled is an important area of research, empowering users to better generate images according to their own fine-grained specifications via e.g. edge maps, poses. Although this task has…

We study Transformers through the perspective of optimal control theory, using tools from continuous-time formulations to derive actionable insights into training and architecture design. This framework improves the performance of existing…

In this paper, modification of the classical PID controller and development of open-loop control mechanisms to improve stability and robustness of a differential wheeled robot are discussed. To deploy the algorithm, a test platform has been…

We consider the problem of optimal sparse output feedback controller synthesis for continuous linear time invariant systems when the feedback gain is static and subject to specified structural constraints. Introducing an additional term…

Overall, in any system, the proportional term, integral term, and derivative term combined to produce a fast response time, less overshoot, no oscillations, increased stability, and no steady-state errors. Eliminating the steady state…

Robust control design is mainly devoted to guarantee closed-loop stability of a model-based control law in presence of parametric and structural uncertainties. The control law is usually a complex feedback law which is derived from a…

A widely cited result by Dong et al. (2021) showed that Transformers built from self-attention alone, without skip connections or feed-forward layers, suffer from rapid rank collapse: all token representations converge to a single…

Transformer models have emerged as fundamental tools across various scientific and engineering disciplines, owing to their outstanding performance in diverse applications. Despite this empirical success, the theoretical foundations of…

Transformers evaluated in a single, fixed-depth pass are provably limited in expressive power to the constant-depth circuit class TC0. Running a Transformer autoregressively removes that ceiling -- first in next-token prediction and, more…

An important development in deep learning from the earliest MLPs has been a move towards architectures with structural inductive biases which enable the model to keep distinct sources of information and routes of processing well-separated.…

Proportional-Integral-Derivative (PID) control has been the workhorse of control technology for about a century. Yet to this day, designing and tuning PID controllers relies mostly on either tabulated rules (Ziegler-Nichols) or on classical…

PID control has been the dominant control strategy in the process industry due to its simplicity in design and effectiveness in controlling a wide range of processes. However, traditional methods on PID tuning often require extensive domain…