Related papers: Decomposition of Small Transformer Models

The modal decomposition techniques of proper orthogonal decomposition (POD) and dynamic mode decomposition (DMD) have become a common method for analysing the spatio-temporal coherence of dynamical systems. In particular, these techniques…

Decoupling is a powerful modeling paradigm for representing multivariate functions as compositions of linear transformations and univariate nonlinear functions. A single-layer decoupling can be viewed as a fully connected neural network…

This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models---including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation---which…

The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function…

In the manufacturing process, sensor data collected from equipment is crucial for building predictive models to manage processes and improve productivity. However, in the field, it is challenging to gather sufficient data to build robust…

Decoder transformers have continued increasing in scale reaching hundreds of billions of parameters. Due to their scale the same decoder sets state-of-the-art results on various language tasks via prompting or fine-tuning. Yet, these large…

Transformer-based models generate hidden states that are difficult to interpret. In this work, we analyze hidden states and modify them at inference, with a focus on motion forecasting. We use linear probing to analyze whether interpretable…

Functional decomposition is the process of breaking down a function $f$ into a composition $f=g(f_1,\dots,f_k)$ of simpler functions $f_1,\dots,f_k$ belonging to some class $\mathcal{F}$. This fundamental notion can be used to model…

Interpretation of deep learning remains a very challenging problem. Although the Class Activation Map (CAM) is widely used to interpret deep model predictions by highlighting object location, it fails to provide insight into the salient…

Dynamic mode decomposition (DMD) is a versatile approach that enables the construction of low-order models from data. Controller design tasks based on such models require estimates and guarantees on predictive accuracy. In this work, we…

Tensors provide a structured representation for multidimensional data, yet discretization can obscure important information when such data originates from continuous processes. We address this limitation by introducing a functional Tucker…

Off-the-shelf Gaussian Process (GP) covariance functions encode smoothness assumptions on the structure of the function to be modeled. To model complex and non-differentiable functions, these smoothness assumptions are often too…

The decomposition of a time series is an essential task that helps to understand its very nature. It facilitates the analysis and forecasting of complex time series expressing various hidden components such as the trend, seasonal…

Accurate and efficient plasma models are essential to understand and control experimental devices. Existing magnetohydrodynamic or kinetic models are nonlinear, computationally intensive, and can be difficult to interpret, while often only…

In this work, we propose a new stochastic domain decomposition method for solving steady-state partial differential equations (PDEs) with random inputs. Based on the efficiency of the Variable-separation (VS) method in simulating stochastic…

In autoregressive modeling for tensor-valued time series, Tucker decomposition, when applied to the coefficient tensor, provides a clear interpretation of supervised factor modeling but loses its efficiency rapidly with increasing tensor…

Empirical Mode Decomposition is an adaptive and local tool that extracts underlying analytical components of a non-linear and non-stationary process, in turn, is the basis of Hilbert Huang transform, however, there are problems such as…

Sentence embeddings are central to modern NLP and AI systems, yet little is known about their internal structure. While we can compare these embeddings using measures such as cosine similarity, the contributing features are not…

Although it is known that transformer language models (LMs) pass features from early layers to later layers, it is not well understood how this information is represented and routed by the model. We analyze a mechanism used in two LMs to…

The problem of modeling ropes arises in many applications, including providing haptic feedback to surgeons who are using surgical robots to realign the distal and proximal ends of split bones. Here, we consider a simplified, 2D variant of…