Related papers: Decomposition of Small Transformer Models

Through considerable effort and intuition, several recent works have reverse-engineered nontrivial behaviors of transformer models. This paper systematizes the mechanistic interpretability process they followed. First, researchers choose a…

We propose a new algorithm to compute a shifted proper orthogonal decomposition (sPOD) for systems dominated by multiple transport velocities. The sPOD is a recently proposed mode decomposition technique which overcomes the poor performance…

Increased training parameters have enabled large pre-trained models to excel in various downstream tasks. Nevertheless, the extensive computational requirements associated with these models hinder their widespread adoption within the…

The history-dependent behaviors of classical plasticity models are often driven by internal variables evolved according to phenomenological laws. The difficulty to interpret how these internal variables represent a history of deformation,…

Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple…

Dynamic Mode Decomposition (DMD) is a data-driven decomposition technique extracting spatio-temporal patterns of time-dependent phenomena. In this paper, we perform a comprehensive theoretical analysis of various variants of DMD. We provide…

Given a demonstration, a robot should be able to generalize a skill to any object it encounters-but existing approaches to skill transfer often fail to adapt to objects with unfamiliar shapes. Motivated by examples of improved transfer from…

Decomposition is the basis of works dedicated to business process modelling at the stage of information and management systems analysis and design. The article shows that the business process decomposition can be represented as a Galton…

Mode decomposition is a prototypical pattern recognition problem that can be addressed from the (a priori distinct) perspectives of numerical approximation, statistical inference and deep learning. Could its analysis through these combined…

Symmetric positive definite~(SPD) matrices have shown important value and applications in statistics and machine learning, such as FMRI analysis and traffic prediction. Previous works on SPD matrices mostly focus on discriminative models,…

It is well known that vision classification models suffer from poor calibration in the face of data distribution shifts. In this paper, we take a geometric approach to this problem. We propose Geometric Sensitivity Decomposition (GSD) which…

Mechanistic interpretability of transformers requires identifying not just which components matter but how they compose into the computational route that produced a prediction. Both attention and MLP follow a shared key-value template…

Mode-based model-reduction is used to reduce the degrees of freedom of high dimensional systems, often by describing the system state by a linear combination of spatial modes. Transport dominated phenomena, ubiquitous in technical and…

Conventional neural network elastoplasticity models are often perceived as lacking interpretability. This paper introduces a two-step machine learning approach that returns mathematical models interpretable by human experts. In particular,…

Mechanistic interpretability seeks to reverse-engineer neural network computations into human-understandable algorithms, yet extracting sparse computational circuits from billion-parameter language models remains challenging due to…

The transmission matrix (TM) is a representation to describe the light scattering process through a scattering medium. The degree of control elements in TM is correlated with the capacity of evaluating enormous equations with tremendous…

This paper proposes a methodology for generating and perturbing detailed derivations of equations at scale, aided by a symbolic engine, to evaluate the generalisability of Transformers to out-of-distribution mathematical reasoning problems.…

Dynamic mode decomposition (DMD) has recently become a popular tool for the non-intrusive analysis of dynamical systems. Exploiting Proper Orthogonal Decomposition (POD) as a dimensionality reduction technique, DMD is able to approximate a…

In the analysis of High-Energy Physics data, it is frequently desired to separate resonant signals from a smooth, non-resonant background. This paper introduces a new technique - functional decomposition (FD) - to accomplish this task. It…

This work proposes an extension of the 1-D Hilbert Huang transform for the analysis of images. The proposed method consists in (i) adaptively decomposing an image into oscillating parts called intrinsic mode functions (IMFs) using a mode…