Related papers: Decomposition of Small Transformer Models

Transformers are widely used to extract semantic meanings from input tokens, yet they usually operate as black-box models. In this paper, we present a simple yet informative decomposition of hidden states (or embeddings) of trained…

Unsupervised multi-object scene decomposition is a fast-emerging problem in representation learning. Despite significant progress in static scenes, such models are unable to leverage important dynamic cues present in video. We propose a…

How to integrate human factors into the motion planning system is of great significance for improving the acceptance of intelligent vehicles. Decomposing motion into primitives and then accurately and smoothly joining the motion primitives…

Dynamic mode decomposition (DMD) is a recently developed tool for the analysis of the behavior of complex dynamical systems. In this paper, we will propose an extension of DMD that exploits low-rank tensor decompositions of potentially…

Dynamic Mode Decomposition (DMD) is an unsupervised machine learning method that has attracted considerable attention in recent years owing to its equation-free structure, ability to easily identify coherent spatio-temporal structures in…

This work proposes a new framework of model reduction for parametric complex systems. The framework employs a popular model reduction technique dynamic mode decomposition (DMD), which is capable of combining data-driven learning and physics…

Decomposition, statically dividing a program into multiple units, is a common programming technique for realizing parallelism and refining programs. The decomposition of a sequential program into components is tedious, due to the…

Deformable object manipulation remains a challenging task in robotics research. Conventional techniques for parameter inference and state estimation typically rely on a precise definition of the state space and its dynamics. While this is…

Articulated objects are ubiquitous and important in robotics, AR/VR, and digital twins. Most self-supervised methods for articulated object modeling reconstruct discrete interaction states and relate them via cross-state geometric…

Representation learning often plays a critical role in reinforcement learning by managing the curse of dimensionality. A representative class of algorithms exploits a spectral decomposition of the stochastic transition dynamics to construct…

Circuit analysis of any certain model behavior is a central task in mechanistic interpretability. We introduce our circuit discovery pipeline with Sparse Autoencoders (SAEs) and a variant called Transcoders. With these two modules inserted…

Signal decomposition and multiscale signal analysis provide many useful tools for time-frequency analysis. We proposed a random feature method for analyzing time-series data by constructing a sparse approximation to the spectrogram. The…

Shapley values have become one of the go-to methods to explain complex models to end-users. They provide a model agnostic post-hoc explanation with foundations in game theory: what is the worth of a player (in machine learning, a feature…

Mechanistic interpretability often uses activation patching, causal tracing, path patching, and steering directions to reveal behaviorally meaningful directions in Transformer activation space. This paper develops a field-theoretic…

Transformers have significantly advanced the field of natural language processing, but comprehending their internal mechanisms remains a challenge. In this paper, we introduce a novel geometric perspective that elucidates the inner…

Reconstructing the surfaces of deformable objects from correspondences between a 3D template and a 2D image is well studied under Shape-from-Template (SfT) methods; however, existing approaches break down when topological changes accompany…

This article presents two novel adaptive-sparse polynomial dimensional decomposition (PDD) methods for solving high-dimensional uncertainty quantification problems in computational science and engineering. The methods entail global…

Signal decomposition (SD) approaches aim to decompose non-stationary signals into their constituent amplitude- and frequency-modulated components. This represents an important preprocessing step in many practical signal processing…

We propose a new methodology for parametric domain decomposition using iterative principal component analysis. Starting with iterative principle component analysis, the high dimension manifold is reduced to the lower dimension manifold.…

Variables in many massive high-dimensional data sets are structured, arising for example from measurements on a regular grid as in imaging and time series or from spatial-temporal measurements as in climate studies. Classical multivariate…