Related papers: Toward Manifest Relationality in Transformers via …
The attention mechanism in its standard implementation contains extraneous rotational degrees of freedom that are carried through computation but do not affect model activations or outputs. We introduce a simple symmetry-breaking protocol…
Many complex engineering systems consist of multiple subsystems that are developed by different teams of engineers. To analyse, simulate and control such complex systems, accurate yet computationally efficient models are required. Modular…
Many complex mechatronic systems consist of multiple interconnected dynamical subsystems, which are designed, developed, analyzed, and manufactured by multiple independent teams. To support such a design approach, a modular model framework…
This paper explores the role of symmetries and reduction in nonlinear control and optimal control systems. The focus of the paper is to give a geometric framework of symmetry reduction of optimal control systems as well as to show how to…
Modern deep learning models are highly overparameterized, resulting in large sets of parameter configurations that yield the same outputs. A significant portion of this redundancy is explained by symmetries in the parameter…
The last two decades have seen major developments in interpolatory methods for model reduction of large-scale linear dynamical systems. Advances of note include the ability to produce (locally) optimal reduced models at modest cost; refined…
Symmetries impose structure on the Hilbert space of a quantum mechanical model. The mathematical units of this structure are the irreducible representations of symmetry groups and I consider how they function as conceptual units of…
We offer a new formalism for global explanations of pairwise feature dependencies and interactions in supervised models. Building upon SHAP values and SHAP interaction values, our approach decomposes feature contributions into synergistic,…
This research endeavors to offer insights into unlocking the further potential of transformer-based architectures. One of the primary motivations is to offer a geometric interpretation for the attention mechanism in transformers. In our…
A geometric approach to integrability and reduction of dynamical system is developed from a modern perspective. The main ingredients in such analysis are the infinitesimal symmetries and the tensor fields that are invariant under the given…
Complex engineering models are typically computationally demanding and defined by a high-dimensional parameter space challenging the comprehensive exploration of parameter effects and design optimization. To overcome this curse of…
Many countries are currently challenged with the extensive integration of renewable energy sources, which necessitates vast capacity expansion measures. These measures in turn require comprehensive power flow studies, which are often…
Many high-dimensional optimisation problems exhibit rich geometric structures in their set of minimisers, often forming smooth manifolds due to over-parametrisation or symmetries. When this structure is known, at least locally, it can be…
Many real-world complex networks contain a significant amount of structural redundancy, in which multiple vertices play identical topological roles. Such redundancy arises naturally from the simple growth processes which form and shape many…
In this paper, we discuss a novel model reduction framework for generalized linear systems. The transfer functions of these systems are assumed to have a special structure, e.g., coming from second-order linear systems and time-delay…
This paper introduces the concept of abstracted model reduction: a framework to improve the tractability of structure-preserving methods for the complexity reduction of interconnected system models. To effectively reduce high-order,…
A novel general framework is proposed in this paper for dimension reduction in regression to fill the gap between linear and fully nonlinear dimension reduction. The main idea is to transform first each of the raw predictors monotonically,…
Transformer architectures are typically described in algorithmic and statistical terms, leaving their internal mechanics without a familiar structural language for researchers trained in physical theories. To bridge this gap, we develop a…
Many applications, such as optimization, uncertainty quantification and inverse problems, require repeatedly performing simulations of large-dimensional physical systems for different choices of parameters. This can be prohibitively…
We survey the role of symmetry in diffeomorphic registration of landmarks, curves, surfaces, images and higher-order data. The infinite dimensional problem of finding correspondences between objects can for a range of concrete data types be…