Related papers: Dynamical Behaviors of the Gradient Flows for In-C…

Assignment flows denote a class of dynamical models for contextual data labeling (classification) on graphs. We derive a novel parametrization of assignment flows that reveals how the underlying information geometry induces two processes…

Dynamical systems that describe the escape from the basins of attraction of stable invariant sets are presented and analyzed. It is shown that the stable fixed points of such dynamical systems are the index-1 saddle points. Generalizations…

At present, the mechanisms of in-context learning in Transformers are not well understood and remain mostly an intuition. In this paper, we suggest that training Transformers on auto-regressive objectives is closely related to…

In this work, we present a new approach to analyze the gradient flow for a positive semi-definite matrix denoising problem in an extensive-rank and high-dimensional regime. We use recent linear pencil techniques of random matrix theory to…

We study a gradient flow on Sobolev diffeomorphisms for the problem of image registration. The energy functional quantifies the effect of transforming a template to a target, while also penalizing deformation of the metric tensor. The main…

We propose a method for learning dynamical systems from high-dimensional empirical data that combines variational autoencoders and (spatio-)temporal attention within a framework designed to enforce certain scientifically-motivated…

Advancements in artificial intelligence call for a deeper understanding of the fundamental mechanisms underlying deep learning. In this work, we propose a theoretical framework to analyze learning dynamics through the lens of dynamical…

Understanding the learning dynamics of neural networks is one of the key issues for the improvement of optimization algorithms as well as for the theoretical comprehension of why deep neural nets work so well today. In this paper, we…

We consider the idealized setting of gradient flow on the population risk for infinitely wide two-layer ReLU neural networks (without bias), and study the effect of symmetries on the learned parameters and predictors. We first describe a…

We introduce manifold-learning flows (M-flows), a new class of generative models that simultaneously learn the data manifold as well as a tractable probability density on that manifold. Combining aspects of normalizing flows, GANs,…

We develop new variational principles to study stability and equilibrium of axisymmetric flows. We show that there is an infinite number of steady state solutions. We show that these steady states maximize a (non-universal) $H$-function. We…

In this paper, we investigate the geometry of a general class of gradient flows with multiple local maxima. we decompose the underlying space into disjoint regions of attraction and establish the adjacency criterion. The criterion states a…

We demonstrate how deep convolutional neural networks can be trained to predict 2+1 D hydrodynamic simulation results for flow coefficients, mean-transverse-momentum and charged particle multiplicity from the initial energy density profile.…

To understand how deep learning works, it is crucial to understand the training dynamics of neural networks. Several interesting hypotheses about these dynamics have been made based on empirically observed phenomena, but there exists a…

We introduce a class of unconditionally energy stable, high order accurate schemes for gradient flows in a very general setting. The new schemes are a high order analogue of the minimizing movements approach for generating a time discrete…

This report is actually the in-class project of MATH489 FA2022, Dynamics and Differential Equations in University of Illinois Urbana-Champaign. The report is mainly about gradient flow of a Morse-type functions (Morse functions, Morse--Bott…

Learning rules -- prescriptions for updating model parameters to improve performance -- are typically assumed rather than derived. Why do some learning rules work better than others, and under what assumptions can a given rule be considered…

In order to better model complex real-world data such as multiphase flow, one approach is to develop pattern recognition techniques and robust features that capture the relevant information. In this paper, we use deep learning methods, and…

Large pretrained language models have shown surprising in-context learning (ICL) ability. With a few demonstration input-label pairs, they can predict the label for an unseen input without parameter updates. Despite the great success in…

Despite the widespread use of gradient-based algorithms for optimizing high-dimensional non-convex functions, understanding their ability of finding good minima instead of being trapped in spurious ones remains to a large extent an open…