Related papers: Improved high-index saddle dynamics for finding sa…
We derive the system of differential equations for the gradient flow characterizing the training process of linear in-context learning in full generality. Next, we explore the geometric structure of the gradient flows in two instances,…
Recently, saddle point problems have received much attention due to their powerful modeling capability for a lot of problems from diverse domains. Applications of these problems occur in many applied areas, such as robust optimization,…
We analyze the variance of stochastic gradients along negative curvature directions in certain non-convex machine learning models and show that stochastic gradients exhibit a strong component along these directions. Furthermore, we show…
Generating high-quality novel view renderings of 3D Gaussian Splatting (3DGS) in scenes featuring transient objects is challenging. We propose a novel hybrid representation, termed as HybridGS, using 2D Gaussians for transient objects per…
We consider isogeometric discretizations of the Poisson model problem, focusing on high polynomial degrees and strong hierarchical refinements. We derive a posteriori error estimates by equilibrated fluxes, i.e., vector-valued mapped…
The study of dynamical systems has long focused on the characterization of their asymptotic dynamics such as fixed points, limit cycles and other types of attractors and how these invariant sets change their properties as systems parameters…
The paper surveys recent progresses in understanding the dynamics and loss landscape of the gradient flow equations associated to deep linear neural networks, i.e., the gradient descent training dynamics (in the limit when the step size…
Nesterov's accelerated gradient descent (AGD), an instance of the general family of "momentum methods", provably achieves faster convergence rate than gradient descent (GD) in the convex setting. However, whether these methods are superior…
The multi-modality and stochastic characteristics of human behavior make motion prediction a highly challenging task, which is critical for autonomous driving. While deep learning approaches have demonstrated their great potential in this…
Recent advances in symbolic dynamic programming (SDP) combined with the extended algebraic decision diagram (XADD) data structure have provided exact solutions for mixed discrete and continuous (hybrid) MDPs with piecewise linear dynamics…
An analytic reversible Hamiltonian system with two degrees of freedom is studied in a neighborhood of its symmetric heteroclinic connection made up of a symmetric saddle-center, a symmetric orientable saddle periodic orbit lying in the same…
Adaptive methods such as Adam and RMSProp are widely used in deep learning but are not well understood. In this paper, we seek a crisp, clean and precise characterization of their behavior in nonconvex settings. To this end, we first…
Multiscale stochastic dynamical systems have been widely adopted to a variety of scientific and engineering problems due to their capability of depicting complex phenomena in many real world applications. This work is devoted to…
In this work we present a novel methodology that combines Higher Order Singular Value Decomposition (HOSVD) with Deep Learning (DL) techniques for super-resolution in computational fluid dynamics (CFD) and sparse experimental datasets. This…
Slow manifolds are important geometric structures in the state spaces of dynamical systems with multiple time scales. This paper introduces an algorithm for computing trajectories on slow manifolds that are normally hyperbolic with both…
We present a novel view of nonlinear manifold learning using derivative-free optimization techniques. Specifically, we propose an extension of the classical multi-dimensional scaling (MDS) method, where instead of performing gradient…
The paper is devoted to the numerical solution of elastoplastic constitutive initial value problems. An improved form of the implicit return-mapping scheme for nonsmooth yield surfaces is proposed that systematically builds on a…
In this paper, we study the problem of escaping from saddle points in smooth nonconvex optimization problems subject to a convex set $\mathcal{C}$. We propose a generic framework that yields convergence to a second-order stationary point of…
This work introduces hybrid stochastic differential equations with memory (mH-SDEs), a new class of stochastic systems where transition rates depend on the joint history of both Euclidean and discrete components. This extends existing…
Recently, the problem of local minima in very high dimensional non-convex optimization has been challenged and the problem of saddle points has been introduced. This paper introduces a dynamic type of normalization that forces the system to…