Related papers: Gradient flow approach to geometric convergence an…

An appealing property of the natural gradient is that it is invariant to arbitrary differentiable reparameterizations of the model. However, this invariance property requires infinitesimal steps and is lost in practical implementations with…

This paper describes a method for solving smooth nonconvex minimization problems subject to bound constraints with good worst-case complexity guarantees and practical performance. The method contains elements of two existing methods: the…

We study gradient flow on the exponential loss for a classification problem with a one-layer softmax attention model, where the key and query weight matrices are trained separately. Under a separability assumption on the data, we show that…

This chapter describes how gradient flows and nonlinear power methods in Banach spaces can be used to solve nonlinear eigenvector-dependent eigenvalue problems, and how convergence of (discretized) approximations can be verified. We review…

Consider the problem of minimizing a convex differentiable function on the probability simplex, spectrahedron, or set of quantum density matrices. We prove that the exponentiated gradient method with Armjo line search always converges to…

We analyze the last-iterate convergence of the Anchored Gradient Descent Ascent algorithm for smooth convex-concave min-max problems. While previous work established a last-iterate rate of $\mathcal{O}(1/t^{2-2p})$ for the squared gradient…

Gradient schemes is a framework which enables the unified convergence analysis of many different methods -- such as finite elements (conforming, non-conforming and mixed) and finite volumes methods -- for $2^{\rm nd}$ order diffusion…

Intertwining analysis, algebra, numerical analysis and optimization, computing conjugate co-gradients of real-valued quotients gives rise to eigenvalue problems. In the linear Hermitian case, by inspecting optimal quotients in terms of…

In this paper two types of multgrid methods, i.e., the Rayleigh quotient iteration and the inverse iteration with fixed shift, are developed for solving the Maxwell eigenvalue problem with discontinuous relative magnetic permeability and…

This paper considers the problem of multi-agent distributed optimization. In this problem, there are multiple agents in the system, and each agent only knows its local cost function. The objective for the agents is to collectively compute a…

We consider the iterative solution of regularized saddle-point systems. When the leading block is symmetric and positive semi-definite on an appropriate subspace, Dollar, Gould, Schilders, and Wathen (2006) describe how to apply the…

Isogeometric analysis was applied very successfully to many problem classes like linear elasticity, heat transfer and incompressible flow problems but its application to compressible flows is very rare. However, its ability to accurately…

This work aims to accelerate the convergence of proximal gradient methods used to solve regularized linear inverse problems. This is achieved by designing a polynomial-based preconditioner that targets the eigenvalue spectrum of the normal…

Gradient methods are widely used in optimization problems. In practice, while the smoothness parameter can be estimated utilizing techniques such as backtracking, estimating the strong convexity parameter remains a challenge; moreover, even…

Direct and large-eddy simulations of turbulence are often solved using explicit temporal schemes. However, this imposes very small time-steps because the eigenvalues of the (linearized) dynamical system, re-scaled by the time-step, must lie…

We provide an interior point method based on quasi-Newton iterations, which only requires first-order access to a strongly self-concordant barrier function. To achieve this, we extend the techniques of Dunagan-Harvey [STOC '07] to maintain…

In this paper, we provide the universal first-order methods of Composite Optimization with new complexity analysis. It delivers some universal convergence guarantees, which are not linked directly to any parametric problem class. However,…

We solve an eigenvalue equation that appears in several papers about a wide range of physical problems. The Frobenius method leads to a three-term recurrence relation for the coefficients of the power series that, under suitable truncation,…

Bilevel optimization has been developed for many machine learning tasks with large-scale and high-dimensional data. This paper considers a constrained bilevel optimization problem, where the lower-level optimization problem is convex with…

We present a Riemannian optimization framework for Hartree-Fock theory formulated directly in the Sobolev space $H^1$. The orthonormality constraints are interpreted geometrically via infinite-dimensional Stiefel and Grassmann manifolds…