Related papers: Gradient Flows and Nonlinear Power Methods for the…
Existing analyses of optimization in deep learning are either continuous, focusing on (variants of) gradient flow, or discrete, directly treating (variants of) gradient descent. Gradient flow is amenable to theoretical analysis, but is…
This paper reviews the gradient sampling methodology for solving nonsmooth, nonconvex optimization problems. An intuitively straightforward gradient sampling algorithm is stated and its convergence properties are summarized. Throughout this…
Optimization is at the heart of machine learning, statistics and many applied scientific disciplines. It also has a long history in physics, ranging from the minimal action principle to finding ground states of disordered systems such as…
In this paper, using generalized metric projection, we propose a new extragradient method for finding a common element of the solutions set of a generalized equilibrium problem and a variational inequality for an $\alpha$-inverse-strongly…
This paper tackles the unconstrained minimization of a class of nonsmooth and nonconvex functions that can be written as finite max-functions. A gradient and function-based sampling method is proposed which, under special circumstances,…
We prove the convergence of a particle method for the approximation of diffusive gradient flows in one dimension. This method relies on the discretisation of the energy via non-overlapping balls centred at the particles and preserves the…
The energetically most efficient way how a deformed red blood cell regains equilibrium is mathematically described by the gradient flow of the Canham-Helfrich functional, including a spontaneous curvature and the conservation of surface…
Asymptotic eigenvalues and eigenfunctions for the Orr-Sommerfeld equation in two and three dimensional incompressible flows on an infinite domain and on a semi-infinite domain are obtained. Two configurations are considered, one in which a…
We consider the problem of optimizing the sum of a smooth convex function and a non-smooth convex function using proximal-gradient methods, where an error is present in the calculation of the gradient of the smooth term or in the proximity…
A stochastic conjugate gradient method for approximation of a function is proposed. The proposed method avoids computing and storing the covariance matrix in the normal equations for the least squares solution. In addition, the method…
This paper deals with nonsmooth convex optimization problems in Euclidean spaces. We identify special elements of the subdifferential of a convex function, called specular gradients. Based on this observation, we propose three numerical…
We study the existence of positive eigenvalues with associated nonnegative mild eigenfunctions for a class of abstract initial value problems in Banach spaces with functional, possibly nonlocal, initial conditions. The framework includes…
Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step…
This article is concerned with the approximation and expressive powers of deep neural networks. This is an active research area currently producing many interesting papers. The results most commonly found in the literature prove that neural…
This article deals with the stationary Gross-Pitaevskii non-linear eigenvalue problem in the presence of a rotating magnetic field that is used to model macroscopic quantum effects such as Bose-Einstein condensates (BECs). In this regime,…
Gradient normalization and soft clipping are two popular techniques for tackling instability issues and improving convergence of stochastic gradient descent (SGD) with momentum. In this article, we study these types of methods through the…
We consider the mathematical analysis and numerical approximation of a system of nonlinear partial differential equations that arises in models that have relevance to steady isochoric flows of colloidal suspensions. The symmetric velocity…
It is well understood that neural networks with carefully hand-picked weights provide powerful function approximation and that they can be successfully trained in over-parametrized regimes. Since over-parametrization ensures zero training…
In this paper, we analyze several methods for approximating gradients of noisy functions using only function values. These methods include finite differences, linear interpolation, Gaussian smoothing and smoothing on a sphere. The methods…
Many problems in machine learning and statistics can be formulated as (generalized) eigenproblems. In terms of the associated optimization problem, computing linear eigenvectors amounts to finding critical points of a quadratic function…