Related papers: Gradient descent in some simple settings
This paper shows that a wide class of effective learning rules -- those that improve a scalar performance measure over a given time window -- can be rewritten as natural gradient descent with respect to a suitably defined loss function and…
The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored. We examine how natural gradients can be used in non-conjugate stochastic settings, together…
We prove that the evolution of weight vectors in online gradient descent can encode arbitrary polynomial-space computations, even in very simple learning settings. Our results imply that, under weak complexity-theoretic assumptions, it is…
Gradients of voltage, pressure, temperature, and salinity can transport objects in micro- and nanofluidic systems by well known mechanisms. Here we report the discovery of a transport effect driven by viscosity gradients, which cause an…
In this paper, we analyze several methods for approximating gradients of noisy functions using only function values. These methods include finite differences, linear interpolation, Gaussian smoothing and smoothing on a sphere. The methods…
Various results for higher-order perturbative calculations in the gradient-flow formalism are reviewed, including the gradient-flow beta function and the small-flow-time expansion of the hadronic vacuum polarization and the energy-momentum…
Point clouds acquired from scanning devices are often perturbed by noise, which affects downstream tasks such as surface reconstruction and analysis. The distribution of a noisy point cloud can be viewed as the distribution of a set of…
Momentum is known to accelerate the convergence of gradient descent in strongly convex settings without stochastic gradient noise. In stochastic optimization, such as training neural networks, folklore suggests that momentum may help deep…
Multivariate functions encountered in high-dimensional uncertainty quantification problems often vary most strongly along a few dominant directions in the input parameter space. We propose a gradient-based method for detecting these…
We provide the first convergence analysis of local gradient descent for minimizing the average of smooth and convex but otherwise arbitrary functions. Problems of this form and local gradient descent as a solution method are of importance…
When training neural networks, it has been widely observed that a large step size is essential in stochastic gradient descent (SGD) for obtaining superior models. However, the effect of large step sizes on the success of SGD is not well…
The adjoint sensitivity method scalably computes gradients of solutions to ordinary differential equations. We generalize this method to stochastic differential equations, allowing time-efficient and constant-memory computation of gradients…
Based on the gradient flow, we propose a new method to determine the bounce configuration for false vacuum decay. Our method is applicable to a large class of models with multiple fields. Since the bounce is a saddle point of an action, a…
This paper develops methodology for local sensitivity analysis based on directional derivatives associated with spatial processes. Formal gradient analysis for spatial processes was elaborated in previous papers, focusing on distribution…
Gradient, chemically modified, flat surfaces enable directed transport of droplets. Calculation of apparent contact angles inherent for gradient surfaces is challenging even for atomically flat ones. Wetting of gradient, flat solid surfaces…
We have analyzed the interplay between an externally added noise and the intrinsic noise of systems that relax fast towards a stationary state, and found that increasing the intensity of the external noise can reduce the total noise of the…
These are lecture notes for various Summer and Winter schools that I have given. The notes describe the methodology called Variational Modelling, and focus on the application to the modelling of gradient-flow systems. I describe the…
The practical performance of online stochastic gradient descent algorithms is highly dependent on the chosen step size, which must be tediously hand-tuned in many applications. The same is true for more advanced variants of stochastic…
Wetting and dewetting dynamics of simple and complex liquids is described by kinetic equations in gradient dynamics form that incorporates the various coupled dissipative processes in a fully thermodynamically consistent manner. After…
It has become customary to use a smoothing algorithm called "gradient flow" to fix the lattice spacing in a simulation, through a parameter called $t_0$. It is shown that in order to keep the length $t_0$ fixed with respect to mesonic or…