Related papers: Gradient descent in some simple settings

Neural networks trained with standard objectives exhibit behaviors characteristic of probabilistic inference: soft clustering, prototype specialization, and Bayesian uncertainty tracking. These phenomena appear across architectures -- in…

A tutorial review is given of some developments and applications of stochastic processes from the point of view of the practicioner physicist. The index is the following: 1.- Introduction 2.- Stochastic Processes 3.- Transient Stochastic…

The starting assumptions to study the convergence and complexity of gradient-type methods may be the smoothness (also called Lipschitz continuity of gradient) and the strong convexity. In this note, we revisit these two basic properties…

Differentially private stochastic gradient descent (DP-SGD) is known to have poorer training and test performance on large neural networks, compared to ordinary stochastic gradient descent (SGD). In this paper, we perform a detailed study…

Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study…

We present a new class of gradient-type optimization methods that extends vanilla gradient descent, mirror descent, Riemannian gradient descent, and natural gradient descent. Our approach involves constructing a surrogate for the objective…

We investigate the effect of time-dependent noise on the shape of a morphogen gradient in a developing embryo. Perturbation theory is used to calculate the deviations from deterministic behavior in a simple reaction-diffusion model of…

We consider a slow passage through a point of loss of stability. If the passage is sufficiently slow, the dynamics are controlled by additive random disturbances, even if they are extremely small. We derive expressions for the `exit value'…

The paper contributes to strengthening the relation between machine learning and the theory of differential equations. In this context, the inverse problem of fitting the parameters, and the initial condition of a differential equation to…

Although the theoretical behavior of one-dimensional random walks in random environments is well understood, the numerical evaluation of various characteristics of such processes has received relatively little attention. This paper develops…

How to find flat minima? We propose running normalized gradient descent, usually reserved for nonsmooth optimization, with sufficiently slowly diminishing step sizes. This induces implicit regularization towards flat minima if an…

In machine learning, stochastic gradient descent (SGD) is widely deployed to train models using highly non-convex objectives with equally complex noise models. Unfortunately, SGD theory often makes restrictive assumptions that fail to…

Additive noise in Partial Differential equations, in particular those of fluid mechanics, has relatively natural motivations. The aim of this work is showing that suitable multiscale arguments lead rigorously, from a model of fluid with…

Noise is often considered to be a nuisance. Here we argue that it can be a useful probe of fluctuating two level systems in glasses. It can be used to: (1) shed light on whether the fluctuations are correlated or independent events; (2)…

In this contribution we present an exploratory study of several novel methods for numerical stochastic perturbation theory. For the investigation we consider observables defined through the gradient flow in the simple {\phi}^4 theory.

In the first part of this paper, we consider a family of continuous-time dynamical systems coupled with diffusion-transmutation processes. Under certain conditions, such randomly perturbed dynamical systems can be interpreted as an averaged…

How much can you say about the gradient of a neural network without computing a loss or knowing the label? This may sound like a strange question: surely the answer is "very little." However, in this paper, we show that gradients are more…

Microcanonical gradient descent is a sampling procedure for energy-based models allowing for efficient sampling of distributions in high dimension. It works by transporting samples from a high-entropy distribution, such as Gaussian white…

Gradient dynamics play a central role in determining the stability and generalization of deep neural networks. In this work, we provide an empirical analysis of how variance and standard deviation of gradients evolve during training,…

We describe an alternative learning method for neural networks, which we call Blind Descent. By design, Blind Descent does not face problems like exploding or vanishing gradients. In Blind Descent, gradients are not used to guide the…