Related papers: Gradient descent in higher codimension

Stochastic gradient descent (SGD) is often applied to train Deep Neural Networks (DNNs), and research efforts have been devoted to investigate the convergent dynamics of SGD and minima found by SGD. The influencing factors identified in the…

We study the effects of disorder on strongly coupled compressible matter in 2+1 dimensions. Our system consists of a D3/D5 intersection at finite temperature and in the presence of a disordered chemical potential. We first study the impact…

It has been shown that perturbing the input during training implicitly regularises the gradient of the learnt function, leading to smoother models and enhancing generalisation. However, previous research mostly considered the addition of…

Using MRI and high-speed video we investigate the motion of a large intruder particle inside a vertically shaken bed of smaller particles. We find a pronounced, non-monotonic density dependence, with both light and heavy intruders moving…

We use particle dynamics simulations to probe the correlations between noise and dynamics in a variety of disordered systems, including superconducting vortices, 2D electron liquid crystals, colloids, domain walls, and granular media. The…

Although gradient descent with Polyak's momentum is widely used in modern machine and deep learning, a concrete understanding of its effects on the training trajectory remains elusive. In this work, we empirically show that for linear…

This article overviews how gradient flows, and discretizations thereof, are useful to design and analyze optimization and sampling algorithms. The interplay between optimization, sampling, and gradient flows is an active research area; our…

We propose new numerical approach to non-conservative dynamical systems. Our method being of low order, enhances qualitative performance of standard discrete gradient algorithm, thank to new concept of a reservoir. Paper is of explanatory…

This paper presents a heuristic derivation of a geometric minimum action method that can be used to determine most-probable transition paths in noise-driven dynamical systems. Particular attention is focused on systems that violate detailed…

Stochastic gradient descent (SGD) is a key ingredient in the training of deep neural networks and yet its geometrical significance appears elusive. We study a deterministic model in which the trajectories of our dynamical systems are…

Establishing a theoretical analysis that explains why deep learning can outperform shallow learning such as kernel methods is one of the biggest issues in the deep learning literature. Towards answering this question, we evaluate excess…

The paper contributes to strengthening the relation between machine learning and the theory of differential equations. In this context, the inverse problem of fitting the parameters, and the initial condition of a differential equation to…

The long term aim is to use modern dynamical systems theory to derive discretisations of noisy, dissipative partial differential equations. As a first step we here consider a small domain and apply stochastic centre manifold techniques to…

We investigate the capability of dynamical decoupling techniques to reduce decoherence from a realistic environment generating 1/f noise. The predominance of low frequency modes in the noise profile allows for decoherence scenarios where…

Randomly-forced fluid flow in the presence of scale-unselective dissipation develops mean currents following topographic contours. Known mechanisms based on the scale-selective action of damping processes are not at work in this situation.…

Gradients of voltage, pressure, temperature, and salinity can transport objects in micro- and nanofluidic systems by well known mechanisms. Here we report the discovery of a transport effect driven by viscosity gradients, which cause an…

Collective dynamics result from interactions among noisy dynamical components. Examples include heartbeats, circadian rhythms, and various pattern formations. Because of noise in each component, collective dynamics inevitably involve…

Stochastic neurons can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic neurons, i.e.,…

Deep neural networks (DNN) are typically optimized using stochastic gradient descent (SGD). However, the estimation of the gradient using stochastic samples tends to be noisy and unreliable, resulting in large gradient variance and bad…

We present a manifestly covariant formulation of the gradient descent method, ensuring consistency across arbitrary coordinate systems and general curved trainable spaces. The optimization dynamics is defined using a covariant force vector…