Related papers: Stochastic Modified Flows, Mean-Field Limits and D…

We discover restrained numerical instabilities in current training practices of deep networks with stochastic gradient descent (SGD), and its variants. We show numerical error (on the order of the smallest floating point bit and thus the…

Asynchronous computation and gradient compression have emerged as two key techniques for achieving scalability in distributed optimization for large-scale machine learning. This paper presents a unified analysis framework for distributed…

The selective frequency damping (SFD) method is an alternative to classical Newton's method to obtain unstable steady-state solutions of dynamical systems. However this method has two main limitations: it does not converge for arbitrary…

We study pathwise approximation of scalar stochastic differential equations at a single time point or globally in time by means of methods that are based on finitely many observations of the driving Brownian motion. We prove lower error…

Stochastic gradient descent (SGD) is of fundamental importance in deep learning. Despite its simplicity, elucidating its efficacy remains challenging. Conventionally, the success of SGD is ascribed to the stochastic gradient noise (SGN)…

Stochastic gradient descent (SGD) has been widely used in machine learning due to its computational efficiency and favorable generalization properties. Recently, it has been empirically demonstrated that the gradient noise in several deep…

In this article, we mainly study stochastic viscosity solutions for a class of semilinear stochastic integral-partial differential equations (SIPDEs). We investigate a new class of generalized backward doubly stochastic differential…

We develop a new computational framework to solve the partial differential equations (PDEs) governing the flow of the joint probability density functions (PDFs) in continuous-time stochastic nonlinear systems. The need for computing the…

We present analytical expressions for the time-dependent and stationary probability distributions corresponding to a stochastically perturbed one-dimensional flow with critical points, in two physically relevant situations: delayed…

Stochastic Gradient Descent (SGD) methods see many uses in optimization problems. Modifications to the algorithm, such as momentum-based SGD methods have been known to produce better results in certain cases. Much of this, however, is due…

Stochastic differential equations (SDEs) are one of the most important representations of dynamical systems. They are notable for the ability to include a deterministic component of the system and a stochastic one to represent random…

Recent work has established an empirically successful framework for adapting learning rates for stochastic gradient descent (SGD). This effectively removes all needs for tuning, while automatically reducing learning rates over time on…

In this paper we study the stochastic control problem of partially observed (multi-dimensional) stochastic system driven by both Brownian motions and fractional Brownian motions. In the absence of the powerful tool of Girsanov…

Sparse inversion and classification problems are ubiquitous in modern data science and imaging. They are often formulated as non-smooth minimisation problems. In sparse inversion, we minimise, e.g., the sum of a data fidelity term and an…

This paper studies the impacts of stochastic load fluctuations, namely the fluctuation intensity and the changing speed of load power, on the size of the voltage stability margin. To this end, Stochastic Differential-Algebraic Equations…

In this work we explore the limiting dynamics of deep neural networks trained with stochastic gradient descent (SGD). As observed previously, long after performance has converged, networks continue to move through parameter space by a…

Accurate and efficient fluid flow models are essential for applications relating to many physical phenomena including geophysical, aerodynamic, and biological systems. While these flows may exhibit rich and multiscale dynamics, in many…

We consider in this work a system of two stochastic differential equations named the perturbed compositional gradient flow. By introducing a separation of fast and slow scales of the two equations, we show that the limit of the slow motion…

Atmospheric models used for weather and climate prediction are traditionally formulated in a deterministic manner. In other words, given a particular state of the resolved scale variables, the most likely forcing from the sub-grid scale…

In this article, we introduce a system of stochastic differential equations (SDEs) consisting of time-dependent covariates and consider both fixed and random effects set-ups. We also allow the functional part associated with the drift…