Related papers: Truncated Stochastic Approximation with Moving Bou…
Stochastic equations play an important role in computational science, due to their ability to treat a wide variety of complex statistical problems. However, current algorithms are strongly limited by their sampling variance, which scales…
Gradient clipping is a popular modification to standard (stochastic) gradient descent, at every iteration limiting the gradient norm to a certain value $c >0$. It is widely used for example for stabilizing the training of deep learning…
In this paper we study approximations for boundary crossing probabilities for the moving sums of i.i.d. normal random variables. We propose approximating a discrete time problem with a continuous time problem allowing us to apply developed…
In this paper we show an alternative approach to the concentration of truncated variation for stochastic processes on a real line. Our method is based on the moments control and can be used to generalize the results to the case of processes…
Stochastic Gradient Descent (SGD) methods see many uses in optimization problems. Modifications to the algorithm, such as momentum-based SGD methods have been known to produce better results in certain cases. Much of this, however, is due…
We introduce a statistical physics inspired supervised machine learning algorithm for classification and regression problems. The method is based on the invariances or stability of predicted results when known data is represented as…
The Alternating Direction Method of Multipliers (ADMM) has been studied for years. The traditional ADMM algorithm needs to compute, at each iteration, an (empirical) expected loss function on all training examples, resulting in a…
Variational inequalities have gained significant attention in machine learning and optimization research. While stochastic methods for solving these problems typically assume independent data sampling, we investigate an alternative approach…
A finite dimensional abstract approximation and convergence theory is developed for estimation of the distribution of random parameters in infinite dimensional discrete time linear systems with dynamics described by regularly dissipative…
In this article we establish exponential moment bounds, moment bounds in fractional order smoothness spaces, a uniform H\"older continuity in time, and strong convergence rates for a class of fully discrete exponential Euler-type numerical…
We consider the problem of approximating the moment generating function (MGF) of a truncated random variable in terms of the MGF of the underlying (i.e., untruncated) random variable. The purpose of approximating the MGF is to enable the…
Variance reduction is a family of powerful mechanisms for stochastic optimization that appears to be helpful in many machine learning tasks. It is based on estimating the exact gradient with some recursive sequences. Previously, many papers…
We study stochastic optimization of nonconvex loss functions, which are typical objectives for training neural networks. We propose stochastic approximation algorithms which optimize a series of regularized, nonlinearized losses on large…
A succesful method to describe the asymptotic behavior of a discrete time stochastic process governed by some recursive formula is to relate it to the limit sets of a well chosen mean differential equation. Under an attainability condition,…
Stochastic gradient descent is one of the most successful approaches for solving large-scale problems, especially in machine learning and statistics. At each iteration, it employs an unbiased estimator of the full gradient computed from one…
It is often the case in Statistics that one needs to compute sums of infinite series, especially in marginalising over discrete latent variables. This has become more relevant with the popularization of gradient-based techniques (e.g.…
We analyze the behavior of stochastic approximation algorithms where iterates, in expectation, progress towards an objective at each step. When progress is proportional to the step size of the algorithm, we prove exponential concentration…
We consider the problem of minimizing the sum of two convex functions: one is the average of a large number of smooth component functions, and the other is a general convex function that admits a simple proximal mapping. We assume the whole…
In this paper, a sample-based procedure for obtaining simple and computable approximations of chance-constrained sets is proposed. The procedure allows to control the complexity of the approximating set, by defining families of…
We present a stochastic setting for optimization problems with nonsmooth convex separable objective functions over linear equality constraints. To solve such problems, we propose a stochastic Alternating Direction Method of Multipliers…