Related papers: Asynchronous Richardson iterations
We study asymptotic synchronization at the level of global attractors in a class of coupled second order in time models which arises in dissipative wave and elastic structure dynamics. Under some conditions we prove that this…
Several works have shown linear speedup is achieved by an asynchronous parallel implementation of stochastic coordinate descent so long as there is not too much parallelism. More specifically, it is known that if all updates are of similar…
Modern large-scale statistical models require to estimate thousands to millions of parameters. This is often accomplished by iterative algorithms such as gradient descent, projected gradient descent or their accelerated versions. What are…
A second order explicit one-step numerical method for the initial value problem of the general ordinary differential equation is proposed. It is obtained by natural modifications of the well-known leapfrog method, which is a second order,…
This study concerns numerical methods for efficiently solving the Richards equation where different weak formulations and computational techniques are analyzed. The spatial discretizations are based on standard or mixed finite element…
Two-time-scale stochastic approximation, a generalized version of the popular stochastic approximation, has found broad applications in many areas including stochastic control, optimization, and machine learning. Despite its popularity,…
We present a novel class of methods to compute functions of matrices or their action on vectors that are suitable for parallel programming. Solving appropriate simple linear systems of equations in parallel (or computing the inverse of…
We propose a first-order method for convex optimization, where instead of being restricted to the gradient from a single parameter, gradients from multiple parameters can be used during each step of gradient descent. This setup is…
Asynchronous methods are widely used in deep learning, but have limited theoretical justification when applied to non-convex problems. We show that running stochastic gradient descent (SGD) in an asynchronous manner can be viewed as adding…
Modern distributed optimization methods mostly rely on traditional synchronous approaches, despite substantial recent progress in asynchronous optimization. We revisit Synchronous SGD and its robust variant, called $m$-Synchronous SGD, and…
It was recently established that for convex optimization problems with sparse optimal solutions (be it entry-wise sparsity or matrix rank-wise sparsity) it is possible to design first-order methods with linear convergence rates that depend…
This paper shows a simple parameter substitution, which makes use of the reciprocal relation of typical objective functions with typical random parameters. Thereby, the accuracy of first-order probabilistic analysis improves significantly…
In this article, we establish a class of new accelerated modulus-based iteration methods for solving the linear complementarity problem. When the system matrix is an $H_+$-matrix, we present appropriate criteria for the convergence…
The choice of the parameter value for regularized inverse problems is critical to the results and remains a topic of interest. This article explores a criterion for selecting a good parameter value by maximizing the probability of the data,…
Quasi-Newton methods refer to a class of algorithms at the interface between first and second order methods. They aim to progress as substantially as second order methods per iteration, while maintaining the computational complexity of…
We present two types of meta-algorithm that can greatly improve the accuracy of existing algorithms for integrating the equations of motion of dynamical systems. The first meta-algorithm takes an integrator that is time-symmetric only for…
Revisionist integral deferred correction (RIDC) methods are a family of parallel--in--time methods to solve systems of initial values problems. The approach is able to bootstrap lower order time integrators to provide high order…
The alternating direction method of multipliers (ADMM) has emerged as a powerful technique for large-scale structured optimization. Despite many recent results on the convergence properties of ADMM, a quantitative characterization of the…
Numerical solutions to the Eikonal equation are computed using variants of the fast marching method, the fast sweeping method, and the fast iterative method. In this paper, we provide a unified view of these algorithms that highlights their…
This paper provides examples of various synchronous and asynchronous signal processing systems for performing optimization, utilizing the framework and elements developed in a preceding paper. The general strategy in that paper was to…