Related papers: Preconditioning via Diagonal Scaling
The alternating direction multiplier method (ADMM) is widely used in computer graphics for solving optimization problems that can be nonsmooth and nonconvex. It converges quickly to an approximate solution, but can take a long time to…
Stochastic collocation methods for approximating the solution of partial differential equations with random input data (e.g., coefficients and forcing terms) suffer from the curse of dimensionality whereby increases in the stochastic…
Statistical preconditioning enables fast methods for distributed large-scale empirical risk minimization problems. In this approach, multiple worker nodes compute gradients in parallel, which are then used by the central node to update the…
The Dynamical Graph Grammar (DGG) formalism can describe complex system dynamics with graphs that are mapped into a master equation. An exact stochastic simulation algorithm may be used, but it is slow for large systems. To overcome this…
Domain decomposition (DD) methods are a natural way to take advantage of parallel computers when solving large scale linear systems. Their scalability depends on the design of the coarse space used in the two-level method. The analysis of…
Electron cryomicroscopy (cryo-EM) is an imaging technique widely used in structural biology to determine the three-dimensional structure of biological molecules from noisy two-dimensional projections with unknown orientations. As the…
Layer-wise preconditioning methods are a family of memory-efficient optimization algorithms that introduce preconditioners per axis of each layer's weight tensors. These methods have seen a recent resurgence, demonstrating impressive…
The proximal gradient method is a generic technique introduced to tackle the non-smoothness in optimization problems, wherein the objective function is expressed as the sum of a differentiable convex part and a non-differentiable…
Establishing a fast rate of convergence for optimization methods is crucial to their applicability in practice. With the increasing popularity of deep learning over the past decade, stochastic gradient descent and its adaptive variants…
We propose a parallel adaptive constraint-tightening approach to solve a linear model predictive control problem for discrete-time systems, based on inexact numerical optimization algorithms and operator splitting methods. The underlying…
We adapt the Gradient Discretisation Method (GDM), originally designed for elliptic and parabolic partial differential equations, to the case of a linear scalar hyperbolic equations. This enables the simultaneous design and convergence…
The paper focuses on numerical solution of parametrized diffusion equations with scalar parameter-dependent coefficient function by the stochastic (spectral) Galerkin method. We study preconditioning of the related discretized problems…
In this paper, we propose a stochastic optimization method that adaptively controls the sample size used in the computation of gradient approximations. Unlike other variance reduction techniques that either require additional storage or the…
Stochastic Gradient Decent (SGD) is one of the core techniques behind the success of deep neural networks. The gradient provides information on the direction in which a function has the steepest rate of change. The main problem with basic…
One of the most widely used methods for solving large-scale stochastic optimization problems is distributed asynchronous stochastic gradient descent (DASGD), a family of algorithms that result from parallelizing stochastic gradient descent…
In this work, we develop proximal preconditioned gradient methods with a focus on spectral gradient methods providing a proximal extension to the Muon and Scion optimizers. We introduce a family of stochastic algorithms that can handle a…
Motivated by applications of large embedding models, we study differentially private (DP) optimization problems under sparsity of individual gradients. We start with new near-optimal bounds for the classic mean estimation problem but with…
Alternating Direction Method of Multipliers (ADMM) has recently been proposed as a potential alternative optimizer to the Stochastic Gradient Descent(SGD) for deep learning problems. This is because ADMM can solve gradient vanishing and…
In this paper, we propose and analyze an efficient preconditioning method for the elliptic problem based on the reconstructed discontinuous approximation method. We reconstruct a high-order piecewise polynomial space that arbitrary order…
In this article we construct and analyze multigrid preconditioners for discretizations of operators of the form D+K* K, where D is the multiplication with a relatively smooth positive function and K is a compact linear operator. These…