Related papers: Stochastic Anderson Mixing for Nonconvex Stochasti…

The MIXANDMIX (mixtures by Anderson mixing) tool for the computation of the empirical spectral distribution of random matrices generated by mixtures of populations is described. Within the population mixture model the mapping between the…

Stochastic nonconvex optimization problems with nonlinear constraints have a broad range of applications in intelligent transportation, cyber-security, and smart grids. In this paper, first, we propose an inexact-proximal accelerated…

We show how one can obtain nonaccelerated randomized coordinate descent method (Yu. Nesterov, 2010) and nonaccelerated method of randomization of sum-type functional (Le Roux-Schmidt-Bach, 2012) from the optimal method for the stochastic…

Improved EM strategies, based on the idea of efficient data augmentation (Meng and van Dyk 1997, 1998), are presented for ML estimation of mixture proportions. The resulting algorithms inherit the simplicity, ease of implementation, and…

Alternating Direction Method of Multipliers (ADMM) is a popular convex optimization algorithm, which can be employed for solving distributed consensus optimization problems. In this setting agents locally estimate the optimal solution of an…

We consider the optimization problem of minimizing the sum-of-nonconvex function, i.e., a convex function that is the average of nonconvex components. The existing stochastic algorithms for such a problem only focus on a single machine and…

Although federated learning has achieved many breakthroughs recently, the heterogeneous nature of the learning environment greatly limits its performance and hinders its real-world applications. The heterogeneous data, time-varying wireless…

In this work, we offer a theoretical analysis of two modern optimization techniques for training large and complex models: (i) adaptive optimization algorithms, such as Adam, and (ii) the model exponential moving average (EMA).…

We consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named…

Two new stochastic variance-reduced algorithms named SARAH and SPIDER have been recently proposed, and SPIDER has been shown to achieve a near-optimal gradient oracle complexity for nonconvex optimization. However, the theoretical advantage…

We propose a new stochastic first-order algorithmic framework to solve stochastic composite nonconvex optimization problems that covers both finite-sum and expectation settings. Our algorithms rely on the SARAH estimator introduced in…

In this work we consider the stochastic minimization of nonsmooth convex loss functions, a central problem in machine learning. We propose a novel algorithm called Accelerated Nonsmooth Stochastic Gradient Descent (ANSGD), which exploits…

We propose a stochastic variance reduced optimization algorithm for solving sparse learning problems with cardinality constraints. Sufficient conditions are provided, under which the proposed algorithm enjoys strong linear convergence…

This paper proposes a provably convergent multiblock ADMM for nonconvex optimization with nonlinear dynamics constraints, overcoming the divergence issue in classical extensions. We consider a class of optimization problems that arise from…

For solving pseudo-convex global optimization problems, we present a novel fully adaptive steepest descent method (or ASDM) without any hard-to-estimate parameters. For the step-size regulation in an $\varepsilon$-normalized direction, we…

Adam is a commonly used stochastic optimization algorithm in machine learning. However, its convergence is still not fully understood, especially in the non-convex setting. This paper focuses on exploring hyperparameter settings for the…

In this paper, we propose an acceleration framework for a class of iterative methods using the Reduced Order Method (ROM). Assuming that the underlying iterative scheme generates a rich basis for the solution space, we construct the next…

In this paper we consider the neural network optimization. We develop Anderson-type acceleration method for the stochastic gradient decent method and it improves the network permanence very much. We demonstrate the applicability of the…

In this paper, we propose a successive pseudo-convex approximation algorithm to efficiently compute stationary points for a large class of possibly nonconvex optimization problems. The stationary points are obtained by solving a sequence of…

Many modern computer vision and machine learning applications rely on solving difficult optimization problems that involve non-differentiable objective functions and constraints. The alternating direction method of multipliers (ADMM) is a…