Related papers: Optimal Shrinkage for Distributed Second-Order Opt…

We consider minimizing a function consisting of a quadratic term and a proximable term which is possibly nonconvex and nonsmooth. This problem is also known as scaled proximal operator. Despite its simple form, existing methods suffer from…

We revisit the asymptotic bias analysis of the distributed Pareto optimization algorithm developed based on the diffusion strategies. We propose an alternative way to analyze the asymptotic bias of this algorithm at small step-sizes and…

Heavy-tailed noise is pervasive in modern machine learning applications, arising from data heterogeneity, outliers, and non-stationary stochastic environments. While second-order methods can significantly accelerate convergence in…

Unconstrained optimization problems become more common in scientific computing and engineering applications with the rapid development of artificial intelligence, and numerical methods for solving them more quickly and efficiently have been…

Motivated by emerging applications in machine learning, we consider an optimization problem in a general form where the gradient of the objective function is available through a biased stochastic oracle. We assume a bias-control parameter…

The emergence of big data has caused a dramatic shift in the operating regime for optimization algorithms. The performance bottleneck, which used to be computations, is now often communications. Several gradient compression techniques have…

This paper studies a distributed stochastic optimization problem over random networks with imperfect communications subject to a global constraint, which is the intersection of local constraint sets assigned to agents. The global cost…

We describe a second-order accurate approach to sparsifying the off-diagonal blocks in the hierarchical approximate factorizations of sparse symmetric positive definite matrices. The norm of the error made by the new approach depends…

In this thesis, we propose new theoretical frameworks for the analysis of stochastic and distributed methods with error compensation and local updates. Using these frameworks, we develop more than 20 new optimization methods, including the…

We consider distributed smooth nonconvex unconstrained optimization over networks, modeled as a connected graph. We examine the behavior of distributed gradient-based algorithms near strict saddle points. Specifically, we establish that (i)…

Shrinkage estimators have profound impacts in statistics and in scientific and engineering applications. In this article, we consider shrinkage estimation in the presence of linear predictors. We formulate two heteroscedastic hierarchical…

We present a derivative-based algorithm for nonlinearly constrained optimization problems that is tolerant of inaccuracies in the data. The algorithm solves a semi-smooth set of nonlinear equations that are equivalent to the first-order…

Recovering a low-rank signal matrix from its noisy observation, commonly known as matrix denoising, is a fundamental inverse problem in statistical signal processing. Matrix denoising methods are generally based on shrinkage or thresholding…

We leverage second-order information for tuning of inverse optimal controllers for a class of discrete-time nonlinear input-affine systems. For this, we select the input penalty matrix, representing a tuning knob, to yield the Hessian of…

Stochastic gradient descent (SGD) has been a go-to algorithm for nonconvex stochastic optimization problems arising in machine learning. Its theory however often requires a strong framework to guarantee convergence properties. We hereby…

We propose a randomized second-order method for optimization known as the Newton Sketch: it is based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian. For self-concordant functions, we prove that…

We tackle covariance estimation in low-sample scenarios, employing a structured covariance matrix with shrinkage methods. These involve convexly combining a low-bias/high-variance empirical estimate with a biased regularization estimator,…

Several approximate inference algorithms have been proposed to minimize an alpha-divergence between an approximating distribution and a target distribution. Many of these algorithms introduce bias, the magnitude of which becomes problematic…

While many distributed optimization algorithms have been proposed for solving smooth or convex problems over the networks, few of them can handle non-convex and non-smooth problems. Based on a proximal primal-dual approach, this paper…

We study a class of zeroth-order distributed optimization problems, where each agent can control a partial vector and observe a local cost that depends on the joint vector of all agents, and the agents can communicate with each other with…