Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence
Abstract
We propose a randomized second-order method for optimization known as the Newton Sketch: it is based on performing an approximate Newton step using a randomly projected or sub-sampled Hessian. For self-concordant functions, we prove that the algorithm has super-linear convergence with exponentially high probability, with convergence and complexity guarantees that are independent of condition numbers and related problem-dependent quantities. Given a suitable initialization, similar guarantees also hold for strongly convex and smooth objectives without self-concordance. When implemented using randomized projections based on a sub-sampled Hadamard basis, the algorithm typically has substantially lower complexity than Newton's method. We also describe extensions of our methods to programs involving convex constraints that are equipped with self-concordant barriers. We discuss and illustrate applications to linear programs, quadratic programs with convex constraints, logistic regression and other generalized linear models, as well as semidefinite programs.
Cite
@article{arxiv.1505.02250,
title = {Newton Sketch: A Linear-time Optimization Algorithm with Linear-Quadratic Convergence},
author = {Mert Pilanci and Martin J. Wainwright},
journal= {arXiv preprint arXiv:1505.02250},
year = {2015}
}