Optimization with First-Order Surrogate Functions
Abstract
In this paper, we study optimization methods consisting of iteratively minimizing surrogates of an objective function. By proposing several algorithmic variants and simple convergence analyses, we make two main contributions. First, we provide a unified viewpoint for several first-order optimization techniques such as accelerated proximal gradient, block coordinate descent, or Frank-Wolfe algorithms. Second, we introduce a new incremental scheme that experimentally matches or outperforms state-of-the-art solvers for large-scale optimization problems typically arising in machine learning.
Cite
@article{arxiv.1305.3120,
title = {Optimization with First-Order Surrogate Functions},
author = {Julien Mairal},
journal= {arXiv preprint arXiv:1305.3120},
year = {2013}
}
Comments
to appear in the proceedings of ICML 2013; the arxiv paper contains the 9 pages main text followed by 26 pages of supplemental material. International Conference on Machine Learning (ICML 2013) (2013)