Analysis of Optimization Algorithms via Sum-of-Squares
Abstract
We introduce a new framework for unifying and systematizing the performance analysis of first-order black-box optimization algorithms for unconstrained convex minimization. The low-cost iteration complexity enjoyed by first-order algorithms renders them particularly relevant for applications in machine learning and large-scale data analysis. Relying on sum-of-squares (SOS) optimization, we introduce a hierarchy of semidefinite programs that give increasingly better convergence bounds for higher levels of the hierarchy. Alluding to the power of the SOS hierarchy, we show that the (dual of the) first level corresponds to the Performance Estimation Problem (PEP) introduced by Drori and Teboulle [Math. Program., 145(1):451--482, 2014], a powerful framework for determining convergence rates of first-order optimization algorithms. Consequently, many results obtained within the PEP framework can be reinterpreted as degree-1 SOS proofs, and thus, the SOS framework provides a promising new approach for certifying improved rates of convergence by means of higher-order SOS certificates. To determine analytical rate bounds, in this work we use the first level of the SOS hierarchy and derive new result{s} for noisy gradient descent with inexact line search methods (Armijo, Wolfe, and Goldstein).
Cite
@article{arxiv.1906.04648,
title = {Analysis of Optimization Algorithms via Sum-of-Squares},
author = {Sandra S. Y. Tan and Antonios Varvitsiotis and Vincent Y. F. Tan},
journal= {arXiv preprint arXiv:1906.04648},
year = {2021}
}
Comments
Extended version of a paper presented at the 2019 Signal Processing with Adaptive Sparse Structured Representations (SPARS) workshop; Code for numerically and symbolically verifying the results can be found at https://github.com/sandratsy/SumsOfSquares; v3: added acknowledgments; v4 Accepted to the Journal of Optimization Theory and Applications (JOTA)