IHT dies hard: Provable accelerated Iterative Hard Thresholding
Optimization and Control
2019-09-17 v2 Data Structures and Algorithms
Machine Learning
Numerical Analysis
Machine Learning
Abstract
We study --both in theory and practice-- the use of momentum motions in classic iterative hard thresholding (IHT) methods. By simply modifying plain IHT, we investigate its convergence behavior on convex optimization criteria with non-convex constraints, under standard assumptions. In diverse scenaria, we observe that acceleration in IHT leads to significant improvements, compared to state of the art projected gradient descent and Frank-Wolfe variants. As a byproduct of our inspection, we study the impact of selecting the momentum parameter: similar to convex settings, two modes of behavior are observed --"rippling" and linear-- depending on the level of momentum.
Cite
@article{arxiv.1712.09379,
title = {IHT dies hard: Provable accelerated Iterative Hard Thresholding},
author = {Rajiv Khanna and Anastasios Kyrillidis},
journal= {arXiv preprint arXiv:1712.09379},
year = {2019}
}
Comments
accepted to AISTATS 2018