Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate
Machine Learning
2026-05-29 v2 Artificial Intelligence
Abstract
Sparse optimization is a fundamental challenge in various practical applications. A popular approach to sparse optimization is regularization. However, it may encounter optimization instability due to the unbounded gradients when . In this paper, we introduce a novel approach to sparse optimization termed ReWA, based on Reparameterization, Weight decay, and Adaptive learning rate. ReWA is closely connected to -regularization, yet it unveils a distinct optimization landscape that helps mitigate instability issues. Experiments on CIFAR-10 and ImageNet with ResNets demonstrate that ReWA leads to significant sparsity improvements over the -regularization approach while preserving test accuracy.
Cite
@article{arxiv.2605.25134,
title = {Theoretical Analysis of Sparse Optimization with Reparameterization, Weight Decay, and Adaptive Learning Rate},
author = {Huangyu Xu and Jingqin Yang and Qianqian Xu and Jiaye Teng},
journal= {arXiv preprint arXiv:2605.25134},
year = {2026}
}
Comments
31 pages, 5 figures. Submitted to ICML 2026