SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems
Abstract
This paper introduces a novel optimization algorithm designed for nonlinear least-squares problems. The method is derived by preconditioning the gradient descent direction using the Singular Value Decomposition (SVD) of the Jacobian. This SVD-based preconditioner is then integrated with the first- and second-moment adaptive learning rate mechanism of the Adam optimizer. We establish the local linear convergence of the proposed method under standard regularity assumptions and prove global convergence for a modified version of the algorithm under suitable conditions. The effectiveness of the approach is demonstrated experimentally across a range of tasks, including function approximation, partial differential equation (PDE) solving, and image classification on the CIFAR-10 dataset. Results show that the proposed method consistently outperforms standard Adam, achieving faster convergence and lower error in both regression and classification settings.
Cite
@article{arxiv.2602.09057,
title = {SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems},
author = {Zhipeng Chang and Wenrui Hao and Nian Liu},
journal= {arXiv preprint arXiv:2602.09057},
year = {2026}
}