Linear Gradient Prediction with Control Variates
Machine Learning
2025-11-10 v1 Machine Learning
Abstract
We propose a new way of training neural networks, with the goal of reducing training cost. Our method uses approximate predicted gradients instead of the full gradients that require an expensive backward pass. We derive a control-variate-based technique that ensures our updates are unbiased estimates of the true gradient. Moreover, we propose a novel way to derive a predictor for the gradient inspired by the theory of the Neural Tangent Kernel. We empirically show the efficacy of the technique on a vision transformer classification task.
Cite
@article{arxiv.2511.05187,
title = {Linear Gradient Prediction with Control Variates},
author = {Kamil Ciosek and Nicolò Felicioni and Juan Elenter Litwin},
journal= {arXiv preprint arXiv:2511.05187},
year = {2025}
}