English

AdamD: Improved bias-correction in Adam

Machine Learning 2021-10-25 v2

Abstract

Here I present a small update to the bias-correction term in the Adam optimizer that has the advantage of making smaller gradient updates in the first several steps of training. With the default bias-correction, Adam may actually make larger than requested gradient updates early in training. By only including the well-justified bias-correction of the second moment gradient estimate, vtv_t, and excluding the bias-correction on the first-order estimate, mtm_t, we attain these more desirable gradient update properties in the first series of steps. The default implementation of Adam may be as sensitive as it is to the hyperparameters β1,β2\beta_1, \beta_2 partially due to the originally proposed bias correction procedure, and its behavior in early steps.

Keywords

Cite

@article{arxiv.2110.10828,
  title  = {AdamD: Improved bias-correction in Adam},
  author = {John St John},
  journal= {arXiv preprint arXiv:2110.10828},
  year   = {2021}
}

Comments

8 pages, 1 figure

R2 v1 2026-06-24T07:03:30.532Z