AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates
Abstract
In the training of large language models (LLMs), updating parameters more efficiently and stably has always been an important challenge. To achieve efficient parameter updates, existing methods usually achieve performance comparable to full parameter updates through methods such as low-dimensional decomposition or layer-wise selective updates. In this work, we propose AlphaAdam, an optimization framework for LLM from the perspective of intra-layer parameter updates. By decoupling parameter updates and dynamically adjusting their strength, AlphaAdam accelerates convergence and improves training stability. We construct parameter masks based on the consistency of historical momentum and gradient direction and combine them with an adaptive mask strength strategy to ensure efficient optimization and theoretical convergence guarantees, which is also applicable to most momentum-based optimizers. Extensive experiments show that AlphaAdam outperforms state-of-the-art methods such as AdamW in terms of convergence speed and computational efficiency across tasks, including GPT-2 pre-trained and fine-tuned RoBERTa and Llama-7B. Our AlphaAdam implements an optimizer enhancement framework for LLMs through intra-layer asynchronous masked adaptive updates. Our code is available in this https://github.com/MaeChd/AlphaAdam.
Cite
@article{arxiv.2501.18094,
title = {AlphaAdam:Asynchronous Masked Optimization with Dynamic Alpha for Selective Updates},
author = {Da Chang and Yu Li and Ganzhao Yuan},
journal= {arXiv preprint arXiv:2501.18094},
year = {2025}
}
Comments
Theorem 3.5 has issues of insufficient rigor. The content "Let $E[g_i^2] = \sigma_i^2$ ... $E[g_im_{t-1,i}] = \rho_i \sigma_i^2$ be the correlation between gradients and historical momentum ...." is a non-standard assumption and may mislead readers. In the spirit of rigor and responsibility, we temporarily withdraw this version of the content