LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization
Abstract
Low-rank adaption (LoRA) is a widely used parameter-efficient finetuning method for LLM that reduces memory requirements. However, current LoRA optimizers lack transformation invariance, meaning the actual updates to the weights depends on how the two LoRA factors are scaled or rotated. This deficiency leads to inefficient learning and sub-optimal solutions in practice. This paper introduces LoRA-RITE, a novel adaptive matrix preconditioning method for LoRA optimization, which can achieve transformation invariance and remain computationally efficient. We provide theoretical analysis to demonstrate the benefit of our method and conduct experiments on various LLM tasks with different models including Gemma 2B, 7B, and mT5-XXL. The results demonstrate consistent improvements against existing optimizers. For example, replacing Adam with LoRA-RITE during LoRA fine-tuning of Gemma-2B yielded 4.6\% accuracy gain on Super-Natural Instructions and 3.5\% accuracy gain across other four LLM benchmarks (HellaSwag, ArcChallenge, GSM8K, OpenBookQA).
Cite
@article{arxiv.2410.20625,
title = {LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization},
author = {Jui-Nan Yen and Si Si and Zhao Meng and Felix Yu and Sai Surya Duvvuri and Inderjit S. Dhillon and Cho-Jui Hsieh and Sanjiv Kumar},
journal= {arXiv preprint arXiv:2410.20625},
year = {2025}
}
Comments
Published as an oral paper at ICLR 2025. The code for our project is available at https://github.com/gkevinyen5418/LoRA-RITE