通过高级初始化策略优化低秩适配

参数高效微调方法的快速发展显著 improve the efficiency of adapting large language models. 在这些方法中，LoRA 因其 strong balance of effectiveness and parameter efficiency 而获得广泛流行。然而，LoRA relies on initializing two low-rank matrices whose product is zero, which limits its ability to effectively activate and leverage the original model weights-creating a potential bottleneck for optimal performance. 为 address this limitation, 我们提出 IniLoRA,一种 novel initialization strategy that initializes the low-rank matrices to closely approximate the original model weights. 实验结果 indicate that IniLoRA achieves better performance than LoRA across a range of models and tasks. 此外，我们引入 two variants, IniLoRA-α and IniLoRA-β, both leveraging distinct initialization methods to enhance performance further.