Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning method for large language models. However, its linear nature limits expressiveness. We propose LoRAN, a non-linear extension of LoRA that applies lightweight transformations to the low-rank updates. We further introduce Sinter, a sine-based activation that adds structured perturbations without increasing parameter count. Experiments across summarization and classification tasks show that LoRAN consistently improves over QLoRA. Ablation studies reveal that Sinter outperforms standard activations such as Sigmoid, ReLU, and Tanh, highlighting the importance of activation design in lowrank tuning.
@article{arxiv.2509.21870,
title = {Enhancing Low-Rank Adaptation with Structured Nonlinear Transformations},
author = {Guanzhi Deng and Mingyang Liu and Dapeng Wu and Yinqiao Li and Linqi Song},
journal= {arXiv preprint arXiv:2509.21870},
year = {2025}
}
Comments
This manuscript has been submitted to IEEE Journal of Selected Topics in Signal Processing (JSTSP) for review. Until the moment I submitted the manuscript to arXiv, we haven't received any review comments from JSTSP