On the Robustness Tradeoff in Fine-Tuning
Abstract
Fine-tuning has become the standard practice for adapting pre-trained models to downstream tasks. However, the impact on model robustness is not well understood. In this work, we characterize the robustness-accuracy trade-off in fine-tuning. We evaluate the robustness and accuracy of fine-tuned models over 6 benchmark datasets and 7 different fine-tuning strategies. We observe a consistent trade-off between adversarial robustness and accuracy. Peripheral updates such as BitFit are more effective for simple tasks -- over 75% above the average measured by the area under the Pareto frontiers on CIFAR-10 and CIFAR-100. In contrast, fine-tuning information-heavy layers, such as attention layers via Compacter, achieves a better Pareto frontier on more complex tasks -- 57.5% and 34.6% above the average on Caltech-256 and CUB-200, respectively. Lastly, we observe that the robustness of fine-tuning against out-of-distribution data closely tracks accuracy. These insights emphasize the need for robustness-aware fine-tuning to ensure reliable real-world deployments.
Cite
@article{arxiv.2503.14836,
title = {On the Robustness Tradeoff in Fine-Tuning},
author = {Kunyang Li and Jean-Charles Noirot Ferrand and Ryan Sheatsley and Blaine Hoak and Yohan Beugin and Eric Pauley and Patrick McDaniel},
journal= {arXiv preprint arXiv:2503.14836},
year = {2025}
}
Comments
Accepted to International Conference on Computer Vision, ICCV 2025