We introduce Context Tuning, a simple and effective method to significantly enhance few-shot adaptation of language models (LLMs) without fine-tuning model parameters. While prompt-based adaptation techniques have demonstrated the effectiveness of lightweight adaptation methods for LLMs, they typically initialize a trainable prompt or prefix with irrelevant tokens for the task at hand. In contrast, Context Tuning initializes the trainable prompt or prefix with task-specific demonstration examples, leveraging the model's inherent In-Context Learning (ICL) ability to extract relevant information for improved few-shot learning performance. Extensive evaluations on benchmarks such as CrossFit, UnifiedQA, MMLU, BIG-Bench Hard, and ARC demonstrate that Context Tuning outperforms traditional prompt-based adaptation methods and achieves competitive accuracy to Test-Time Training with significantly higher training efficiency.
@article{arxiv.2507.04221,
title = {Context Tuning for In-Context Optimization},
author = {Jack Lu and Ryan Teehan and Zhenbang Yang and Mengye Ren},
journal= {arXiv preprint arXiv:2507.04221},
year = {2025}
}
Comments
A short version of this paper was accepted at ICML 2025 Workshop on Test-Time Adaptation