Fine tuning has been regarded as a de facto approach for adapting large language models (LLMs) to downstream tasks, but the high training memory consumption inherited from LLMs makes this process inefficient. Among existing memory efficient approaches, activation-related optimization has proven particularly effective, as activations consistently dominate overall memory consumption. Although prior arts offer various activation optimization strategies, their data-agnostic nature ultimately results in ineffective and unstable fine tuning. In this paper, we propose TokenSeek, a universal plugin solution for various transformer-based models through instance-aware token seeking and ditching, achieving significant fine-tuning memory savings (e.g., requiring only 14.8% of the memory on Llama3.2 1B) with on-par or even better performance. Furthermore, our interpretable token seeking process reveals the underlying reasons for its effectiveness, offering valuable insights for future research on token efficiency. Homepage: https://runjia.tech/iclr_tokenseek/
@article{arxiv.2601.19739,
title = {TokenSeek: Memory Efficient Fine Tuning via Instance-Aware Token Ditching},
author = {Runjia Zeng and Qifan Wang and Qiang Guan and Ruixiang Tang and Lifu Huang and Zhenting Wang and Xueling Zhang and Cheng Han and Dongfang Liu},
journal= {arXiv preprint arXiv:2601.19739},
year = {2026}
}