Visual-language models (VLMs) like CLIP exhibit strong generalization but struggle with distribution shifts at test time. Existing training-free test-time adaptation (TTA) methods operate strictly within CLIP's original feature space, relying on high-confidence samples while overlooking the potential of low-confidence ones. We propose MS-TTA, a training-free approach that enhances feature representations beyond CLIP's space using a single-step k-nearest neighbors (kNN) Mean-Shift. By refining all test samples, MS-TTA improves feature compactness and class separability, leading to more stable adaptation. Additionally, a cache of refined embeddings further enhances inference by providing Mean Shift enhanced logits. Extensive evaluations on OOD and cross-dataset benchmarks demonstrate that MS-TTA consistently outperforms state-of-the-art training-free TTA methods, achieving robust adaptation without requiring additional training.
@article{arxiv.2507.00462,
title = {Unleashing the Potential of All Test Samples: Mean-Shift Guided Test-Time Adaptation},
author = {Jizhou Han and Chenhao Ding and SongLin Dong and Yuhang He and Xinyuan Gao and Yihong Gong},
journal= {arXiv preprint arXiv:2507.00462},
year = {2026}
}
Comments
Accepted by IEEE TCSVT. This is the author's version which has not been fully edited and content may change prior to final publication