Reinforcement Learning with Verifiable Rewards (RLVR) has improved the reasoning abilities of large language models (LLMs) on mathematics and programming tasks, but standard approaches that optimize single-attempt accuracy can inadvertently suppress response diversity across repeated attempts, narrowing exploration and overlooking underrepresented strategies. We introduce UpSkill, a training time method that adapts Mutual Information Skill Learning (MISL) to LLMs for optimizing pass@k correctness. We propose a novel reward that we implement within Group Relative Policy Optimization (GRPO): a token-level mutual information (MI) reward that encourages trajectory specificity to z. Experiments on GSM8K with three open-weight models, Llama 3.1-8B, Qwen 2.5-7B, and R1-Distilled-Qwen2.5-Math-1.5B, show that UpSkill improves multi-attempt metrics on the stronger base models, yielding mean gains of ~3% in pass@k for both Qwen and Llama without degrading pass@1. Additionally, we find both empirical and theoretical evidence that improvements in pass@k are closely tied to the mutual information objective.
@article{arxiv.2602.22296,
title = {UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMs},
author = {Devan Shah and Owen Yang and Daniel Yang and Chongyi Zheng and Benjamin Eysenbach},
journal= {arXiv preprint arXiv:2602.22296},
year = {2026}
}
Comments
First two authors equal contribution. 29 pages total (11 pages main text), 10 figures, 10 tables. Project website: https://dshah.io/upskill/