We present Frost Training, a method for improving Monte Carlo-based policy optimization for a large family of LLM-as-a-judge tasks called Cross-Entropy Games. The key idea is to exploit the gradient of the reward function in embedding space. This signal is used in the Greedy Coordinate Gradient (GCG) jailbreaking technique; we demonstrate for the first time that it can also be used to boost model training. We validate our method using GRPO training for maximum-likelihood infilling. Frost Training improves the model's ability to generate high-scoring outputs, reaching higher maximum scores in a best-of-k setting, and does so at an increased speed.
@article{arxiv.2605.27701,
title = {Cross-Entropy Games and Frost Training},
author = {Arthur Renard and Franck Gabriel and Valentin Hartmann and Clément Hongler},
journal= {arXiv preprint arXiv:2605.27701},
year = {2026}
}