The Structure Gap between probabilistic LLM generation and deterministic schema requirements hinders automated workflows. We propose RL-Struct, a lightweight framework using Gradient Regularized Policy Optimization (GRPO) with a hierarchical reward function to align LLMs with structural constraints. This approach eliminates the critic network, reducing peak VRAM by 38% compared to PPO. On complex JSON tasks, RL-Struct achieves 89.7% structural accuracy and 92.1% validity, significantly outperforming SFT and zero-shot baselines. We also report an emergent curriculum--a self-organized learning process where the model prioritizes syntax before semantics. Our model is publicly available at https://huggingface.co/Freakz3z/Qwen-JSON.
@article{arxiv.2512.00319,
title = {RL-Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMs},
author = {Ruike Hu and Shulei Wu},
journal= {arXiv preprint arXiv:2512.00319},
year = {2025}
}
Comments
13 pages, 9 figures. Model is available at https://huggingface.co/Freakz3z/Qwen-JSON