Recent agentic workflows automate professional document generation but focus narrowly on textual quality, overlooking structural and stylistic professionalism, which is equally critical for readability. This gap stems mainly from a lack of effective reward models capable of guiding agents toward producing documents with high structural and stylistic professionalism. We introduce DocReward, a document reward model that evaluates documents based on their structure and style. To achieve this, we propose a textual-quality-agnostic framework that ensures assessments are not confounded by content quality, and construct DocPair, a dataset of 117K paired documents covering 32 domains and 267 types. Each pair shares identical content but differs in structural and stylistic professionalism. DocReward is trained using the Bradley-Terry loss. On a manually annotated benchmark, DocReward outperforms GPT-5 by 14.6 percentage points in the same setting. Reinforcement learning experiments further show that DocReward effectively guides agents toward generating documents with consistently higher structural and stylistic professionalism, highlighting its practical utility.
@article{arxiv.2510.11391,
title = {DocReward: A Document Reward Model for Structuring and Stylizing},
author = {Junpeng Liu and Yuzhong Zhao and Bowen Cao and Jiayu Ding and Yilin Jia and Tengchao Lv and Yupan Huang and Wenshan Wu and Shaohan Huang and Nan Yang and Li Dong and Lei Cui and Tao Ge and Xun Wang and Huitian Jiao and Sun Mao and FNU Kartik and Si-Qing Chen and Wai Lam and Furu Wei},
journal= {arXiv preprint arXiv:2510.11391},
year = {2026}
}