Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
Abstract
When training artificial intelligence (AI) to perform tasks, humans often care not only about whether a task is completed but also how it is performed. As AI agents tackle increasingly complex tasks, aligning their behavior with human-provided specifications becomes critical for responsible AI deployment. Reward design provides a direct channel for such alignment by translating human expectations into reward functions that guide reinforcement learning (RL). However, existing methods are often too limited to capture nuanced human preferences that arise in long-horizon tasks. Hence, we introduce Hierarchical Reward Design from Language (HRDL): a problem formulation that extends classical reward design to encode richer behavioral specifications for hierarchical RL agents. We further propose Language to Hierarchical Rewards (L2HR) as a solution to HRDL. Experiments show that AI agents trained with rewards designed via L2HR not only complete tasks effectively but also better adhere to human specifications. Together, HRDL and L2HR advance the research on human-aligned AI agents.
Keywords
Cite
@article{arxiv.2602.18582,
title = {Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications},
author = {Zhiqin Qian and Ryan Diaz and Sangwon Seo and Vaibhav Unhelkar},
journal= {arXiv preprint arXiv:2602.18582},
year = {2026}
}
Comments
Extended version of an identically-titled paper accepted at AAMAS 2026