基于稀疏安全反馈的轨迹信用分配 (TraCeS)

在安全强化学习（RL）中，使用辅助安全成本来对齐代理的安全决策。在实际应用中，安全约束（包括成本函数和预算）难以指定，因为这需要预见所有可能的不安全行为。因此，我们 addressed a general setting where the true safety definition is unknown, and has to be learned from sparsely labeled data. Our key contributions are: first, we design a safety model that performs credit assignment to estimate each decision step's impact on the overall safety using a dataset of diverse trajectories and their corresponding binary safety labels (i.e., whether the corresponding trajectory is safe/unsafe). Second, we illustrate the architecture of our safety model to demonstrate its ability to learn a separate safety score for each timestep. Third, we reformulate the safe RL problem using the proposed safety model and derive an effective algorithm to optimize a safe yet rewarding policy. Finally, our empirical results corroborate our findings and show that this approach is effective in satisfying unknown safety definition, and scalable to various continuous control tasks.