Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle
Abstract
Recent innovations in language model training have demonstrated that it is possible to create highly performant models that are small enough to run on a smartphone. As these models are deployed in an increasing number of domains, it is critical to ensure that they are aligned with human preferences and safety considerations. In this report, we present our methodology for safety aligning the Phi-3 series of language models. We utilized a "break-fix" cycle, performing multiple rounds of dataset curation, safety post-training, benchmarking, red teaming, and vulnerability identification to cover a variety of harm areas in both single and multi-turn scenarios. Our results indicate that this approach iteratively improved the performance of the Phi-3 models across a wide range of responsible AI benchmarks. Finally, we include additional red teaming strategies and evaluations that were used to test the safety behavior of Phi-3.5-mini and Phi-3.5-MoE, which were optimized for multilingual capabilities.
Keywords
Cite
@article{arxiv.2407.13833,
title = {Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle},
author = {Emman Haider and Daniel Perez-Becker and Thomas Portet and Piyush Madan and Amit Garg and Atabak Ashfaq and David Majercak and Wen Wen and Dongwoo Kim and Ziyi Yang and Jianwen Zhang and Hiteshi Sharma and Blake Bullwinkel and Martin Pouliot and Amanda Minnich and Shiven Chawla and Solianna Herrera and Shahed Warreth and Maggie Engler and Gary Lopez and Nina Chikanov and Raja Sekhar Rao Dheekonda and Bolor-Erdene Jagdagdorj and Roman Lutz and Richard Lundeen and Tori Westerhoff and Pete Bryan and Christian Seifert and Ram Shankar Siva Kumar and Andrew Berkley and Alex Kessler},
journal= {arXiv preprint arXiv:2407.13833},
year = {2024}
}