Complex instruction-following with elaborate constraints is imperative for Large Language Models (LLMs). While existing methods have constructed data for complex instruction alignment, they all rely on a more advanced model, especially GPT-4, limiting their application. In this paper, we propose a Multi-granularity Self-Contrastive Training (MuSC) framework, to improve the complex instruction alignment without relying on a stronger model. Our method is conducted on both coarse and fine granularity. On coarse-granularity, we construct constraint-aware preference data based on instruction decomposition and recombination. On fine-granularity, we perform token-aware preference optimization with dynamic token-level supervision. Our method is evaluated on open-sourced models, and experiment results show our method achieves significant improvement on both complex and general instruction-following benchmarks, surpassing previous self-alignment methods.
@article{arxiv.2502.11541,
title = {MuSC: Improving Complex Instruction Following with Multi-granularity Self-Contrastive Training},
author = {Hui Huang and Jiaheng Liu and Yancheng He and Shilong Li and Bing Xu and Conghui Zhu and Muyun Yang and Tiejun Zhao},
journal= {arXiv preprint arXiv:2502.11541},
year = {2025}
}