TrustLLM：大型语言模型的可信度
计算与语言
2024-10-01 v6
摘要
以ChatGPT为代表的大型语言模型（LLM）因其出色的自然语言处理能力而受到广泛关注。然而，这些LLM带来了许多挑战，尤其是在可信度方面。因此，确保LLM的可信度成为一个重要课题。本文介绍了TrustLLM，一项关于LLM可信度的全面研究，包括不同可信度维度的原则、已建立的基准、对主流LLM可信度的评估和分析，以及对开放挑战和未来方向的讨论。具体来说，我们首先提出了一套涵盖八个不同维度的可信LLM原则。基于这些原则，我们进一步建立了一个涵盖六个维度的基准，包括真实性、安全性、公平性、鲁棒性、隐私性和机器伦理。然后，我们展示了一项评估TrustLLM中16个主流LLM的研究，涉及超过30个数据集。我们的研究结果首先表明，总体而言，可信度和效用（即功能有效性）呈正相关。其次，我们的观察显示，专有LLM在可信度方面通常优于大多数开源LLM，这引发了对广泛可用的开源LLM潜在风险的担忧。然而，少数开源LLM非常接近专有LLM。第三，值得注意的是，一些LLM可能过度校准以表现出可信度，以至于它们错误地将良性提示视为有害而拒绝响应，从而损害了效用。最后，我们强调不仅要在模型本身中确保透明度，还要在支撑可信度的技术中确保透明度。了解具体采用了哪些可信技术对于分析其有效性至关重要。
引用
@article{arxiv.2401.05561,
title = {TrustLLM: Trustworthiness in Large Language Models},
author = {Yue Huang and Lichao Sun and Haoran Wang and Siyuan Wu and Qihui Zhang and Yuan Li and Chujie Gao and Yixin Huang and Wenhan Lyu and Yixuan Zhang and Xiner Li and Zhengliang Liu and Yixin Liu and Yijue Wang and Zhikun Zhang and Bertie Vidgen and Bhavya Kailkhura and Caiming Xiong and Chaowei Xiao and Chunyuan Li and Eric Xing and Furong Huang and Hao Liu and Heng Ji and Hongyi Wang and Huan Zhang and Huaxiu Yao and Manolis Kellis and Marinka Zitnik and Meng Jiang and Mohit Bansal and James Zou and Jian Pei and Jian Liu and Jianfeng Gao and Jiawei Han and Jieyu Zhao and Jiliang Tang and Jindong Wang and Joaquin Vanschoren and John Mitchell and Kai Shu and Kaidi Xu and Kai-Wei Chang and Lifang He and Lifu Huang and Michael Backes and Neil Zhenqiang Gong and Philip S. Yu and Pin-Yu Chen and Quanquan Gu and Ran Xu and Rex Ying and Shuiwang Ji and Suman Jana and Tianlong Chen and Tianming Liu and Tianyi Zhou and William Wang and Xiang Li and Xiangliang Zhang and Xiao Wang and Xing Xie and Xun Chen and Xuyu Wang and Yan Liu and Yanfang Ye and Yinzhi Cao and Yong Chen and Yue Zhao},
journal= {arXiv preprint arXiv:2401.05561},
year = {2024}
}
备注
This work is still under work and we welcome your contribution