像大金鱼一样，不要记忆！缓解生成式大语言模型中的记忆问题
计算与语言
2024-11-05 v2
摘要
大型语言模型可以记忆并重复其训练数据，导致隐私和版权风险。为缓解记忆问题，我们引入一种对下一标记训练目标的微妙修改，称为金鱼损失。在训练期间，排除随机抽样的子集标记。这些被丢弃的标记未被模型记忆，从而防止完整标记链的逐字重现。我们对训练规模为十亿级的 Llama-2 模型进行了大量实验，包括预训练和从头训练，显著减少了可提取的记忆，同时对下游基准几乎没有影响。
引用
@article{arxiv.2406.10209,
title = {Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs},
author = {Abhimanyu Hans and Yuxin Wen and Neel Jain and John Kirchenbauer and Hamid Kazemi and Prajwal Singhania and Siddharth Singh and Gowthami Somepalli and Jonas Geiping and Abhinav Bhatele and Tom Goldstein},
journal= {arXiv preprint arXiv:2406.10209},
year = {2024}
}
备注
10 pages, 8 figures, and 1 table in the main body. Code available at https://github.com/ahans30/goldfish-loss and checkpoints at https://huggingface.co/collections/tomg-group-umd/goldfish-loss-mitigating-memorization-in-llms-66c175becb6aab07744f7272