Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

Zihan Qiu; Zeyu Huang; Youcheng Huang; Jie Fu

Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

Computation and Language 2024-02-20 v1

Authors: Zihan Qiu , Zeyu Huang , Youcheng Huang , Jie Fu

Abstract

The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empirical ablation study on updating keys (the 1st layer in the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those two methods in various knowledge editing and fine-tuning tasks of large language models to draw insights to understand FFNs further. Code is available at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$ .

Keywords

transformers feed-forward networks key-value memories knowledge editing fine-tuning large language models empirical study

Cite

@article{arxiv.2402.12233,
  title  = {Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers},
  author = {Zihan Qiu and Zeyu Huang and Youcheng Huang and Jie Fu},
  journal= {arXiv preprint arXiv:2402.12233},
  year   = {2024}
}

Comments

Accepted to Tiny Paper @ ICLR 2024. Codes available at this $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$

Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

Abstract

Keywords

Cite

Comments

Related papers