Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models
Abstract
Large language models (LLMs) encode vast amounts of knowledge during pre-training (parametric knowledge, or PK) and can further be enhanced by incorporating contextual knowledge (CK). Can LLMs effectively integrate their internal PK with external CK to solve complex problems? In this paper, we investigate the dynamic interaction between PK and CK, categorizing their relationships into four types: Supportive, Complementary, Conflicting, and Irrelevant. To support this investigation, we introduce ECHOQA, a benchmark spanning scientific, factual, and commonsense knowledge. Our results show that LLMs tend to suppress their PK when contextual information is available, even when it is complementary or irrelevant. While tailored instructions can encourage LLMs to rely more on their PK, they still struggle to fully leverage it. These findings reveal a key vulnerability in LLMs, raising concerns about their reliability in knowledge-intensive tasks. Resources are available at https://github.com/sitaocheng/Knowledge_Interplay
Cite
@article{arxiv.2410.08414,
title = {Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models},
author = {Sitao Cheng and Liangming Pan and Xunjian Yin and Xinyi Wang and William Yang Wang},
journal= {arXiv preprint arXiv:2410.08414},
year = {2024}
}
Comments
27 pages, 8 figures and 17 tables