English

BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge

Computation and Language 2019-10-18 v1

Abstract

We introduce a simple yet effective method of integrating contextual embeddings with commonsense graph embeddings, dubbed BERT Infused Graphs: Matching Over Other embeDdings. First, we introduce a preprocessing method to improve the speed of querying knowledge bases. Then, we develop a method of creating knowledge embeddings from each knowledge base. We introduce a method of aligning tokens between two misaligned tokenization methods. Finally, we contribute a method of contextualizing BERT after combining with knowledge base embeddings. We also show BERTs tendency to correct lower accuracy question types. Our model achieves a higher accuracy than BERT, and we score fifth on the official leaderboard of the shared task and score the highest without any additional language model pretraining.

Keywords

Cite

@article{arxiv.1910.07713,
  title  = {BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge},
  author = {Jeff Da},
  journal= {arXiv preprint arXiv:1910.07713},
  year   = {2019}
}

Comments

Accepted to EMNLP Commonsense (COIN)

R2 v1 2026-06-23T11:46:15.242Z