Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords
Abstract
We present a method for exploring regions around individual points in a contextualized vector space (particularly, BERT space), as a way to investigate how these regions correspond to word senses. By inducing a contextualized "pseudoword" as a stand-in for a static embedding in the input layer, and then performing masked prediction of a word in the sentence, we are able to investigate the geometry of the BERT-space in a controlled manner around individual instances. Using our method on a set of carefully constructed sentences targeting ambiguous English words, we find substantial regularity in the contextualized space, with regions that correspond to distinct word senses; but between these regions there are occasionally "sense voids" -- regions that do not correspond to any intelligible sense.
Keywords
Cite
@article{arxiv.2109.11491,
title = {Putting Words in BERT's Mouth: Navigating Contextualized Vector Spaces with Pseudowords},
author = {Taelin Karidi and Yichu Zhou and Nathan Schneider and Omri Abend and Vivek Srikumar},
journal= {arXiv preprint arXiv:2109.11491},
year = {2021}
}
Comments
EMNLP 2021 camera-ready version