An Interpretability Illusion for BERT
Computation and Language
2021-04-16 v1 Machine Learning
Abstract
We describe an "interpretability illusion" that arises when analyzing the BERT model. Activations of individual neurons in the network may spuriously appear to encode a single, simple concept, when in fact they are encoding something far more complex. The same effect holds for linear combinations of activations. We trace the source of this illusion to geometric properties of BERT's embedding space as well as the fact that common text corpora represent only narrow slices of possible English sentences. We provide a taxonomy of model-learned concepts and discuss methodological implications for interpretability research, especially the importance of testing hypotheses on multiple data sets.
Cite
@article{arxiv.2104.07143,
title = {An Interpretability Illusion for BERT},
author = {Tolga Bolukbasi and Adam Pearce and Ann Yuan and Andy Coenen and Emily Reif and Fernanda Viégas and Martin Wattenberg},
journal= {arXiv preprint arXiv:2104.07143},
year = {2021}
}