Multiple Word Embeddings for Increased Diversity of Representation
Abstract
Most state-of-the-art models in natural language processing (NLP) are neural models built on top of large, pre-trained, contextual language models that generate representations of words in context and are fine-tuned for the task at hand. The improvements afforded by these "contextual embeddings" come with a high computational cost. In this work, we explore a simple technique that substantially and consistently improves performance over a strong baseline with negligible increase in run time. We concatenate multiple pre-trained embeddings to strengthen our representation of words. We show that this concatenation technique works across many tasks, datasets, and model types. We analyze aspects of pre-trained embedding similarity and vocabulary coverage and find that the representational diversity between different pre-trained embeddings is the driving force of why this technique works. We provide open source implementations of our models in both TensorFlow and PyTorch.
Cite
@article{arxiv.2009.14394,
title = {Multiple Word Embeddings for Increased Diversity of Representation},
author = {Brian Lester and Daniel Pressel and Amy Hemmeter and Sagnik Ray Choudhury and Srinivas Bangalore},
journal= {arXiv preprint arXiv:2009.14394},
year = {2020}
}
Comments
arXiv admin note: text overlap with arXiv:2001.01167