Learning the Dimensionality of Word Embeddings
Machine Learning
2017-04-14 v3 Computation and Language
Machine Learning
Abstract
We describe a method for learning word embeddings with data-dependent dimensionality. Our Stochastic Dimensionality Skip-Gram (SD-SG) and Stochastic Dimensionality Continuous Bag-of-Words (SD-CBOW) are nonparametric analogs of Mikolov et al.'s (2013) well-known 'word2vec' models. Vector dimensionality is made dynamic by employing techniques used by Cote & Larochelle (2016) to define an RBM with an infinite number of hidden units. We show qualitatively and quantitatively that SD-SG and SD-CBOW are competitive with their fixed-dimension counterparts while providing a distribution over embedding dimensionalities, which offers a window into how semantics distribute across dimensions.
Keywords
Cite
@article{arxiv.1511.05392,
title = {Learning the Dimensionality of Word Embeddings},
author = {Eric Nalisnick and Sachin Ravi},
journal= {arXiv preprint arXiv:1511.05392},
year = {2017}
}