Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
Computation and Language
2018-03-06 v4 Machine Learning
Abstract
We formulate language modeling as a matrix factorization problem, and show that the expressiveness of Softmax-based models (including the majority of neural language models) is limited by a Softmax bottleneck. Given that natural language is highly context-dependent, this further implies that in practice Softmax with distributed word embeddings does not have enough capacity to model natural language. We propose a simple and effective method to address this issue, and improve the state-of-the-art perplexities on Penn Treebank and WikiText-2 to 47.69 and 40.68 respectively. The proposed method also excels on the large-scale 1B Word dataset, outperforming the baseline by over 5.6 points in perplexity.
Cite
@article{arxiv.1711.03953,
title = {Breaking the Softmax Bottleneck: A High-Rank RNN Language Model},
author = {Zhilin Yang and Zihang Dai and Ruslan Salakhutdinov and William W. Cohen},
journal= {arXiv preprint arXiv:1711.03953},
year = {2018}
}
Comments
ICLR Oral 2018