Dynamic Evaluation of Neural Sequence Models
Neural and Evolutionary Computing
2017-10-27 v2 Computation and Language
Abstract
We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.
Keywords
Cite
@article{arxiv.1709.07432,
title = {Dynamic Evaluation of Neural Sequence Models},
author = {Ben Krause and Emmanuel Kahembwe and Iain Murray and Steve Renals},
journal= {arXiv preprint arXiv:1709.07432},
year = {2017}
}