In statistical dialogue management, the dialogue manager learns a policy that maps a belief state to an action for the system to perform. Efficient exploration is key to successful policy optimisation. Current deep reinforcement learning methods are very promising but rely on epsilon-greedy exploration, thus subjecting the user to a random choice of action during learning. Alternative approaches such as Gaussian Process SARSA (GPSARSA) estimate uncertainties and are sample efficient, leading to better user experience, but on the expense of a greater computational complexity. This paper examines approaches to extract uncertainty estimates from deep Q-networks (DQN) in the context of dialogue management. We perform an extensive benchmark of deep Bayesian methods to extract uncertainty estimates, namely Bayes-By-Backprop, dropout, its concrete variation, bootstrapped ensemble and alpha-divergences, combining it with DQN algorithm.
@article{arxiv.1711.11486,
title = {Uncertainty Estimates for Efficient Neural Network-based Dialogue Policy Optimisation},
author = {Christopher Tegho and Paweł Budzianowski and Milica Gašić},
journal= {arXiv preprint arXiv:1711.11486},
year = {2017}
}
Comments
Accepted at the Bayesian Deep Learning Workshop, 31st Conference on Neural Information Processing Systems (NIPS 2017)