It seems to be a pearl of conventional wisdom that parameter learning in deep sum-product networks is surprisingly fast compared to shallow mixture models. This paper examines the effects of overparameterization in sum-product networks on the speed of parameter optimisation. Using theoretical analysis and empirical experiments, we show that deep sum-product networks exhibit an implicit acceleration compared to their shallow counterpart. In fact, gradient-based optimisation in deep tree-structured sum-product networks is equal to gradient ascend with adaptive and time-varying learning rates and additional momentum terms.
@article{arxiv.1905.08196,
title = {Optimisation of Overparametrized Sum-Product Networks},
author = {Martin Trapp and Robert Peharz and Franz Pernkopf},
journal= {arXiv preprint arXiv:1905.08196},
year = {2019}
}
Comments
Workshop on Tractable Probabilistic Models (TPM) at ICML 2019