Asymptotic Log-loss of Prequential Maximum Likelihood Codes
Machine Learning
2007-07-16 v1 Information Theory
math.IT
Abstract
We analyze the Dawid-Rissanen prequential maximum likelihood codes relative to one-parameter exponential family models M. If data are i.i.d. according to an (essentially) arbitrary P, then the redundancy grows at rate c/2 ln n. We show that c=v1/v2, where v1 is the variance of P, and v2 is the variance of the distribution m* in M that is closest to P in KL divergence. This shows that prequential codes behave quite differently from other important universal codes such as the 2-part MDL, Shtarkov and Bayes codes, for which c=1. This behavior is undesirable in an MDL model selection setting.
Keywords
Cite
@article{arxiv.cs/0502004,
title = {Asymptotic Log-loss of Prequential Maximum Likelihood Codes},
author = {Peter Grunwald and Steven de Rooij},
journal= {arXiv preprint arXiv:cs/0502004},
year = {2007}
}
Comments
22 pages, an abstract has been submitted to COLT 2005