$k$-means as a variational EM approximation of Gaussian mixture models
Abstract
We show that -means (Lloyd's algorithm) is obtained as a special case when truncated variational EM approximations are applied to Gaussian Mixture Models (GMM) with isotropic Gaussians. In contrast to the standard way to relate -means and GMMs, the provided derivation shows that it is not required to consider Gaussians with small variances or the limit case of zero variances. There are a number of consequences that directly follow from our approach: (A) -means can be shown to increase a free energy associated with truncated distributions and this free energy can directly be reformulated in terms of the -means objective; (B) -means generalizations can directly be derived by considering the 2nd closest, 3rd closest etc. cluster in addition to just the closest one; and (C) the embedding of -means into a free energy framework allows for theoretical interpretations of other -means generalizations in the literature. In general, truncated variational EM provides a natural and rigorous quantitative link between -means-like clustering and GMM clustering algorithms which may be very relevant for future theoretical and empirical studies.
Keywords
Cite
@article{arxiv.1704.04812,
title = {$k$-means as a variational EM approximation of Gaussian mixture models},
author = {Jörg Lücke and Dennis Forster},
journal= {arXiv preprint arXiv:1704.04812},
year = {2019}
}