Count-Min Tree Sketch: Approximate counting for NLP
Abstract
The Count-Min Sketch is a widely adopted structure for approximate event counting in large scale processing. In a previous work we improved the original version of the Count-Min-Sketch (CMS) with conservative update using approximate counters instead of linear counters. These structures are computationaly efficient and improve the average relative error (ARE) of a CMS at constant memory footprint. These improvements are well suited for NLP tasks, in which one is interested by the low-frequency items. However, if Log counters allow to improve ARE, they produce a residual error due to the approximation. In this paper, we propose the Count-Min Tree Sketch (Copyright 2016 eXenSa. All rights reserved) variant with pyramidal counters, which are focused toward taking advantage of the Zipfian distribution of text data.
Cite
@article{arxiv.1604.05492,
title = {Count-Min Tree Sketch: Approximate counting for NLP},
author = {Guillaume Pitel and Geoffroy Fouquier and Emmanuel Marchand and Abdul Mouhamadsultane},
journal= {arXiv preprint arXiv:1604.05492},
year = {2016}
}
Comments
submitted to the second International Symposium on Web Algorithms (iSwag'2016). arXiv admin note: text overlap with arXiv:1502.04885, In the proceedings of the Second International Symposium on Web Algorithms (iSWAG 2016), June 9-10, 2016, Deauville, Normandy, France