English

Estimating small moments of data stream in nearly optimal space-time

Data Structures and Algorithms 2010-06-21 v2 Machine Learning

Abstract

For each p(0,2]p \in (0,2], we present a randomized algorithm that returns an ϵ\epsilon-approximation of the ppth frequency moment of a data stream Fp=i=1n\absfipF_p = \sum_{i = 1}^n \abs{f_i}^p. The algorithm requires space O(ϵ2log(mM)(logn))O(\epsilon^{-2} \log (mM)(\log n)) and processes each stream update using time O((logn)(logϵ1))O((\log n) (\log \epsilon^{-1})). It is nearly optimal in terms of space (lower bound O(ϵ2log(mM))O(\epsilon^{-2} \log (mM)) as well as time and is the first algorithm with these properties. The technique separates heavy hitters from the remaining items in the stream using an appropriate threshold and estimates the contribution of the heavy hitters and the light elements to FpF_p separately. A key component is the design of an unbiased estimator for \absfip\abs{f_i}^p whose data structure has low update time and low variance.

Keywords

Cite

@article{arxiv.1005.1120,
  title  = {Estimating small moments of data stream in nearly optimal space-time},
  author = {Sumit Ganguly},
  journal= {arXiv preprint arXiv:1005.1120},
  year   = {2010}
}

Comments

Withdrawn due to error in analysis