Approaching Optimal Duplicate Detection in a Sliding Window
Abstract
Duplicate detection is the problem of identifying whether a given item has previously appeared in a (possibly infinite) stream of data, when only a limited amount of memory is available. Unfortunately the infinite stream setting is ill-posed, and error rates of duplicate detection filters turn out to be heavily constrained: consequently they appear to provide no advantage, asymptotically, over a biased coin toss [8]. In this paper we formalize the sliding window setting introduced by [13,16], and show that a perfect (zero error) solution can be used up to a maximal window size . Above this threshold we show that some existing duplicate detection filters (designed for the setting) perform better that those targeting the windowed problem. Finally, we introduce a "queuing construction" that improves on the performance of some duplicate detection filters in the windowed setting. We also analyse the security of our filters in an adversarial setting.
Cite
@article{arxiv.2005.04740,
title = {Approaching Optimal Duplicate Detection in a Sliding Window},
author = {Rémi Géraud-Stewart and Marius Lombard-Platet and David Naccache},
journal= {arXiv preprint arXiv:2005.04740},
year = {2020}
}