Quotient Hash Tables - Efficiently Detecting Duplicates in Streaming Data
Data Structures and Algorithms
2019-01-15 v1
Abstract
This article presents the Quotient Hash Table (QHT) a new data structure for duplicate detection in unbounded streams. QHTs stem from a corrected analysis of streaming quotient filters (SQFs), resulting in a 33\% reduction in memory usage for equal performance. We provide a new and thorough analysis of both algorithms, with results of interest to other existing constructions. We also introduce an optimised version of our new data structure dubbed Queued QHT with Duplicates (QQHTD). Finally we discuss the effect of adversarial inputs for hash-based duplicate filters similar to QHT.
Keywords
Cite
@article{arxiv.1901.04358,
title = {Quotient Hash Tables - Efficiently Detecting Duplicates in Streaming Data},
author = {Rémi Géraud and Marius Lombard-Platet and David Naccache},
journal= {arXiv preprint arXiv:1901.04358},
year = {2019}
}
Comments
Shorter version was accepted at SIGAPP SAC '19