Non-minimal k-perfect hashing: Tight lower bounds and an application to fast static hash tables

A minimal perfect hash function (minimal PHF) is a data structure mapping a static set of $n$ keys to $n$ bins without collisions. Two natural generalizations are minimal $k$-PHFs where $n$ keys are mapped to $n/k$ bins of capacity $k$ each, and (non-minimal) PHFs with load factor ${\alpha} < 1$ where the number of bins is increased by a factor of $1/{\alpha}$, resulting in spare capacity. While there has been a recent surge of interest in perfect hashing generally, non-minimal $k$-PHFs have not been systematically studied despite a natural use case of speeding up static hash tables: The idea is that a small cache-resident $k$-PHF maps each key $x$ to a cache-line-sized bin of capacity $k$ where $x$ resides. Ideally, this yields a branchless lookup operation with a single cache miss working at high load factors for positive and negative queries alike. Our main theoretical contribution is to determine tight space lower bounds for $k$-PHFs for all pairs of ${\alpha} \in (0,1]$ and $k \geq 1$. It turns out that combining ${\alpha} < 1$ and $k \geq 2$ drastically reduces the space of $k$-PHFs, e.g. for $(k,{\alpha}) = (16,0.8)$ the space lower bound is $0.027$ bits per key while for $(k,{\alpha}) = (16,1.0)$ and $(k,{\alpha}) = (1,0.8)$ the lower bounds are higher by factors of $\approx 8$ and $\approx 32$, respectively. On the practical side, we develop a $k$-PHF based on PtrHash and tune it for use in static hash tables. Empirically, our implementation produces $k$-PHFs of size roughly $50\%$ above the lower bound. A static hash set based on this $k$-PHF is consistently at least as fast as other hash sets for negative and mixed queries. On two of the three tested architectures it achieves up to $1.5\times$ speedup for large $n\geq 30M$ where a $1$-PHF does not fit in cache.