On the Problem of $p_1^{-1}$ in Locality-Sensitive Hashing

Abstract

A Locality-Sensitive Hash (LSH) function is called (r, cr, $p_1,p_2)$-sensitive, if two data-points with a distance less than r collide with probability at least $p_1$ while data points with a distance greater than cr collide with probability at most $p_2$. These functions form the basis of the successful Indyk-Motwani algorithm (STOC 1998) for nearest neighbour problems. In particular one may build a c-approximate nearest neighbour data structure with query time $\tilde{O}(n^\rho /p_1)$ where $\rho =\frac{\log 1/p_1}{\log 1/p_2}\in (0,1)$. This is sub-linear as long as $p_1$ is not too small. Such an algorithm is significant, since most high dimensional nearest neighbour problems suffer from the curse of dimensionality, and can’t be solved exact, faster than a brute force linear-time scan of the database.

Unfortunately many of the best LSH functions tend to have very low collision probabilities, including the best functions for Cosine and Jaccard Similarity. This means that the $n^\rho /p_1$ query time of LSH is often not sub-linear after all, even for approximate nearest neighbours!

In this paper, we improve the general Indyk-Motwani algorithm to reduce the query time of LSH to $\tilde{O}(n^\rho /p_1^{1-\rho })$ (and the space usage correspondingly.) Since $n^\rho /p_1^{1-\rho } < n \Leftrightarrow p_1 > n^{-1}$, our algorithm always obtains sublinear query time, for all collision probabilities at least 1/n. For $p_1$ and $p_2$ small enough, our improvement over all previous methods can be up to a factor n in both query time and space.

The improvement comes from a simple change to the Indyk-Motwani algorithm, which we call “LSH with High-Low Tables”. This technique can easily be implemented in existing software packages.