A fast and efficient Hamming LSH-based scheme for accurate linkage

Abstract

In this paper, we propose an efficient scheme for privacy-preserving record linkage by using the Hamming locality-sensitive hashing technique as the blocking mechanism and the Bloom filter-based encoding method for anonymizing the data sets at hand. We achieve highly accurate results and simultaneously reduce significantly the computational cost by minimizing the number of distance computations performed. Our scheme provides theoretical guarantees for identifying the similar anonymized record pairs by conducting redundant blocking and by performing a distance computation only if the corresponding anonymized record pair is formulated a specified number of times. A series of experiments illustrate the efficacy of our scheme in identifying the similar record pairs, while simultaneously keeping the running time exceptionally low.

Appendix: Solving for \(L_{{\mathcal {C}}}\) in (9)

We bound above the right-hand side of (9) in order to guarantee that the probability for a pair with $d_H=\vartheta$ to exhibit less collisions than ${\mathcal {C}}$ is bounded by $\delta$ as follows:

$$\begin{aligned} \exp \left\{ -\frac{(L_{{\mathcal {C}}} \, p^{K}_{\vartheta }-({\mathcal {C}}-1))^2}{2 \, L_{{\mathcal {C}}} \, p^{K}_{\vartheta }}\right\} \le \delta \Leftrightarrow -\frac{(L_{{\mathcal {C}}} \, p^{K}_{\vartheta }-({\mathcal {C}}-1))^2}{2 \, L_{{\mathcal {C}}} \, p^{K}_{\vartheta }} \le \ln (\delta ). \end{aligned}$$

(11)

We expand $(L_{{\mathcal {C}}}\, p^{K}_{\vartheta }-({\mathcal {C}}-1))^2$ and derive the following quadratic equation:

$$\begin{aligned} (p^{K}_{\vartheta } \, L_{{\mathcal {C}}})^2 + (2 \, p^{K}_{\vartheta } \, \ln (\delta ) - 2 \, p^{K}_{\vartheta } ({\mathcal {C}}-1) )L_{{\mathcal {C}}}+({\mathcal {C}}-1)^2 \ge 0, \end{aligned}$$

(12)

where we finally solve for $L_{{\mathcal {C}}}$ and keep only the equality, since we want to be as optimal as possible.