M$^2$M: A General Method to Perform Various Data Analysis Tasks from a Differentially Private Sketch

Abstract

Differential privacy is the standard privacy definition for performing analyses over sensitive data. Yet, its privacy budget bounds the number of tasks an analyst can perform with reasonable accuracy, which makes it challenging to deploy in practice. This can be alleviated by private sketching, where the dataset is compressed into a single noisy sketch vector which can be shared with the analysts and used to perform arbitrarily many analyses. However, the algorithms to perform specific tasks from sketches must be developed on a case-by-case basis, which is a major impediment to their use. In this paper, we introduce the generic moment-to-moment ($\textrm{M}^2\textrm{M}$) method to perform a wide range of data exploration tasks from a single private sketch. Among other things, this method can be used to estimate empirical moments of attributes, the covariance matrix, counting queries (including histograms), and regression models. Our method treats the sketching mechanism as a black-box operation, and can thus be applied to a wide variety of sketches from the literature, widening their ranges of applications without further engineering or privacy loss, and removing some of the technical barriers to the wider adoption of sketches for data exploration under differential privacy. We validate our method with data exploration tasks on artificial and real-world data, and show that it can be used to reliably estimate statistics and train classification models from private sketches.

F. Houssiau and V. Schellekens—These authors contributed equally.

Appendices

A Proof of Theorem 1

Let $J_\varSigma$, the left-hand side of the inequality, the mean squared error between the empirical mean $\overline{f}$ and the estimation from the sketch $\widetilde{f}$. Denoting $X=(X_1,\dots ,X_n)$, we have

$$ \begin{array}{ll} J_\varSigma &{}= \mathbb {E}_{X,\xi }\left[ \left( \frac{1}{n}\sum _{i=1}^n f(X_i) - \langle a, \frac{1}{n}\left( \sum _{i=1}^n \varPhi (X_i) + \xi \right) \rangle \right) ^2\right] \\ &{}= \mathbb {E}_{X,\xi }\left[ \left( \frac{1}{n}\sum _{i=1}^n \left( f(X_i)-\langle a,\varPhi (X_i)\rangle \right) - \frac{1}{n}\langle a, \xi \rangle \right) ^2\right] \\ &{}{\mathop {=}\limits ^{(i)}} \mathbb {E}_{X}\left[ \left( \frac{1}{n}\sum _{i=1}^n \left( f(X_i)-\langle a,\varPhi (X_i)\rangle \right) \right) ^2\right] + \frac{1}{n^2}\mathbb {E}_{\xi }\left[ \langle a,\xi \rangle ^2\right] \\ &{}{\mathop {=}\limits ^{(ii)}} \frac{n (n-1)}{n^2} \cdot \left( \mathbb {E}_{X}\left[ f(X)\right] -\langle a,\mathbb {E}_{X}\left[ \varPhi (X)\right] \rangle \right) ^2\\ &{}\quad \quad ~+~\frac{n}{n^2}\cdot \mathbb {E}_{X}\left[ \left( f(X) - \langle a,\varPhi (X)\rangle \right) ^2\right] + ||a||_2^2 \frac{\mathbb {V}[\xi ]}{n^2} \end{array} $$

where we used in (i) the independence from $\xi$ and X and the fact that $\mathbb {E}\left[ \xi \right] = 0$, and in (ii) the fact that samples $(X_i)_{1\le i\le n}$ are independent (and $\mathbb {V}[\cdot ]$ denotes the variance of a random variable). Finally, we use Jensen’s inequality (since $x\mapsto x^2$ is convex) to show that $\left( \mathbb {E}_{X}\left[ f(X)\right] -\langle a,\mathbb {E}_{X}\left[ \varPhi (X)\right] \rangle \right) ^2 \le \mathbb {E}_{X}\left[ \left( f(X) - \langle a,\varPhi (X)\rangle \right) ^2\right]$, which concludes the proof.

B $\textrm{M}^2\textrm{M}$ Learning Procedure