# Exact MCMC with differentially private moves

- 216 Downloads

## Abstract

We view the penalty algorithm of Ceperley and Dewing (J Chem Phys 110(20):9812–9820, 1999), a Markov chain Monte Carlo algorithm for Bayesian inference, in the context of data privacy. Specifically, we studied differential privacy of the penalty algorithm and advocate its use for data privacy. The algorithm can be made differentially private while remaining exact in the sense that its target distribution is the true posterior distribution conditioned on the private data. We also show that in a model with independent observations the algorithm has desirable convergence and privacy properties that scale with data size. Two special cases are also investigated and privacy-preserving schemes are proposed for those cases: (i) Data are distributed among several users who are interested in the inference of a common parameter while preserving their data privacy. (ii) The data likelihood belongs to an exponential family. The results of our numerical experiments on the Beta-Bernoulli and the logistic regression models agree with the theoretical results.

## Keywords

Markov chain Monte Carlo Differential privacy Penalty algorithm## Notes

## References

- Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: Conference on Computer and Communications Security, pp. 308–318. ACM SIGSAC (2016)Google Scholar
- Andrieu, C., Doucet, A., Yıldırım, S., Chopin, N.: On the utility of Metropolis–Hastings with asymmetric acceptance ratio. Technical report (2018). arXiv:1803.09527
- Atchadé, Y.F., Perron, F.: On the geometric ergodicity of Metropolis–Hastings algorithms. Statistics
**41**(1), 77–84 (2007)MathSciNetCrossRefGoogle Scholar - Bierkens, J.: Non-reversible Metropolis–Hastings. Stat. Comput.
**26**(6), 1213–1228 (2016)MathSciNetCrossRefGoogle Scholar - Bun, M., Steinke, T.: Concentrated differential privacy: simplifications, extensions, and lower bounds. In: Proceedings, Part I, of the 14th International Conference on Theory of Cryptography, vol. 9985, pp. 635–658. Springer, New York (2016)Google Scholar
- Ceperley, D.M., Dewing, M.: The penalty method for random walks with uncertain energies. J. Chem. Phys.
**110**(20), 9812–9820 (1999)CrossRefGoogle Scholar - Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) Theory of Cryptography. TCC 2006. Lecture Notes in Computer Science, vol. 3876, pp. 265–284. Springer, Berlin, Heidelberg (2006)Google Scholar
- Dwork, C., Roth, A.: The algorithmic foundations of differential privacy. Theor. Comput. Sci.
**9**(3–4), 211–407 (2013)MathSciNetzbMATHGoogle Scholar - Dwork, C., Rothblum, G.N.: Concentrated differential privacy. Technical report (2016). arXiv:1603.01887v2
- Dwork, C., Rothblum, G.N., Vadhan, S.: Boosting and differential privacy. In: 2010 51st Annual IEEE Symposium on Foundations of Computer Science (FOCS), pp. 51–60 (2010)Google Scholar
- Foulds, J., Geumlek, J., an Kamalika Chaudhuri, M.W.: On the theory and practice of privacy-preserving Bayesian data analysis. Technical report (2016). arxiv:1603.07294
- Geumlek, J., Song, S., Chaudhuri, K.: Renyi differential privacy mechanisms for posterior sampling. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 5289–5298. Curran Associates Inc, Red Hook (2017)Google Scholar
- Gustafson, P.: A guided walk Metropolis algorithm. Stat. Comput.
**8**(4), 357–364 (1998)CrossRefGoogle Scholar - Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika
**52**(1), 97–109 (1970)MathSciNetCrossRefGoogle Scholar - Heikkilä, M., Lagerspetz, E., Kaski, S., Shimizu, K., Tarkoma, S., Honkela, A.: Differentially private Bayesian learning on distributed data. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 3226–3235. Curran Associates Inc, Red Hook (2017)Google Scholar
- Minami, K., Arai, H., Sato, I., Nakagawa, H.: Differential privacy without sensitivity. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 956–964. Curran Associates Inc, Red Hook (2016)Google Scholar
- Mironov, I.: Rényi differential privacy. In: 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263–275 (2017)Google Scholar
- Nicholls, G., Fox, C., Watt, A.: Coupled MCMC with a randomised acceptance probability. Technical report (2012). arXiv:1205.6857
- Park, M., Foulds, J.R., Chaudhuri, K., Welling, M.: Variational Bayes in private settings (VIPS). Technical report (2016). arXiv:1611.00340v3
- Robert, C.P., Casella, G.: Monte Carlo Statistical Methods, 2nd edn. Springer, New York (2004)CrossRefGoogle Scholar
- Roberts, G., Tweedie, R.: Geometric convergence and central limit theorems for multidimensional Hastings and Metropolis algorithms. Biometrika
**83**, 95–110 (1996)MathSciNetCrossRefGoogle Scholar - Wang, Y.-X., Fienberg, S., Smola, A.: Privacy for free: posterior sampling and stochastic gradient Monte Carlo. In: Blei, D., Bach, F. (eds), Proceedings of the 32nd International Conference on Machine Learning (ICML-15), Workshop and Conference Proceedings, pp. 2493–2502. JMLR (2015)Google Scholar
- Welling, M., Teh, Y.W.: Bayesian learning via stochastic gradient langevin dynamics. In: Getoor, L., Scheffer, T. (eds), Proceedings of 28th International Conference on Machine Learning (ICML 2011), ICML ’11, pp. 681–688. ACM (2011)Google Scholar