Exact MCMC with differentially private moves

Revisiting the penalty algorithm in a data privacy framework
  • Sinan YıldırımEmail author
  • Beyza Ermiş


We view the penalty algorithm of Ceperley and Dewing (J Chem Phys 110(20):9812–9820, 1999), a Markov chain Monte Carlo algorithm for Bayesian inference, in the context of data privacy. Specifically, we studied differential privacy of the penalty algorithm and advocate its use for data privacy. The algorithm can be made differentially private while remaining exact in the sense that its target distribution is the true posterior distribution conditioned on the private data. We also show that in a model with independent observations the algorithm has desirable convergence and privacy properties that scale with data size. Two special cases are also investigated and privacy-preserving schemes are proposed for those cases: (i) Data are distributed among several users who are interested in the inference of a common parameter while preserving their data privacy. (ii) The data likelihood belongs to an exponential family. The results of our numerical experiments on the Beta-Bernoulli and the logistic regression models agree with the theoretical results.


Markov chain Monte Carlo Differential privacy Penalty algorithm 



  1. Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., Zhang, L.: Deep learning with differential privacy. In: Conference on Computer and Communications Security, pp. 308–318. ACM SIGSAC (2016)Google Scholar
  2. Andrieu, C., Doucet, A., Yıldırım, S., Chopin, N.: On the utility of Metropolis–Hastings with asymmetric acceptance ratio. Technical report (2018). arXiv:1803.09527
  3. Atchadé, Y.F., Perron, F.: On the geometric ergodicity of Metropolis–Hastings algorithms. Statistics 41(1), 77–84 (2007)MathSciNetCrossRefGoogle Scholar
  4. Bierkens, J.: Non-reversible Metropolis–Hastings. Stat. Comput. 26(6), 1213–1228 (2016)MathSciNetCrossRefGoogle Scholar
  5. Bun, M., Steinke, T.: Concentrated differential privacy: simplifications, extensions, and lower bounds. In: Proceedings, Part I, of the 14th International Conference on Theory of Cryptography, vol. 9985, pp. 635–658. Springer, New York (2016)Google Scholar
  6. Ceperley, D.M., Dewing, M.: The penalty method for random walks with uncertain energies. J. Chem. Phys. 110(20), 9812–9820 (1999)CrossRefGoogle Scholar
  7. Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) Theory of Cryptography. TCC 2006. Lecture Notes in Computer Science, vol. 3876, pp. 265–284. Springer, Berlin, Heidelberg (2006)Google Scholar
  8. Dwork, C., Roth, A.: The algorithmic foundations of differential privacy. Theor. Comput. Sci. 9(3–4), 211–407 (2013)MathSciNetzbMATHGoogle Scholar
  9. Dwork, C., Rothblum, G.N.: Concentrated differential privacy. Technical report (2016). arXiv:1603.01887v2
  10. Dwork, C., Rothblum, G.N., Vadhan, S.: Boosting and differential privacy. In: 2010 51st Annual IEEE Symposium on Foundations of Computer Science (FOCS), pp. 51–60 (2010)Google Scholar
  11. Foulds, J., Geumlek, J., an Kamalika Chaudhuri, M.W.: On the theory and practice of privacy-preserving Bayesian data analysis. Technical report (2016). arxiv:1603.07294
  12. Geumlek, J., Song, S., Chaudhuri, K.: Renyi differential privacy mechanisms for posterior sampling. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 5289–5298. Curran Associates Inc, Red Hook (2017)Google Scholar
  13. Gustafson, P.: A guided walk Metropolis algorithm. Stat. Comput. 8(4), 357–364 (1998)CrossRefGoogle Scholar
  14. Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika 52(1), 97–109 (1970)MathSciNetCrossRefGoogle Scholar
  15. Heikkilä, M., Lagerspetz, E., Kaski, S., Shimizu, K., Tarkoma, S., Honkela, A.: Differentially private Bayesian learning on distributed data. In: Guyon, I., Luxburg, U.V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 30, pp. 3226–3235. Curran Associates Inc, Red Hook (2017)Google Scholar
  16. Minami, K., Arai, H., Sato, I., Nakagawa, H.: Differential privacy without sensitivity. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 29, pp. 956–964. Curran Associates Inc, Red Hook (2016)Google Scholar
  17. Mironov, I.: Rényi differential privacy. In: 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263–275 (2017)Google Scholar
  18. Nicholls, G., Fox, C., Watt, A.: Coupled MCMC with a randomised acceptance probability. Technical report (2012). arXiv:1205.6857
  19. Park, M., Foulds, J.R., Chaudhuri, K., Welling, M.: Variational Bayes in private settings (VIPS). Technical report (2016). arXiv:1611.00340v3
  20. Robert, C.P., Casella, G.: Monte Carlo Statistical Methods, 2nd edn. Springer, New York (2004)CrossRefGoogle Scholar
  21. Roberts, G., Tweedie, R.: Geometric convergence and central limit theorems for multidimensional Hastings and Metropolis algorithms. Biometrika 83, 95–110 (1996)MathSciNetCrossRefGoogle Scholar
  22. Wang, Y.-X., Fienberg, S., Smola, A.: Privacy for free: posterior sampling and stochastic gradient Monte Carlo. In: Blei, D., Bach, F. (eds), Proceedings of the 32nd International Conference on Machine Learning (ICML-15), Workshop and Conference Proceedings, pp. 2493–2502. JMLR (2015)Google Scholar
  23. Welling, M., Teh, Y.W.: Bayesian learning via stochastic gradient langevin dynamics. In: Getoor, L., Scheffer, T. (eds), Proceedings of 28th International Conference on Machine Learning (ICML 2011), ICML ’11, pp. 681–688. ACM (2011)Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Faculty of Engineering and Natural SciencesSabancı UniversityİstanbulTurkey
  2. 2.Department of Computer EngineeringBoğaziçi UniversityİstanbulTurkey

Personalised recommendations