Robust and Private Bayesian Inference

  • Christos Dimitrakakis
  • Blaine Nelson
  • Aikaterini Mitrokotsa
  • Benjamin I. P. Rubinstein
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8776)


We examine the robustness and privacy of Bayesian inference, under assumptions on the prior, and with no modifications to the Bayesian framework. First, we generalise the concept of differential privacy to arbitrary dataset distances, outcome spaces and distribution families. We then prove bounds on the robustness of the posterior, introduce a posterior sampling mechanism, show that it is differentially private and provide finite sample bounds for distinguishability-based privacy under a strong adversarial model. Finally, we give examples satisfying our assumptions.


Posterior Distribution Prior Distribution Bayesian Inference Posterior Sampling Distribution Family 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berger, J.O.: Statistical Decision Theory and Bayesian Analysis. Springer (1985)Google Scholar
  2. 2.
    Bickel, P.J., Doksum, K.A.: Mathematical Statistics: Basic Ideas and Selected Topics, vol. 1. Holden-Day Company (2001)Google Scholar
  3. 3.
    Bousquet, O., Elisseeff, A.: Stability and generalization. Journal of Machine Learning Research 2, 499–526 (2002)MathSciNetzbMATHGoogle Scholar
  4. 4.
    Chatzikokolakis, K., Andrés, M.E., Bordenabe, N.E., Palamidessi, C.: Broadening the scope of differential privacy using metrics. In: De Cristofaro, E., Wright, M. (eds.) PETS 2013. LNCS, vol. 7981, pp. 82–102. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  5. 5.
    Chaudhuri, K., Hsu, D.: Convergence rates for differentially private statistical estimation. In: ICML (2012)Google Scholar
  6. 6.
    Chaudhuri, K., Monteleoni, C., Sarwate, A.D.: Differentially private empirical risk minimization. Journal of Machine Learning Research 12, 1069–1109 (2011)MathSciNetzbMATHGoogle Scholar
  7. 7.
    DeGroot, M.H.: Optimal Statistical Decisions. John Wiley & Sons (1970)Google Scholar
  8. 8.
    Dimitrakakis, C., Nelson, B., Mitrokotsa, A., Rubinstein, B.: Robust and private Bayesian inference. Technical report, arXiv:1306.1066 (2014)Google Scholar
  9. 9.
    Duchi, J.C., Jordan, M.I., Wainwright, M.J.: Local privacy and statistical minimax rates. Technical report, arXiv:1302.3203 (2013)Google Scholar
  10. 10.
    Dwork, C.: Differential privacy. In: Bugliesi, M., Preneel, B., Sassone, V., Wegener, I. (eds.) ICALP 2006. LNCS, vol. 4052, pp. 1–12. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  11. 11.
    Dwork, C., Lei, J.: Differential privacy and robust statistics. In: STOC, pp. 371–380 (2009)Google Scholar
  12. 12.
    Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 265–284. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  13. 13.
    Dwork, C., Smith, A.: Differential privacy for statistics: What we know and what we want to learn. Journal of Privacy and Confidentiality 1(2), 135–154 (2009)Google Scholar
  14. 14.
    Fedotov, A.A., Harremoës, P., Topsoe, F.: Refinements of Pinsker’s inequality. IEEE Transactions on Information Theory 49(6), 1491–1498 (2003)CrossRefzbMATHGoogle Scholar
  15. 15.
    Grünwald, P.D., Dawid, A.P.: Game theory, maximum entropy, minimum discrepancy, and robust bayesian decision theory. The Annals of Statistics 32(4), 1367–1433 (2004)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Hall, R., Rinaldo, A., Wasserman, L.: Differential privacy for functions and functional data. Journal of Machine Learning Research 14, 703–727 (2013)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust Statistics: The Approach Based on Influence Functions. John Wiley and Sons (1986)Google Scholar
  18. 18.
    Huber, P.J.: Robust Statistics. John Wiley and Sons (1981)Google Scholar
  19. 19.
    McSherry, F., Talwar, K.: Mechanism design via differential privacy. In: FOCS, pp. 94–103 (2007)Google Scholar
  20. 20.
    Mir, D.: Differentially-private learning and information theory. In: Proceedings of the 2012 Joint EDBT/ICDT Workshops, pp. 206–210. ACM (2012)Google Scholar
  21. 21.
    Norkin, V.: Stochastic Lipschitz functions. Cybernetics and Systems Analysis 22(2), 226–233 (1986)MathSciNetzbMATHGoogle Scholar
  22. 22.
    Rubinstein, B.I.P., Bartlett, P.L., Huang, L., Taft, N.: Learning in a large function space: Privacy-preserving mechanisms for SVM learning. Journal of Privacy and Confidentiality 4(1) (2012)Google Scholar
  23. 23.
    Wasserman, L., Zhou, S.: A statistical framework for differential privacy. Journal of the American Statistical Association 105(489), 375–389 (2010)MathSciNetCrossRefGoogle Scholar
  24. 24.
    Weissman, T., Ordentlich, E., Seroussi, G., Verdu, S., Weinberger, M.J.: Inequalities for the L1 deviation of the empirical distribution. Technical report, Hewlett-Packard Labs (2003)Google Scholar
  25. 25.
    Williams, O., McSherry, F.: Probabilistic inference and differential privacy. In: NIPS, pp. 2451–2459 (2010)Google Scholar
  26. 26.
    Xiao, Y., Xiong, L.: Bayesian inference under differential privacy. Technical report, arXiv:1203.0617 (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Christos Dimitrakakis
    • 1
  • Blaine Nelson
    • 2
  • Aikaterini Mitrokotsa
    • 1
  • Benjamin I. P. Rubinstein
    • 3
  1. 1.Chalmers University of TechnologySweden
  2. 2.University of PotsdamGermany
  3. 3.The University of MelbourneAustralia

Personalised recommendations