Model Checking Differentially Private Properties

  • Depeng Liu
  • Bow-Yaw WangEmail author
  • Lijun Zhang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11275)


We introduce the branching time temporal logic Open image in new window for specifying differential privacy. Several differentially private mechanisms are formalized as Markov chains or Markov decision processes. Using our formal models, subtle privacy conditions are specified by Open image in new window . In order to verify privacy properties automatically, model checking problems are investigated. We give a model checking algorithm for Markov chains. Model checking Open image in new window properties on Markov decision processes however is shown to be undecidable.


  1. 1.
    Alvim, M.S., Andrés, M.E., Chatzikokolakis, K., Degano, P., Palamidessi, C.: On the information leakage of differentially-private mechanisms. J. Comput. Secur. 23(4), 427–469 (2015)CrossRefGoogle Scholar
  2. 2.
    Baier, C., Katoen, J.P.: Principles of Model Checking. The MIT Press, Cambridge (2008)zbMATHGoogle Scholar
  3. 3.
    Baier, C., Kiefer, S., Klein, J., Klüppelholz, S., Müller, D., Worrell, J.: Markov chains and unambiguous Büchi automata. In: Chaudhuri, S., Farzan, A. (eds.) CAV 2016. LNCS, vol. 9779, pp. 23–42. Springer, Cham (2016). Scholar
  4. 4.
    Barthe, G., Danezis, G., Grégoire, B., Kunz, C., Zanella-Béguelin, S.: Verified computational differential privacy with applications to smart metering. In: CSF, pp. 287–301. IEEE (2013)Google Scholar
  5. 5.
    Barthe, G., et al.: Differentially private Bayesian programming. In: CCS, pp. 68–79. ACM (2016)Google Scholar
  6. 6.
    Barthe, G., Fong, N., Gaboardi, M., Grégoire, B., Hsu, J., Strub, P.Y.: Advanced probabilistic couplings for differential privacy. In: CCS, pp. 55–67. ACM (2016)Google Scholar
  7. 7.
    Barthe, G., Gaboardi, M., Arias, E.J.G., Hsu, J., Kunz, C., Strub, P.Y.: Proving differential privacy in Hoare logic. In: CSF, pp. 411–424. IEEE (2014)Google Scholar
  8. 8.
    Barthe, G., Gaboardi, M., Arias, E.J.G., Hsu, J., Roth, A., Strub, P.: Higher-order approximate relational refinement types for mechanism design and differential privacy. In: POPL, pp. 68–79. ACM (2015)Google Scholar
  9. 9.
    Barthe, G., Gaboardi, M., Gregoire, B., Hsu, J., Strub, P.Y.: Proving differential privacy via probabilistic couplings. In: LICS. IEEE (2016)Google Scholar
  10. 10.
    Barthe, G., Köpf, B., Olmedo, F., Zanella-Béguelin, S.: Probabilistic relational reasoning for differential privacy. In: POPL, pp. 97–110. ACM (2012)Google Scholar
  11. 11.
    Bianco, A., de Alfaro, L.: Model checking of probabilistic and nondeterministic systems. In: Thiagarajan, P.S. (ed.) FSTTCS 1995. LNCS, vol. 1026, pp. 499–513. Springer, Heidelberg (1995). Scholar
  12. 12.
    Clarke, E.M., Grumberg, O., Peled, D.: Model Checking. The MIT Press, Cambridge (1999)Google Scholar
  13. 13.
    Courcoubetis, C., Yannakakis, M.: The complexity of probabilistic verification. J. ACM 42(4), 857–907 (1995)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Couvreur, J.-M., Saheb, N., Sutre, G.: An optimal automata approach to LTL model checking of probabilistic systems. In: Vardi, M.Y., Voronkov, A. (eds.) LPAR 2003. LNCS (LNAI), vol. 2850, pp. 361–375. Springer, Heidelberg (2003). Scholar
  15. 15.
    Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 265–284. Springer, Heidelberg (2006). Scholar
  16. 16.
    Dwork, C., Roth, A.: The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. 9(3–4), 211–407 (2014)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Dwork, C.: Differential privacy. In: Bugliesi, M., Preneel, B., Sassone, V., Wegener, I. (eds.) ICALP 2006. LNCS, vol. 4052, pp. 1–12. Springer, Heidelberg (2006). Scholar
  18. 18.
    Fung, B.C.M., Wang, K., Chen, R., Yu, P.S.: Privacy-preserving data publish: a survey of recent developments. ACM Comput. Surv. 42(4), 14:1–14:53 (2010)CrossRefGoogle Scholar
  19. 19.
    Gaboardi, M., Haeberlen, A., Hsu, J., Narayan, A., Pierce, B.C.: Linear dependent types for differential privacy. In: POPL, pp. 357–370 (2013)CrossRefGoogle Scholar
  20. 20.
    Gazeau, I., Miller, D., Palamidessi, C.: Preserving differential privacy under finite-precision semantics. Theor. Comput. Sci. 655, 92–108 (2016)MathSciNetCrossRefGoogle Scholar
  21. 21.
    Ghosh, A., Roughgarden, T., Sundararajan, M.: Universally utility-maximizing privacy mechanisms. In: STOC, pp. 351–360. ACM, New York (2009)Google Scholar
  22. 22.
    Ghosh, A., Roughgarden, T., Sundararajan, M.: Universally utility-maximizing privacy mechanisms. SIAM J. Comput. 41(6), 1673–1693 (2012)MathSciNetCrossRefGoogle Scholar
  23. 23.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Form. Asp. Comput. 6(5), 512–535 (1994)CrossRefGoogle Scholar
  24. 24.
    Ji, Z., Lipton, Z.C., Elkan, C.: Differential privacy and machine learning: a survey and review. CoRR abs/1412.7584 (2014).
  25. 25.
    Manna, Z., Pnueli, A.: The Temporal Logic of Reactive and Concurrent Systems: Specification. Springer, New York (1992). Scholar
  26. 26.
    Mironov, I.: On significance of the least significant bits for differential privacy. In: Yu, T., Danezis, G., Gligor, V.D. (eds.) ACM CCS, pp. 650–661 (2012)Google Scholar
  27. 27.
    Paz, A.: Introduction to Probabilistic Automata: Computer Science and Applied Mathematics. Academic Press, Inc., Orlando (1971)Google Scholar
  28. 28.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics, vol. 594. Wiley, Hoboken (2005)Google Scholar
  29. 29.
    Rabin, M.: Probabilistic automata. Inf. Control. 6(3), 230–245 (1963)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Reed, J., Pierce, B.C.: Distance makes the types grow stronger: a calculus for differential privacy. In: ICFP, pp. 157–168. ACM (2010)Google Scholar
  31. 31.
    Tang, J., Korolova, A., Bai, X., Wang, X., Wang, X.: Privacy loss in apple’s implementation of differential privacy on MacOS 10.12. CoRR abs/1709.02753 (2017).
  32. 32.
    Tschantz, M.C., Kaynar, D., Datta, A.: Formal verification of differential privacy for interactive systems (extended abstract). In: Mathematical Foundations of Programming Semantics. ENTCS, vol. 276, pp. 61–79 (2011)MathSciNetCrossRefGoogle Scholar
  33. 33.
    Tzeng, W.: A polynomial-time algorithm for the equivalence of probabilistic automata. SIAM J. Comput. 21(2), 216–227 (1992)MathSciNetCrossRefGoogle Scholar
  34. 34.
    Winograd-Cort, D., Haeberlen, A., Roth, A., Pierce, B.C.: A framework for adaptive differential privacy. Proc. ACM Program. Lang. 1(ICFP), 10:1–10:29 (2017)CrossRefGoogle Scholar
  35. 35.
    WWDC: Engineering privacy for your users (2016).
  36. 36.
    Zhang, D., Kifer, D.: LightDP: towards automating differential privacy proofs. In: POPL, pp. 888–901. ACM (2017)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.State Key Laboratory of Computer ScienceInstitute of Software, Chinese Academy of SciencesBeijingChina
  2. 2.University of Chinese Academy of SciencesBeijingChina
  3. 3.Institute of Information ScienceAcademia SinicaTaipeiTaiwan
  4. 4.Institute of Intelligent SoftwareGuangzhouChina

Personalised recommendations