Understanding and Protecting Privacy: Formal Semantics and Principled Audit Mechanisms

  • Anupam Datta
  • Jeremiah Blocki
  • Nicolas Christin
  • Henry DeYoung
  • Deepak Garg
  • Limin Jia
  • Dilsun Kaynar
  • Arunesh Sinha
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7093)


Privacy has become a significant concern in modern society as personal information about individuals is increasingly collected, used, and shared, often using digital technologies, by a wide range of organizations. Certain information handling practices of organizations that monitor individuals’ activities on the Web, data aggregation companies that compile massive databases of personal information, cell phone companies that collect and use location data about individuals, online social networks and search engines—while enabling useful services—have aroused much indignation and protest in the name of privacy. Similarly, as healthcare organizations are embracing electronic health record systems and patient portals to enable patients, employees, and business affiliates more efficient access to personal health information, there is trepidation that the privacy of patients may not be adequately protected if information handling practices are not carefully designed and enforced.

Given this state of affairs, it is very important to arrive at a general understanding of (a) why certain information handling practices arouse moral indignation, what practices or policies are appropriate in a given setting, and (b) how to represent and enforce such policies using information processing systems. This article summarizes progress on a research program driven by goal (b). We describe a semantic model and logic of privacy that formalizes privacy as a right to appropriate flows of personal information—a position taken by contextual integrity, a philosphical theory of privacy for answering questions of the form identified in (a). The logic is designed with the goal of enabling specification and enforcement of practical privacy policies. It has been used to develop the first complete formalization of two US privacy laws—the HIPAA Privacy Rule that prescribes and proscribes flows of personal health information, and the Gramm-Leach-Bliley Act that similarly governs flows of personal financial information. Observing that preventive access control mechanisms are not sufficient to enforce such privacy policies, we develop two complementary audit mechanisms for policy enforcement. These mechanisms enable auditing of practical privacy policies, including the entire HIPAA Privacy Rule. The article concludes with a vision for further research in this area.


Privacy Policy Semantic Model Policy Enforcement Protected Health Information Personal Health Information 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abadi, M., Burrows, M., Lampson, B.W., Plotkin, G.D.: A calculus for access control in distributed systems. ACM Trans. Program. Lang. Syst. 15(4), 706–734 (1993)CrossRefGoogle Scholar
  2. 2.
    Apt, K.R., Marchiori, E.: Reasoning about Prolog programs: From modes through types to assertions. Formal Aspects of Computing 6(6), 743–765 (1994)CrossRefzbMATHGoogle Scholar
  3. 3.
    Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, R.: The nonstochastic multiarmed bandit problem. SIAM Journal on Computing 32(1), 48–77 (2003)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Awerbuch, B., Kleinberg, R.: Online linear optimization and adaptive routing. Journal of Computer and System Sciences 74(1), 97–114 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Baader, F., Bauer, A., Lippmann, M.: Runtime Verification Using a Temporal Description Logic. In: Ghilardi, S., Sebastiani, R. (eds.) FroCoS 2009. LNCS, vol. 5749, pp. 149–164. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  6. 6.
    Backes, M., Pfitzmann, B., Schunter, M.: A Toolkit for Managing Enterprise Privacy Policies. In: Snekkenes, E., Gollmann, D. (eds.) ESORICS 2003. LNCS, vol. 2808, pp. 162–180. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  7. 7.
    Barringer, H., Goldberg, A., Havelund, K., Sen, K.: Rule-Based Runtime Verification. In: Steffen, B., Levi, G. (eds.) VMCAI 2004. LNCS, vol. 2937, pp. 44–57. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  8. 8.
    Barth, A., Datta, A., Mitchell, J.C., Nissenbaum, H.: Privacy and contextual integrity: Framework and applications. In: Proceedings of the 27th IEEE Symposium on Security and Privacy, Oakland, pp. 184–198 (2006)Google Scholar
  9. 9.
    Barth, A., Datta, A., Mitchell, J.C., Sundaram, S.: Privacy and utility in business processes. In: Proceedings of the 20th IEEE Computer Security Foundations Symposium (CSF), pp. 279–294 (2007)Google Scholar
  10. 10.
    Barth, A., Rubinstein, B.I.P., Sundararajan, M., Mitchell, J.C., Song, D., Bartlett, P.L.: A Learning-Based Approach to Reactive Security. In: Sion, R. (ed.) FC 2010. LNCS, vol. 6052, pp. 192–206. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Basin, D., Klaedtke, F., Müller, S.: Monitoring security policies with metric first-order temporal logic. In: Proceeding of the 15th ACM Symposium on Access Control Models and Technologies (SACMAT), pp. 23–34 (2010)Google Scholar
  12. 12.
    Basin, D., Klaedtke, F., Müller, S.: Policy Monitoring in First-Order Temporal Logic. In: Touili, T., Cook, B., Jackson, P. (eds.) CAV 2010. LNCS, vol. 6174, pp. 1–18. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  13. 13.
    Bettini, C., Jajodia, S., Wang, X.S., Wijesekera, D.: Provisions and obligations in policy rule management. Journal of Network and Systems Management 11, 351–372 (2003)CrossRefGoogle Scholar
  14. 14.
    Blocki, J., Christin, N., Datta, A., Sinha, A.: Regret minimizing audits: A learning-theoretic basis for privacy protection. In: Proceedings of the 24th IEEE Computer Security Foundations Symposium (CSF), pp. 312–327 (2011)Google Scholar
  15. 15.
    Blum, A., Mansour, Y.: From External to Internal Regret. In: Auer, P., Meir, R. (eds.) COLT 2005. LNCS (LNAI), vol. 3559, pp. 621–636. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  16. 16.
    Blum, A., Mansour, Y.: Learning, regret minimization, and equilibria. Algorithmic Game Theory, 79–102 (2007)Google Scholar
  17. 17.
    Bruns, G., Godefroid, P.: Generalized Model Checking: Reasoning About Partial State Spaces. In: Palamidessi, C. (ed.) CONCUR 2000. LNCS, vol. 1877, pp. 168–182. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  18. 18.
    Cederquist, J.G., Corin, R., Dekker, M.A.C., Etalle, S., den Hartog, J.I., Lenzini, G.: Audit-based compliance control. International Journal of Information Security 6(2), 133–151 (2007)CrossRefGoogle Scholar
  19. 19.
    Cheng, P.-C., Rohatgi, P.: IT Security as Risk Management: A Research Perspective. IBM Research Report RC24529 (April 2008)Google Scholar
  20. 20.
    Cheng, P.-C., Rohatgi, P., Keser, C., Karger, P.A., Wagner, G.M., Reninger, A.S.: Fuzzy Multi-Level Security: An Experiment on Quantified Risk-Adaptive Access Control. In: Proceedings of the IEEE Symposium on Security and Privacy (2007)Google Scholar
  21. 21.
    Dani, V., Hayes, T.: Robbing the bandit: Less regret in online geometric optimization against an adaptive adversary. In: Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete algorithm, p. 943. ACM (2006)Google Scholar
  22. 22.
    De Young, H., Garg, D., Jia, L., Kaynar, D., Datta, A.: Experiences in the logical specification of the HIPAA and GLBA privacy laws. In: Proceedings of the 9th Annual ACM Workshop on Privacy in the Electronic Society (WPES) (2010), Full version: Carnegie Mellon University Technical Report CMU-CyLab-10-007Google Scholar
  23. 23.
    Dougherty, D.J., Fisler, K., Krishnamurthi, S.: Obligations and their Interaction with Programs. In: Biskup, J., López, J. (eds.) ESORICS 2007. LNCS, vol. 4734, pp. 375–389. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  24. 24.
    Fudenberg, D., Tirole, J.: Game theory. MIT Press (1991)Google Scholar
  25. 25.
    Garg, D., Jia, L., Datta, A.: Policy auditing over incomplete logs: Theory, implementation and applications. In: Proceedings of the 18th ACM Conference on Computer and Communications Security, CCS (2011)Google Scholar
  26. 26.
    Giblin, C., Liu, A.Y., Müller, S., Pfitzmann, B., Zhou, X.: Regulations expressed as logical models (REALM). In: Proceeding of the 18th Annual Conference on Legal Knowledge and Information Systems (JURIX), pp. 37–48 (2005)Google Scholar
  27. 27.
    Godefroid, P., Huth, M.: Model checking vs. generalized model checking: Semantic minimizations for temporal logics. In: Proceedings of the 20th Annual IEEE Symposium on Logic in Computer Science (LICS), pp. 158–167 (2005)Google Scholar
  28. 28.
    Hilty, M., Basin, D., Pretschner, A.: On Obligations. In: di Vimercati, S.d.C., Syverson, P.F., Gollmann, D. (eds.) ESORICS 2005. LNCS, vol. 3679, pp. 98–117. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  29. 29.
    Hulme, G.: Steady Bleed: State of HealthCare Data Breaches. Information Week (September 2010),
  30. 30.
    Irwin, K., Yu, T., Winsborough, W.H.: On the modeling and analysis of obligations. In: Proceedings of the 13th ACM Conference on Computer and Communications Security (CCS), pp. 134–143 (2006)Google Scholar
  31. 31.
    Jagadeesan, R., Jeffrey, A., Pitcher, C., Riely, J.: Towards a Theory of Accountability and Audit. In: Backes, M., Ning, P. (eds.) ESORICS 2009. LNCS, vol. 5789, pp. 152–167. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  32. 32.
    Küsters, R., Truderung, T., Vogt, A.: Accountability: definition and relationship to verifiability. In: ACM Conference on Computer and Communications Security, pp. 526–535 (2010)Google Scholar
  33. 33.
    Lam, P.E., Mitchell, J.C., Sundaram, S.: A Formalization of HIPAA for a Medical Messaging System. In: Fischer-Hübner, S., Lambrinoudakis, C., Pernul, G. (eds.) TrustBus 2009. LNCS, vol. 5695, pp. 73–85. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  34. 34.
    Lampson, B.W.: Computer security in the real world. IEEE Computer 37(6), 37–46 (2004)CrossRefGoogle Scholar
  35. 35.
    Littlestone, N., Warmuth, M.K.: The weighted majority algorithm. Inf. Comput. 108(2), 212–261 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    Liu, Y., Müller, S., Xu, K.: A static compliance-checking framework for business process models. IBM Systems Journal 46, 335–361 (2007)CrossRefGoogle Scholar
  37. 37.
    Manna, Z., Pnueli, A.: Temporal Verification of Reactive Systems: Safety. Springer, Heidelberg (1995)CrossRefzbMATHGoogle Scholar
  38. 38.
    May, M.J., Gunter, C.A., Lee, I.: Privacy APIs: Access control techniques to analyze and verify legal privacy policies. In: Proceedings of the 19th IEEE Workshop on Computer Security Foundations (CSFW), pp. 85–97 (2006)Google Scholar
  39. 39.
    Ni, Q., Bertino, E., Lobo, J.: An obligation model bridging access control policies and privacy policies. In: Proceedings of the 13th ACM Symposium on Access Control Models and Technologies (SACMAT), pp. 133–142 (2008)Google Scholar
  40. 40.
    Nissenbaum, H.: Privacy in Context: Technology, Policy, and the Integrity of Social Life. Stanford University Press (2010)Google Scholar
  41. 41.
    OASIS XACML Committee. Extensible access control markup language (XACML) v2.0 (2004),
  42. 42.
    Park, J., Sandhu, R.: Towards usage control models: beyond traditional access control. In: Proceedings of the 7th ACM Symposium on Access Control Models and Technologies (SACMAT), pp. 57–64 (2002)Google Scholar
  43. 43.
    Robertson, J.: New data spill shows risk of online health records. Yahoo News (August 2011),
  44. 44.
    Roşu, G., Havelund, K.: Rewriting-based techniques for runtime verification. Automated Software Engineering 12, 151–197 (2005)CrossRefGoogle Scholar
  45. 45.
    Roger, M., Goubault-Larrecq, J.: Log auditing through model-checking. In: Proceedings of the 14th IEEE Workshop on Computer Security Foundations (CSF), pp. 220–236 (2001)Google Scholar
  46. 46.
    Sokolsky, O., Sammapun, U., Lee, I., Kim, J.: Run-time checking of dynamic properties. Electronic Notes in Theoretical Computer Science 144, 91–108 (2006)CrossRefGoogle Scholar
  47. 47.
    Thati, P., Roşu, G.: Monitoring algorithms for metric temporal logic specifications. Electronic Notes in Theoretical Computer Science 113, 145–162 (2005)CrossRefGoogle Scholar
  48. 48.
    Tschantz, M. C., Datta, A., Wing, J.: On the semantics of purpose requirements in privacy policies. Tech. Rep. CMU-CS-11-102, Carnegie Mellon University (2010)Google Scholar
  49. 49.
    US Health and Human Services. HIPAA enforcement, (accessed November 19, 2010)
  50. 50.
    Vose, M.D., Wright, A.H., Rowe, J.E.: Implicit Parallelism. In: Cantú-Paz, E., Foster, J.A., Deb, K., Davis, L., Roy, R., O’Reilly, U.-M., Beyer, H.-G., Kendall, G., Wilson, S.W., Harman, M., Wegener, J., Dasgupta, D., Potter, M.A., Schultz, A., Dowsland, K.A., Jonoska, N., Miller, J., Standish, R.K. (eds.) GECCO 2003. LNCS, vol. 2724, pp. 1505–1517. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  51. 51.
    Wall Street Journal. What they know, (accessed on September 8, 2011)
  52. 52.
    Weitzner, D.J., Abelson, H., Berners-Lee, T., Feigenbaum, J., Hendler, J.A., Sussman, G.J.: Information accountability. Commun. ACM 51(6), 82–87 (2008)CrossRefGoogle Scholar
  53. 53.
    Zhao, X., Johnson, M.E.: Access governance: Flexibility with escalation and audit. In: HICSS, pp. 1–13 (2010)Google Scholar
  54. 54.
    Zinkevich, M., Johanson, M., Bowling, M., Piccione, C.: Regret minimization in games with incomplete information. Advances in Neural Information Processing Systems 20, 1729–1736 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Anupam Datta
    • 1
  • Jeremiah Blocki
    • 1
  • Nicolas Christin
    • 1
  • Henry DeYoung
    • 1
  • Deepak Garg
    • 2
  • Limin Jia
    • 1
  • Dilsun Kaynar
    • 1
  • Arunesh Sinha
    • 1
  1. 1.Carnegie Mellon UniversityUSA
  2. 2.Max Planck Institute for Software SystemsGermany

Personalised recommendations