Defining and Measuring Policy Coverage in Testing Access Control Policies

  • Evan Martin
  • Tao Xie
  • Ting Yu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4307)


To facilitate managing access control in a system, security officers increasingly write access control policies in specification languages such as XACML, and use a dedicated software component called a Policy Decision Point (PDP). To increase confidence on written policies, certain types of policy testing (often in an ad hoc way) are usually conducted, which probe the PDP with some typical requests and check PDP’s responses against expected ones. This paper develops a first step toward systematic policy testing by defining and measuring policy coverage when testing policies. We have developed a coverage-measurement tool to measure policy coverage given a set of XACML policies and a set of requests. We have developed a tool for request generation, which randomly generates requests for a given set of policies, and a tool for request reduction, which greedily selects a nearly minimal set of requests for achieving the same coverage as the originally generated requests. To evaluate coverage-based request reduction and its effect on fault detection, we have conducted an experiment with mutation testing on a set of real policies. Our experimental results show that the coveragebased test reduction can substantially reduce the size of generated requests and incur only relatively low loss on fault detection. We also conduct a study on the policy coverage achieved by manually generated requests.


Access Control Fault Detection Test Suite Policy Language Access Control Policy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    OASIS eXtensible Access Control Markup Language (XACML) (2005),
  2. 2.
    Sun’s XACML implementation (2005),
  3. 3.
  4. 4.
    Amla, N., Ammann, P.: Using Z specifications in category partition testing. In: Proc. 7th Annual Conference on Computer Assurance, June 1992, pp. 3–10 (1992)Google Scholar
  5. 5.
    Ammann, P., Offutt, J.: Using formal methods to derive test frames in category-partition testing. In: Proc. 9th Annual Conference on Computer Assurance, June 1994, pp. 69–80 (1994)Google Scholar
  6. 6.
    Anderson, A.: XACML 1.1 committee specification conformance tests (2002),
  7. 7.
    Anderson, R.J.: A security policy model for clinical information systems. In: Proc. IEEE Symposium on Security and Privacy, pp. 30–43 (1996)Google Scholar
  8. 8.
    Bertino, E., Buccafurri, F., Ferrari, E., Rullo, P.: A logical framework for reasoning on data access control policies. In: Proc. 12th IEEE Computer Security Foundations Workshop, pp. 175–189 (1999)Google Scholar
  9. 9.
    Bertino, E., Castano, S., Ferrari, E.: On specifying security policies for web documents with an XML-based language. In: Proc. 6th ACM Symposium on Access Control Models and Technologies, Chantilly, VA, May 2001, pp. 57–65 (2001)Google Scholar
  10. 10.
    Bonatti, P., Vimercati, S., Samarati, P.: A modular approach to composing access control policies. In: Proc. ACM Conference on Computer and Communication Security, Athens, Greece, November 2000, pp. 164–173 (2000)Google Scholar
  11. 11.
    Bussler, C., Jablonski, S.: Policy resolution for workflow management systems. In: Proc. Hawaii International Conference on System Science, Maui, Hawaii, January 1995, pp. 831–840 (1995)Google Scholar
  12. 12.
    Chang, J., Richardson, D.J.: Structural specification-based testing: automated support and experimental evaluation. In: Proc. 7th ESEC/FSE, pp. 285–302 (1999)Google Scholar
  13. 13.
    Clarke, E., Fujita, M., McGeer, P., Yang, J., Zhao, X.: Multi-terminal binary decision diagrams: An efficient data structure for matrix representation. In: Proc. International Workshop on Logic Synthesis, pp. 1–15 (1993)Google Scholar
  14. 14.
    Damianou, N., Dulay, N., Lupu, E., Sloman, M.: The Ponder policy specification language. In: Sloman, M., Lobo, J., Lupu, E.C. (eds.) POLICY 2001. LNCS, vol. 1995, pp. 18–38. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  15. 15.
    DeMillo, R.A., Lipton, R.J., Sayward, F.G.: Hints on test data selection: Help for the practicing programmer. IEEE Computer 11(4), 34–41 (1978)Google Scholar
  16. 16.
    Fisler, K., Krishnamurthi, S., Meyerovich, L.A., Tschantz, M.C.: Verification and change-impact analysis of access-control policies. In: Proc. 27th International Conference on Software Engineering, pp. 196–205 (2005)Google Scholar
  17. 17.
    Geist, R., Offutt, A.J., Harris, F.: Estimation and enhancement of real-time software reliability through mutation analysis. IEEE Transactions on Computers 41(5), 550–558 (1992)CrossRefGoogle Scholar
  18. 18.
    Greenberg, M.M., Marks, C., Meyerovich, L.A., Tschantz, M.C.: The soundness and completeness of Margrave with respect to a subset of XACML. Technical Report CS-05-05, Department of Computer Science, Brown University (2005)Google Scholar
  19. 19.
    Griffiths, P., Wade, B.: An authorization mechanism for a relational database systems. ACM Transactions on Database Systems 1(3) (1976)Google Scholar
  20. 20.
    Harrold, M.J., Gupta, R., Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2(3), 270–285 (1993)CrossRefGoogle Scholar
  21. 21.
    Hennessy, M., Power, J.F.: An analysis of rule coverage as a criterion in generating minimal test suites for grammar-based software. In: Proc. 20th IEEE/ACM International Conference on Automated Software Engineering, November 2005, pp. 104–113 (2005)Google Scholar
  22. 22.
    Hughes, G., Bultan, T.: Automated verification of access control policies. Technical Report 2004-22, Department of Computer Science, University of California, Santa Barbara (2004)Google Scholar
  23. 23.
    Jackson, D., Shlyakhter, I., Sridharan, M.: A micromodularity mechanism. In: Proc. 8th ESEC/FSE, pp. 62–73 (2001)Google Scholar
  24. 24.
    Jaeger, T., Zhang, X., Cacheda, F.: Policy management using access control spaces. ACM Transactions on Information and System Security 6(3) (2003)Google Scholar
  25. 25.
    Jajodia, S., Samarati, P., Subrahmanian, V.S.: A logical language for expressing authorizations. In: Proc. 1997 IEEE Symposium on Security and Privacy, pp. 31–42 (1997)Google Scholar
  26. 26.
    Jajodia, S., Samarati, P., Subrahmanian, V.S., Bertino, E.: A unified framework for enforcing multiple access control policies. In: Proc. ACM SIGMOD International Conference on Management of Data, pp. 474–485 (1997)Google Scholar
  27. 27.
    Johnson, D.S.: Approximation algorithms for combinatorial problems. J. Comput. System Sci. 9, 256–278 (1974)MATHMathSciNetCrossRefGoogle Scholar
  28. 28.
    Kapfhammer, G.M., Soffa, M.L.: A family of test adequacy criteria for database-driven applications. In: Proceedings of the 9th ESEC/FSE, pp. 98–107 (2003)Google Scholar
  29. 29.
    Kudo, M., Hada, S.: XML document security based on provisional authorization. In: Proc. ACM Conference on Computer and Communication Security, Athens, Greece, November 2000, pp. 87–96 (2000)Google Scholar
  30. 30.
    Lorch, M., Kafura, D., Shah, S.: An XACML-based policy management and authorization service for globus resources. In: Proc. International Workshop on Grid Computing, Phoenix, AZ, November 2003, pp. 208–212 (2003)Google Scholar
  31. 31.
    Lupu, E.C., Sloman, M.: Conflict in policy-based distributed systems management. IEEE Transaction on Software Engineering 25(6), 852–869 (1999)CrossRefGoogle Scholar
  32. 32.
    Moses, T., Anderson, A., Proctor, S., Godik, S.: XACML Profile for Web-Services (WSPL). OASIS Working Draft (September 2003)Google Scholar
  33. 33.
    Myers, G.J.: Art of Software Testing. John Wiley & Sons, Inc., Chichester (1979)Google Scholar
  34. 34.
    Offutt, J., Untch, R.H.: Mutation 2000: Uniting the orthogonal. In: Mutation 2000: Mutation Testing in the Twentieth and the Twenty First Centuries, October 2000, pp. 45–55 (2000)Google Scholar
  35. 35.
    Rothermel, G., Harrold, M.J., Ostrin, J., Hong, C.: An empirical study of the effects of minimization on the fault detection capabilities of test suites. In: Proc. International Conference on Software Maintenance, pp. 34–43 (1998)Google Scholar
  36. 36.
    Ryutov, T., Neuman, C.: Representation and evaluation of security policies for distributed system services. In: Proc. DARPA Information Survivability Conference and Exposition, January 2000, pp. 172–183 (2000)Google Scholar
  37. 37.
    Sandhu, R., Bhamidipati, V., Munawer, Q.: The ARBAC97 model for role-based aministration of roles. ACM Transactions on Information and Systems Security 2(1), 105–135 (1999)CrossRefGoogle Scholar
  38. 38.
    Sirer, E., Wang, K.: An access control language for web services. In: Proc. 7th ACM Symposium on Access Control Models and Technologies, Monterey, CA, June 2002, pp. 23–30 (2002)Google Scholar
  39. 39.
    Suarez-Cabal, M.J., Tuya, J.: Using an SQL coverage measurement for testing database applications. In: Proc. ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 253–262 (2004)Google Scholar
  40. 40.
    Voas, J.M.: PIE: A dynamic failure-based technique. IEEE Transactions on Software Engineering 18(8), 717–727 (1992)CrossRefGoogle Scholar
  41. 41.
    Zhang, N., Ryan, M., Guelev, D.P.: Synthesising verified access control systems in XACML. In: Proc. 2004 ACM workshop on Formal Methods in Security Engineering, pp. 56–65 (2004)Google Scholar
  42. 42.
    Zhang, N., Ryan, M., Guelev, D.P.: Evaluating access control policies through model checking. In: Proc. 8th International Conference on Information Security, September 2005, pp. 446–460 (2005)Google Scholar
  43. 43.
    Zhu, H., Hall, P.A.V., May, J.H.R.: Software unit test coverage and adequacy. ACM Comput. Surv. 29(4), 366–427 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Evan Martin
    • 1
  • Tao Xie
    • 1
  • Ting Yu
    • 1
  1. 1.Department of Computer ScienceNorth Carolina State UniversityRaleigh

Personalised recommendations