Skip to main content
SpringerLink
Log in

Measuring and achieving test coverage of attack simulations extended version

  • Special Section Paper
  • Published:
Software and Systems Modeling Aims and scope Submit manuscript

Abstract

Designing secure and reliable systems is a difficult task. Threat modeling is a process that supports the secure design of systems by easing the understanding of the system’s complexity, as well as identifying and modeling potential threats. These threat models can serve as input for attack simulations, which are used to analyze the behavior of attackers within the system. To ensure the correct functionality of these attack simulations, automated tests are designed that check if an attacker can reach a certain point in the threat model. Currently, there is no way for developers to estimate the degree to which their tests cover the attack simulations and, thus, they cannot determine the quality of their tests. To resolve this shortcoming, we analyze structural testing methods from the software engineering domain and transfer them to the threat modeling domain by following an action design research approach. Further, we develop a first prototype, which is able to assess the test coverage in an automated way and provide a first approach to achieve full coverage. This will enable threat modeler to determine the quality of their tests and, simultaneously, increase the quality of the threat models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. We are aware that defenses might be effective by a probability. However, testing non-deterministic behavior is challenging and as we are evaluating a threat modeling language and not an actual organizational structure, the testing would just reveal if the probability distributions were correctly implemented in the compiler and not if the values are correctly chosen. Thus, we make the assumption that a defense is either effective or not.

  2. https://github.com/nicklashersen/malcompiler.

  3. https://github.com/nicklashersen/mal-coverage-viewer.

  4. https://github.com/mal-lang/.

References

  1. Aldea, A., Vaicekauskaitė, E., Daneva, M., Sebastian Piest, J.P.: Assessing resilience in enterprise architecture: a systematic review. In: EDOC, pp. 1–10 (2020). https://doi.org/10.1109/EDOC49727.2020.00011

  2. Ammann, P., Offutt, J.: Introduction to Software Testing, 2nd edn. Cambridge University Press, New York (2016)

    Book  Google Scholar 

  3. Bach-Nutman, M.: Understanding the top 10 owasp vulnerabilities. arXiv: 2012.09960 (2020)

  4. Band, I., Engelsman, W., Feltus, C., Paredes, S.G., Diligens, D.: Modeling Enterprise Risk Management and Security with the Archimate®. Language, The Open Group (2015)

  5. Bertolino, A., Marré, M.: Automatic generation of path covers based on the control flow analysis of computer programs. IEEE Trans. Softw. Eng. 20(12), 885–899 (1994)

    Article  Google Scholar 

  6. Bishop, M.: About penetration testing. IEEE Secur. Priv. 5(6), 84–87 (2007)

    Article  Google Scholar 

  7. Caralli, R.A., Stevens, J.F., Young, L.R., Wilson, W.R.: Introducing octave allegro: improving the information security risk assessment process. Carnegie-Mellon Univ Pittsburgh PA Software Engineering Inst, Technical report (2007)

  8. Chi, S.D., Park, J.S., Jung, K.C., Lee, J.S.: Network security modeling and cyber attack simulation methodology. In: Australasian Conference on Information Security and Privacy, pp. 320–333. Springer (2001)

  9. Choo, K.K.R.: The cyber threat landscape: challenges and future research directions. Comput. Secur. 30(8), 719–731 (2011)

    Article  Google Scholar 

  10. Cohen, F.: Simulating cyber attacks, defences and consequences. Comput. Secur. 18(6), 479–518 (1999)

    Article  Google Scholar 

  11. Committee of Sponsoring Organizations of the Treadway Commission, et al.: Enterprise risk management-integrated framework: executive summary and framework. American Institute of Certified Public Accountants (AICPA) (2004)

  12. Deng, M., Wuyts, K., Scandariato, R., Preneel, B., Joosen, W.: A privacy threat analysis framework: supporting the elicitation and fulfillment of privacy requirements. Req. Eng. 16(1), 3–32 (2011)

    Article  Google Scholar 

  13. Ekstedt, M., Johnson, P., Lagerström, R., Gorton, D., Nydrén, J., Shahzad, K.: securiCAD by foreseeti: a CAD tool for enterprise cyber security management. In: 19th International EDOC Workshop, pp. 152–155. IEEE (2015)

  14. Fögen, K.: Combinatorial robustness testing based on error-constraints. Ph.D. thesis, Dissertation, RWTH Aachen University (2021)

  15. Freund, J., Jones, J.: Measuring and Managing Information Risk. Butterworth-Heinemann, Waltham (2015). https://doi.org/10.1016/C2013-0-09966-5

    Book  Google Scholar 

  16. Geers, K.: The cyber threat to national critical infrastructures: beyond theory. Inf. Secur. J. A Glob. Perspect. 18(1), 1–7 (2009)

    Article  Google Scholar 

  17. Goodenough, J.B., Gerhart, S.L.: Toward a theory of test data selection. IEEE Trans. Softw. Eng. 2, 156–173 (1975)

    Article  MathSciNet  Google Scholar 

  18. Grandry, E., Feltus, C., Dubois, E.: Conceptual integration of enterprise architecture management and security risk management. In: EDOCW, pp. 114–123 (2013). https://doi.org/10.1109/EDOCW.2013.19

  19. Grov, G., Mancini, F., Mestl, E.M.S.: Challenges for risk and security modelling in enterprise architecture. In: PoEM, pp. 215–225. Springer (2019)

  20. Gulati, S., Sharma, R.: JUnit 5 Extension Model, pp. 121–137. Apress, Berkeley (2017)

  21. Hacks, S., Hacks, A., Katsikeas, S., Klaer, B., Lagerström, R.: Creating meta attack language instances using archimate: applied to electric power and energy system cases. In: 23rd International EDOC, pp. 88–97 (2019)

  22. Hacks, S., Katsikeas, S., Ling, E., Lagerström, R., Ekstedt, M.: powerlang: a probabilistic attack simulation language for the power domain. Energy Inform 3(1) (2020)

  23. Hersén, N., Hacks, S., Fögen, K.: Towards measuring test coverage of attack simulations. In: Augusto, A., Gill, A., Nurcan, S., Reinhartz-Berger, I., Schmidt, R., Zdravkovic, J. (eds.) Enterprise, Business-Process and Information Systems Modeling, pp. 303–317. Springer, Cham (2021)

    Chapter  Google Scholar 

  24. Hevner, A.R., March, S.T., Park, J., Ram, S.: Design science in information systems research. MIS Q. 28(1), 75–105 (2004)

    Article  Google Scholar 

  25. Holm, H., Shahzad, K., Buschle, M., Ekstedt, M.: P\(^2\)CySeMoL: predictive, probabilistic cyber security modeling language. IEEE Trans. Depend. Secure Comput. 12(6), 626–639 (2015)

    Article  Google Scholar 

  26. ISO: ISO 31000:2018 Risk management-Guidelines. Standard, International Organization for Standardization (2018)

  27. ISO/IEC: ISO/IEC 27005:2018 Information technology-Security techniques-Information security risk management. Standard, International Organization for Standardization/International Electrotechnical Commission (2018)

  28. Johnson, P., Lagerström, R., Ekstedt, M.: A meta language for threat modeling and attack simulations. In: 13th ARES Conference, pp. 1–8 (2018)

  29. Jürjens, J.: Umlsec: extending uml for secure systems development. In: International Conference on the Unified Modeling Language, pp. 412–425. Springer (2002)

  30. Jürjens, J., Schreck, J., Yu, Y.: Automated analysis of permission-based security using umlsec. In: International Conference on Fundamental Approaches to Software Engineering, pp. 292–295. Springer (2008)

  31. Katsikeas, S., Hacks, S., Johnson, P., Ekstedt, M., Lagerström, R., Jacobsson, J., Wällstedt, M., Eliasson, P.: An attack simulation language for the it domain. In: Graphical Models for Security, pp. 67–86. Springer, Cham (2020)

  32. Katsikeas, S., Johnson, P., Hacks, S., Lagerström, R.: Probabilistic modeling and simulation of vehicular cyber attacks : an application of the meta attack language. In: 5th ICISSP (2019)

  33. Kordy, B., Mauw, S., Radomirović, S., Schweitzer, P.: Foundations of attack-defense trees. In: Degano, P., Etalle, S., Guttman, J. (eds.) Formal Aspects of Security and Trust, pp. 80–95. Springer, Berlin Heidelberg, Berlin, Heidelberg (2011)

    Chapter  Google Scholar 

  34. Limited, A.T.: Hostile risk decisions and capabilities-based analysis (2020). https://www.amenaza.com/downloads/docs/Hostile%20Risk%20Decisions.pdf

  35. Lodderstedt, T., Basin, D., Doser, J.: Secureuml: A uml-based modeling language for model-driven security, pp. 426–441. Springer (2002)

  36. Manzur, L., Ulloa, J.M., Sánchez, M., Villalobos, J.: xarchimate: enterprise architecture simulation, experimentation and analysis. Simulation 91(3), 276–301 (2015)

    Article  Google Scholar 

  37. Mauw, S., Oostdijk, M.: Foundations of attack trees. In: International Conference on Information Security and Cryptology, pp. 186–198. Springer (2005)

  38. Miranda, B., Bertolino, A.: Testing relative to usage scope: Revisiting software coverage criteria. ACM Trans. Softw. Eng. Methodol. 29(3) (2020)

  39. Morana, M.M., Uceda Vélez, T.: Risk Centric Threat Modeling: Process for Attack Simulation and Threat Analysis. Wiley, Hoboken (2015)

    Google Scholar 

  40. Myagmar, S., Lee, A.J., Yurcik, W.: Threat modeling as a basis for security requirements. In: SREIS, vol. 2005, pp. 1–8. Citeseer (2005)

  41. Nascimento, D.L.M., Viana, D.L., Neto, P., Martins, D., Garcia, V.C., Meira, S.: A systematic mapping study on domain-specific languages. In: ICSEA 2012, pp. 179–187 (2012)

  42. Ntafos, S., Hakimi, S.: On path cover problems in digraphs and applications to program testing. IEEE Trans. Softw. Eng. SE–5(5), 520–529 (1979). https://doi.org/10.1109/TSE.1979.234213

    Article  MathSciNet  MATH  Google Scholar 

  43. Offutt, J., Liu, S., Abdurazik, A., Ammann, P.: Generating test data from state-based specifications. Softw. Test. Verif. Reliab. 13(1), 25–53 (2003)

    Article  Google Scholar 

  44. Oladimeji, E.A., Supakkul, S., Chung, L.: Security threat modeling and analysis: a goal-oriented approach. In: SEA 2006, pp. 13–15. Citeseer (2006)

  45. Rencelj Ling, E., Ekstedt, M.: Generating threat models and attack graphs based on the iec 61850 system configuration description language. In: Proceedings of the 2021 ACM Workshop on Secure and Trustworthy Cyber-Physical Systems, SAT-CPS ’21, pp. 98–103. Association for Computing Machinery, New York (2021). https://doi.org/10.1145/3445969.3450421

  46. Ross, R.: Guide for conducting risk assessments. NIST Special Publication 800-30 Revision 1, National Institute of Standard and Technology (2012)

  47. Ruhroth, T., Jürjens, J.: Supporting security assurance in the context of evolution: Modular modeling and analysis with umlsec. In: 2012 IEEE 14th International Symposium on High-Assurance Systems Engineering, pp. 177–184. IEEE (2012)

  48. Saini, V., Duan, Q., Paruchuri, V.: Threat modeling using attack trees. J. Comput. Sci. Coll. 23(4), 124–131 (2008)

    Google Scholar 

  49. Saitta, P., Larcom, B., Eddington, M.: Trike v1 methodology document (2005)

  50. Schneier, B.: Attack trees. Dr. Dobb’s J 24(12), 21–29 (1999)

  51. Sein, M.K., Henfridsson, O., Purao, S., Rossi, M., Lindgren, R.: Action design research. MIS Q. 35(1), 37–56 (2011)

    Article  Google Scholar 

  52. Sheyner, O., Haines, J., Jha, S., Lippmann, R., Wing, J.M.: Automated generation and analysis of attack graphs. In: Proceedings 2002 IEEE Symposium on Security and Privacy, pp. 273–284. IEEE (2002)

  53. Shostack, A.: Experiences threat modeling at microsoft. Technical report, Microsoft (2008)

  54. Shostack, A.: Threat Modeling: Designing for Security. Wiley, Indianapolis (2014)

    Google Scholar 

  55. Smith, M., Szongott, C., Henne, B., Von Voigt, G.: Big data privacy issues in public social media. In: 2012 6th IEEE International Conference on Digital Ecosystems and Technologies (DEST), pp. 1–6. IEEE (2012)

  56. Tuglular, T., Kaya, Ö., Müftüoglu, C.A., Belli, F.: Directed acyclic graph modeling of security policies for firewall testing. In: International Conference on Secure Software Integration and Reliability Improvement, pp. 393–398. IEEE (2009)

  57. Venable, J.R., Pries-Heje, J., Baskerville, R.: Choosing a design science research methodology. In: ACIS 2017 (2017)

  58. Xiong, W., Hacks, S., Lagerström, R.: A method for assigning probability distributions in attack simulation languages. Complex Syst. Inf. Model. Q. 26, 55–77 (2021)

    Google Scholar 

  59. Xiong, W., Hacks, S., Lagerström, R.: A method for quality assessment of threat modeling languages: the case of enterpriselang. In: B. Barn, K. Sandkuhl, E.S. Asensio, J. Stirna (eds.) Proceedings of the Forum at Practice of Enterprise Modeling 2021 (PoEM-Forum 2021) (PoEM 2021), Riga, Latvia, November 24–26, 2021, CEUR Workshop Proceedings, vol. 3045, pp. 49–58. CEUR-WS.org (2021)

  60. Xiong, W., Lagerström, R.: Threat modeling—a systematic literature review. Comput. Secur. 84, 53–69 (2019)

    Article  Google Scholar 

  61. Xiong, W., Legrand, E., Åberg, O., Lagerström, R.: Cyber security threat modeling based on the mitre enterprise att &ck matrix. In: Software and Systems Modeling, pp. 1–21 (2021)

Download references

Author information

Authors and Affiliations

  1. The Maersk Mc-Kinney Moller Institute, University of Southern Denmark, Odense, Denmark

    Simon Hacks

  2. Uppsala University, Uppsala, Sweden

    Linus Persson

  3. KTH Royal Institute of Technology, Stockholm, Sweden

    Nicklas Hersén

Authors
  1. Simon Hacks
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Linus Persson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicklas Hersén
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simon Hacks.

Additional information

Communicated by Iris Reinhartz-Berger, Jelena Zdravkovic, and Asif Gill.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hacks, S., Persson, L. & Hersén, N. Measuring and achieving test coverage of attack simulations extended version. Softw Syst Model 22, 31–46 (2023). https://doi.org/10.1007/s10270-022-01042-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10270-022-01042-9

Keywords

Search

Navigation