Skip to main content
SpringerLink
Log in

Towards Measuring Test Coverage of Attack Simulations

  • Conference paper
  • First Online:
Enterprise, Business-Process and Information Systems Modeling (BPMDS 2021, EMMSAD 2021)

Part of the book series: Lecture Notes in Business Information Processing ((LNBIP,volume 421))

Abstract

Designing secure and reliable systems is a difficult task. Threat modeling is a process that supports the secure design of systems by easing the understanding of the system’s complexity, as well as identifying and modeling potential threats. These threat models can serve as input for attack simulations, which are used to analyze the behavior of attackers within the system. To ensure the correct functionality of these attack simulations, automated tests are designed that check if an attacker can reach a certain point in the threat model. Currently, there is no way for developers to estimate the degree to which their tests cover the attack simulations and, thus, they cannot the determine the quality of their tests. To resolve this shortcoming, we analyze structural testing methods from the software engineering domain and transfer them to the threat modeling domain by following an Action Design Research approach. Further, we develop a first prototype, which is able to assess the test coverage in an automated way. This will enable threat modeler to determine the quality of their tests and, simultaneously, increase the quality of the threat models.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    We are aware that defenses might be effective by a probability. However, testing non-deterministic behavior is challenging and, thus, we make the assumption that a defense is either effective or not.

  2. 2.

    https://github.com/nicklashersen/malcompiler.

  3. 3.

    https://github.com/nicklashersen/mal-coverage-viewer.

References

  1. Ammann, P., Offutt, J.: Introduction to Software Testing, 2nd edn. Cambridge University Press, New York (2016)

    Book  Google Scholar 

  2. Chi, S.-D., Park, J.S., Jung, K.-C., Lee, J.-S.: Network security modeling and cyber attack simulation methodology. In: Varadharajan, V., Mu, Y. (eds.) ACISP 2001. LNCS, vol. 2119, pp. 320–333. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-47719-5_26

    Chapter  Google Scholar 

  3. Cohen, F.: Simulating cyber attacks, defences and consequences. Comput. Secur. 18(6), 479–518 (1999)

    Article  Google Scholar 

  4. Ekstedt, M., Johnson, P., Lagerström, R., Gorton, D., Nydrén, J., Shahzad, K.: SecuriCAD by Foreseeti: a CAD tool for enterprise cyber security management. In: 19th International EDOC Workshop, pp. 152–155. IEEE (2015)

    Google Scholar 

  5. Goodenough, J.B., Gerhart, S.L.: Toward a theory of test data selection. IEEE Trans. Softw. Eng. 2, 156–173 (1975)

    Article  MathSciNet  Google Scholar 

  6. Gulati, S., Sharma, R.: JUnit 5 Extension Model, pp. 121–137. Apress, Berkeley (2017)

    Google Scholar 

  7. Hacks, S., Hacks, A., Katsikeas, S., Klaer, B., Lagerström, R.: Creating meta attack language instances using ArchiMate: applied to electric power and energy system cases. In: 23rd International EDOC, pp. 88–97 (2019)

    Google Scholar 

  8. Hacks, S., Katsikeas, S., Ling, E., Lagerström, R., Ekstedt, M.: powerlang: a probabilistic attack simulation language for the power domain. Energy Inform. 3(1), 30 (2020). https://doi.org/10.1186/s42162-020-00134-4

  9. Holm, H., Shahzad, K., Buschle, M., Ekstedt, M.: P\(^2\)CySeMoL: predictive, probabilistic cyber security modeling language. IEEE Trans. Dependable Secure Comput. 12(6), 626–639 (2015)

    Article  Google Scholar 

  10. Johnson, P., Lagerström, R., Ekstedt, M.: A meta language for threat modeling and attack simulations. In: 13th ARES Conference, pp. 1–8 (2018)

    Google Scholar 

  11. Jürjens, J.: UMLsec: extending UML for secure systems development. In: Jézéquel, J.-M., Hussmann, H., Cook, S. (eds.) UML 2002. LNCS, vol. 2460, pp. 412–425. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-45800-X_32

    Chapter  MATH  Google Scholar 

  12. Katsikeas, S., et al.: An attack simulation language for the IT domain. In: Eades III, H., Gadyatskaya, O. (eds.) GraMSec 2020. LNCS, vol. 12419, pp. 67–86. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62230-5_4

    Chapter  Google Scholar 

  13. Katsikeas, S., Johnson, P., Hacks, S., Lagerström, R.: Probabilistic modeling and simulation of vehicular cyber attacks: an application of the meta attack language. In: 5th ICISSP (2019)

    Google Scholar 

  14. Kordy, B., Mauw, S., Radomirović, S., Schweitzer, P.: Foundations of attack–defense trees. In: Degano, P., Etalle, S., Guttman, J. (eds.) FAST 2010. LNCS, vol. 6561, pp. 80–95. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19751-2_6

    Chapter  Google Scholar 

  15. Limited, A.T.: Hostile risk decisions and capabilities-based analysis (2020). https://www.amenaza.com/downloads/docs/Hostile

  16. Mauw, S., Oostdijk, M.: Foundations of attack trees. In: Won, D.H., Kim, S. (eds.) ICISC 2005. LNCS, vol. 3935, pp. 186–198. Springer, Heidelberg (2006). https://doi.org/10.1007/11734727_17

    Chapter  Google Scholar 

  17. Miranda, B., Bertolino, A.: Testing relative to usage scope: Revisiting software coverage criteria. ACM Trans. Softw. Eng. Methodol. 29(3), 1–24 (2020)

    Article  Google Scholar 

  18. Myagmar, S., Lee, A.J., Yurcik, W.: Threat modeling as a basis for security requirements. In: SREIS, vol. 2005, pp. 1–8. Citeseer (2005)

    Google Scholar 

  19. do Nascimento, L.M., Viana, D.L., Neto, P., Martins, D., Garcia, V.C., Meira, S.: A systematic mapping study on domain-specific languages. In: ICSEA 2012, pp. 179–187 (2012)

    Google Scholar 

  20. Offutt, J., Liu, S., Abdurazik, A., Ammann, P.: Generating test data from state-based specifications. Softw. Test. Verif. Reliab. 13(1), 25–53 (2003)

    Article  Google Scholar 

  21. Oladimeji, E.A., Supakkul, S., Chung, L.: Security threat modeling and analysis: a goal-oriented approach. In: SEA 2006, pp. 13–15. Citeseer (2006)

    Google Scholar 

  22. Saini, V., Duan, Q., Paruchuri, V.: Threat modeling using attack trees. J. Comput. Sci. Coll. 23(4), 124–131 (2008)

    Google Scholar 

  23. Schneier, B.: Attack trees. Dr. Dobb’s J. 24(12), 21–29 (1999)

    Google Scholar 

  24. Sein, M.K., Henfridsson, O., Purao, S., Rossi, M., Lindgren, R.: Action design research. MIS Q. 35(1), 37–56 (2011)

    Article  Google Scholar 

  25. Sheyner, O., Haines, J., Jha, S., Lippmann, R., Wing, J.M.: Automated generation and analysis of attack graphs. In: Proceedings 2002 IEEE Symposium on Security and Privacy, pp. 273–284. IEEE (2002)

    Google Scholar 

  26. Shostack, A.: Threat Modeling: Designing for Security. Wiley, Indianapolis (2014)

    Google Scholar 

  27. Tuglular, T., Kaya, Ö., Müftüoglu, C.A., Belli, F.: Directed acyclic graph modeling of security policies for firewall testing. In: International Conference on Secure Software Integration and Reliability Improvement, pp. 393–398. IEEE (2009)

    Google Scholar 

Download references

Acknowledgement

This project has received funding from the European Union’s H2020 research and innovation programme under the Grant Agreement No. 832907.

Author information

Authors and Affiliations

  1. Division of Network and Systems Engineering, KTH Royal Institute of Technology, Stockholm, Sweden

    Nicklas Hersén & Simon Hacks

  2. Research Group Software Construction, RWTH Aachen University, Aachen, Germany

    Konrad Fögen

Authors
  1. Nicklas Hersén
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simon Hacks
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Konrad Fögen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Simon Hacks .

Editor information

Editors and Affiliations

  1. University of Melbourne, Parkville, VIC, Australia

    Adriano Augusto

  2. University of Technology Sydney, Ultimo, NSW, Australia

    Asif Gill

  3. University Paris 1 Panthéon-Sorbonne, Paris, France

    Selmin Nurcan

  4. University of Haifa, Haifa, Israel

    Iris Reinhartz-Berger

  5. Munich University of Applied Sciences, Munich, Germany

    Rainer Schmidt

  6. Stockholm University, Kista, Sweden

    Jelena Zdravkovic

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hersén, N., Hacks, S., Fögen, K. (2021). Towards Measuring Test Coverage of Attack Simulations. In: Augusto, A., Gill, A., Nurcan, S., Reinhartz-Berger, I., Schmidt, R., Zdravkovic, J. (eds) Enterprise, Business-Process and Information Systems Modeling. BPMDS EMMSAD 2021 2021. Lecture Notes in Business Information Processing, vol 421. Springer, Cham. https://doi.org/10.1007/978-3-030-79186-5_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-79186-5_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-79185-8

  • Online ISBN: 978-3-030-79186-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation