The Role of Catalogues of Threats and Security Controls in Security Risk Assessment: An Empirical Study with ATM Professionals

  • Martina de Gramatica
  • Katsiaryna Labunets
  • Fabio Massacci
  • Federica Paci
  • Alessandra Tedeschi
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9013)


[Context and motivation] To remedy the lack of security expertise, industrial security risk assessment methods come with catalogues of threats and security controls. [Question/problem] We investigate in both qualitative and quantitative terms whether the use of catalogues of threats and security controls has an effect on the actual and perceived effectiveness of a security risk assessment method. In particular, we assessed the effect of using domain-specific versus domain-general catalogues on the actual and perceived efficacy of a security risk assessment method conducted by non-experts and compare it with the effect of running the same method by security experts but without catalogues.

[Principal ideas/results] The quantitative analysis shows that non-security experts who applied the method with catalogues identified threats and controls of the same quality of security experts without catalogues. The perceived ease of use was higher when participants used method without catalogues albeit only at 10 % significance level. The qualitative analysis indicates that security experts have different expectations from a catalogue than non-experts. Non-experts are mostly worried about the difficulty of navigating through the catalogue (the larger and less specific the worse it was) while expert users found it mostly useful to get a common terminology and a checklist that nothing was forgotten.

[Contribution] This paper sheds light on the important features of the catalogues and discuss how they contribute into risk assessment process.


Empirical study Security risk assessment methods MEM 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Information System Audit and Control Association: COBIT 5: A Business Framework for the Governance and Management of Enterprise IT (2012)Google Scholar
  2. 2.
    Barnum, S., McGraw, G.: Knowledge for software security. IEEE Security & Privacy 3(2), 74–78 (2005)CrossRefGoogle Scholar
  3. 3.
    BSI: IT-Grundschutz Catalogues (2005)Google Scholar
  4. 4.
    COBIT: Control Practices: Guidance to Achieve Control Objective for Successful IT Governance, 2nd edn. IT Governance Institute (2007)Google Scholar
  5. 5.
    Cysneiros, L.M.: Evaluating the effectiveness of using catalogues to elicit non-functional requirements. In: WER, pp. 107–115 (2007)Google Scholar
  6. 6.
    EATM: Threats, pre-controls and post-controls catalogues. European Organisation for the Safety of Air Navigation (2009)Google Scholar
  7. 7.
    ISO: Iso/iec 27005: Information technology security techniques - information security risk management (2012)Google Scholar
  8. 8.
    ISO: IEC 27002: 2013 (EN) Information technology-Security techniques-Code of practice for information security controls Switzerland. ISO/IEC (2013)Google Scholar
  9. 9.
    Jung, J., Hoefig, K., Domis, D., Jedlitschka, A., Hiller, M.: Experimental comparison of two safety analysis methods and its replication. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 223–232. IEEE (2013)Google Scholar
  10. 10.
    Juristo, N., Moreno, A.M.: Basics of software engineering experimentation. Springer Publishing Company, Incorporated (2010)Google Scholar
  11. 11.
    Karpati, P., Redda, Y., Opdahl, A.L., Sindre, G.: Comparing attack trees and misuse cases in an industrial setting. Inf. Soft. Technology 56(3), 294–308 (2014)CrossRefGoogle Scholar
  12. 12.
    Labunets, K., Massacci, F., Paci, F., Tran, L.M.: An experimental comparison of two risk-based security methods. In: Proc. of ESEM 2013, pp. 163–172 (2013)Google Scholar
  13. 13.
    Labunets, K., Paci, F., Massacci, F., Ruprai, R.: An experiment on comparing textual vs. visual industrial methods for security risk assessment. In: 2014 IEEE Fourth International Workshop on Empirical Requirements Engineering (EmpiRE), pp. 28–35. IEEE (2014)Google Scholar
  14. 14.
    Maiden, N., Robertson, S.: Integrating creativity into requirements processes: experiences with an air traffic management system. In: Proceedings of the 13th IEEE International Conference on Requirements Engineering, pp. 105–114. IEEE (2005)Google Scholar
  15. 15.
    Maiden, N.A.M., Jones, S.V., Manning, S., Greenwood, J., Renou, L.: Model-driven requirements engineering: synchronising models in an air traffic management case study. In: Persson, A., Stirna, J. (eds.) CAiSE 2004. LNCS, vol. 3084, pp. 368–383. Springer, Heidelberg (2004) CrossRefGoogle Scholar
  16. 16.
    Massacci, F., Paci, F., Tran, L.M.S., Tedeschi, A.: Assessing a requirements evolution approach: Empirical studies in the air traffic management domain. Journal of Systems and Software (2013)Google Scholar
  17. 17.
    Mavin, A., Maiden, N.: Determining socio-technical systems requirements: experiences with generating and walking through scenarios. In: Proceedings of the 11th IEEE International on Requirements Engineering Conference, pp. 213–222. IEEE (2003)Google Scholar
  18. 18.
    Meyer, J.P., Seaman, M.A.: A comparison of the exact kruskal-wallis distribution to asymptotic approximations for all sample sizes up to 105. The Journal of Experimental Education 81(2), 139–156 (2013)CrossRefGoogle Scholar
  19. 19.
    Moody, D.L.: The method evaluation model: a theoretical model for validating information systems design methods. In: Proceedings of the 11th European Conference of Information Systems (ECIS), pp. 1327–1336 (2003)Google Scholar
  20. 20.
    NIST: SP. 800–53. Recommended Security Controls for Federal Information Systems, 800-53 (2013)Google Scholar
  21. 21.
    Opdahl, A.L., Sindre, G.: Experimental comparison of attack trees and misuse cases for security threat identification. Inf. Soft. Technology 51(5), 916–932 (2009)CrossRefGoogle Scholar
  22. 22.
    OWASP: The Ten Most Critical Web Application Security Risks 2013. The Open Web Application Security Project (2013)Google Scholar
  23. 23.
    PCI DSS: Payment Card Industry Data Security Standards.
  24. 24.
    Scandariato, R., Wuyts, K., Joosen, W.: A descriptive study of microsoft’s threat modeling technique. REJ, pp. 1–18 (2014)Google Scholar
  25. 25.
    SESAR: ATM Security Risk Assessment Methodology. SESAR WP16.02.03: ATM Security, February 2003Google Scholar
  26. 26.
    SESAR: Single Remote Tower Technical Specification Remotely Operated Tower Multiple Controlled Airports with Integrated Working Position - project P12.04.07 (2012)Google Scholar
  27. 27.
    SESAR: OSED for Remote Provision of ATS to Aerodromes - project P06.09.03 (2013)Google Scholar
  28. 28.
    Stoneburner, G., Goguen, A., Feringa, A.: Risk management guide for information technology systems. NIST special publication, 800-30 (2002)Google Scholar
  29. 29.
    Strauss, A., Corbin, J.M.: Basics of qualitative research: Grounded theory procedures and techniques. Sage Publications, Inc (1990)Google Scholar
  30. 30.
    Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in software engineering. Springer (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Martina de Gramatica
    • 1
  • Katsiaryna Labunets
    • 1
  • Fabio Massacci
    • 1
  • Federica Paci
    • 1
  • Alessandra Tedeschi
    • 2
  1. 1.DISIUniversity of TrentoTrentoItaly
  2. 2.Deep Blue srlRomaItaly

Personalised recommendations