The Role of Catalogues of Threats and Security Controls in Security Risk Assessment: An Empirical Study with ATM Professionals
[Context and motivation] To remedy the lack of security expertise, industrial security risk assessment methods come with catalogues of threats and security controls. [Question/problem] We investigate in both qualitative and quantitative terms whether the use of catalogues of threats and security controls has an effect on the actual and perceived effectiveness of a security risk assessment method. In particular, we assessed the effect of using domain-specific versus domain-general catalogues on the actual and perceived efficacy of a security risk assessment method conducted by non-experts and compare it with the effect of running the same method by security experts but without catalogues.
[Principal ideas/results] The quantitative analysis shows that non-security experts who applied the method with catalogues identified threats and controls of the same quality of security experts without catalogues. The perceived ease of use was higher when participants used method without catalogues albeit only at 10 % significance level. The qualitative analysis indicates that security experts have different expectations from a catalogue than non-experts. Non-experts are mostly worried about the difficulty of navigating through the catalogue (the larger and less specific the worse it was) while expert users found it mostly useful to get a common terminology and a checklist that nothing was forgotten.
[Contribution] This paper sheds light on the important features of the catalogues and discuss how they contribute into risk assessment process.
KeywordsEmpirical study Security risk assessment methods MEM
Unable to display preview. Download preview PDF.
- 1.Information System Audit and Control Association: COBIT 5: A Business Framework for the Governance and Management of Enterprise IT (2012)Google Scholar
- 3.BSI: IT-Grundschutz Catalogues (2005)Google Scholar
- 4.COBIT: Control Practices: Guidance to Achieve Control Objective for Successful IT Governance, 2nd edn. IT Governance Institute (2007)Google Scholar
- 5.Cysneiros, L.M.: Evaluating the effectiveness of using catalogues to elicit non-functional requirements. In: WER, pp. 107–115 (2007)Google Scholar
- 6.EATM: Threats, pre-controls and post-controls catalogues. European Organisation for the Safety of Air Navigation (2009)Google Scholar
- 7.ISO: Iso/iec 27005: Information technology security techniques - information security risk management (2012)Google Scholar
- 8.ISO: IEC 27002: 2013 (EN) Information technology-Security techniques-Code of practice for information security controls Switzerland. ISO/IEC (2013)Google Scholar
- 9.Jung, J., Hoefig, K., Domis, D., Jedlitschka, A., Hiller, M.: Experimental comparison of two safety analysis methods and its replication. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 223–232. IEEE (2013)Google Scholar
- 10.Juristo, N., Moreno, A.M.: Basics of software engineering experimentation. Springer Publishing Company, Incorporated (2010)Google Scholar
- 12.Labunets, K., Massacci, F., Paci, F., Tran, L.M.: An experimental comparison of two risk-based security methods. In: Proc. of ESEM 2013, pp. 163–172 (2013)Google Scholar
- 13.Labunets, K., Paci, F., Massacci, F., Ruprai, R.: An experiment on comparing textual vs. visual industrial methods for security risk assessment. In: 2014 IEEE Fourth International Workshop on Empirical Requirements Engineering (EmpiRE), pp. 28–35. IEEE (2014)Google Scholar
- 14.Maiden, N., Robertson, S.: Integrating creativity into requirements processes: experiences with an air traffic management system. In: Proceedings of the 13th IEEE International Conference on Requirements Engineering, pp. 105–114. IEEE (2005)Google Scholar
- 16.Massacci, F., Paci, F., Tran, L.M.S., Tedeschi, A.: Assessing a requirements evolution approach: Empirical studies in the air traffic management domain. Journal of Systems and Software (2013)Google Scholar
- 17.Mavin, A., Maiden, N.: Determining socio-technical systems requirements: experiences with generating and walking through scenarios. In: Proceedings of the 11th IEEE International on Requirements Engineering Conference, pp. 213–222. IEEE (2003)Google Scholar
- 19.Moody, D.L.: The method evaluation model: a theoretical model for validating information systems design methods. In: Proceedings of the 11th European Conference of Information Systems (ECIS), pp. 1327–1336 (2003)Google Scholar
- 20.NIST: SP. 800–53. Recommended Security Controls for Federal Information Systems, 800-53 (2013)Google Scholar
- 22.OWASP: The Ten Most Critical Web Application Security Risks 2013. The Open Web Application Security Project (2013)Google Scholar
- 23.PCI DSS: Payment Card Industry Data Security Standards. http://www.pcisecuritystandards.org
- 24.Scandariato, R., Wuyts, K., Joosen, W.: A descriptive study of microsoft’s threat modeling technique. REJ, pp. 1–18 (2014)Google Scholar
- 25.SESAR: ATM Security Risk Assessment Methodology. SESAR WP16.02.03: ATM Security, February 2003Google Scholar
- 26.SESAR: Single Remote Tower Technical Specification Remotely Operated Tower Multiple Controlled Airports with Integrated Working Position - project P12.04.07 (2012)Google Scholar
- 27.SESAR: OSED for Remote Provision of ATS to Aerodromes - project P06.09.03 (2013)Google Scholar
- 28.Stoneburner, G., Goguen, A., Feringa, A.: Risk management guide for information technology systems. NIST special publication, 800-30 (2002)Google Scholar
- 29.Strauss, A., Corbin, J.M.: Basics of qualitative research: Grounded theory procedures and techniques. Sage Publications, Inc (1990)Google Scholar
- 30.Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in software engineering. Springer (2012)Google Scholar