Abstract
[Context and motivation] To remedy the lack of security expertise, industrial security risk assessment methods come with catalogues of threats and security controls. [Question/problem] We investigate in both qualitative and quantitative terms whether the use of catalogues of threats and security controls has an effect on the actual and perceived effectiveness of a security risk assessment method. In particular, we assessed the effect of using domain-specific versus domain-general catalogues on the actual and perceived efficacy of a security risk assessment method conducted by non-experts and compare it with the effect of running the same method by security experts but without catalogues.
[Principal ideas/results] The quantitative analysis shows that non-security experts who applied the method with catalogues identified threats and controls of the same quality of security experts without catalogues. The perceived ease of use was higher when participants used method without catalogues albeit only at 10 % significance level. The qualitative analysis indicates that security experts have different expectations from a catalogue than non-experts. Non-experts are mostly worried about the difficulty of navigating through the catalogue (the larger and less specific the worse it was) while expert users found it mostly useful to get a common terminology and a checklist that nothing was forgotten.
[Contribution] This paper sheds light on the important features of the catalogues and discuss how they contribute into risk assessment process.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Information System Audit and Control Association: COBIT 5: A Business Framework for the Governance and Management of Enterprise IT (2012)
Barnum, S., McGraw, G.: Knowledge for software security. IEEE Security & Privacy 3(2), 74–78 (2005)
BSI: IT-Grundschutz Catalogues (2005)
COBIT: Control Practices: Guidance to Achieve Control Objective for Successful IT Governance, 2nd edn. IT Governance Institute (2007)
Cysneiros, L.M.: Evaluating the effectiveness of using catalogues to elicit non-functional requirements. In: WER, pp. 107–115 (2007)
EATM: Threats, pre-controls and post-controls catalogues. European Organisation for the Safety of Air Navigation (2009)
ISO: Iso/iec 27005: Information technology security techniques - information security risk management (2012)
ISO: IEC 27002: 2013 (EN) Information technology-Security techniques-Code of practice for information security controls Switzerland. ISO/IEC (2013)
Jung, J., Hoefig, K., Domis, D., Jedlitschka, A., Hiller, M.: Experimental comparison of two safety analysis methods and its replication. In: 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 223–232. IEEE (2013)
Juristo, N., Moreno, A.M.: Basics of software engineering experimentation. Springer Publishing Company, Incorporated (2010)
Karpati, P., Redda, Y., Opdahl, A.L., Sindre, G.: Comparing attack trees and misuse cases in an industrial setting. Inf. Soft. Technology 56(3), 294–308 (2014)
Labunets, K., Massacci, F., Paci, F., Tran, L.M.: An experimental comparison of two risk-based security methods. In: Proc. of ESEM 2013, pp. 163–172 (2013)
Labunets, K., Paci, F., Massacci, F., Ruprai, R.: An experiment on comparing textual vs. visual industrial methods for security risk assessment. In: 2014 IEEE Fourth International Workshop on Empirical Requirements Engineering (EmpiRE), pp. 28–35. IEEE (2014)
Maiden, N., Robertson, S.: Integrating creativity into requirements processes: experiences with an air traffic management system. In: Proceedings of the 13th IEEE International Conference on Requirements Engineering, pp. 105–114. IEEE (2005)
Maiden, N.A.M., Jones, S.V., Manning, S., Greenwood, J., Renou, L.: Model-driven requirements engineering: synchronising models in an air traffic management case study. In: Persson, A., Stirna, J. (eds.) CAiSE 2004. LNCS, vol. 3084, pp. 368–383. Springer, Heidelberg (2004)
Massacci, F., Paci, F., Tran, L.M.S., Tedeschi, A.: Assessing a requirements evolution approach: Empirical studies in the air traffic management domain. Journal of Systems and Software (2013)
Mavin, A., Maiden, N.: Determining socio-technical systems requirements: experiences with generating and walking through scenarios. In: Proceedings of the 11th IEEE International on Requirements Engineering Conference, pp. 213–222. IEEE (2003)
Meyer, J.P., Seaman, M.A.: A comparison of the exact kruskal-wallis distribution to asymptotic approximations for all sample sizes up to 105. The Journal of Experimental Education 81(2), 139–156 (2013)
Moody, D.L.: The method evaluation model: a theoretical model for validating information systems design methods. In: Proceedings of the 11th European Conference of Information Systems (ECIS), pp. 1327–1336 (2003)
NIST: SP. 800–53. Recommended Security Controls for Federal Information Systems, 800-53 (2013)
Opdahl, A.L., Sindre, G.: Experimental comparison of attack trees and misuse cases for security threat identification. Inf. Soft. Technology 51(5), 916–932 (2009)
OWASP: The Ten Most Critical Web Application Security Risks 2013. The Open Web Application Security Project (2013)
PCI DSS: Payment Card Industry Data Security Standards. http://www.pcisecuritystandards.org
Scandariato, R., Wuyts, K., Joosen, W.: A descriptive study of microsoft’s threat modeling technique. REJ, pp. 1–18 (2014)
SESAR: ATM Security Risk Assessment Methodology. SESAR WP16.02.03: ATM Security, February 2003
SESAR: Single Remote Tower Technical Specification Remotely Operated Tower Multiple Controlled Airports with Integrated Working Position - project P12.04.07 (2012)
SESAR: OSED for Remote Provision of ATS to Aerodromes - project P06.09.03 (2013)
Stoneburner, G., Goguen, A., Feringa, A.: Risk management guide for information technology systems. NIST special publication, 800-30 (2002)
Strauss, A., Corbin, J.M.: Basics of qualitative research: Grounded theory procedures and techniques. Sage Publications, Inc (1990)
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in software engineering. Springer (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
de Gramatica, M., Labunets, K., Massacci, F., Paci, F., Tedeschi, A. (2015). The Role of Catalogues of Threats and Security Controls in Security Risk Assessment: An Empirical Study with ATM Professionals. In: Fricker, S., Schneider, K. (eds) Requirements Engineering: Foundation for Software Quality. REFSQ 2015. Lecture Notes in Computer Science(), vol 9013. Springer, Cham. https://doi.org/10.1007/978-3-319-16101-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-319-16101-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-16100-6
Online ISBN: 978-3-319-16101-3
eBook Packages: Computer ScienceComputer Science (R0)