Abstract
There is growing interest in the use of systems-based risk assessment (RA) methods in human factors and ergonomics (HFE). Despite this, there has been a lack of formal reliability and validity testing undertaken to determine whether systems-based RA methods have the capacity to reliably and accurately identify potential risks within complex systems. The purpose of this study was to test the intra-rater reliability (within subject stability) and criterion-related validity (‘gold standard’ performance) of the Networked Hazard Analysis and Risk Management System (Net-HARMS). Net-HARMS is a new and innovative systems-based RA method that supports analysts with the identification of emergent risks. Emergent risks represent new risks that are created when risks from across a complex system interact with one another. Reliability and validity measures for Net-HARMS were obtained using the Signal Detection Theory (SDT) paradigm. Matthews Correlation Coefficient (MCC) was used to analyze the complete SDT data to measure the strength of the correlation between risks. Findings indicate a weak to moderate level of reliability and validity for Net-HARMS based on the MCC score. The results suggest that there is merit to the continued use of Net-HARMS following a series of methodological recommendations that aim to enhance the reliability and validity of future applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dallat, C., Salmon, P.M., Goode, N.: Risky systems versus risky people: to what extent do risk assessment methods consider the systems approach to accident causation? A review of the literature. Saf Sci 119, 266–279 (2017)
Leveson, N.: Engineering a Safer World: Systems Thinking Applied to Safety. MIT Press, Cambridge (2011)
Hollnagel, E.: FRAM, the Functional Resonance Analysis Method: Modelling Complex Socio-Technical Systems. Ashgate Publishing Ltd, Surrey (2012)
Stanton, N.A., Harvey, C.: Beyond human error taxonomies in assessment of risk in sociotechnical systems: a new paradigm with the EAST ‘broken-links’ approach. Ergonomics 60(2), 221–233 (2017)
Dallat, C., Salmon, P.M., Goode, N.: Identifying risks and emergent risks across sociotechnical systems: The NETworked hazard analysis and risk management system (NET-HARMS). Theoret. Issues Ergon Sci 19(4), 456–482 (2018)
Hulme, A., McLean, S., Dallat, C., Walker, G.H., Waterson, P., Stanton, N.A., Salmon, P.M. Systems thinking-based risk assessment methods applied to sports performance: A comparison of STPA, EAST-BL, and Net-HARMS in the context of elite women’s road cycling. Applied Ergonomics 91 (2020, in press)
Kirwan, B., Ainsworth, L.K.: A Guide to Task Analysis: The Task Analysis Working Group. CRC Press , Boca Raton (1992)
Stanton, N.A.: Hierarchical task analysis: Developments, applications, and extensions. Appl. Ergon. 37(1), 55–79 (2006)
Salmon, P.M., Read, G.J.M., Walker, G.H., Goode, N., Grant, E., Dallat, C., Carden, T., Naweed, A., Stanton, N.A.: STAMP goes EAST: integrating systems ergonomics methods for the analysis of railway level crossing safety management. Saf. Sci. 110, 31–46 (2018)
Green, D.M., Swets, J.A.: Signal Detection Theory and Psychophysics. Wiley , Cambridge (1966)
Chicco, D., Jurman, G.: The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genom. 21(1), 6 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hulme, A., Stanton, N.A., Walker, G.H., Waterson, P., Salmon, P.M. (2021). Testing the Reliability and Validity of Net-HARMS: A New Systems-Based Risk Assessment Method in HFE. In: Black, N.L., Neumann, W.P., Noy, I. (eds) Proceedings of the 21st Congress of the International Ergonomics Association (IEA 2021). IEA 2021. Lecture Notes in Networks and Systems, vol 219. Springer, Cham. https://doi.org/10.1007/978-3-030-74602-5_51
Download citation
DOI: https://doi.org/10.1007/978-3-030-74602-5_51
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-74601-8
Online ISBN: 978-3-030-74602-5
eBook Packages: EngineeringEngineering (R0)