Abstract
Studies of performance in medicine are often based on observation. Videotape provides a valuable tool for recording events from both real environments and simulators. When analysing observational data it is important that robust tools are used, particularly when investigating non-technical (cognitive and social) skills. This paper describes the method used to identify the key non-technical skills required in anaesthesia and to develop a behavioural marker system for their measurement. A prototype taxonomy was designed on the basis of a literature review; an examination of existing marker systems; cognitive task analysis interviews; an iterative development process involving workshops; and cross-checking in theatre. The resulting anaesthetists’ non-technical skills (ANTS) system comprises four skill categories (task management, team working, situation awareness, and decision making) that divide into 15 elements, each with example behaviours. Preliminary evaluation using ratings of videotaped scenarios indicated that the skills were observable and could be rated with reasonable agreement.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Altmaier E, From, R, Pearson K, Gorbatenko-Roth K, Ugolini K (1997) A prospective study to select and evaluate anesthesiology residents: phase 1, the critical incident technique. J Clin Anesth 9:629–636
Anastasi A, Urbina S (1997) Psychological testing. Prentice Hall, Upper Saddle River
Avermaete JAG van, Kruijsen EAC (eds) (1998) NOTECHS. The evaluation of non-technical skills of multi-pilot aircrew in relation to the JAR-FCL requirements. EC NOTECHS project final report: NLR-CR-98443, Amsterdam
Baker DP, Dismukes RK (2002) A framework for understanding crew performance assessment issues. Int J Aviat Psychol 12(3):205–222
Baker D, Mulqueen C, Dismukes R (2001) Training raters to assess resource management skills. In: Salas E, Bowers C, Edens E (eds) Improving teamwork in organisations: applications of resource management training. Laurence Erlbaum Associates, Mahwah, pp 131–145
Bogner M (ed) (2004) Misadventures in health care. Lawrence Erlbaum Associates, Mahwah
Byrne AJ, Jones JG (1997) Responses to simulated anaesthetic emergencies by anaesthetists with different durations of clinical experience. Br J Anaesth 78:553–556
Cannon-Bowers J, Tannenbaum S, Salas E, Volpe C (1995) Defining competencies and establishing team training requirements. In: Guzzo R, Salas E, et al (eds) Team effectiveness and decision making in organisations. Jossey-Bass, San Francisco
Carthey J (2003) The role of structured observational research in healthcare. Qual Saf Health Care 12(Suppl ii):13–16
Charmaz K (1995) Grounded theory. In: Smith J, Harré R, van Langenhove L (eds) Rethinking methods in psychology. Sage, London, pp 25–49
Chipman S, Schraagen J, Shalin V (2000) Introduction to cognitive task analysis. In: Schraagen J, Chipman S, Shalin V (eds) Cognitive task analysis. Lawrence Erlbaum Associates, Mahwah, pp 3–23
Department of Anaesthesia, University of Basel-Kantonspital (1997) Kommunikations-Status (KOMSTAT), Operationssal-Beobachtungen, Ver. 2.0, 07/97 (in German). Personal communication, 2000. Available from Swiss anaesthesia server: http://www.medana.unibas.ch . Cited 16 March 2004
Endsley M (1995) Toward a theory of situation awareness in dynamic systems. Hum Factors 37:32–64
Farmer E, van Rooji J, Riemersma J, Jorna P, Moraal J (1999) Handbook of simulator-based training. Ashgate, Aldershot
Flanagan J (1954) The critical incident technique. Psychol Bull 51:327–358
Fletcher G, McGeorge P, Flin R, Glavin R, Maran N (2002) The identification and measurement of anaesthetists’ non-technical skills: a review of current literature. Br J Anaesth 88:418–429
Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R (2003) Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 90:580–588
Flin R, Martin L (2001) Behavioural markers for crew resource management: a survey of current practice. Int J Aviat Psychol 11:95–118
Flin R, Fletcher G, McGeorge P, Sutherland A, Patey R (2003a) Anaesthetists’ attitudes to teamwork and safety. Anaesthesia 58:233–242
Flin R, Fletcher G, McGeorge P, Glavin R, Maran N, Patey R (2003b) Rating anaesthetists’ non-technical skills. The ANTS system. In: Proceedings of the Human Factors and Ergonomics Society, 47th meeting, Denver. HFES, San Diego
Flin R, Martin L, Goeters K, Hormann J, Amalberti R, Valot C, Nuihjsen H (2003c) Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Hum Factors Aerospace Saf 3:95–117
Flin R, Yute S, Paterson-Brown S, Maran N (2004) Surgeons’ non-technical skills. In Proceedings of the International Meeting of Medical Simulation, Albuquerque, January
Forrest FC, Taylor MA, Postlethwaite K, Aspinall R (2002) Use of a high-fidelity simulator to develop testing of technical performance of novice anaesthetists. Br J Anaesth 88(3):338–344
Gaba D, Howard S, Small S (1995). Situational awareness in anesthesiology. Hum Factors 5(1):20–31
Gaba D, Howard S, Flanagan B, Smith B, Fish K, Botney R (1998) Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 89:123–147
Gaba D, Rall M (2004) Patient simulators (chap 84). In: Miller JD (ed) Anesthesia 6th edn. Elsevier, Philadelphia, in press
Helmreich R, Schaefer H-G, Sexton J (1995) Operating room checklist (aerospace crew resource project technical report 95–4). University of Texas, Austin
Howard S, Gaba D, Fish K, Yang G, Sarnquist F (1992) Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med 63:763–770
Joint Aviation Authorities (1999) JAA-OPS. 1, subpart N (NPA-OPS-16) Crew resource management-flight crew. Author, Hoofddorp
Klampfer B, Flin R, Helmreich RL, Hausler R, Sexton B, Fletcher G et al (2001) Enhancing performance in high risk environments: recommendations for the use of behavioural markers. Daimler-Benz Shiftung, Ladenburg. Downloadable version available from: http://www.abdn.ac.uk/iprc . Cited 16 March 2004
Klein G, Calderwood R, MacGregor D (1989) Critical decision method for eliciting knowledge. IEEE Trans Syst Man Cybern 19:462–472
Klein G, Militello L (2000) The knowledge audit as a method for cognitive task analysis. In: 5th conference on naturalistic decision making, Stockholm, 26–29 May 2000
Klein G, Militello L (2001) Some guidelines for conducting a cognitive task analysis. In: Salas E (ed) Advances in human performance and cognitive engineering research. JAI Press, New York
Klemola U-M, Norros L (1997) Analysis of the clinical behaviour of anaesthetists: recognition of uncertainty as a basis for practice. Med Educ 31:449–456
Mackenzie C, Xiao Y (2003) Video techniques and data compared with observation in emergency trauma care. Qual Saf Health Care 12(Suppl ii):51–57
Mackenzie C, Xiao Y, and the LOTAS group (1999) Video analysis for performance modelling in real environments: methods and lessons learnt. In: Proceedings of the human factors and ergonomics society, 43rd meeting, Houston. HFES, San Diego
Manser T, Wehner T (2002) Analysing action sequences: variations in action density in the administration of anaesthesia. Cogn Technol Work 4(2):71–81
Muniz E, Stout R, Bowers C, Salas E (1998) A methodology for measuring team situational awareness: situational awareness linked indicators adapted to novel tasks (SALIENT). In: NATO human factors and medicine panel on collaborative crew performance in complex systems, Edinburgh, North Atlantic Treaties Organisation, Neuilly-sur-Seine, 20–24 April 1998
Nunnally J, Bernstein I (1993) Psychometric theory, 3rd edn. McGraw-Hill, New York
O’Connor P, Hörmann H-J, Flin R, Lodge M, Goeters K-M, and the JARTEL Group (2002) Developing a method for evaluating crew resource management skills: a European perspective. Int J Aviat Psychol 12(3):263–285
Rugg G, McGeorge P (1997) The sorting techniques: a tutorial paper on card sorts, picture sorts and item sorts. Expert Syst 14(2):80–93
Salas E, Bowers C, Edens E (eds) (2001) Improving teamwork in organisations: applications of resource management training. Lawrence Erlbaum Associates, Mahwah
Seagull J, Xiao Y, Mackenzie C, Jaberi M, Dutton R (1999) Monitoring behavior: a pilot study using an ambulatory eye-tracker in surgical operating rooms. In: Proceedings of the human factors and ergonomics society, 43rd annual meeting. HFES, San Diego
Seamster T, Edens E (1993) Cognitive modelling of CRM assessment expertise: identification of the primary assessors. In: Proceedings of the human factors and ergonomics society, 37th annual meeting. HFES, San Diego
Seamster T, Kaempf G (2001) Identifying resource management skills for airline pilots. In: Salas E, Bowers C, Edens E (eds) Improving teamwork in organisations: applications of resource management training. Lawrence Erlbaum Associates, Mahwah, pp 9–30
Seamster T, Redding R, Kaempf G (1997) Applied cognitive task analysis in aviation. Avebury Aviation, Aldershot
Small S, Wuerz R, Simon R, Shapiro N, Conn A, Setnik G (1999) Demonstration of high-fidelity simulation team training for emergency medicine. Acad Emerg Med 6:312–323
Strauss A, Corbin J (1990) Basics of qualitative research: grounded theory procedures and techniques. Sage, Newbury Park
Weigner M, et al (2004) Video capture and analysis. Qual Saf Health Care, in press
Weiner E, Kanki B, Helmreich R (eds) (1993) Cockpit resource management. Academic, San Diego
Williams D, Holt R, Boehm-Davis D (1997) Training for inter-rater reliability: baselines and benchmarks. In: Jensen R (ed) Proceedings of the 9th international symposium on aviation psychology, Columbus, Ohio. Ohio State University, Columbus, 27 April–1 May 1997
Xiao Y, Mackenzie CF (1998) Collaboration in complex medical systems. In: NATO human factors and medicine panel on collaborative crew performance in complex systems, Edinburgh. North Atlantic Treaties Organisation, Neuilly-sur-Seine, 20–24 April 1998
Xiao Y, Hunter W, Mackenzie C, Jefferies N, Horst R, and the LOTAS Group (1996) Task complexity and emergency medical care and its implications for team coordination. Hum Factors 38(4):636–645
Acknowledgements
The ANTS system was developed with research funding from NHS Education for Scotland. The views presented in this paper are those of the authors and should not be taken to represent the position or policy of the funding body. We would like to thank all the anaesthetists who have taken part in the research for the ANTS system.
Author information
Authors and Affiliations
Corresponding author
Additional information
Georgina Fletcher is now with the National Patient Safety Agency, London.
Rights and permissions
About this article
Cite this article
Fletcher, G., Flin, R., McGeorge, P. et al. Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cogn Tech Work 6, 165–171 (2004). https://doi.org/10.1007/s10111-004-0158-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10111-004-0158-y