Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Rating non-technical skills: developing a behavioural marker system for use in anaesthesia


Studies of performance in medicine are often based on observation. Videotape provides a valuable tool for recording events from both real environments and simulators. When analysing observational data it is important that robust tools are used, particularly when investigating non-technical (cognitive and social) skills. This paper describes the method used to identify the key non-technical skills required in anaesthesia and to develop a behavioural marker system for their measurement. A prototype taxonomy was designed on the basis of a literature review; an examination of existing marker systems; cognitive task analysis interviews; an iterative development process involving workshops; and cross-checking in theatre. The resulting anaesthetists’ non-technical skills (ANTS) system comprises four skill categories (task management, team working, situation awareness, and decision making) that divide into 15 elements, each with example behaviours. Preliminary evaluation using ratings of videotaped scenarios indicated that the skills were observable and could be rated with reasonable agreement.

This is a preview of subscription content, log in to check access.


  1. Altmaier E, From, R, Pearson K, Gorbatenko-Roth K, Ugolini K (1997) A prospective study to select and evaluate anesthesiology residents: phase 1, the critical incident technique. J Clin Anesth 9:629–636

  2. Anastasi A, Urbina S (1997) Psychological testing. Prentice Hall, Upper Saddle River

  3. Avermaete JAG van, Kruijsen EAC (eds) (1998) NOTECHS. The evaluation of non-technical skills of multi-pilot aircrew in relation to the JAR-FCL requirements. EC NOTECHS project final report: NLR-CR-98443, Amsterdam

  4. Baker DP, Dismukes RK (2002) A framework for understanding crew performance assessment issues. Int J Aviat Psychol 12(3):205–222

  5. Baker D, Mulqueen C, Dismukes R (2001) Training raters to assess resource management skills. In: Salas E, Bowers C, Edens E (eds) Improving teamwork in organisations: applications of resource management training. Laurence Erlbaum Associates, Mahwah, pp 131–145

  6. Bogner M (ed) (2004) Misadventures in health care. Lawrence Erlbaum Associates, Mahwah

  7. Byrne AJ, Jones JG (1997) Responses to simulated anaesthetic emergencies by anaesthetists with different durations of clinical experience. Br J Anaesth 78:553–556

  8. Cannon-Bowers J, Tannenbaum S, Salas E, Volpe C (1995) Defining competencies and establishing team training requirements. In: Guzzo R, Salas E, et al (eds) Team effectiveness and decision making in organisations. Jossey-Bass, San Francisco

  9. Carthey J (2003) The role of structured observational research in healthcare. Qual Saf Health Care 12(Suppl ii):13–16

  10. Charmaz K (1995) Grounded theory. In: Smith J, Harré R, van Langenhove L (eds) Rethinking methods in psychology. Sage, London, pp 25–49

  11. Chipman S, Schraagen J, Shalin V (2000) Introduction to cognitive task analysis. In: Schraagen J, Chipman S, Shalin V (eds) Cognitive task analysis. Lawrence Erlbaum Associates, Mahwah, pp 3–23

  12. Department of Anaesthesia, University of Basel-Kantonspital (1997) Kommunikations-Status (KOMSTAT), Operationssal-Beobachtungen, Ver. 2.0, 07/97 (in German). Personal communication, 2000. Available from Swiss anaesthesia server: http://www.medana.unibas.ch . Cited 16 March 2004

  13. Endsley M (1995) Toward a theory of situation awareness in dynamic systems. Hum Factors 37:32–64

  14. Farmer E, van Rooji J, Riemersma J, Jorna P, Moraal J (1999) Handbook of simulator-based training. Ashgate, Aldershot

  15. Flanagan J (1954) The critical incident technique. Psychol Bull 51:327–358

  16. Fletcher G, McGeorge P, Flin R, Glavin R, Maran N (2002) The identification and measurement of anaesthetists’ non-technical skills: a review of current literature. Br J Anaesth 88:418–429

  17. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R (2003) Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 90:580–588

  18. Flin R, Martin L (2001) Behavioural markers for crew resource management: a survey of current practice. Int J Aviat Psychol 11:95–118

  19. Flin R, Fletcher G, McGeorge P, Sutherland A, Patey R (2003a) Anaesthetists’ attitudes to teamwork and safety. Anaesthesia 58:233–242

  20. Flin R, Fletcher G, McGeorge P, Glavin R, Maran N, Patey R (2003b) Rating anaesthetists’ non-technical skills. The ANTS system. In: Proceedings of the Human Factors and Ergonomics Society, 47th meeting, Denver. HFES, San Diego

  21. Flin R, Martin L, Goeters K, Hormann J, Amalberti R, Valot C, Nuihjsen H (2003c) Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Hum Factors Aerospace Saf 3:95–117

  22. Flin R, Yute S, Paterson-Brown S, Maran N (2004) Surgeons’ non-technical skills. In Proceedings of the International Meeting of Medical Simulation, Albuquerque, January

  23. Forrest FC, Taylor MA, Postlethwaite K, Aspinall R (2002) Use of a high-fidelity simulator to develop testing of technical performance of novice anaesthetists. Br J Anaesth 88(3):338–344

  24. Gaba D, Howard S, Small S (1995). Situational awareness in anesthesiology. Hum Factors 5(1):20–31

  25. Gaba D, Howard S, Flanagan B, Smith B, Fish K, Botney R (1998) Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 89:123–147

  26. Gaba D, Rall M (2004) Patient simulators (chap 84). In: Miller JD (ed) Anesthesia 6th edn. Elsevier, Philadelphia, in press

  27. Helmreich R, Schaefer H-G, Sexton J (1995) Operating room checklist (aerospace crew resource project technical report 95–4). University of Texas, Austin

  28. Howard S, Gaba D, Fish K, Yang G, Sarnquist F (1992) Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med 63:763–770

  29. Joint Aviation Authorities (1999) JAA-OPS. 1, subpart N (NPA-OPS-16) Crew resource management-flight crew. Author, Hoofddorp

  30. Klampfer B, Flin R, Helmreich RL, Hausler R, Sexton B, Fletcher G et al (2001) Enhancing performance in high risk environments: recommendations for the use of behavioural markers. Daimler-Benz Shiftung, Ladenburg. Downloadable version available from: http://www.abdn.ac.uk/iprc . Cited 16 March 2004

  31. Klein G, Calderwood R, MacGregor D (1989) Critical decision method for eliciting knowledge. IEEE Trans Syst Man Cybern 19:462–472

  32. Klein G, Militello L (2000) The knowledge audit as a method for cognitive task analysis. In: 5th conference on naturalistic decision making, Stockholm, 26–29 May 2000

  33. Klein G, Militello L (2001) Some guidelines for conducting a cognitive task analysis. In: Salas E (ed) Advances in human performance and cognitive engineering research. JAI Press, New York

  34. Klemola U-M, Norros L (1997) Analysis of the clinical behaviour of anaesthetists: recognition of uncertainty as a basis for practice. Med Educ 31:449–456

  35. Mackenzie C, Xiao Y (2003) Video techniques and data compared with observation in emergency trauma care. Qual Saf Health Care 12(Suppl ii):51–57

  36. Mackenzie C, Xiao Y, and the LOTAS group (1999) Video analysis for performance modelling in real environments: methods and lessons learnt. In: Proceedings of the human factors and ergonomics society, 43rd meeting, Houston. HFES, San Diego

  37. Manser T, Wehner T (2002) Analysing action sequences: variations in action density in the administration of anaesthesia. Cogn Technol Work 4(2):71–81

  38. Muniz E, Stout R, Bowers C, Salas E (1998) A methodology for measuring team situational awareness: situational awareness linked indicators adapted to novel tasks (SALIENT). In: NATO human factors and medicine panel on collaborative crew performance in complex systems, Edinburgh, North Atlantic Treaties Organisation, Neuilly-sur-Seine, 20–24 April 1998

  39. Nunnally J, Bernstein I (1993) Psychometric theory, 3rd edn. McGraw-Hill, New York

  40. O’Connor P, Hörmann H-J, Flin R, Lodge M, Goeters K-M, and the JARTEL Group (2002) Developing a method for evaluating crew resource management skills: a European perspective. Int J Aviat Psychol 12(3):263–285

  41. Rugg G, McGeorge P (1997) The sorting techniques: a tutorial paper on card sorts, picture sorts and item sorts. Expert Syst 14(2):80–93

  42. Salas E, Bowers C, Edens E (eds) (2001) Improving teamwork in organisations: applications of resource management training. Lawrence Erlbaum Associates, Mahwah

  43. Seagull J, Xiao Y, Mackenzie C, Jaberi M, Dutton R (1999) Monitoring behavior: a pilot study using an ambulatory eye-tracker in surgical operating rooms. In: Proceedings of the human factors and ergonomics society, 43rd annual meeting. HFES, San Diego

  44. Seamster T, Edens E (1993) Cognitive modelling of CRM assessment expertise: identification of the primary assessors. In: Proceedings of the human factors and ergonomics society, 37th annual meeting. HFES, San Diego

  45. Seamster T, Kaempf G (2001) Identifying resource management skills for airline pilots. In: Salas E, Bowers C, Edens E (eds) Improving teamwork in organisations: applications of resource management training. Lawrence Erlbaum Associates, Mahwah, pp 9–30

  46. Seamster T, Redding R, Kaempf G (1997) Applied cognitive task analysis in aviation. Avebury Aviation, Aldershot

  47. Small S, Wuerz R, Simon R, Shapiro N, Conn A, Setnik G (1999) Demonstration of high-fidelity simulation team training for emergency medicine. Acad Emerg Med 6:312–323

  48. Strauss A, Corbin J (1990) Basics of qualitative research: grounded theory procedures and techniques. Sage, Newbury Park

  49. Weigner M, et al (2004) Video capture and analysis. Qual Saf Health Care, in press

  50. Weiner E, Kanki B, Helmreich R (eds) (1993) Cockpit resource management. Academic, San Diego

  51. Williams D, Holt R, Boehm-Davis D (1997) Training for inter-rater reliability: baselines and benchmarks. In: Jensen R (ed) Proceedings of the 9th international symposium on aviation psychology, Columbus, Ohio. Ohio State University, Columbus, 27 April–1 May 1997

  52. Xiao Y, Mackenzie CF (1998) Collaboration in complex medical systems. In: NATO human factors and medicine panel on collaborative crew performance in complex systems, Edinburgh. North Atlantic Treaties Organisation, Neuilly-sur-Seine, 20–24 April 1998

  53. Xiao Y, Hunter W, Mackenzie C, Jefferies N, Horst R, and the LOTAS Group (1996) Task complexity and emergency medical care and its implications for team coordination. Hum Factors 38(4):636–645

Download references


The ANTS system was developed with research funding from NHS Education for Scotland. The views presented in this paper are those of the authors and should not be taken to represent the position or policy of the funding body. We would like to thank all the anaesthetists who have taken part in the research for the ANTS system.

Author information

Correspondence to R. Flin.

Additional information

Georgina Fletcher is now with the National Patient Safety Agency, London.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Fletcher, G., Flin, R., McGeorge, P. et al. Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cogn Tech Work 6, 165–171 (2004). https://doi.org/10.1007/s10111-004-0158-y

Download citation


  • Non-technical skills
  • Behavioural marker system
  • Anaesthesia
  • Performance assessment
  • Cognitive task analysis