Rating non-technical skills: developing a behavioural marker system for use in anaesthesia


Studies of performance in medicine are often based on observation. Videotape provides a valuable tool for recording events from both real environments and simulators. When analysing observational data it is important that robust tools are used, particularly when investigating non-technical (cognitive and social) skills. This paper describes the method used to identify the key non-technical skills required in anaesthesia and to develop a behavioural marker system for their measurement. A prototype taxonomy was designed on the basis of a literature review; an examination of existing marker systems; cognitive task analysis interviews; an iterative development process involving workshops; and cross-checking in theatre. The resulting anaesthetists’ non-technical skills (ANTS) system comprises four skill categories (task management, team working, situation awareness, and decision making) that divide into 15 elements, each with example behaviours. Preliminary evaluation using ratings of videotaped scenarios indicated that the skills were observable and could be rated with reasonable agreement.

This is a preview of subscription content, access via your institution.


  1. Altmaier E, From, R, Pearson K, Gorbatenko-Roth K, Ugolini K (1997) A prospective study to select and evaluate anesthesiology residents: phase 1, the critical incident technique. J Clin Anesth 9:629–636

    Article  CAS  PubMed  Google Scholar 

  2. Anastasi A, Urbina S (1997) Psychological testing. Prentice Hall, Upper Saddle River

  3. Avermaete JAG van, Kruijsen EAC (eds) (1998) NOTECHS. The evaluation of non-technical skills of multi-pilot aircrew in relation to the JAR-FCL requirements. EC NOTECHS project final report: NLR-CR-98443, Amsterdam

    Google Scholar 

  4. Baker DP, Dismukes RK (2002) A framework for understanding crew performance assessment issues. Int J Aviat Psychol 12(3):205–222

    Article  Google Scholar 

  5. Baker D, Mulqueen C, Dismukes R (2001) Training raters to assess resource management skills. In: Salas E, Bowers C, Edens E (eds) Improving teamwork in organisations: applications of resource management training. Laurence Erlbaum Associates, Mahwah, pp 131–145

  6. Bogner M (ed) (2004) Misadventures in health care. Lawrence Erlbaum Associates, Mahwah

  7. Byrne AJ, Jones JG (1997) Responses to simulated anaesthetic emergencies by anaesthetists with different durations of clinical experience. Br J Anaesth 78:553–556

    CAS  PubMed  Google Scholar 

  8. Cannon-Bowers J, Tannenbaum S, Salas E, Volpe C (1995) Defining competencies and establishing team training requirements. In: Guzzo R, Salas E, et al (eds) Team effectiveness and decision making in organisations. Jossey-Bass, San Francisco

  9. Carthey J (2003) The role of structured observational research in healthcare. Qual Saf Health Care 12(Suppl ii):13–16

    PubMed  Google Scholar 

  10. Charmaz K (1995) Grounded theory. In: Smith J, Harré R, van Langenhove L (eds) Rethinking methods in psychology. Sage, London, pp 25–49

  11. Chipman S, Schraagen J, Shalin V (2000) Introduction to cognitive task analysis. In: Schraagen J, Chipman S, Shalin V (eds) Cognitive task analysis. Lawrence Erlbaum Associates, Mahwah, pp 3–23

  12. Department of Anaesthesia, University of Basel-Kantonspital (1997) Kommunikations-Status (KOMSTAT), Operationssal-Beobachtungen, Ver. 2.0, 07/97 (in German). Personal communication, 2000. Available from Swiss anaesthesia server: http://www.medana.unibas.ch . Cited 16 March 2004

  13. Endsley M (1995) Toward a theory of situation awareness in dynamic systems. Hum Factors 37:32–64

    Google Scholar 

  14. Farmer E, van Rooji J, Riemersma J, Jorna P, Moraal J (1999) Handbook of simulator-based training. Ashgate, Aldershot

  15. Flanagan J (1954) The critical incident technique. Psychol Bull 51:327–358

    CAS  PubMed  Google Scholar 

  16. Fletcher G, McGeorge P, Flin R, Glavin R, Maran N (2002) The identification and measurement of anaesthetists’ non-technical skills: a review of current literature. Br J Anaesth 88:418–429

    Article  CAS  PubMed  Google Scholar 

  17. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R (2003) Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 90:580–588

    Article  CAS  PubMed  Google Scholar 

  18. Flin R, Martin L (2001) Behavioural markers for crew resource management: a survey of current practice. Int J Aviat Psychol 11:95–118

    Article  Google Scholar 

  19. Flin R, Fletcher G, McGeorge P, Sutherland A, Patey R (2003a) Anaesthetists’ attitudes to teamwork and safety. Anaesthesia 58:233–242

    Article  CAS  PubMed  Google Scholar 

  20. Flin R, Fletcher G, McGeorge P, Glavin R, Maran N, Patey R (2003b) Rating anaesthetists’ non-technical skills. The ANTS system. In: Proceedings of the Human Factors and Ergonomics Society, 47th meeting, Denver. HFES, San Diego

  21. Flin R, Martin L, Goeters K, Hormann J, Amalberti R, Valot C, Nuihjsen H (2003c) Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Hum Factors Aerospace Saf 3:95–117

    Google Scholar 

  22. Flin R, Yute S, Paterson-Brown S, Maran N (2004) Surgeons’ non-technical skills. In Proceedings of the International Meeting of Medical Simulation, Albuquerque, January

  23. Forrest FC, Taylor MA, Postlethwaite K, Aspinall R (2002) Use of a high-fidelity simulator to develop testing of technical performance of novice anaesthetists. Br J Anaesth 88(3):338–344

    Article  CAS  PubMed  Google Scholar 

  24. Gaba D, Howard S, Small S (1995). Situational awareness in anesthesiology. Hum Factors 5(1):20–31

    Google Scholar 

  25. Gaba D, Howard S, Flanagan B, Smith B, Fish K, Botney R (1998) Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 89:123–147

    Google Scholar 

  26. Gaba D, Rall M (2004) Patient simulators (chap 84). In: Miller JD (ed) Anesthesia 6th edn. Elsevier, Philadelphia, in press

  27. Helmreich R, Schaefer H-G, Sexton J (1995) Operating room checklist (aerospace crew resource project technical report 95–4). University of Texas, Austin

  28. Howard S, Gaba D, Fish K, Yang G, Sarnquist F (1992) Anesthesia crisis resource management training: teaching anesthesiologists to handle critical incidents. Aviat Space Environ Med 63:763–770

    CAS  PubMed  Google Scholar 

  29. Joint Aviation Authorities (1999) JAA-OPS. 1, subpart N (NPA-OPS-16) Crew resource management-flight crew. Author, Hoofddorp

  30. Klampfer B, Flin R, Helmreich RL, Hausler R, Sexton B, Fletcher G et al (2001) Enhancing performance in high risk environments: recommendations for the use of behavioural markers. Daimler-Benz Shiftung, Ladenburg. Downloadable version available from: http://www.abdn.ac.uk/iprc . Cited 16 March 2004

  31. Klein G, Calderwood R, MacGregor D (1989) Critical decision method for eliciting knowledge. IEEE Trans Syst Man Cybern 19:462–472

    Article  Google Scholar 

  32. Klein G, Militello L (2000) The knowledge audit as a method for cognitive task analysis. In: 5th conference on naturalistic decision making, Stockholm, 26–29 May 2000

  33. Klein G, Militello L (2001) Some guidelines for conducting a cognitive task analysis. In: Salas E (ed) Advances in human performance and cognitive engineering research. JAI Press, New York

  34. Klemola U-M, Norros L (1997) Analysis of the clinical behaviour of anaesthetists: recognition of uncertainty as a basis for practice. Med Educ 31:449–456

    CAS  PubMed  Google Scholar 

  35. Mackenzie C, Xiao Y (2003) Video techniques and data compared with observation in emergency trauma care. Qual Saf Health Care 12(Suppl ii):51–57

    PubMed  Google Scholar 

  36. Mackenzie C, Xiao Y, and the LOTAS group (1999) Video analysis for performance modelling in real environments: methods and lessons learnt. In: Proceedings of the human factors and ergonomics society, 43rd meeting, Houston. HFES, San Diego

  37. Manser T, Wehner T (2002) Analysing action sequences: variations in action density in the administration of anaesthesia. Cogn Technol Work 4(2):71–81

    Article  Google Scholar 

  38. Muniz E, Stout R, Bowers C, Salas E (1998) A methodology for measuring team situational awareness: situational awareness linked indicators adapted to novel tasks (SALIENT). In: NATO human factors and medicine panel on collaborative crew performance in complex systems, Edinburgh, North Atlantic Treaties Organisation, Neuilly-sur-Seine, 20–24 April 1998

  39. Nunnally J, Bernstein I (1993) Psychometric theory, 3rd edn. McGraw-Hill, New York

  40. O’Connor P, Hörmann H-J, Flin R, Lodge M, Goeters K-M, and the JARTEL Group (2002) Developing a method for evaluating crew resource management skills: a European perspective. Int J Aviat Psychol 12(3):263–285

    Article  Google Scholar 

  41. Rugg G, McGeorge P (1997) The sorting techniques: a tutorial paper on card sorts, picture sorts and item sorts. Expert Syst 14(2):80–93

    Article  Google Scholar 

  42. Salas E, Bowers C, Edens E (eds) (2001) Improving teamwork in organisations: applications of resource management training. Lawrence Erlbaum Associates, Mahwah

    Google Scholar 

  43. Seagull J, Xiao Y, Mackenzie C, Jaberi M, Dutton R (1999) Monitoring behavior: a pilot study using an ambulatory eye-tracker in surgical operating rooms. In: Proceedings of the human factors and ergonomics society, 43rd annual meeting. HFES, San Diego

  44. Seamster T, Edens E (1993) Cognitive modelling of CRM assessment expertise: identification of the primary assessors. In: Proceedings of the human factors and ergonomics society, 37th annual meeting. HFES, San Diego

  45. Seamster T, Kaempf G (2001) Identifying resource management skills for airline pilots. In: Salas E, Bowers C, Edens E (eds) Improving teamwork in organisations: applications of resource management training. Lawrence Erlbaum Associates, Mahwah, pp 9–30

  46. Seamster T, Redding R, Kaempf G (1997) Applied cognitive task analysis in aviation. Avebury Aviation, Aldershot

  47. Small S, Wuerz R, Simon R, Shapiro N, Conn A, Setnik G (1999) Demonstration of high-fidelity simulation team training for emergency medicine. Acad Emerg Med 6:312–323

    CAS  PubMed  Google Scholar 

  48. Strauss A, Corbin J (1990) Basics of qualitative research: grounded theory procedures and techniques. Sage, Newbury Park

    Google Scholar 

  49. Weigner M, et al (2004) Video capture and analysis. Qual Saf Health Care, in press

  50. Weiner E, Kanki B, Helmreich R (eds) (1993) Cockpit resource management. Academic, San Diego

  51. Williams D, Holt R, Boehm-Davis D (1997) Training for inter-rater reliability: baselines and benchmarks. In: Jensen R (ed) Proceedings of the 9th international symposium on aviation psychology, Columbus, Ohio. Ohio State University, Columbus, 27 April–1 May 1997

  52. Xiao Y, Mackenzie CF (1998) Collaboration in complex medical systems. In: NATO human factors and medicine panel on collaborative crew performance in complex systems, Edinburgh. North Atlantic Treaties Organisation, Neuilly-sur-Seine, 20–24 April 1998

  53. Xiao Y, Hunter W, Mackenzie C, Jefferies N, Horst R, and the LOTAS Group (1996) Task complexity and emergency medical care and its implications for team coordination. Hum Factors 38(4):636–645

    CAS  PubMed  Google Scholar 

Download references


The ANTS system was developed with research funding from NHS Education for Scotland. The views presented in this paper are those of the authors and should not be taken to represent the position or policy of the funding body. We would like to thank all the anaesthetists who have taken part in the research for the ANTS system.

Author information



Corresponding author

Correspondence to R. Flin.

Additional information

Georgina Fletcher is now with the National Patient Safety Agency, London.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Fletcher, G., Flin, R., McGeorge, P. et al. Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cogn Tech Work 6, 165–171 (2004). https://doi.org/10.1007/s10111-004-0158-y

Download citation


  • Non-technical skills
  • Behavioural marker system
  • Anaesthesia
  • Performance assessment
  • Cognitive task analysis