Cognition, Technology & Work

, Volume 17, Issue 1, pp 63–77 | Cite as

Development of instruments for assessment of individuals’ and teams’ non-technical skills in healthcare: a critical review

  • Rikke M. H. G. Jepsen
  • Doris Østergaard
  • Peter Dieckmann
Original Article


Focus on patient safety has increased over the last decades; however, patient safety still relies on technology and the performance of healthcare professionals. Technology improves rapidly but despite numerous reports on how to improve the performance of healthcare professionals, this improvement has been more slow than expected. The performance of healthcare professionals is a product of each individual’s medical knowledge, technical and non-technical skills as well as the settings in which these skills are used. Training and the assessment of non-technical skills for individuals or teams in healthcare have shown to improve a safe and efficient performance. However, the implementation has been slow. One reason for this might be the shortage of assessment instrument to assess individuals’ and teams’ non-technical skills. In this paper, we review the development process of 23 instruments for assessment of non-technical skills for individuals or teams within healthcare. The instruments are mainly for use in the operating room or for teams handling emergency situations. Several of the instruments are developed based on a thorough needs analysis by a team having the same professional background as the target group. Most of the instruments consist of almost the same categories of non-technical skills and many use behavioural markers. Overall, the instruments have been validated to some extent, but there is room for improvement. However, there seems to be a lack of training of the raters. The importance of providing feedback after the assessment is emphasised. The criteria on which the behavioural markers are developed should undergo continuous changes, representing the development of the patient safety culture in healthcare organisations.


Non-technical skills Human factors Assessment Healthcare Performance Observation Patient safety 


  1. Andersen PO, Jensen MK, Lippert A, Østergaard D, Klausen TW (2010) Development of a formative assessment tool for measurement of performance in multi-professional resuscitation teams. Resuscitation 81(6):703–711CrossRefGoogle Scholar
  2. Antonovsky A (1996) The salutogenic model as a theory to guide health promotion. Health Promot Int 11(1):11–18CrossRefGoogle Scholar
  3. Bateson G, Bateson MC (2000) Steps to an ecology of mind (vol 1972). University of Chicago Press Chicago. Accessed 20 May 2014
  4. Bested KM, Malling B, Skjelsager K, Østergaard D, Østergaard HT, Ringsted C (2011) Rater bias in postgraduate medical education. Ugeskr Laeger 173(44):2788–2790Google Scholar
  5. Cooper JB, Newbower RS, Long CD, McPeek B (2002) Preventable anesthesia mishaps: a study of human factors*. Qual Saf Health Care 11(3):277–282. doi:10.1136/qhc.11.3.277 CrossRefGoogle Scholar
  6. Cooper S, Cant R, Porter J, Sellick K, Somers G, Kinsman L, Nestel D (2010) Rating medical emergency teamwork performance: development of the Team Emergency Assessment Measure (TEAM). Resuscitation 81(4):446–452CrossRefGoogle Scholar
  7. Corrigan JM (2005) Crossing the quality chasm. In: Reid PP, Compton WD, Grossman JH et al (eds) Building a better delivery system: a new engineering/health care partnership. National Academies Press, Washington, DC. Crossing the Quality Chasm. Accessed 20 Apr 2014
  8. Crossingham GV, Sice PJA, Roberts MJ, Lam WH, Gale TCE (2012) Development of workplace‐based assessments of non‐technical skills in anaesthesia*. Anaesthesia 67(2):158–164. doi:10.1111/j.1365-2044.2011.06977.x
  9. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R (2003) Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 90(5):580–588CrossRefGoogle Scholar
  10. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R (2004) Rating non-technical skills: developing a behavioural marker system for use in anaesthesia. Cognit Technol Work 6(3). doi:10.1007/s10111-004-0158-y
  11. Flin R, Patey R (2011) Non-technical skills for anaesthetists: developing and applying ANTS. Best Pract Res Clin Anaesthesiol 25(2):215–227. doi:10.1016/j.bpa.2011.02.005 CrossRefGoogle Scholar
  12. Flin R, Martin L, Goeters K-M, Hörmann H-J, Amalberti R, Valot C, Nijhuis H (2005) Development of the NOTECHS (Non-Technical Skills) system for assessing pilots’ CRM skills. In: Harris D, Muir HC (eds) Contemporary issues in human factors and aviation safety. Ashgate, Aldershot, pp 133–154Google Scholar
  13. Fuhrmann L, Østergaard D, Lippert A, Perner A (2009) A multi-professional full-scale simulation course in the recognition and management of deteriorating hospital patients. Resuscitation 80(6):669–673CrossRefGoogle Scholar
  14. Gaba David M (2000) Anaesthesiology as a model for patient safety in health care. BMJ 320(7237):785–788CrossRefGoogle Scholar
  15. Gaba David M, DeAnda A (1988) A comprehensive anesthesia simulation environment: re-creating the operating room for research and training. Anesthesiology 69(3):387CrossRefGoogle Scholar
  16. Gaba David M, Maxwell M, DeAnda A (1987) Anesthetic mishaps: breaking the chain of accident evolution. Anesthesiology 66(5):670CrossRefGoogle Scholar
  17. Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R (1998) Assessment of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 89(1):8CrossRefGoogle Scholar
  18. Glavin RJ (2011) Skills, training, and education. Simul Healthc 6(1):4–7. doi:10.1097/SIH.0b013e31820aa1ee Google Scholar
  19. Graham J, Hocking G, Giles E (2010) Anaesthesia Non-Technical Skills: can anaesthetists be trained to reliably use this behavioural marker system in 1 day? Br J Anaesth 104(4):440–445. doi:10.1093/bja/aeq032 CrossRefGoogle Scholar
  20. Guise JM, Deering SH, Kanki BG, Osterweil P, Li H, Mori M, Lowe NK (2008) Validation of a tool to measure and promote clinical teamwork. Simul Healthc 3(4):217–223CrossRefGoogle Scholar
  21. Healey A, Undre S, Vincent C (2004) Developing observational measures of performance in surgical teams. Qual Saf Health Care 13(Suppl 1):i33–i40. doi:10.1136/qshc.2004.009936 CrossRefGoogle Scholar
  22. Hodges B (2006) Medical education and the maintenance of incompetence. Med Teach 28(8):690–696CrossRefGoogle Scholar
  23. Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N (2011) Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg 212(2):234–243.e5. doi:10.1016/j.jamcollsurg.2010.11.001
  24. Jepsen RMHG, Lyk-Jensen HT, Spanager L, Dieckmann P, Østergaard D (2012) Anaesthetists’ non-technical skills—adapting the system to another setting. In: Short communications, AMEE 2012Google Scholar
  25. Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J (2006) A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Crit Care Med 34(8):2167–2174. doi:10.1097/01.CCM.0000229877.45125.CC CrossRefGoogle Scholar
  26. Klampfer B, Flin R, Helmreich RL, Häusler R, Sexton B, Fletcher G, Field P, Staender S, Lauche K, Dieckmann P, Amacher A (2001) Group interaction in high risk environments. Enhancing performance in high risk environments: recommendations for the use of behavioural markers. Retrieved from Accessed 20 May 2014
  27. Klinect JR, Murray P, Merritt A, Helmreich R (2003) Line operations safety audit (LOSA): definition and operating characteristics. In: Proceedings of the 12th international symposium on aviation psychology, pp 663–668. Retrieved from Accessed 20 Apr 2014
  28. Kohn LT, Corrigan JM, Donaldson MS (1999) To err is human. Building a safer health system. National Academy Press, Washington, DC. Accessed 20 Apr 2014
  29. Kontogiannis T, Malakis S (2013) Strategies in coping with complexity: development of a behavioural marker system for air traffic controllers. Saf Sci 57:27–34CrossRefGoogle Scholar
  30. Lambden S, DeMunter C, Dowson A, Cooper M, Gautama S, Sevdalis N (2013) The imperial paediatric emergency training toolkit (IPETT) for use in paediatric emergency training: development and evaluation of feasibility and validity. Resuscitation 84(6):831–836CrossRefGoogle Scholar
  31. Lingard L, Reznick R, Espin S, Regehr G, DeVito I (2002) Team communications in the operating room: talk patterns, sites of tension, and implications for novices. Acad Med 77(3):232–237CrossRefGoogle Scholar
  32. Lyk-Jensen H, Jepsen RM, Spanager L, Dieckmann P, Østergaard D (2014) Assessing nurse anaesthetists’ non-technical skills in the operating room. Acta Anaesthesiol Scand 58:794–801CrossRefGoogle Scholar
  33. Malec JF, Torsher LC, Dunn WF, Wiegmann DA, Arnold JJ, Brown DA, Phatak V (2007) The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc 2(1):4–10CrossRefGoogle Scholar
  34. Manser T (2009) Teamwork and patient safety in dynamic domains of healthcare: a review of the literature. Acta Anaesthesiol Scand 53(2):143–151. doi:10.1111/j.1399-6576.2008.01717.x CrossRefGoogle Scholar
  35. Mishra A, Catchpole K, McCulloch P (2009) The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care 18(2):104–108CrossRefGoogle Scholar
  36. Morey JC, Simon R, Jay GD, Wears RL, Salisbury M, Dukes KA, Berns SD (2002) Error reduction and performance improvement in the emergency department through formal teamwork training: evaluation results of the MedTeams project. Health Serv Res 37(6):1553–1581CrossRefGoogle Scholar
  37. Morgan PJ, Tregunno D, Pittini R, Tarshis J, Regehr G, Desousa S et al (2012) Determination of the psychometric properties of a behavioural marking system for obstetrical team training using high-fidelity simulation. BMJ Qual Saf 21(1):78–82. doi:10.1136/bmjqs-2011-000296 CrossRefGoogle Scholar
  38. Neily J, Mills PD, Young-Xu Y, Carney BT, West P, Berger DH et al (2010) Association between implementation of a medical team training program and surgical mortality. JAMA 304(15):1693–1700. doi:10.1001/jama.2010.1506
  39. Nestel D, Walker K, Simon R, Aggarwal R, Andreatta P (2011) Nontechnical skills: an inaccurate and unhelpful descriptor? Simul Healthc 6(1):2Google Scholar
  40. Parker SH, Flin R, McKinley A, Yule S (2013) The Surgeons’ Leadership Inventory (SLI): a taxonomy and rating system for surgeons’ intraoperative leadership skills. Am J Surg 205(6):745–751CrossRefGoogle Scholar
  41. Plant JL, Van Schaik SM, Sliwka DC, Boscardin CK, O’Sullivan PS (2011) Validation of a self-efficacy instrument and its relationship to performance of crisis resource management skills. Adv Health Sci Educ 16(5):579–590CrossRefGoogle Scholar
  42. Rasmussen MB, Dieckmann P, Barry Issenberg S, Ostergaard D, Søreide E, Ringsted CV (2012) Long-term intended and unintended experiences after advanced life support training. Resuscitation. doi:10.1016/j.resuscitation.2012.07.030 Google Scholar
  43. Schraagen JM, Schouten T, Smit M, Haas F, van der Beek D, van de Ven J, Barach P (2010) Assessing and improving teamwork in cardiac surgery. Qual Saf Health Care 19(6):1–6. doi:10.1136/qshc.2009.040105 Google Scholar
  44. Sevdalis N, Davis R, Koutantji M, Undre S, Darzi A, Vincent CA (2008) Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg 196(2):184CrossRefGoogle Scholar
  45. Sevdalis N, Hull L, Birnbach DJ (2012) Improving patient safety in the operating theatre and perioperative care: obstacles, interventions, and priorities for accelerating progress. Br J Anaesth 109(Suppl 1):i3–i16. doi:10.1093/bja/aes391 CrossRefGoogle Scholar
  46. Spanager L, Lyk-Jensen HT, Dieckmann P, Wettergren A, Rosenberg J, Ostergaard D (2012) Customization of a tool to assess Danish surgeons’ non-technical skills in the operating room. Dan Med J 59(11):A4526Google Scholar
  47. Spanager L, Beier-Holgersen R, Dieckmann P, Konge L, Rosenberg J, Oestergaard D (2013) Reliable assessment of general surgeons’ non-technical skills based on video-recordings of patient simulated scenarios. Am J Surg 206(5):810–817. doi:10.1016/j.amjsurg.2013.04.002
  48. Steinemann S, Berg B, DiTullio A, Skinner A, Terada K, Anzelon K, Ho HC (2012) Assessing teamwork in the trauma bay: introduction of a modified “NOTECHS” scale for trauma. Am J Surg 203(1):69–75. doi:10.1016/j.amjsurg.2011.08.004 CrossRefGoogle Scholar
  49. Thomas EJ, Sexton JB, Helmreich RL (2004) Translating teamwork behaviours from aviation to healthcare: development of behavioural markers for neonatal resuscitation. Qual Saf Health Care 13(Suppl_1):i57–i64. doi:10.1136/qshc.2004.009811
  50. Undre S, Sevdalis N, Healey A, Darzi A, Vincent C (2007) Observational teamwork assessment for surgery (OTAS): refinement and application in urological surgery. World J Surg 31(7):1373–1381. doi:10.1007/s00268-007-9053-z CrossRefGoogle Scholar
  51. Walker S, Brett S, McKay A, Lambden S, Vincent C, Sevdalis N (2011) Observational skill-based clinical assessment tool for resuscitation (OSCAR): development and validation. Resuscitation 82(7):835–844. doi:10.1016/j.resuscitation.2011.03.009 CrossRefGoogle Scholar
  52. Weller J, Frengley R, Torrie J, Shulruf B, Jolly B, Hopley L et al (2011) Evaluation of an instrument to measure teamwork in multidisciplinary critical care teams. BMJ Qual Saf 20(3):216–222. doi:10.1136/bmjqs.2010.041913 CrossRefGoogle Scholar
  53. Westli HK, Johnsen BH, Eid J, Rasten I, Brattebø G (2010) Teamwork skills, shared mental models, and performance in simulated trauma teams: an independent group design. Scand J Trauma, Resusc Emerg Med 18(1):47. doi:10.1186/1757-7241-18-47 CrossRefGoogle Scholar
  54. Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D (2006) Development of a rating system for surgeons’ non-technical skills. Med Educ 40(11):1098–1104. doi:10.1111/j.1365-2929.2006.02610.x CrossRefGoogle Scholar
  55. Yule Steven, Flin R, Maran N, Rowley D, Youngson G, Paterson-Brown S (2008) Surgeons’ non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg 32(4):548–556. doi:10.1007/s00268-007-9320-z CrossRefGoogle Scholar
  56. Yule S, Rowley D, Flin R, Maran N, Youngson G, Duncan J, Paterson-Brown S (2009) Experience matters: comparing novice and expert ratings of non-technical skills using the NOTSS system. ANZ J Surg 79(3):154–160CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  • Rikke M. H. G. Jepsen
    • 1
  • Doris Østergaard
    • 1
  • Peter Dieckmann
    • 1
  1. 1.Danish Institute for Medical SimulationCapital Region of Denmark and University of CopenhagenHerlevDenmark

Personalised recommendations