Previous research has shown that surgeons’ intraoperative non-technical skills are related to surgical outcomes. The aim of this study was to evaluate the reliability of the NOTSS (Non-technical Skills for Surgeons) behavior rating system. Based on task analysis, the system incorporates five categories of skills for safe surgical practice (Situation Awareness, Decision Making, Task Management, Communication & Teamwork, and Leadership).
Consultant (attending) surgeons (n = 44) from five Scottish hospitals attended one of six experimental sessions and were trained to use the NOTSS system. They then used the system to rate consultant surgeons’ behaviors in six simulated operating room scenarios that were presented using video. Surgeons’ ratings of the behaviors demonstrated in each scenario were compared to expert ratings (“accuracy”), and assessed for inter-rater reliability and internal consistency.
The NOTSS system had a consistent internal structure. Although raters had minimal training, rating “accuracy” for acceptable/unacceptable behavior was above 60% for all categories, with mean of 0.67 scale points difference from reference (expert) ratings (on 4-point scale). For inter-rater reliability, the mean values of within-group agreement (r wg) were acceptable for the categories Communication & Teamwork (.70), and Leadership (.72), but below a priori criteria for other categories. Intra-class correlation coefficients (ICC) indicated high agreement using average measures (values were .95–.99).
With the requisite training, the prototype NOTSS system could be used reliably by surgeons to observe and rate surgeons’ behaviors. The instrument should now be tested for usability in the operating room.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Gawande AA, Zinner MJ, Studdert DM, et al. (2003) Analysis of errors reported by surgeons at three teaching hospitals. Surgery 133:614–621
Yule S, Flin R, Maran N, et al. (2006) Non-technical skills for surgeons in the operating room. A review of the literature. Surgery 39:140–149
Stevenson KS, Gibson SC, Rogers PN, et al. Process of care in acute surgical admissions: room for improvement. Br J Surg in press
Christian C, Gustafon M, Roth E, et al. (2006) A prospective study of patient safety in the operating room. Surgery 139:159–173
Studdert DM, Mello MM, Gawande AA, et al. (2006) Claims, errors, and compensation payments in medical malpractice litigation. N Engl J Med 354:2024–2033
Carthey J, de Leval MR, Wright DJ, et al. (2003) Behavioral markers of surgical excellence. Safety Sci 41:409–425
Moorthy K, Munz Y, Adams S, et al. (2005) A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg 242:631–641
Templeton S, Feinmann J (2006) Arrogant surgeons ‘risk another Bristol babies scandal’. The Sunday Times London September 3
Davidson P (2002) The surgeon of the future and implications for training. Aust N Z J Surg 72:822–828
Flin R, Maran N (2004) Identifying and training non-technical skills in acute medicine. Qual Safety Healthcare i180–i184
Flin R, Martin L, Goeters K, et al. (2003) Pilots’ non-technical skills: NOTECHS. Hum Factors Aerospace Safety 3:95–117
Fletcher G, Flin R, McGeorge P, et al. (2003) Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioral marker system. Br J Anaesthesia 90:580–588
Klampfer B, Flin R, Helmreich RL, et al. (2001) Enhancing performance in high risk environments: recommendations for the use of behavioral markers. Berlin: GIHRE
Baker D, Mulqueen C, Dismukes R (2001) Training raters to assess resource management skills. In Salas E, Bowers C, Edens E, editors, Improving Teamwork in Organizations. Mahwah, NJ, Lawrence Earlbaum, 131–145
Flin R, Yule S, Paterson-Brown S, et al. (2005) Surgeons’ non-technical skills. Surg News 4:83–85
Yule S, Flin R, Paterson-Brown S, et al. (2006) Development of a rating system for surgeons’ non-technical skills. Med Ed 40:1098–1104
Goldsmith T, Johnson P (2002) Assessing and improving evaluation of aircrew performance. Int J Aviation Psychology 12:223–240
James L, Demaree R, Wolf G (1993) rwg: an assessment of within-group inter-rater agreement. J Appl Psychology 78:306–309
Nunnally J, Bernstein I (1993) Psychometric Theory. New York, McGraw Hill
Bliese P (2000) Within-group agreement, non-independence, and reliability. Implications for data aggregation and analysis. In Klein K, Kozlowski S, editors, Multilevel theory, research, and methods in organizations. San Francisco, Jossey-Bass
Shrout PE, Fleiss JL (1979) Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86:420–428
O’Connor P, Hormann HJ, Flin R, et al. (2002) Developing a method for evaluating Crew Resource Management skills: a European perspective. Int J Aviation Psychology 12:265–288
Holt R, Boehm-Davis D, Beaubien J (2001) Evaluating resource management training. In Salas E, Bowers C, Edens E, editors. Improving Teamwork in Organizations. Applications of Resource Management Training. Mahwah, NJ, Lawrence Earlbaum Associates
Yule S, Flin R, Rowley D, et al. Debriefing surgeons on non-technical skills (NOTSS). Cognition, Technology & Work (in press)
Davis DA, Mazamanian PE, Fordis M, et al. (2006) Accuracy of physician self-assessment compared with observed measures of competence. JAMA 296:1094–1102
Donaldson LJ (2006) Good doctors, safer patients: proposals to strengthen the system to assure and improve the performance of doctors and to protect the safety of patients. London, Department of Health
Flin R, O’Connor P, Crichton M (2007) Safety at the Sharp End. A Guide to Non-Technical Skills. Aldershot, UK, Ashgate, in press
The NOTSS system was developed under funding from the Royal College of Surgeons of Edinburgh and NHS Education Scotland. The views presented are those of the authors and should not be taken to represent the position or policy of the funding bodies. The authors thank the surgeons who took part in this study.
Appendix 1. Description of video scenarios
Appendix 1. Description of video scenarios
Scenario 1. General surgery
A 62-year-old obese man with symptomatic gallstones is about to undergo emergency laparoscopic cholecystectomy. The patient is in the operating room and has been anesthetised. The surgeon arrives late and does not appear to know the patient. There are no long ports in the operating room or day surgery, so the surgeon decides to operate with short ports instead, exposing the patient to risk.
Scenario 2. Orthopaedic surgery
A 69-year-old woman is undergoing a foot operation. The surgeon appears to be in control but does not communicate with the rest of the team (radiographer, trainee surgeon). He experiences technical difficulties with a drill.
Scenario 3. General surgery
A 51-year-old man is undergoing elective mesh repair of an inguinal hernia. The consultant surgeon is bad-tempered, breaks operating room protocol, and openly questions the competence of operating room team members in a hostile manner.
Scenario 4. Orthopedic surgery
An 86-year-old woman is undergoing a trial reduction of femur. The consultant surgeon tells the trainee to be careful but is chatting with another team member when the trainee breaks the patient’s femur. The consultant surgeon blames the trainee and is initially aggressive, but rapidly regains control of the situation.
Scenario 5. General surgery
A 40-year-old man has been anesthetized and is about to undergo repair of an inguinal hernia. A junior member of staff has taken the patient’s consent for the operation, and there is confusion over which side should be operated on. The trainee surgeon is keen to proceed and is sure that it is the left side, but the consultant surgeon decides to stop the operation and wake the patient up to make sure, involving other team members in the decision-making process.
Scenario 6. Orthopedic surgery
A 68-year-old woman is undergoing a knee replacement. The consultant surgeon treats the trainee and scrub nurse differently—he is friendly and encouraging with the trainee and not very communicative with the nurse. She makes a minor error as a direct result of the surgeons’ ambiguous communication style.
About this article
Cite this article
Yule, S., Flin, R., Maran, N. et al. Surgeons’ Non-technical Skills in the Operating Room: Reliability Testing of the NOTSS Behavior Rating System. World J Surg 32, 548–556 (2008). https://doi.org/10.1007/s00268-007-9320-z
- Situation Awareness
- Task Management
- Reference Rating
- Element Rating
- Crew Resource Management