Skip to main content
SpringerLink
Log in

Surgeons’ Non-technical Skills in the Operating Room: Reliability Testing of the NOTSS Behavior Rating System

  • Published:
World Journal of Surgery Aims and scope Submit manuscript

Abstract

Background

Previous research has shown that surgeons’ intraoperative non-technical skills are related to surgical outcomes. The aim of this study was to evaluate the reliability of the NOTSS (Non-technical Skills for Surgeons) behavior rating system. Based on task analysis, the system incorporates five categories of skills for safe surgical practice (Situation Awareness, Decision Making, Task Management, Communication & Teamwork, and Leadership).

Methods

Consultant (attending) surgeons (n = 44) from five Scottish hospitals attended one of six experimental sessions and were trained to use the NOTSS system. They then used the system to rate consultant surgeons’ behaviors in six simulated operating room scenarios that were presented using video. Surgeons’ ratings of the behaviors demonstrated in each scenario were compared to expert ratings (“accuracy”), and assessed for inter-rater reliability and internal consistency.

Results

The NOTSS system had a consistent internal structure. Although raters had minimal training, rating “accuracy” for acceptable/unacceptable behavior was above 60% for all categories, with mean of 0.67 scale points difference from reference (expert) ratings (on 4-point scale). For inter-rater reliability, the mean values of within-group agreement (r wg) were acceptable for the categories Communication & Teamwork (.70), and Leadership (.72), but below a priori criteria for other categories. Intra-class correlation coefficients (ICC) indicated high agreement using average measures (values were .95–.99).

Conclusions

With the requisite training, the prototype NOTSS system could be used reliably by surgeons to observe and rate surgeons’ behaviors. The instrument should now be tested for usability in the operating room.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Gawande AA, Zinner MJ, Studdert DM, et al. (2003) Analysis of errors reported by surgeons at three teaching hospitals. Surgery 133:614–621

    Article  PubMed  Google Scholar 

  2. Yule S, Flin R, Maran N, et al. (2006) Non-technical skills for surgeons in the operating room. A review of the literature. Surgery 39:140–149

    Article  Google Scholar 

  3. Stevenson KS, Gibson SC, Rogers PN, et al. Process of care in acute surgical admissions: room for improvement. Br J Surg in press

  4. Christian C, Gustafon M, Roth E, et al. (2006) A prospective study of patient safety in the operating room. Surgery 139:159–173

    Article  PubMed  Google Scholar 

  5. Studdert DM, Mello MM, Gawande AA, et al. (2006) Claims, errors, and compensation payments in medical malpractice litigation. N Engl J Med 354:2024–2033

    Article  PubMed  CAS  Google Scholar 

  6. Carthey J, de Leval MR, Wright DJ, et al. (2003) Behavioral markers of surgical excellence. Safety Sci 41:409–425

    Article  Google Scholar 

  7. Moorthy K, Munz Y, Adams S, et al. (2005) A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Ann Surg 242:631–641

    Article  PubMed  Google Scholar 

  8. Templeton S, Feinmann J (2006) Arrogant surgeons ‘risk another Bristol babies scandal’. The Sunday Times London September 3

  9. Davidson P (2002) The surgeon of the future and implications for training. Aust N Z J Surg 72:822–828

    Article  Google Scholar 

  10. Flin R, Maran N (2004) Identifying and training non-technical skills in acute medicine. Qual Safety Healthcare i180–i184

  11. Flin R, Martin L, Goeters K, et al. (2003) Pilots’ non-technical skills: NOTECHS. Hum Factors Aerospace Safety 3:95–117

    Google Scholar 

  12. Fletcher G, Flin R, McGeorge P, et al. (2003) Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioral marker system. Br J Anaesthesia 90:580–588

    Article  CAS  Google Scholar 

  13. Klampfer B, Flin R, Helmreich RL, et al. (2001) Enhancing performance in high risk environments: recommendations for the use of behavioral markers. Berlin: GIHRE

  14. Baker D, Mulqueen C, Dismukes R (2001) Training raters to assess resource management skills. In Salas E, Bowers C, Edens E, editors, Improving Teamwork in Organizations. Mahwah, NJ, Lawrence Earlbaum, 131–145

    Google Scholar 

  15. Flin R, Yule S, Paterson-Brown S, et al. (2005) Surgeons’ non-technical skills. Surg News 4:83–85

    Google Scholar 

  16. Yule S, Flin R, Paterson-Brown S, et al. (2006) Development of a rating system for surgeons’ non-technical skills. Med Ed 40:1098–1104

    Article  CAS  Google Scholar 

  17. Goldsmith T, Johnson P (2002) Assessing and improving evaluation of aircrew performance. Int J Aviation Psychology 12:223–240

    Article  Google Scholar 

  18. James L, Demaree R, Wolf G (1993) rwg: an assessment of within-group inter-rater agreement. J Appl Psychology 78:306–309

    Article  Google Scholar 

  19. Nunnally J, Bernstein I (1993) Psychometric Theory. New York, McGraw Hill

    Google Scholar 

  20. Bliese P (2000) Within-group agreement, non-independence, and reliability. Implications for data aggregation and analysis. In Klein K, Kozlowski S, editors, Multilevel theory, research, and methods in organizations. San Francisco, Jossey-Bass

    Google Scholar 

  21. Shrout PE, Fleiss JL (1979) Intraclass correlations: uses in assessing rater reliability. Psychol Bull 86:420–428

    Article  Google Scholar 

  22. O’Connor P, Hormann HJ, Flin R, et al. (2002) Developing a method for evaluating Crew Resource Management skills: a European perspective. Int J Aviation Psychology 12:265–288

    Google Scholar 

  23. Holt R, Boehm-Davis D, Beaubien J (2001) Evaluating resource management training. In Salas E, Bowers C, Edens E, editors. Improving Teamwork in Organizations. Applications of Resource Management Training. Mahwah, NJ, Lawrence Earlbaum Associates

    Google Scholar 

  24. Yule S, Flin R, Rowley D, et al. Debriefing surgeons on non-technical skills (NOTSS). Cognition, Technology & Work (in press)

  25. Davis DA, Mazamanian PE, Fordis M, et al. (2006) Accuracy of physician self-assessment compared with observed measures of competence. JAMA 296:1094–1102

    Article  PubMed  CAS  Google Scholar 

  26. Donaldson LJ (2006) Good doctors, safer patients: proposals to strengthen the system to assure and improve the performance of doctors and to protect the safety of patients. London, Department of Health

  27. Flin R, O’Connor P, Crichton M (2007) Safety at the Sharp End. A Guide to Non-Technical Skills. Aldershot, UK, Ashgate, in press

Download references

Acknowledgments

The NOTSS system was developed under funding from the Royal College of Surgeons of Edinburgh and NHS Education Scotland. The views presented are those of the authors and should not be taken to represent the position or policy of the funding bodies. The authors thank the surgeons who took part in this study.

Author information

Authors and Affiliations

  1. School of Psychology, University of Aberdeen, Aberdeen, AB24 2UB, Scotland, United Kingdom

    Steven Yule & Rhona Flin

  2. Department of Anaesthesia, Royal Infirmary of Edinburgh, 51 Little France Crescent, Old Dalkeith Road, Edinburgh, Scotland, EH16 4SA, United Kingdom

    Nicola Maran

  3. Department of Orthopaedic and Trauma Surgery, University of Dundee, Ninewells Hospital, Dundee, Scotland, DD2 9SY, United Kingdom

    David Rowley

  4. Royal Aberdeen Children’s Hospital, Cornhill Road, Aberdeen, Scotland, AB25 2ZD, United Kingdom

    George Youngson

  5. Department of Surgery, Royal Infirmary Edinburgh, 51 Little France Crescent, Old Dalkeith Road, Edinburgh, Scotland, EH16 4SA, United Kingdom

    Simon Paterson-Brown

Authors
  1. Steven Yule
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rhona Flin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicola Maran
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Rowley
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. George Youngson
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Simon Paterson-Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Steven Yule.

Appendix 1. Description of video scenarios

Appendix 1. Description of video scenarios

Scenario 1. General surgery

A 62-year-old obese man with symptomatic gallstones is about to undergo emergency laparoscopic cholecystectomy. The patient is in the operating room and has been anesthetised. The surgeon arrives late and does not appear to know the patient. There are no long ports in the operating room or day surgery, so the surgeon decides to operate with short ports instead, exposing the patient to risk.

Scenario 2. Orthopaedic surgery

A 69-year-old woman is undergoing a foot operation. The surgeon appears to be in control but does not communicate with the rest of the team (radiographer, trainee surgeon). He experiences technical difficulties with a drill.

Scenario 3. General surgery

A 51-year-old man is undergoing elective mesh repair of an inguinal hernia. The consultant surgeon is bad-tempered, breaks operating room protocol, and openly questions the competence of operating room team members in a hostile manner.

Scenario 4. Orthopedic surgery

An 86-year-old woman is undergoing a trial reduction of femur. The consultant surgeon tells the trainee to be careful but is chatting with another team member when the trainee breaks the patient’s femur. The consultant surgeon blames the trainee and is initially aggressive, but rapidly regains control of the situation.

Scenario 5. General surgery

A 40-year-old man has been anesthetized and is about to undergo repair of an inguinal hernia. A junior member of staff has taken the patient’s consent for the operation, and there is confusion over which side should be operated on. The trainee surgeon is keen to proceed and is sure that it is the left side, but the consultant surgeon decides to stop the operation and wake the patient up to make sure, involving other team members in the decision-making process.

Scenario 6. Orthopedic surgery

A 68-year-old woman is undergoing a knee replacement. The consultant surgeon treats the trainee and scrub nurse differently—he is friendly and encouraging with the trainee and not very communicative with the nurse. She makes a minor error as a direct result of the surgeons’ ambiguous communication style.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Yule, S., Flin, R., Maran, N. et al. Surgeons’ Non-technical Skills in the Operating Room: Reliability Testing of the NOTSS Behavior Rating System. World J Surg 32, 548–556 (2008). https://doi.org/10.1007/s00268-007-9320-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00268-007-9320-z

Keywords

Search

Navigation