Skip to main content

Measuring Decision-Making During Thyroidectomy: Validity Evidence for a Web-Based Assessment Tool

World Journal of Surgery volume 42, pages 376–383 (2018)Cite this article

Abstract

Background

Errors in judgment during thyroidectomy can lead to recurrent laryngeal nerve injury and other complications. Despite the strong link between patient outcomes and intraoperative decision-making, methods to evaluate these complex skills are lacking. The purpose of this study was to develop objective metrics to evaluate advanced cognitive skills during thyroidectomy and to obtain validity evidence for them.

Methods

An interactive online learning platform was developed (www.thinklikeasurgeon.com). Trainees and surgeons from four institutions completed a 33-item assessment, developed based on a cognitive task analysis and expert Delphi consensus. Sixteen items required subjects to make annotations on still frames of thyroidectomy videos, and accuracy scores were calculated based on an algorithm derived from experts’ responses (“visual concordance test,” VCT). Seven items were short answer (SA), requiring users to type their answers, and scores were automatically calculated based on their similarity to a pre-populated repertoire of correct responses. Test–retest reliability, internal consistency, and correlation of scores with self-reported experience and training level (novice, intermediate, expert) were calculated.

Results

Twenty-eight subjects (10 endocrine surgeons and otolaryngologists, 18 trainees) participated. There was high test–retest reliability (intraclass correlation coefficient = 0.96; n = 10) and internal consistency (Cronbach’s α = 0.93). The assessment demonstrated significant differences between novices, intermediates, and experts in total score (p < 0.01), VCT score (p < 0.01) and SA score (p < 0.01). There was high correlation between total case number and total score (ρ = 0.95, p < 0.01), between total case number and VCT score (ρ = 0.93, p < 0.01), and between total case number and SA score (ρ = 0.83, p < 0.01).

Conclusion

This study describes the development of novel metrics and provides validity evidence for an interactive Web-based platform to objectively assess decision-making during thyroidectomy.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  1. Chan WF, Lo CY (2006) Pitfalls of intraoperative neuromonitoring for predicting postoperative recurrent laryngeal nerve function during thyroidectomy. World J Surg 30:806–812. https://doi.org/10.1007/s00268-005-0355-8

    Article  PubMed  Google Scholar 

  2. Jeannon JP, Orabi AA, Bruch GA et al (2009) Diagnosis of recurrent laryngeal nerve palsy after thyroidectomy: a systematic review. Int J Clin Pract 63:624–629

    Article  PubMed  Google Scholar 

  3. Lo CY, Kwok KF, Yuen PW (2000) A prospective evaluation of recurrent laryngeal nerve paralysis during thyroidectomy. Arch Surg 135:204–207

    CAS  Article  PubMed  Google Scholar 

  4. Serpell JW, Yeung MJ, Grodski S (2009) The motor fibers of the recurrent laryngeal nerve are located in the anterior extralaryngeal branch. Ann Surg 249:648–652

    Article  PubMed  Google Scholar 

  5. Steurer M, Passler C, Denk DM et al (2002) Advantages of recurrent laryngeal nerve identification in thyroidectomy and parathyroidectomy and the importance of preoperative and postoperative laryngoscopic examination in more than 1000 nerves at risk. Laryngoscope 112:124–133

    Article  PubMed  Google Scholar 

  6. Bononi M, de Cesare A, Atella F et al (2000) Surgical treatment of multinodular goiter: incidence of lesions of the recurrent nerves after total thyroidectomy. Int Surg 85:190–193

    CAS  PubMed  Google Scholar 

  7. Erbil Y, Barbaros U, Issever H et al (2007) Predictive factors for recurrent laryngeal nerve palsy and hypoparathyroidism after thyroid surgery. Clin Otolaryngol 32:32–37

    CAS  Article  PubMed  Google Scholar 

  8. Abadin SS, Kaplan EL, Angelos P (2010) Malpractice litigation after thyroid surgery: the role of recurrent laryngeal nerve injuries, 1989–2009. Surgery 148:718–722

    Article  PubMed  Google Scholar 

  9. Kern KA (1993) Medicolegal analysis of errors in diagnosis and treatment of surgical endocrine disease. Surgery 114:1167–1173

    CAS  PubMed  Google Scholar 

  10. Strasberg SM, Sanabria JR, Clavien PA (1992) Complications of laparoscopic cholecystectomy. Can J Surg 35:275–280

    CAS  PubMed  Google Scholar 

  11. Strasberg SM, Hertl M, Soper NJ (1995) An analysis of the problem of biliary injury during laparoscopic cholecystectomy. J Am Coll Surg 180:101–125

    CAS  PubMed  Google Scholar 

  12. Way LW, Stewart L, Gantert W et al (2003) Causes and prevention of laparoscopic bile duct injuries: analysis of 252 cases from a human factors and cognitive psychology perspective. Ann Surg 237:460–469

    PubMed  PubMed Central  Google Scholar 

  13. Strasberg SM, Eagon CJ, Drebin JA (2000) The “hidden cystic duct” syndrome and the infundibular technique of laparoscopic cholecystectomy–the danger of the false infundibulum. J Am Coll Surg 191:661–667

    CAS  Article  PubMed  Google Scholar 

  14. Strasberg SM, Gouma DJ (2012) ‘Extreme’ vasculobiliary injuries: association with fundus-down cholecystectomy in severely inflamed gallbladders. HPB 14:1–8

    Article  PubMed  PubMed Central  Google Scholar 

  15. Chiang FY, Lu IC, Kuo WR et al (2008) The mechanism of recurrent laryngeal nerve injury during thyroid surgery–the application of intraoperative neuromonitoring. Surgery 143:743–749

    Article  PubMed  Google Scholar 

  16. Rulli F, Ambrogi V, Dionigi G et al (2014) Meta-analysis of recurrent laryngeal nerve injury in thyroid surgery with or without intraoperative nerve monitoring. Acta Otorhinolaryngol Ital 34:223–229

    CAS  PubMed  PubMed Central  Google Scholar 

  17. Wheeler MH (1999) Thyroid surgery and the recurrent laryngeal nerve. Br J Surg 86:291–292

    CAS  Article  PubMed  Google Scholar 

  18. Rogers SO Jr, Gawande AA, Kwaan M et al (2006) Analysis of surgical errors in closed malpractice claims at 4 liability insurers. Surgery 140:25–33

    Article  PubMed  Google Scholar 

  19. Guru V, Tu JV, Etchells E et al (2008) Relationship between preventability of death after coronary artery bypass graft surgery and all-cause risk-adjusted mortality rates. Circulation 117:2969–2976

    Article  PubMed  Google Scholar 

  20. Madani A, Watanabe Y, Vassiliou M et al (2016) Defining competencies for safe thyroidectomy: an international Delphi consensus. Surgery 159(86–94):96–101

    Google Scholar 

  21. Madani A, Watanabe Y, Bilgic E et al (2017) Measuring intra-operative decision-making during laparoscopic cholecystectomy: validity evidence for a novel interactive Web-based assessment tool. Surg Endosc 31:1203–1212

    Article  PubMed  Google Scholar 

  22. Maertens H, Madani A, Landry T et al (2016) Systematic review of e-learning for surgical training. Br J Surg 103:1428–1437

    CAS  Article  PubMed  Google Scholar 

  23. Charlin B, Roy L, Brailovsky C et al (2000) The script concordance test: a tool to assess the reflective clinician. Teach Learn Med 12:189–195

    CAS  Article  PubMed  Google Scholar 

  24. Meterissian SH (2006) A novel method of assessing clinical reasoning in surgical residents. Surg Innov 13:115–119

    Article  PubMed  Google Scholar 

  25. Nouh T, Boutros M, Gagnon R et al (2012) The script concordance test as a measure of clinical reasoning: a national validation study. Am J Surg 203:530–534

    Article  PubMed  Google Scholar 

  26. Lubarsky S, Dory V, Duggan P et al (2013) Script concordance testing: from theory to practice: AMEE guide no. 75. Med Teach 35:184–193

    Article  PubMed  Google Scholar 

  27. Downing SM (2003) Validity: on meaningful interpretation of assessment data. Med Educ 37:830–837

    Article  PubMed  Google Scholar 

  28. Adam MA, Thomas S, Youngwirth L et al (2017) Is there a minimum number of thyroidectomies a surgeon should perform to optimize patient outcomes? Ann Surg 265:402–407

    Article  PubMed  Google Scholar 

  29. Flin R, Youngson G, Yule S (2007) How do surgeons make intraoperative decisions? Qual Saf Health Care 16:235–239

    Article  PubMed  PubMed Central  Google Scholar 

  30. Frank JR, Snell LS, Cate OT et al (2010) Competency-based medical education: theory to practice. Med Teach 32:638–645

    Article  PubMed  Google Scholar 

  31. Singh P, Aggarwal R, Tahir M et al (2015) A randomized controlled study to evaluate the role of video-based coaching in training laparoscopic skills. Ann Surg 261:862–869

    Article  PubMed  Google Scholar 

  32. Ericsson KA (2014) Necessity is the mother of invention: video recording firsthand perspectives of critical medical procedures to make simulated training more effective. Acad Med 89:17–20

    Article  PubMed  Google Scholar 

  33. Andersen DK (2012) How can educators use simulation applications to teach and assess surgical judgment? Acad Med 87:934–941

    Article  PubMed  Google Scholar 

  34. Al-Qurayshi Z, Robins R, Hauch A et al (2016) Association of surgeon volume with outcomes and cost savings following thyroidectomy: a national forecast. JAMA Otolaryngol Head Neck Surg 142:32–39

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We acknowledge the work of Robert Messina for developing Think Like A Surgeon©, and all the surgeons and trainees who participated in this study.

Author information

Authors and Affiliations

  1. Department of Surgery, McGill University Health Centre, 1650 Cedar Avenue, Rm D6-257, Montreal, QC, H3G 1A4, Canada

    Amin Madani, Jordan Gornitsky, Cassandre Benay, Roger Tabah & Elliot J. Mitmaker

  2. Department of Gastroenterological Surgery II, Hokkaido University, Sapporo, Japan

    Yusuke Watanabe

  3. Department of Surgery, Stony Brook University Medical Center, Stony Brook, NY, USA

    Maria S. Altieri

  4. Department of Surgery and Cancer, Imperial College London, London, UK

    Philip H. Pucher

Authors
  1. Amin Madani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jordan Gornitsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yusuke Watanabe
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Cassandre Benay
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Maria S. Altieri
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Philip H. Pucher
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Roger Tabah
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Elliot J. Mitmaker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amin Madani.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Madani, A., Gornitsky, J., Watanabe, Y. et al. Measuring Decision-Making During Thyroidectomy: Validity Evidence for a Web-Based Assessment Tool. World J Surg 42, 376–383 (2018). https://doi.org/10.1007/s00268-017-4322-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00268-017-4322-y