Abstract
Simulation-based surgical skills training during preclinical education is a persistent challenge due to time constraints of trainees and instructors alike. Self-directed practice is resource-efficient and flexible; however, insight into technical proficiency among trainees is often lacking. The purpose of this study is to prospectively assess the accuracy of self-assessments among medical students learning basic surgical suturing. Over seven weekly practice sessions, preclinical medical students performed serial repetitions of a simulation-based suturing task under one-on-one observation by one of four trainers. Following each task repetition, self- and trainer-assessments (SA–TA) were performed using a 36-point weighted checklist of technical standards developed a priori by expert consensus. Upon study completion, agreement between SA and TA was measured using weighted Cohen’s kappa coefficients. Twenty-nine medical students each performed a median of 25 suture task repetitions (IQR 21.5–28). Self-assessments tended to overestimate proficiency during the first tertile of practice attempts. Agreement between SA and TA improved with experience, such that the weighted kappa statistics for the two-handed and instrument ties were >0.81 after 18–21 task attempts. Inexperienced trainees frequently overestimate technical proficiency through self-assessments. However, this bias diminishes with repetitive practice. Only after trainees have attained the capacity to accurately self-assess can effective self-directed learning take place.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aggarwal, R., Hance, J., Undre, S., Ratnasothy, J., Moorthy, K., Chang, A., et al. (2006). Training junior operative residents in laparoscopic suturing skills is feasible and efficacious. Surgery, 139(6), 729–734.
Anthoney, T. R. (1986). A discrepancy in objective and subjective measures of knowledge: Do some medical students with learning problems delude themselves? Medical Education, 20(1), 17–22.
Arnold, L., Willoughby, T. L., & Calkins, E. V. (1985). Self-evaluation in undergraduate medical education: A longitudinal perspective. Journal of Medical Education, 60(1), 21–28.
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191–215.
Brennan, P., & Silman, A. (1992). Statistical methods for assessing observer variability in clinical measures. British Medical Journal, 304(6840), 1491–1494.
Brunt, L. M., Halpin, V. J., Klingensmith, M. E., Tiemann, D., Matthews, B. D., Spitler, J. A., et al. (2008). Accelerated skills preparation and assessment for senior medical students entering surgical internship. Journal of the American College of Surgeons, 206(5), 897–904. discussion 904–907.
Brydges, R., Carnahan, H., & Dubrowski, A. (2009). Assessing suturing skills in a self-guided learning setting: Absolute symmetry error. Advances in Health Sciences Education : Theory and Practice, 14(5), 685–695.
Brydges, R., Carnahan, H., Rose, D., & Dubrowski, A. (2010). Comparing self-guided learning and educator-guided learning formats for simulation-based clinical training. Journal of Advanced Nursing, 66(8), 1832–1844.
Chipman, J. G., & Schmitz, C. C. (2009). Using objective structured assessment of technical skills to evaluate a basic skills simulation curriculum for first-year surgical residents. Journal of the American College of Surgeons, 209(3), 364–370.
Datta, V., Bann, S., Mandalia, M., & Darzi, A. (2006). The surgical efficiency score: A feasible, reliable, and valid method of skills assessment. American Journal of Surgery, 192(3), 372–378.
Derossis, A. M., Fried, G. M., Abrahamowicz, M., Sigman, H. H., Barkun, J. S., & Meakins, J. L. (1998). Development of a model for training and evaluation of laparoscopic skills. American Journal of Surgery, 175(6), 482–487.
Eubanks, T. R., Clements, R. H., Pohl, D., Williams, N., Schaad, D. C., Horgan, S., et al. (1999). An objective scoring system for laparoscopic cholecystectomy. Journal of the American College of Surgeons, 189(6), 566–574.
Fitzgerald, J. T., White, C. B., & Gruppen, L. D. (2003). A longitudinal study of self-assessment accuracy. Medical Education, 37(7), 645–649.
Fraser, S. A., Klassen, D. R., Feldman, L. S., Ghitulescu, G. A., Stanbridge, D., & Fried, G. M. (2003). Evaluating laparoscopic skills: Setting the pass/fail score for the MISTELS system. Surgical Endoscopy, 17(6), 964–967.
Gordon, M. J. (1992). Self-assessment programs and their implications for health professions training. Academic Medicine, 67(10), 672–679.
Gow, K. W. (2013). Self-evaluation: How well do surgery residents judge performance on a rotation? The American Journal of Surgery, 205(5), 557–562. discussion 562.
Grow, G. (1991). Teaching learners to be self-directed. Adult Education Quarterly, 41(3), 125–149.
Hildebrand, C., Trowbridge, E., Roach, M. A., Sullivan, A. G., Broman, A. T., & Vogelman, B. (2009). Resident self-assessment and self-reflection: University of Wisconsin-Madison’s Five-Year Study. Journal of General Internal Medicine, 24(3), 361–365.
Hu, Y., Tiemann, D., & Michael Brunt, L. (2013). Video self-assessment of basic suturing and knot tying skills by novice trainees. Journal of Surgical Education, 70(2), 279–283.
Klingensmith, M. E., & Brunt, L. M. (2010). Focused surgical skills training for senior medical students and interns. Surgical Clinics of North America, 90(3), 505–518.
Korndorffer, J. R, Jr, Dunne, J. B., Sierra, R., Stefanidis, D., Touchard, C. L., & Scott, D. J. (2005). Simulator training for laparoscopic suturing using performance goals translates to the operating room. Journal of the American College of Surgeons, 201(1), 23–29.
Krueger, J., & Mueller, R. A. (2002). Unskilled, unaware, or both? The better-than-average heuristic and statistical regression predict errors in estimates of own performance. Journal of Personality and Social Psychology, 82(2), 180–188.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174.
Lipsett, P. A., Harris, I., & Downing, S. (2011). Resident self-other assessor agreement: Influence of assessor, competency, and performance level. Archives of Surgery, 146(8), 901–906.
MacDonald, J., Williams, R. G., & Rogers, D. A. (2003). Self-assessment in simulation-based surgical skills training. American Journal of Surgery, 185(4), 319–322.
Mandel, L. S., Goff, B. A., & Lentz, G. M. (2005). Self-assessment of resident surgical skills: Is it feasible? American Journal of Obstetrics and Gynecology, 193(5), 1817–1822.
Martin, J. A., Regehr, G., Reznick, R., MacRae, H., Murnaghan, J., Hutchison, C., et al. (1997). Objective structured assessment of technical skill (OSATS) for surgical residents. British Journal of Surgery, 84(2), 273–278.
Miller, S. H. (2005). American Board of Medical Specialties and repositioning for excellence in lifelong learning: Maintenance of certification. Journal of Continuing Education in the Health Professions, 25(3), 151–156.
Nakayama, D. K., & Steiber, A. (1990). Surgery interns’ experience with surgical procedures as medical students. The American Journal of Surgery, 159(3), 341–343. discussion 344.
Pearson, A. M., Gallagher, A. G., Rosser, J. C., & Satava, R. M. (2002). Evaluation of structured and quantitative training methods for teaching intracorporeal knot tying. Surgical Endoscopy, 16(1), 130–137.
Safir, O., Williams, C. K., Dubrowski, A., Backstein, D., & Carnahan, H. (2013). Self-directed practice schedule enhances learning of suturing skills. Canadian Journal of Surgery, 56(6), E142–E147.
Sargeant, J. M., Mann, K. V., van der Vleuten, C. P., & Metsemakers, J. F. (2009). Reflection: A link between receiving and using assessment feedback. Advances in Health Sciences Education, 14(3), 399–410.
Scott, D. J., Goova, M. T., & Tesfay, S. T. (2007). A cost-effective proficiency-based knot-tying and suturing curriculum for residency programs. Journal of Surgical Research, 141(1), 7–15.
Stewart, R. A., Hauge, L. S., Stewart, R. D., Rosen, R. L., Charnot-Katsikas, A., Prinz, R. A., et al. (2007). A CRASH course in procedural skills improves medical students’ self-assessment of proficiency, confidence, and anxiety. American Journal of Surgery, 193(6), 771–773.
Stewart, J., O’Halloran, C., Barton, J. R., Singleton, S. J., Harrigan, P., & Spencer, J. (2000). Clarifying the concepts of confidence and competence to produce appropriate self-evaluation measurement scales. Medical Education, 34(11), 903–909.
Taffinder, N., Sutton, C., Fishwick, R. J., McManus, I. C., & Darzi, A. (1998). Validation of virtual reality to teach and assess psychomotor skills in laparoscopic surgery: Results from randomised controlled studies using the MIST VR laparoscopic simulator. Studies in Health Technology and Informatics, 50, 124–130.
van Hove, P. D., Tuijthof, G. J., Verdaasdonk, E. G., Stassen, L. P., & Dankelman, J. (2010). Objective assessment of technical surgical skills. British Journal of Surgery, 97(7), 972–987.
Vassiliou, M. C., Ghitulescu, G. A., Feldman, L. S., Stanbridge, D., Leffondre, K., Sigman, H. H., et al. (2006). The MISTELS program to measure technical skill in laparoscopic surgery: Evidence for reliability. Surgical Endoscopy, 20(5), 744–747.
Ward, M., MacRae, H., Schlachta, C., Mamazza, J., Poulin, E., Reznick, R., et al. (2003). Resident self-assessment of operative performance. American Journal of Surgery, 185(6), 521–524.
Xeroulis, G. J., Park, J., Moulton, C. A., Reznick, R. K., Leblanc, V., & Dubrowski, A. (2007). Teaching suturing and knot-tying skills to medical students: A randomized controlled study comparing computer-based video instruction and (concurrent and summary) expert feedback. Surgery, 141(4), 442–449.
Acknowledgements
University of Virginia Academy of Distinguished Educators: Undergraduate Medical Education Research and Innovation Grant.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hu, Y., Kim, H., Mahmutovic, A. et al. Verification of accurate technical insight: a prerequisite for self-directed surgical training. Adv in Health Sci Educ 20, 181–191 (2015). https://doi.org/10.1007/s10459-014-9519-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-014-9519-3