Skip to main content
SpringerLink
Log in

Crowdsourcing and Large-Scale Evaluation

  • Chapter
  • First Online:
Surgeons as Educators

Abstract

Traditional methods of surgical education have been based on an apprenticeship model and the Halsteadian approach of graduated responsibility through residency training. However, current paradigms of surgical skills assessment are limited by subjectivity, poor timeliness, and cost, as they rely on faculty mentors and their pre-existing relationships with trainees. With the increasing use of simulation in surgical education, the challenge of providing appropriate feedback to trainees on a large scale remains unsolved. Crowdsourcing or crowd-based evaluation of technical skills may be one possible solution. The capability of this technology to deliver similarly equivalent feedback as that provided by expert surgeons has been demonstrated across a myriad of simulated and live surgical tasks in a variety of surgical specialties through multiple studies. The potential of this technology has been further explored in its application to surgical quality improvement and identifying surgically precocious trainees and may become even more widely applied in the current model of competency-based surgical education as additional research in this field progresses. Moreover, with the continued introduction of new surgical technology, it may come to play a significant role in enhancing new skills acquisition for surgeons already in practice.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Reznick RK, MacRae H. Teaching surgical skills – changes in the wind. N Engl J Med. 2006;355(25):2664–9.

    Article  CAS  PubMed  Google Scholar 

  2. Polavarapu HV, Kulaylat AN, Sun SHO. 100 years of surgical education: the past, present, and future. Bull Am Coll Surg. 2013;98(7):22–7.

    PubMed  Google Scholar 

  3. Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, D.C.: National Academy Press; 2000. 360 p.

    Google Scholar 

  4. Altman DE, Clancy C, Blendon RJ. Improving patient safety — five years after the IOM report. N Engl J Med. 2004;351(20):2041–3.

    Article  CAS  PubMed  Google Scholar 

  5. Leape LL, Berwick DM. Five years after to err is human what have we learned? JAMA. 2005;293(19):2384–90.

    Article  CAS  PubMed  Google Scholar 

  6. Rogers SO Jr, Gawande AA, Kwaan M, Puopolo AL, Yoon C, Brennan TA, et al. Analysis of surgical errors in closed malpractice claims at 4 liability insurers. Surgery. 2006;140(1):25–33.

    Article  PubMed  Google Scholar 

  7. Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, et al. Surgical skill and complication rates after bariatric surgery. N Engl J Med. 2013;369(15):1434–42.

    Article  CAS  PubMed  Google Scholar 

  8. Bridges M, Diamond DL. The financial impact of teaching surgical residents in the operating room. Am J Surg. 1999;177(1):28–32.

    Article  CAS  PubMed  Google Scholar 

  9. Babineau TJ, Becker J, Gibbons G, Sentovich S, Hess D, Robertson S, et al. The “cost” of operative training for surgical residents. Arch Surg. 2004;139(4):366–70.

    Article  PubMed  Google Scholar 

  10. Papandria D, Rhee D, Ortega G, Zhang Y, Gorgy A, Makary MA, et al. Assessing trainee impact on operative time for common general surgical procedures in ACS-NSQIP. J Surg Educ. 2012;69(2):149–55.

    Article  PubMed  Google Scholar 

  11. Macario A. What does one minute of operating room time cost? J Clin Anesth. 2010;22(4):233–6.

    Article  PubMed  Google Scholar 

  12. Fitts PM, Posner MI. Human performance. Belmont: Brooks/Cole; 1967. p. 469–98.

    Google Scholar 

  13. Ericsson KAK, Krampe RRT, Tesch-Romer C, Ashworth C, Carey G, Grassia J, et al. The role of deliberate practice in the Acquisition of Expert Performance. Psychol Rev. 1993;100(3):363–406.

    Article  Google Scholar 

  14. Boyle E, Al-Akash M, Gallagher AG, Traynor O, Hill ADNP. Optimising surgical training: use of feedback to reduce errors during a simulated surgical procedure. Postgr Med J. 2011;87(1030):524–8.

    Article  Google Scholar 

  15. Bosse HM, Mohr J, Buss B, Krautter M, Weyrich P, Herzog W, et al. The benefit of repetitive skills training and frequency of expert feedback in the early acquisition of procedural skills. BMC Med Educ [Internet]. 2015;15(1):22. Available from: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=4339240&tool=pmcentrez&rendertype=abstract.

  16. Stefanidis D, Korndorffer JJ, Heniford B, Scott D. Limited feedback and video tutorials optimize learning and resource utilization during laparoscopic simulator training. Surgery. 2007;142(2):202–6.

    Article  PubMed  Google Scholar 

  17. Perone J, Fankhauser G, Adhikari D, Mehta H, Woods M, Strohmeyer J, et al. Who did the case? Perceptions on resident operative participation. Am J Surg. 2017;213(4):821–6.

    Article  PubMed  Google Scholar 

  18. Morgan R, Kauffman DF, Doherty G, Sachs T. Resident and attending perceptions of resident involvement: an analysis of ACGME reporting guidelines. J Surg Educ. 2016;74(3):415–22.

    Article  PubMed  Google Scholar 

  19. Reznick RK. Teaching and testing technical skills. Am J Surg. 1993;165(3):358–61.

    Article  CAS  PubMed  Google Scholar 

  20. Williams RG, Klamen DA, McGaghie WC. Cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med. 2003;15(4):270–92.

    Article  PubMed  Google Scholar 

  21. Gundle K, Mickelson D, Hanel D. Reflections in a time of transition: orthopaedic faculty and resident understanding of accreditation schemes and opinions on surgical skills feedback. Med Educ Online. 2016;21(1):30584.

    Article  Google Scholar 

  22. Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, et al. Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84(2):273–8.

    Article  CAS  PubMed  Google Scholar 

  23. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondré K, Stanbridge D, et al. A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg. 2005;190(1):107–13.

    Article  PubMed  Google Scholar 

  24. Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ. Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol. 2012;187(1):247–52.

    Article  PubMed  Google Scholar 

  25. Hogg M, Zenati M, Novak S, Chen Y, Jun Y, Steve J, et al. Grading of surgeon technical performance predicts postoperative pancreatic fistula for Pancreaticoduodenectomy independent of patient-related variables. Ann Surg. 2016;264(3):482–91.

    Article  PubMed  Google Scholar 

  26. Shah J, Darzi A. Surgical skills assessment: an ongoing debate. BJU Int. 2001;88(7):655–60.

    Article  CAS  PubMed  Google Scholar 

  27. Holst D, Kowalewski TM, White LW, Brand TC, Harper JD, Sorensen MD, et al. Crowd-sourced assessment of technical skills: differentiating animate surgical skill through the wisdom of crowds. J Endourol. 2015;29(10):1183–8. 150413093359007

    Article  PubMed  Google Scholar 

  28. Stefanidis D, Arora S, Parrack DM, Hamad GG, Capella J, Grantcharov T, et al. Research priorities in surgical simulation for the 21st century. Am J Surg. 2012;203(1):49–53.

    Article  PubMed  PubMed Central  Google Scholar 

  29. Greenberg CC, Ghousseini HN, Pavuluri Quamme SR, Beasley HL, Wiegmann DA. Surgical coaching for individual performance improvement. Ann Surg [Internet]. 2015;261(1). Available from: http://journals.lww.com/annalsofsurgery/Fulltext/2015/01000/Surgical_Coaching_for_Individual_Performance.8.aspx.

  30. Greenberg C, Dombrowski J, Dimick J. Video-based surgical coaching: an emerging approach to performance improvement. JAMA Surg [Internet]. 2016;151(3):282–3. Available from: https://doi.org/10.1001/jamasurg.2015.4442.

  31. Howe J. The rise of crowdsourcing. Wired Mag [Internet]. 2006;14(6):1–5. Available from: http://www.clickadvisor.com/downloads/Howe_The_Rise_of_Crowdsourcing.pdf.

    Google Scholar 

  32. Garrigos-Simon FJ, Gil-Pechuán I, Estelles-Miguel S. Advances in crowdsourcing. Cham: Springer; 2015. p. 1–183.

    Book  Google Scholar 

  33. Brabham DC. Crowdsourcing as a model for problem solving: an introduction and cases. Converg Int J Res into New Media Technol. 2008;14(1):75–90.

    Article  Google Scholar 

  34. Lévy P. Collective intelligence: Mankind’s emerging world in cyberspace [internet]. Challenges. 1997:277 p. Available from: http://portal.acm.org/citation.cfm?id=550283

  35. Ipeirotis P. Demographics of mechanical turk. Working Paper CeDER-10-01. 2010. http://hdl.handle.net/2451/29585

  36. Ranard BL, Ha YP, Meisel ZF, Asch DA, Hill SS, Becker LB, et al. Crowdsourcing – harnessing the masses to advance health and medicine, a systematic review. J Gen Intern Med. 2014;29:187–203.

    Article  PubMed  Google Scholar 

  37. Cooper S, Khatib F, Treuille A, Barbero J, Lee J, Beenen M, et al. Predicting protein structures with a multiplayer online game. Nature. 2010;466:756–60.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  38. Kawrykow A, Roumanis G, Kam A, Kwak D, Leung C, Wu C, et al. Phylo: a citizen science approach for improving multiple sequence alignment. PLoS One. 2012;7(3):e31362.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  39. Nguyen TB, Wang S, Anugu V, Rose N, McKenna M, Petrick N, et al. Distributed human intelligence for colonic polyp classification in computer-aided detection for CT colonography. Radiology. 2012;262(3):824–33.

    Article  PubMed  PubMed Central  Google Scholar 

  40. Mavandadi S, Dimitrov S, Feng S, Yu F, Sikora U, Yaglidere O, et al. Distributed medical image analysis and diagnosis through crowd-sourced games: a malaria case study. PLoS One. 2012;7(5):e37245.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Freifeld CC, Chunara R, Mekaru SR, Chan EH, Kass-Hout T, Iacucci AA, et al. Participatory epidemiology: use of mobile phones for community-based health reporting. PLoS Med. 2010;7(12):e1000376.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Chen SP, Kirsch S, Zlatev DV, Chang TC, Comstock B, Lendvay TS, et al. Optical biopsy of bladder cancer using crowd-sourced assessment. JAMA Surg. 2016;151(1):90–2.

    Article  PubMed  Google Scholar 

  43. Brady CJ, Villanti AC, Pearson JL, Kirchner TR, Gupta OP, Shah CP. Rapid grading of fundus photographs for diabetic retinopathy using crowdsourcing. J Med Internet Res. 2014;16(10):e233.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Mitry D, Peto T, Hayat S, Blows P, Morgan J, Khaw KT, et al. Crowdsourcing as a screening tool to detect clinical features of glaucomatous optic neuropathy from digital photography. PLoS One. 2015;10(2):e0117401.

    Article  PubMed  PubMed Central  Google Scholar 

  45. Bow HC, Dattilo JR, Jonas AM, Lehmann CU. A crowdsourcing model for creating preclinical medical education study tools. Acad Med. 2013;88(6):766–70.

    Article  PubMed  Google Scholar 

  46. Blackwell KA, Travis MJ, Arbuckle MR, Ross DA. Crowdsourcing medical education. Med Educ. 2016;50(5):576.

    Article  PubMed  PubMed Central  Google Scholar 

  47. Chen C, White L, Kowalewski T, Aggarwal R, Lintott C, Comstock B, et al. Crowd-sourced assessment of technical skills: a novel method to evaluate surgical performance. J Surg Res. 2014;187(1):65–71.

    Article  PubMed  Google Scholar 

  48. Malpani A, Vedula SS, Chen CCG, Hager GD. A study of crowdsourced segment-level surgical skill assessment using pairwise rankings. Int J Comput Assist Radiol Surg. 2015;10(9):1435–47.

    Article  PubMed  Google Scholar 

  49. Aghdasi N, Bly R, White LW, Hannaford B, Moe K, Lendvay TS. Crowd-sourced assessment of surgical skills in cricothyrotomy procedure. J Surg Res. 2015;196(2):302–6.

    Article  PubMed  Google Scholar 

  50. White LW, Kowalewski TM, Dockter RL, Comstock B, Hannaford B, Lendvay TS. Crowd-sourced assessment of technical skill: a valid method for discriminating basic robotic surgery skills. J Endourol. 2015;29(11):1295–301.

    Article  PubMed  Google Scholar 

  51. Powers MK, Boonjindasup A, Pinsky M, Dorsey P, Maddox M, Su L-M, et al. Crowdsourcing assessment of surgeon dissection of renal artery and vein during robotic partial nephrectomy: a novel approach for quantitative assessment of surgical performance. J Endourol. 2016;30(4):447–52.

    Article  PubMed  Google Scholar 

  52. Polin MR, Siddiqui NY, Comstock BA, Hesham H, Brown C, Lendvay TS, et al. Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. Am J Obstet Gynecol. 2016;215:644.e1–7.

    Article  Google Scholar 

  53. Ghani KR, Miller DC, Linsell S, Brachulis A, Lane B, Sarle R, et al. Measuring to improve: peer and crowd-sourced assessments of technical skill with robot-assisted radical prostatectomy. Am J Obstet Gynecol. 2016;69:547–50.

    Google Scholar 

  54. Deal SB, Lendvay TS, Haque MI, Brand T, Comstock B, Warren J, et al. Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills. Am J Surg. 2016;211(2):398–404.

    Article  PubMed  Google Scholar 

  55. Holst D, Kowalewski TM, White LW, Brand TC, Harper JD, Sorenson MD, et al. Crowd-sourced assessment of technical skills: an adjunct to urology resident surgical simulation training. J Endourol. 2015;29(5):604–9.

    Article  PubMed  Google Scholar 

  56. Vernez SL, Huynh V, Osann K, Okhunov Z, Landman J, Clayman R V. C-SATS: assessing surgical skills among urology residency applicants. J Endourol. 2016;31(S1):S-95-S-100. doi: https://doi.org/10.1089/end.2016.0569.

  57. Katz AJ. The role of crowdsourcing in assessing surgical skills. Surg Laparosc Endosc Percutan Tech. 2016;26(4):271–7.

    Article  PubMed  Google Scholar 

  58. Sturm LP, Windsor JA, Cosman PH, Cregan PC, Hewett PJ, Maddern GJ. A systematic review of surgical skills transfer after simulation-based training. Ann Surg. 2008;248(2):166–79.

    Article  PubMed  Google Scholar 

  59. Moglia A, Ferrari V, Morelli L, Ferrari M, Mosca F, Cuschieri A. A systematic review of virtual reality simulators for robot-assisted surgery. Eur Urol. 2016;69(2):1065–80.

    Article  PubMed  Google Scholar 

  60. Lee JY, Andonian S, Pace KT, Grober E. Basic laparoscopic skills assessment study – validation and standard setting among Canadian urology trainees. Eur Urol. 2017;197(6):1539–44.

    Google Scholar 

  61. Lendvay TS, White L, Kowalewski T. Crowdsourcing to assess surgical skill. JAMA Surg. 2015;150(11):1086–7.

    Article  PubMed  Google Scholar 

  62. Phillips A, Matthan J, Bookless L, Whitehead I, Madhavan A, Rodham P, et al. Individualised expert feedback is not essential for improving basic clinical skills performance in novice learners: a randomized trial. J Surg Educ. 2017;74(4):612–20.

    Article  PubMed  Google Scholar 

  63. Kulasegaram KM, Grierson LEM, Norman GR. The roles of deliberate practice and innate ability in developing expertise: evidence and implications. Med Educ [Internet]. 2013;47(10):979–89. Available from: https://doi.org/10.1111/medu.12260.

  64. Cuschieri A. Lest we forget the surgeon. Semin Laparosc Surg. 2003;10(3):141–8.

    Google Scholar 

  65. Mattar SG, Alseidi AA, Jones DB, Jeyarajah DR, Swanstrom LL, Aye RW, et al. General surgery residency inadequately prepares trainees for fellowship: results of a survey of fellowship program directors. Ann Surg [Internet]. 2013;258(3):440–9. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24022436.

  66. Louridas M, Szasz P, de Montbrun S, Harris KA, Grantcharov TP. Original reports: optimizing the selection of general surgery residents: a national consensus. J Surg Educ [Internet]. 2017; Available from: http://10.0.3.248/j.jsurg.2016.06.015%5Cnhttps://ezp.lib.unimelb.edu.au/login?url=https://search.ebscohost.com/login.aspx?direct=true&db=edselp&AN=S1931720416300915&site=eds-live&scope=site.

  67. Moore EJ, Price DL, Van Abel KM, Carlson ML. Still under the microscope: can a surgical aptitude test predict otolaryngology resident performance? Laryngoscope. 2015;125(2):E57–61.

    Article  PubMed  Google Scholar 

  68. Iramaneerat C. Instruction and assessment of professionalism for surgery residents. J Surg Educ. 2009;66(3):158–62.

    Article  PubMed  Google Scholar 

  69. Hochberg MS, Kalet A, Zabar S, Kachur E, Gillespie C, Berman RS. Can professionalism be taught? Encouraging evidence. Am J Surg. 2010;199(1):86–93.

    Article  PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Washington, Department of Urology, 1959 NE Pacific St, HSB BB 1121, Seattle, WA, USA

    Jessica C. Dai & Mathew D. Sorensen

Authors
  1. Jessica C. Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mathew D. Sorensen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mathew D. Sorensen .

Editor information

Editors and Affiliations

  1. Urology, Mayo Clinic, Rochester, Minnesota, USA

    Tobias S. Köhler

  2. Department of Surgery, Urology, Southern Illinois University School of Medicine, Springfield, Illinois, USA

    Bradley Schwartz

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Cite this chapter

Dai, J.C., Sorensen, M.D. (2018). Crowdsourcing and Large-Scale Evaluation. In: Köhler, T., Schwartz, B. (eds) Surgeons as Educators . Springer, Cham. https://doi.org/10.1007/978-3-319-64728-9_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-64728-9_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-64727-2

  • Online ISBN: 978-3-319-64728-9

  • eBook Packages: MedicineMedicine (R0)

Publish with us

Policies and ethics

Search

Navigation