Surgical Endoscopy

, Volume 32, Issue 12, pp 4923–4931 | Cite as

Development and validation of the TOCO–TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour

  • Anna H. de Vries
  • Arno. M. M. Muijtjens
  • Hilde G. J. van Genugten
  • Ad. J. M. Hendrikx
  • Evert L. Koldewijn
  • Barbara M. A. Schout
  • Cees P. M. van der Vleuten
  • Cordula Wagner
  • Irene M. Tjiam
  • Jeroen J. G. van Merriënboer



The current shift towards competency-based residency training has increased the need for objective assessment of skills. In this study, we developed and validated an assessment tool that measures technical and non-technical competency in transurethral resection of bladder tumour (TURBT).


The ‘Test Objective Competency’ (TOCO)–TURBT tool was designed by means of cognitive task analysis (CTA), which included expert consensus. The tool consists of 51 items, divided into 3 phases: preparatory (n = 15), procedural (n = 21), and completion (n = 15). For validation of the TOCO–TURBT tool, 2 TURBT procedures were performed and videotaped by 25 urologists and 51 residents in a simulated setting. The participants’ degree of competence was assessed by a panel of eight independent expert urologists using the TOCO–TURBT tool. Each procedure was assessed by two raters. Feasibility, acceptability and content validity were evaluated by means of a quantitative cross-sectional survey. Regression analyses were performed to assess the strength of the relation between experience and test scores (construct validity). Reliability was analysed by generalizability theory.


The majority of assessors and urologists indicated the TOCO–TURBT tool to be a valid assessment of competency and would support the implementation of the TOCO–TURBT assessment as a certification method for residents. Construct validity was clearly established for all outcome measures of the procedural phase (all r > 0.5, p < 0.01). Generalizability-theory analysis showed high reliability (coefficient Phi ≥ 0.8) when using the format of two assessors and two cases.


This study provides first evidence that the TOCO–TURBT tool is a feasible, valid and reliable assessment tool for measuring competency in TURBT. The tool has the potential to be used for future certification of competencies for residents and urologists. The methodology of CTA might be valuable in the development of assessment tools in other areas of clinical practice.


Assessment Cognitive task analysis Certification Transurethral resection of bladder tumour Urology Validation 



The authors gratefully acknowledge all the experts who participated in the development and validation of the TOCO–TURBT tool, the residents and urologists who were willing to participate in this study, Ron Hoogeboom for statistical support, and Lisette van Hulst for editorial assistance.

Compliance with ethical standards


A.H. de Vries, H.G.J van Genugten, A.J.M. Hendrikx, E.L. Koldewijn, B.M.A. Schout, A.M.M. Muijtjens, C.P.M. van der Vleuten, C. Wagner, Irene M. Tjiam, and J.J.G. van Merriënboer have no conflicts of interest or financial ties to disclose.

Ethical approval

Ethical approval was sought from the institution’s research and ethics committee. Ethical approval was not required according to the Dutch Medical Research (Human Subjects) Act, since no patients or patient details were involved.

Informed consent

Informed consent with assurance of anonymity and confidentiality was obtained from all participants.

Supplementary material

464_2018_6251_MOESM1_ESM.tiff (1.5 mb)
Online Appendix 1a (TIFF 1521 KB)
464_2018_6251_MOESM2_ESM.tiff (1.5 mb)
Online Appendix 1b (TIFF 1521 KB)
464_2018_6251_MOESM3_ESM.tiff (1.5 mb)
Online Appendix 1c (TIFF 1521 KB)
464_2018_6251_MOESM4_ESM.docx (3.2 mb)
Online Appendix 2 (DOCX 3228 KB)
464_2018_6251_MOESM5_ESM.docx (19 kb)
Online Appendix 3 (DOCX 19 KB)
464_2018_6251_MOESM6_ESM.docx (53 kb)
Online Appendix 4 (DOCX 53 KB)
464_2018_6251_MOESM7_ESM.docx (18 kb)
Online Appendix 5 (DOCX 17 KB)


  1. 1.
    Aggarwal R, Grantcharov T, Moorthy K, Milland T, Darzi A (2008) Toward feasible, valid, and reliable video-based assessments of technical surgical skills in the operating room. Ann Surg 247:372–379. CrossRefPubMedGoogle Scholar
  2. 2.
    Zevin B, Bonrath EM, Aggarwal R, Dedy NJ, Ahmed N, Grantcharov TP, ATLAS group (2013) Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg 216:955–965.e8. (quiz 1029-31, 1033)CrossRefPubMedGoogle Scholar
  3. 3.
    Hampton T (2015) Efforts seek to develop systematic ways to objectively assess surgeons’ skills. JAMA 313:782–784. CrossRefPubMedGoogle Scholar
  4. 4.
    de Montbrun S, Satterthwaite L, Grantcharov TP (2016) Setting pass scores for assessment of technical performance by surgical trainees. Br J Surg 103:300–306. CrossRefPubMedGoogle Scholar
  5. 5.
    Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, Etchells E, Ghali WA, Hebert P, Majumdar SR, O’Beirne M, Palacios-Derflingher L, Reid RJ, Sheps S, Tamblyn R (2004) The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ 170:1678–1686CrossRefGoogle Scholar
  6. 6.
    Vincent C, Moorthy K, Sarker SK, Chang A, Darzi AW (2004) Systems approaches to surgical quality and safety: from concept to measurement. Ann Surg 239:475–482CrossRefGoogle Scholar
  7. 7.
    Miskovic D, Ni M, Wyles SM, Kennedy RH, Francis NK, Parvaiz A, Cunningham C, Rockall TA, Gudgeon AM, Coleman MG, Hanna GB, National Training Programme in Laparoscopic Colorectal Surgery in England (2013) Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg 257:476–482. CrossRefPubMedGoogle Scholar
  8. 8.
    Fried GM, Feldman LS (2008) Objective assessment of technical performance. World J Surg 32:156–160. CrossRefPubMedGoogle Scholar
  9. 9.
    Gallagher AG, Ritter EM, Satava RM (2003) Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc 17:1525–1529. CrossRefPubMedGoogle Scholar
  10. 10.
    Larson JL, Williams RG, Ketchum J, Boehler ML, Dunnington GL (2005) Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents. Surgery 138:640–647 (discussion 647–9)CrossRefGoogle Scholar
  11. 11.
    Hull L, Arora S, Aggarwal R, Darzi A, Vincent C, Sevdalis N (2012) The impact of nontechnical skills on technical performance in surgery: a systematic review. J Am Coll Surg 214:214–230. CrossRefGoogle Scholar
  12. 12.
    Shepherd W, Arora KS, Abboudi H, Shamim Khan M, Dasgupta P, Ahmed K (2014) A review of the available urology skills training curricula and their validation. J Surg Educ 71:289–296. CrossRefPubMedGoogle Scholar
  13. 13.
    Clark R, Feldon D, van Merrienboer J, Yates K (2008) Cognitive task analysis. Anonymous Handbook of research on educational communications and technology. Taylor and Francis Group, Boca Raton, pp 577–593Google Scholar
  14. 14.
    Yates K, Feldon D (2011) Advancing the practice of cognitive task analysis: a call for taxonomic research. Theor Issues Ergon Sci 11:1464–1536Google Scholar
  15. 15.
    Sullivan ME, Ortega A, Wasserberg N, Kaufman H, Nyquist J, Clark R (2008) Assessing the teaching of procedural skills: can cognitive task analysis add to our traditional teaching methods? Am J Surg 195:20–23CrossRefGoogle Scholar
  16. 16.
    Babjuk M, Burger M, Zigeuner R, Shariat SF, van Rhijn BW, Comperat E, Sylvester RJ, Kaasinen E, Bohle A, Palou Redorta J, Roupret M, European Association of Urology (2013) EAU guidelines on non-muscle-invasive urothelial carcinoma of the bladder: update 2013. Eur Urol 64:639–653. CrossRefPubMedPubMedCentralGoogle Scholar
  17. 17.
    Brausi M, Collette L, Kurth K, van der Meijden AP, Oosterlinck W, Witjes JA, Newling D, Bouffioux C, Sylvester RJ, EORTC Genito-Urinary Tract Cancer Collaborative Group (2002) Variability in the recurrence rate at first follow-up cystoscopy after TUR in stage Ta T1 transitional cell carcinoma of the bladder: a combined analysis of seven EORTC studies. Eur Urol 41:523–531CrossRefGoogle Scholar
  18. 18.
    Allard CB, Meyer CP, Gandaglia G, Chang SL, Chun FK, Gelpi-Hammerschmidt F, Hanske J, Kibel AS, Preston MA, Trinh QD (2015) The effect of resident involvement on perioperative outcomes in transurethral urologic surgeries. J Surg Educ 72:1018–1025CrossRefGoogle Scholar
  19. 19.
    Jancke G, Rosell J, Jahnson S (2014) Impact of surgical experience on recurrence and progression after transurethral resection of bladder tumour in non-muscle-invasive bladder cancer. Scand J Urol 48:276–283. CrossRefPubMedGoogle Scholar
  20. 20.
    Barton JR, Corbett S, van der Vleuten CP, English Bowel Cancer Screening Programme, UK Joint Advisory Group for Gastrointestinal Endoscopy (2012) The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 75:591–597. CrossRefPubMedGoogle Scholar
  21. 21.
    de Vries AH, van Genugten HG, Hendrikx AJ, Koldewijn EL, Schout BM, Tjiam IM, van Merrienboer JJ, Muijtjens AM, Wagner C (2016) The Simbla TURBT simulator in urological residency training: from needs analysis to validation. J Endourol. CrossRefPubMedGoogle Scholar
  22. 22.
    Luker KR, Sullivan ME, Peyre SE, Sherman R, Grunwald T (2008) The use of a cognitive task analysis-based multimedia program to teach surgical decision making in flexor tendon repair. Am J Surg 195:11–15CrossRefGoogle Scholar
  23. 23.
    Crossley J, Davies H, Humphris G, Jolly B (2002) Generalisability: a key to unlock professional assessment. Med Educ 36:972–978CrossRefGoogle Scholar
  24. 24.
    Fraenkel JR, Wallen NE (2009) How to design and evaluate research in education. McGraw Hill, BostonGoogle Scholar
  25. 25.
    Cohen J (1988) Statistical power analysis for the behavioural sciences. Lawrence Erlbaum, LondonGoogle Scholar
  26. 26.
    Walsh CM, Ling SC, Khanna N, Cooper MA, Grover SC, May G, Walters TD, Rabeneck L, Reznick R, Carnahan H (2014) Gastrointestinal endoscopy competency assessment tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 79(e5):798–807. CrossRefPubMedGoogle Scholar
  27. 27.
    Tjiam IM, Persoon MC, Hendrikx AJ, Muijtjens AM, Witjes JA, Scherpbier AJ (2012) Program for laparoscopic urologic skills: a newly developed and validated educational program. Urology 79:815–820. CrossRefPubMedGoogle Scholar
  28. 28.
    Ten Cate O (2013) Nuts and bolts of entrustable professional activities. J Grad Med Educ 5:157–158. CrossRefPubMedPubMedCentralGoogle Scholar
  29. 29.
    Miller GE (1990) The assessment of clinical skills/competence/performance. Acad Med 65:S63-7CrossRefGoogle Scholar
  30. 30.
    Ram P, van der Vleuten C, Rethans JJ, Grol R, Aretz K (1999) Assessment of practicing family physicians: comparison of observation in a multiple-station examination using standardized patients with observation of consultations in daily practice. Acad Med 74:62–69CrossRefGoogle Scholar
  31. 31.
    Tjiam IM, Schout BM, Hendrikx AJ, Muijtjens AM, Scherpbier AJ, Witjes JA, Van Der Vleuten CP (2013) Program for laparoscopic urological skills assessment: setting certification standards for residents. Minim Invasive Ther Allied Technol 22:26–32. CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Anna H. de Vries
    • 1
  • Arno. M. M. Muijtjens
    • 2
  • Hilde G. J. van Genugten
    • 1
  • Ad. J. M. Hendrikx
    • 1
  • Evert L. Koldewijn
    • 1
    • 3
  • Barbara M. A. Schout
    • 4
    • 5
  • Cees P. M. van der Vleuten
    • 2
  • Cordula Wagner
    • 5
    • 6
  • Irene M. Tjiam
    • 7
  • Jeroen J. G. van Merriënboer
    • 3
  1. 1.Department of UrologyCatharina HospitalEindhovenThe Netherlands
  2. 2.Department of Educational Development & Research, Faculty of Health, Medicine and Life SciencesMaastricht UniversityMaastrichtThe Netherlands
  3. 3.School of Health Professions EducationMaastricht UniversityMaastrichtThe Netherlands
  4. 4.Department of UrologyAlrijne HospitalLeidenThe Netherlands
  5. 5.Netherlands Institute for Health Services Research (NIVEL)UtrechtThe Netherlands
  6. 6.Department of Public and Occupational HealthEMGO Institute for Health and Care ResearchAmsterdamThe Netherlands
  7. 7.Department of UrologyCanisius Wilhelmina HospitalNijmegenThe Netherlands

Personalised recommendations