Development and validation of the TOCO–TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour
The current shift towards competency-based residency training has increased the need for objective assessment of skills. In this study, we developed and validated an assessment tool that measures technical and non-technical competency in transurethral resection of bladder tumour (TURBT).
The ‘Test Objective Competency’ (TOCO)–TURBT tool was designed by means of cognitive task analysis (CTA), which included expert consensus. The tool consists of 51 items, divided into 3 phases: preparatory (n = 15), procedural (n = 21), and completion (n = 15). For validation of the TOCO–TURBT tool, 2 TURBT procedures were performed and videotaped by 25 urologists and 51 residents in a simulated setting. The participants’ degree of competence was assessed by a panel of eight independent expert urologists using the TOCO–TURBT tool. Each procedure was assessed by two raters. Feasibility, acceptability and content validity were evaluated by means of a quantitative cross-sectional survey. Regression analyses were performed to assess the strength of the relation between experience and test scores (construct validity). Reliability was analysed by generalizability theory.
The majority of assessors and urologists indicated the TOCO–TURBT tool to be a valid assessment of competency and would support the implementation of the TOCO–TURBT assessment as a certification method for residents. Construct validity was clearly established for all outcome measures of the procedural phase (all r > 0.5, p < 0.01). Generalizability-theory analysis showed high reliability (coefficient Phi ≥ 0.8) when using the format of two assessors and two cases.
This study provides first evidence that the TOCO–TURBT tool is a feasible, valid and reliable assessment tool for measuring competency in TURBT. The tool has the potential to be used for future certification of competencies for residents and urologists. The methodology of CTA might be valuable in the development of assessment tools in other areas of clinical practice.
KeywordsAssessment Cognitive task analysis Certification Transurethral resection of bladder tumour Urology Validation
The authors gratefully acknowledge all the experts who participated in the development and validation of the TOCO–TURBT tool, the residents and urologists who were willing to participate in this study, Ron Hoogeboom for statistical support, and Lisette van Hulst for editorial assistance.
Compliance with ethical standards
A.H. de Vries, H.G.J van Genugten, A.J.M. Hendrikx, E.L. Koldewijn, B.M.A. Schout, A.M.M. Muijtjens, C.P.M. van der Vleuten, C. Wagner, Irene M. Tjiam, and J.J.G. van Merriënboer have no conflicts of interest or financial ties to disclose.
Ethical approval was sought from the institution’s research and ethics committee. Ethical approval was not required according to the Dutch Medical Research (Human Subjects) Act, since no patients or patient details were involved.
Informed consent with assurance of anonymity and confidentiality was obtained from all participants.
- 2.Zevin B, Bonrath EM, Aggarwal R, Dedy NJ, Ahmed N, Grantcharov TP, ATLAS group (2013) Development, feasibility, validity, and reliability of a scale for objective assessment of operative performance in laparoscopic gastric bypass surgery. J Am Coll Surg 216:955–965.e8. https://doi.org/10.1016/j.jamcollsurg.2013.01.003 (quiz 1029-31, 1033)CrossRefPubMedGoogle Scholar
- 5.Baker GR, Norton PG, Flintoft V, Blais R, Brown A, Cox J, Etchells E, Ghali WA, Hebert P, Majumdar SR, O’Beirne M, Palacios-Derflingher L, Reid RJ, Sheps S, Tamblyn R (2004) The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ 170:1678–1686CrossRefGoogle Scholar
- 7.Miskovic D, Ni M, Wyles SM, Kennedy RH, Francis NK, Parvaiz A, Cunningham C, Rockall TA, Gudgeon AM, Coleman MG, Hanna GB, National Training Programme in Laparoscopic Colorectal Surgery in England (2013) Is competency assessment at the specialist level achievable? A study for the national training programme in laparoscopic colorectal surgery in England. Ann Surg 257:476–482. https://doi.org/10.1097/SLA.0b013e318275b72a CrossRefPubMedGoogle Scholar
- 13.Clark R, Feldon D, van Merrienboer J, Yates K (2008) Cognitive task analysis. Anonymous Handbook of research on educational communications and technology. Taylor and Francis Group, Boca Raton, pp 577–593Google Scholar
- 14.Yates K, Feldon D (2011) Advancing the practice of cognitive task analysis: a call for taxonomic research. Theor Issues Ergon Sci 11:1464–1536Google Scholar
- 16.Babjuk M, Burger M, Zigeuner R, Shariat SF, van Rhijn BW, Comperat E, Sylvester RJ, Kaasinen E, Bohle A, Palou Redorta J, Roupret M, European Association of Urology (2013) EAU guidelines on non-muscle-invasive urothelial carcinoma of the bladder: update 2013. Eur Urol 64:639–653. https://doi.org/10.1016/j.eururo.2013.06.003 CrossRefPubMedPubMedCentralGoogle Scholar
- 17.Brausi M, Collette L, Kurth K, van der Meijden AP, Oosterlinck W, Witjes JA, Newling D, Bouffioux C, Sylvester RJ, EORTC Genito-Urinary Tract Cancer Collaborative Group (2002) Variability in the recurrence rate at first follow-up cystoscopy after TUR in stage Ta T1 transitional cell carcinoma of the bladder: a combined analysis of seven EORTC studies. Eur Urol 41:523–531CrossRefGoogle Scholar
- 20.Barton JR, Corbett S, van der Vleuten CP, English Bowel Cancer Screening Programme, UK Joint Advisory Group for Gastrointestinal Endoscopy (2012) The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc 75:591–597. https://doi.org/10.1016/j.gie.2011.09.053 CrossRefPubMedGoogle Scholar
- 21.de Vries AH, van Genugten HG, Hendrikx AJ, Koldewijn EL, Schout BM, Tjiam IM, van Merrienboer JJ, Muijtjens AM, Wagner C (2016) The Simbla TURBT simulator in urological residency training: from needs analysis to validation. J Endourol. https://doi.org/10.1089/end.2015.0723 CrossRefPubMedGoogle Scholar
- 24.Fraenkel JR, Wallen NE (2009) How to design and evaluate research in education. McGraw Hill, BostonGoogle Scholar
- 25.Cohen J (1988) Statistical power analysis for the behavioural sciences. Lawrence Erlbaum, LondonGoogle Scholar
- 26.Walsh CM, Ling SC, Khanna N, Cooper MA, Grover SC, May G, Walters TD, Rabeneck L, Reznick R, Carnahan H (2014) Gastrointestinal endoscopy competency assessment tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc 79(e5):798–807. https://doi.org/10.1016/j.gie.2013.10.035 CrossRefPubMedGoogle Scholar
- 31.Tjiam IM, Schout BM, Hendrikx AJ, Muijtjens AM, Scherpbier AJ, Witjes JA, Van Der Vleuten CP (2013) Program for laparoscopic urological skills assessment: setting certification standards for residents. Minim Invasive Ther Allied Technol 22:26–32. https://doi.org/10.3109/13645706.2012.686918 CrossRefPubMedGoogle Scholar