Abstract
Purpose
This study aims to develop and establish the content validity of multiple expert rating instruments to assess performance in endoscopic third ventriculostomy (ETV), collectively called the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT).
Methods
The important aspects of ETV were identified through a review of current literature, ETV videos, and discussion with neurosurgeons, fellows, and residents. Three assessment measures were subsequently developed: a procedure-specific checklist (CL), a CL of surgical errors, and a global rating scale (GRS). Neurosurgeons from various countries, all identified as experts in ETV, were then invited to participate in a modified Delphi survey to establish the content validity of these instruments. In each Delphi round, experts rated their agreement including each procedural step, error, and GRS item in the respective instruments on a 5-point Likert scale.
Results
Seventeen experts agreed to participate in the study and completed all Delphi rounds. After item generation, a total of 27 procedural CL items, 26 error CL items, and 9 GRS items were posed to Delphi panelists for rating. An additional 17 procedural CL items, 12 error CL items, and 1 GRS item were added by panelists. After three rounds, strong consensus (>80 % agreement) was achieved on 35 procedural CL items, 29 error CL items, and 10 GRS items. Moderate consensus (50–80 % agreement) was achieved on an additional 7 procedural CL items and 1 error CL item. The final procedural and error checklist contained 42 and 30 items, respectively (divided into setup, exposure, navigation, ventriculostomy, and closure). The final GRS contained 10 items.
Conclusions
We have established the content validity of three ETV assessment measures by iterative consensus of an international expert panel. Each measure provides unique assessment information and thus can be used individually or in combination, depending on the characteristics of the learner and the purpose of the assessment. These instruments must now be evaluated in both the simulated and operative settings, to determine their construct validity and reliability. Ultimately, the measures contained in the NEVAT may prove suitable for formative assessment during ETV training and potentially as summative assessment measures during certification.
Similar content being viewed by others
References
Reznick RK, MacRae H (2006) Teaching surgical skills—changes in the wind. N Engl J Med 355:2664–2669
Epstein RM (2007) Assessment in medical education. N Engl J Med 356:387–396
Frank JR, Snell LS, Ten Cate O et al (2010) Competency-based medical education: theory to practice. Med Teach 32:638–645
American Board of Neurological Surgery (2013) Primary Certification Process: Training in Neurological Surgery. http://www.abns.org/content/primary_certification_process_pr.asp
Burchiel K (2013) Commentary: simulation training in neurological surgery. Neurosurgery 73:6–7
Dagi TF (2013) Commentary: the roles and future of simulation in neurosurgery. Neurosurgery 73(Suppl 1):4–5
Fallah A, Ebrahim S, Haji F et al (2010) Surgical activity of first-year Canadian neurosurgical residents. Can J Neurol Sci 37:855–860
Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165:358–361
Memon MA, Brigden D, Subramanya MS, Memon B (2010) Assessing the surgeon’s technical skills: analysis of the available tools. Acad Med 85:869–880
Beard JD (2007) Assessment of surgical competence. Br J Surg 94:1315–1316
Cristancho S, Moussa F, Dubrowski A (2012) Simulation-augmented training program for off-pump coronary artery bypass surgery: developing and validating performance assessments. Surgery 151:785–795
Scavone BM, Sproviero MT, McCarthy RJ et al (2006) Development of an objective scoring system for measurement of resident performance on the human patient simulator. Anesthesiology 105:260–266
Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. BMJ 327:1032–1037
Epstein RM, Hundert EM (2002) Defining and assessing professional competence. JAMA 287:226–235
Sugden C, Aggarwal R (2010) Assessment and feedback in the skills laboratory and operating room. Surg Clin N Am 90:519–533
Sanfey H, Dunnington G (2010) Verification of proficiency: a prerequisite for clinical experience. Surg Clin N Am 90:559–567
Ahmed K, Miskovic D, Darzi A et al (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg. doi:10.1016/j.amjsurg.2010.10.020
Eubanks TR, Clements RH, Pohl D et al (1999) An objective scoring system for laparoscopic cholecystectomy. J Am Coll Surg 189:566–574
Haase J, Boisen E (2009) Neurosurgical training: more hours needed or a new learning culture? Surg Neurol 72:89–95, discussion 95–7
Navarro R, Gil-Parra R, Reitman AJ et al (2006) Endoscopic third ventriculostomy in children: early and late complications and their avoidance. Childs Nerv Syst 22:506–513
Schroeder HWS, Niendorf W-R, Gaab MR (2002) Complications of endoscopic third ventriculostomy. J Neurosurg 96:1032–1040
Furlanetti LL, Santos MV, de Oliveira RS (2012) The success of endoscopic third ventriculostomy in children: analysis of prognostic factors. Pediatr Neurosurg. doi:10.1159/000353619
Jones RF, Kwok BC, Stening WA, Vonau M (1994) The current status of endoscopic third ventriculostomy in the management of non-communicating hydrocephalus. Minim Invasive Neurosurg 37:28–36
Cohen AR, Lohani S, Manjila S et al (2013) Virtual reality simulation: basic concepts and use in endoscopic neurosurgery training. Childs Nerv Syst 29:1235–1244
Mokkink LB, Terwee CB, Patrick DL et al (2010) The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol 63:737–745
Boogaard S, Heymans MW, Patijn J et al (2011) Predictors for persistent neuropathic pain—a Delphi survey. Pain Physician 14:559–568
Hasson F, Keeney S, McKenna H (2000) Research guidelines for the Delphi survey technique. J Adv Nurs 32:1008–1015
Hsu C, Sandford BA (2007) The Delphi technique: making sense of consensus. Pract Assess Res Eval 12:1–8
Clayton MJ (1997) Delphi: a technique to harness expert opinion for critical decision‐making tasks in education. Educ Psychol 17:373–386
Walsh CM, Ling SC, Khanna N et al (2013) Gastrointestinal endoscopy competency assessment tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc. doi:10.1016/j.gie.2013.10.035
Cheung JJH, Chen EW, Darani R et al (2012) The creation of an objective assessment tool for ultrasound-guided regional anesthesia using the Delphi method. Reg Anesth Pain Med 37:329–333
Kessler DO, Auerbach M, Pusic M et al (2011) A randomized trial of simulation-based deliberate practice for infant lumbar puncture skills. Simul Healthc 6:197–203
Lin SY, Laeeq K, Ishii M et al (2009) Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy 23:354–359
Peyre SE, Peyre CG, Hagen JA et al (2009) Laparoscopic Nissen fundoplication assessment: task analysis as a model for the development of a procedural checklist. Surg Endosc 23:1227–1232
Palter VN, MacRae HM, Grantcharov TP (2011) Development of an objective evaluation tool to assess technical skill in laparoscopic colorectal surgery: a Delphi methodology. Am J Surg 201:251–259
Morgan PJ, Lam-McCulloch J, Herold-McIlroy J, Tarshis J (2007) Simulation performance checklist generation using the Delphi technique. Can J Anaesth 54:992–997
De Villiers MR, de Villiers PJT, Kent AP (2005) The Delphi technique in health sciences education research. Med Teach 27:639–643
Linstone HA, Turoff M (1975) The Delphi method—techniques and applications. Addison-Wesley, Reading
Martin J, Regehr G, Reznick R et al (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278
Powell C (2003) The Delphi technique: myths and realities. J Adv Nurs 41:376–382
Beems T, Grotenhuis JA (2004) Long-term complications and definition of failure of neuroendoscopic procedures. Childs Nerv Syst 20:868–877
Bouras T, Sgouros S (2011) Complications of endoscopic third ventriculostomy. J Neurosurg Pediatr 7:643–649
Brockmeyer D, Abtin K, Carey L, Walker ML (1998) Endoscopic third ventriculostomy: an outcome analysis. Pediatr Neurosurg 28:236–240
Chowdhry SA, Cohen AR (2013) Intraventricular neuroendoscopy: complication avoidance and management. World Neurosurg. doi:10.1016/j.wneu.2012.02.030
Peretta P, Ragazzi P, Galarza M et al (2006) Complications and pitfalls of neuroendoscopic surgery in children. J Neurosurg 105:187–193
Teo C (2005) Complications of endoscopic third ventriculostomy. In: Cinalli G, Sainte-Rose C, Maixner WJ (eds) Pediatric hydrocephalus. Springer Milan, Milano, pp 411–420
Dusick JR, McArthur DL, Bergsneider M (2008) Success and complication rates of endoscopic third ventriculostomy for adult hydrocephalus: a series of 108 patients. Surg Neurol 69:5–15
Erşahin Y, Arslan D (2008) Complications of endoscopic third ventriculostomy. Childs Nerv Syst 24:943–948
Haji FA, Dubrowski A, Drake J, de Ribaupierre S (2013) Needs assessment for simulation training in neuroendoscopy: a Canadian national survey. J Neurosurg 118:250–257
Jones RF, Stening WA, Brydon M (1990) Endoscopic third ventriculostomy. Neurosurgery 26:86–91, discussion 91–2
García LG, López BR, Botella GI et al (2012) Endoscopic third ventriculostomy success score (ETVSS) predicting success in a series of 50 pediatric patients. Are the outcomes of our patients predictable? Childs Nerv Syst 28:1157–1162
Hopf NJ, Grunert P, Fries G et al (1999) Endoscopic third ventriculostomy: outcome analysis of 100 consecutive procedures. Neurosurgery 44:795–804, discussion 804–6
Bouramas D, Paidakakos N, Sotiriou F et al (2012) Endoscopic third ventriculostomy in obstructive hydrocephalus: surgical technique and pitfalls. Acta Neurochir Suppl 113:135–139
Van Hove PD, Tuijthof GJM, Verdaasdonk EGG et al (2010) Objective assessment of technical surgical skills. Br J Surg 97:972–987
Mattei TA, Frank C, Bailey J et al (2013) Design of a synthetic simulator for pediatric lumbar spine pathologies. J Neurosurg Pediatr 12:192–201
Choudhury N, Gélinas-Phaneuf N, Delorme S, Del Maestro R (2012) Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg. doi:10.1016/j.wneu.2012.08.022
Miranpuri AS, Nickele CM, Akture E et al (2013) Neuroangiography simulation using a silicone model in the angiography suite improves trainee skills. J Neurointerv Surg. doi:10.1136/neurintsurg-2013-010826
Banerjee PP, Luciano CJ, Lemole GM et al (2007) Accuracy of ventriculostomy catheter placement using a head- and hand-tracked high-resolution virtual reality simulator with haptic feedback. J Neurosurg 107:515–521
Luciano CJ, Banerjee PP, Bellotte B et al (2011) Learning retention of thoracic pedicle screw placement using a high-resolution augmented reality simulator with haptic feedback. Neurosurgery 69:14–19, discussion 19
Downing SM (2006) Face validity of assessments: faith-based interpretations or evidence-based science? Med Educ 40:7–8
Bjorkman D, Popp JJ (2006) Measuring the quality of endoscopy. Am J Gastroenterol 101:864–865
Ma IWY, Zalunardo N, Pachev G et al (2012) Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ Theory Pract 17:457–470
Regehr G, MacRae H, Reznick RK, Szalay D (1998) Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73:993–997
Hodges B, Regehr G, McNaughton N et al (1999) OSCE checklists do not capture increasing levels of expertise. Acad Med 74:1129–1134
Gélinas-Phaneuf N, Del Maestro RF (2013) Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery 73(Suppl 1):30–38
El Ahmadieh TY, Aoun SG, El Tecle NE et al (2013) A didactic and hands-on module enhances resident microsurgical knowledge and technical skill. Neurosurgery 73(Suppl 1):51–56
Lobel D, Elder J, Schirmer C et al (2013) A novel craniotomy simulator provides a validated method to enhance education in the management of traumatic brain injury. Neurosurgery 73:57–65
Ray WZ, Ganju A, Harrop JS, Hoh DJ (2013) Developing an anterior cervical diskectomy and fusion simulator for neurosurgical resident training. Neurosurgery 73(Suppl 1):100–106
Harrop J, Rezai AR, Hoh DJ et al (2013) Neurosurgical training with a novel cervical spine simulator: posterior foraminotomy and laminectomy. Neurosurgery 73(Suppl 1):94–99
Guyatt G, Bombardier C, Tugwell P (1986) Measuring disease-specific quality of life in clinical trials. CMAJ 134:889–895
Downing SM (2003) Validity: on meaningful interpretation of assessment data. Med Educ 37:830–837
American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (1994) Standards for educational and psychological testing. American Educational Research Association, Washington, DC
American Psychologoy Association (1966) Standards for educational and psychological tests and manuals. American Psychological Association, Washington, DC
Kane MT (2006) Validation. In: Brennan RL (ed) Educational measurement, 4th edn. Praeger, Westport
De Vet HCW, Terwee CB, Mokkink LB, Knol DL (2011) Measurement in medicine, 1st edn. Cambridge University Press, New York
Hogle NJ, Liu Y, Ogden RT, Fowler DL (2014) Evaluation of surgical fellows’ laparoscopic performance using Global Operative Assessment of Laparoscopic Skills (GOALS). Surg Endosc. doi:10.1007/s00464-013-3324-6
Van der Vleuten C (1996) The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ 1:41–67
Acknowledgments
Thanks to all the experts for their valuable contribution to this Delphi study: Dr. G. Cinalli, Italy; Dr. S. Constantini, Israel; Dr. P. Decq, France; Dr. S. deRibaupierre, Canada; Dr. C. DiRocco, Italy; Dr. Y. Ersahin, Turkey; Dr. J. Grotenhuis, the Netherlands; Dr. N. Gupta, United States of America; Dr. E. Hoving, the Netherlands; Dr. I. Pollack, United States of America; Dr. C. Sainte-Rose, France; Dr. S. Santoreneos, Australia; Dr. H. Schroeder, Germany; Dr. D. Thompson, United Kingdom; Dr. B. Warf, United States of America; Dr. J. Wellons, United States of America; and Dr. S. Zymberg, Brazil.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Breimer, G.E., Haji, F.A., Hoving, E.W. et al. Development and content validation of performance assessments for endoscopic third ventriculostomy. Childs Nerv Syst 31, 1247–1259 (2015). https://doi.org/10.1007/s00381-015-2716-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00381-015-2716-4