Skip to main content

Advertisement

SpringerLink
Log in

Development and content validation of performance assessments for endoscopic third ventriculostomy

  • Original Paper
  • Published:
Child's Nervous System Aims and scope Submit manuscript

Abstract

Purpose

This study aims to develop and establish the content validity of multiple expert rating instruments to assess performance in endoscopic third ventriculostomy (ETV), collectively called the Neuro-Endoscopic Ventriculostomy Assessment Tool (NEVAT).

Methods

The important aspects of ETV were identified through a review of current literature, ETV videos, and discussion with neurosurgeons, fellows, and residents. Three assessment measures were subsequently developed: a procedure-specific checklist (CL), a CL of surgical errors, and a global rating scale (GRS). Neurosurgeons from various countries, all identified as experts in ETV, were then invited to participate in a modified Delphi survey to establish the content validity of these instruments. In each Delphi round, experts rated their agreement including each procedural step, error, and GRS item in the respective instruments on a 5-point Likert scale.

Results

Seventeen experts agreed to participate in the study and completed all Delphi rounds. After item generation, a total of 27 procedural CL items, 26 error CL items, and 9 GRS items were posed to Delphi panelists for rating. An additional 17 procedural CL items, 12 error CL items, and 1 GRS item were added by panelists. After three rounds, strong consensus (>80 % agreement) was achieved on 35 procedural CL items, 29 error CL items, and 10 GRS items. Moderate consensus (50–80 % agreement) was achieved on an additional 7 procedural CL items and 1 error CL item. The final procedural and error checklist contained 42 and 30 items, respectively (divided into setup, exposure, navigation, ventriculostomy, and closure). The final GRS contained 10 items.

Conclusions

We have established the content validity of three ETV assessment measures by iterative consensus of an international expert panel. Each measure provides unique assessment information and thus can be used individually or in combination, depending on the characteristics of the learner and the purpose of the assessment. These instruments must now be evaluated in both the simulated and operative settings, to determine their construct validity and reliability. Ultimately, the measures contained in the NEVAT may prove suitable for formative assessment during ETV training and potentially as summative assessment measures during certification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Reznick RK, MacRae H (2006) Teaching surgical skills—changes in the wind. N Engl J Med 355:2664–2669

    Article  CAS  PubMed  Google Scholar 

  2. Epstein RM (2007) Assessment in medical education. N Engl J Med 356:387–396

    Article  CAS  PubMed  Google Scholar 

  3. Frank JR, Snell LS, Ten Cate O et al (2010) Competency-based medical education: theory to practice. Med Teach 32:638–645

    Article  PubMed  Google Scholar 

  4. American Board of Neurological Surgery (2013) Primary Certification Process: Training in Neurological Surgery. http://www.abns.org/content/primary_certification_process_pr.asp

  5. Burchiel K (2013) Commentary: simulation training in neurological surgery. Neurosurgery 73:6–7

    Article  PubMed  Google Scholar 

  6. Dagi TF (2013) Commentary: the roles and future of simulation in neurosurgery. Neurosurgery 73(Suppl 1):4–5

    Article  PubMed  Google Scholar 

  7. Fallah A, Ebrahim S, Haji F et al (2010) Surgical activity of first-year Canadian neurosurgical residents. Can J Neurol Sci 37:855–860

    PubMed  Google Scholar 

  8. Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165:358–361

    Article  CAS  PubMed  Google Scholar 

  9. Memon MA, Brigden D, Subramanya MS, Memon B (2010) Assessing the surgeon’s technical skills: analysis of the available tools. Acad Med 85:869–880

    Article  PubMed  Google Scholar 

  10. Beard JD (2007) Assessment of surgical competence. Br J Surg 94:1315–1316

    Article  CAS  PubMed  Google Scholar 

  11. Cristancho S, Moussa F, Dubrowski A (2012) Simulation-augmented training program for off-pump coronary artery bypass surgery: developing and validating performance assessments. Surgery 151:785–795

    Article  PubMed  Google Scholar 

  12. Scavone BM, Sproviero MT, McCarthy RJ et al (2006) Development of an objective scoring system for measurement of resident performance on the human patient simulator. Anesthesiology 105:260–266

    Article  PubMed  Google Scholar 

  13. Moorthy K, Munz Y, Sarker SK, Darzi A (2003) Objective assessment of technical skills in surgery. BMJ 327:1032–1037

    Article  PubMed Central  PubMed  Google Scholar 

  14. Epstein RM, Hundert EM (2002) Defining and assessing professional competence. JAMA 287:226–235

    Article  PubMed  Google Scholar 

  15. Sugden C, Aggarwal R (2010) Assessment and feedback in the skills laboratory and operating room. Surg Clin N Am 90:519–533

    Article  PubMed  Google Scholar 

  16. Sanfey H, Dunnington G (2010) Verification of proficiency: a prerequisite for clinical experience. Surg Clin N Am 90:559–567

    Article  PubMed  Google Scholar 

  17. Ahmed K, Miskovic D, Darzi A et al (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg. doi:10.1016/j.amjsurg.2010.10.020

    Google Scholar 

  18. Eubanks TR, Clements RH, Pohl D et al (1999) An objective scoring system for laparoscopic cholecystectomy. J Am Coll Surg 189:566–574

    Article  CAS  PubMed  Google Scholar 

  19. Haase J, Boisen E (2009) Neurosurgical training: more hours needed or a new learning culture? Surg Neurol 72:89–95, discussion 95–7

    Article  PubMed  Google Scholar 

  20. Navarro R, Gil-Parra R, Reitman AJ et al (2006) Endoscopic third ventriculostomy in children: early and late complications and their avoidance. Childs Nerv Syst 22:506–513

    Article  PubMed  Google Scholar 

  21. Schroeder HWS, Niendorf W-R, Gaab MR (2002) Complications of endoscopic third ventriculostomy. J Neurosurg 96:1032–1040

    Article  PubMed  Google Scholar 

  22. Furlanetti LL, Santos MV, de Oliveira RS (2012) The success of endoscopic third ventriculostomy in children: analysis of prognostic factors. Pediatr Neurosurg. doi:10.1159/000353619

    PubMed  Google Scholar 

  23. Jones RF, Kwok BC, Stening WA, Vonau M (1994) The current status of endoscopic third ventriculostomy in the management of non-communicating hydrocephalus. Minim Invasive Neurosurg 37:28–36

    Article  CAS  PubMed  Google Scholar 

  24. Cohen AR, Lohani S, Manjila S et al (2013) Virtual reality simulation: basic concepts and use in endoscopic neurosurgery training. Childs Nerv Syst 29:1235–1244

    Article  PubMed  Google Scholar 

  25. Mokkink LB, Terwee CB, Patrick DL et al (2010) The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol 63:737–745

    Article  PubMed  Google Scholar 

  26. Boogaard S, Heymans MW, Patijn J et al (2011) Predictors for persistent neuropathic pain—a Delphi survey. Pain Physician 14:559–568

    PubMed  Google Scholar 

  27. Hasson F, Keeney S, McKenna H (2000) Research guidelines for the Delphi survey technique. J Adv Nurs 32:1008–1015

    CAS  PubMed  Google Scholar 

  28. Hsu C, Sandford BA (2007) The Delphi technique: making sense of consensus. Pract Assess Res Eval 12:1–8

    Google Scholar 

  29. Clayton MJ (1997) Delphi: a technique to harness expert opinion for critical decision‐making tasks in education. Educ Psychol 17:373–386

    Article  Google Scholar 

  30. Walsh CM, Ling SC, Khanna N et al (2013) Gastrointestinal endoscopy competency assessment tool: development of a procedure-specific assessment tool for colonoscopy. Gastrointest Endosc. doi:10.1016/j.gie.2013.10.035

    PubMed  Google Scholar 

  31. Cheung JJH, Chen EW, Darani R et al (2012) The creation of an objective assessment tool for ultrasound-guided regional anesthesia using the Delphi method. Reg Anesth Pain Med 37:329–333

    Article  PubMed  Google Scholar 

  32. Kessler DO, Auerbach M, Pusic M et al (2011) A randomized trial of simulation-based deliberate practice for infant lumbar puncture skills. Simul Healthc 6:197–203

    Article  PubMed  Google Scholar 

  33. Lin SY, Laeeq K, Ishii M et al (2009) Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy 23:354–359

    Article  PubMed  Google Scholar 

  34. Peyre SE, Peyre CG, Hagen JA et al (2009) Laparoscopic Nissen fundoplication assessment: task analysis as a model for the development of a procedural checklist. Surg Endosc 23:1227–1232

    Article  PubMed  Google Scholar 

  35. Palter VN, MacRae HM, Grantcharov TP (2011) Development of an objective evaluation tool to assess technical skill in laparoscopic colorectal surgery: a Delphi methodology. Am J Surg 201:251–259

    Article  PubMed  Google Scholar 

  36. Morgan PJ, Lam-McCulloch J, Herold-McIlroy J, Tarshis J (2007) Simulation performance checklist generation using the Delphi technique. Can J Anaesth 54:992–997

    Article  PubMed  Google Scholar 

  37. De Villiers MR, de Villiers PJT, Kent AP (2005) The Delphi technique in health sciences education research. Med Teach 27:639–643

    Article  PubMed  Google Scholar 

  38. Linstone HA, Turoff M (1975) The Delphi method—techniques and applications. Addison-Wesley, Reading

    Google Scholar 

  39. Martin J, Regehr G, Reznick R et al (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278

    Article  CAS  PubMed  Google Scholar 

  40. Powell C (2003) The Delphi technique: myths and realities. J Adv Nurs 41:376–382

    Article  PubMed  Google Scholar 

  41. Beems T, Grotenhuis JA (2004) Long-term complications and definition of failure of neuroendoscopic procedures. Childs Nerv Syst 20:868–877

    Article  PubMed  Google Scholar 

  42. Bouras T, Sgouros S (2011) Complications of endoscopic third ventriculostomy. J Neurosurg Pediatr 7:643–649

    Article  PubMed  Google Scholar 

  43. Brockmeyer D, Abtin K, Carey L, Walker ML (1998) Endoscopic third ventriculostomy: an outcome analysis. Pediatr Neurosurg 28:236–240

    Article  CAS  PubMed  Google Scholar 

  44. Chowdhry SA, Cohen AR (2013) Intraventricular neuroendoscopy: complication avoidance and management. World Neurosurg. doi:10.1016/j.wneu.2012.02.030

    PubMed  Google Scholar 

  45. Peretta P, Ragazzi P, Galarza M et al (2006) Complications and pitfalls of neuroendoscopic surgery in children. J Neurosurg 105:187–193

    PubMed  Google Scholar 

  46. Teo C (2005) Complications of endoscopic third ventriculostomy. In: Cinalli G, Sainte-Rose C, Maixner WJ (eds) Pediatric hydrocephalus. Springer Milan, Milano, pp 411–420

    Chapter  Google Scholar 

  47. Dusick JR, McArthur DL, Bergsneider M (2008) Success and complication rates of endoscopic third ventriculostomy for adult hydrocephalus: a series of 108 patients. Surg Neurol 69:5–15

    Article  PubMed  Google Scholar 

  48. Erşahin Y, Arslan D (2008) Complications of endoscopic third ventriculostomy. Childs Nerv Syst 24:943–948

    Article  PubMed  Google Scholar 

  49. Haji FA, Dubrowski A, Drake J, de Ribaupierre S (2013) Needs assessment for simulation training in neuroendoscopy: a Canadian national survey. J Neurosurg 118:250–257

    Article  PubMed  Google Scholar 

  50. Jones RF, Stening WA, Brydon M (1990) Endoscopic third ventriculostomy. Neurosurgery 26:86–91, discussion 91–2

    Article  CAS  PubMed  Google Scholar 

  51. García LG, López BR, Botella GI et al (2012) Endoscopic third ventriculostomy success score (ETVSS) predicting success in a series of 50 pediatric patients. Are the outcomes of our patients predictable? Childs Nerv Syst 28:1157–1162

    Article  PubMed  Google Scholar 

  52. Hopf NJ, Grunert P, Fries G et al (1999) Endoscopic third ventriculostomy: outcome analysis of 100 consecutive procedures. Neurosurgery 44:795–804, discussion 804–6

    Article  CAS  PubMed  Google Scholar 

  53. Bouramas D, Paidakakos N, Sotiriou F et al (2012) Endoscopic third ventriculostomy in obstructive hydrocephalus: surgical technique and pitfalls. Acta Neurochir Suppl 113:135–139

    Article  CAS  PubMed  Google Scholar 

  54. Van Hove PD, Tuijthof GJM, Verdaasdonk EGG et al (2010) Objective assessment of technical surgical skills. Br J Surg 97:972–987

    Article  PubMed  Google Scholar 

  55. Mattei TA, Frank C, Bailey J et al (2013) Design of a synthetic simulator for pediatric lumbar spine pathologies. J Neurosurg Pediatr 12:192–201

    Article  PubMed  Google Scholar 

  56. Choudhury N, Gélinas-Phaneuf N, Delorme S, Del Maestro R (2012) Fundamentals of neurosurgery: virtual reality tasks for training and evaluation of technical skills. World Neurosurg. doi:10.1016/j.wneu.2012.08.022

    PubMed  Google Scholar 

  57. Miranpuri AS, Nickele CM, Akture E et al (2013) Neuroangiography simulation using a silicone model in the angiography suite improves trainee skills. J Neurointerv Surg. doi:10.1136/neurintsurg-2013-010826

    Google Scholar 

  58. Banerjee PP, Luciano CJ, Lemole GM et al (2007) Accuracy of ventriculostomy catheter placement using a head- and hand-tracked high-resolution virtual reality simulator with haptic feedback. J Neurosurg 107:515–521

    Article  PubMed  Google Scholar 

  59. Luciano CJ, Banerjee PP, Bellotte B et al (2011) Learning retention of thoracic pedicle screw placement using a high-resolution augmented reality simulator with haptic feedback. Neurosurgery 69:14–19, discussion 19

    Article  Google Scholar 

  60. Downing SM (2006) Face validity of assessments: faith-based interpretations or evidence-based science? Med Educ 40:7–8

    Article  PubMed  Google Scholar 

  61. Bjorkman D, Popp JJ (2006) Measuring the quality of endoscopy. Am J Gastroenterol 101:864–865

    PubMed  Google Scholar 

  62. Ma IWY, Zalunardo N, Pachev G et al (2012) Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation. Adv Health Sci Educ Theory Pract 17:457–470

    Article  PubMed  Google Scholar 

  63. Regehr G, MacRae H, Reznick RK, Szalay D (1998) Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73:993–997

    Article  CAS  PubMed  Google Scholar 

  64. Hodges B, Regehr G, McNaughton N et al (1999) OSCE checklists do not capture increasing levels of expertise. Acad Med 74:1129–1134

    Article  CAS  PubMed  Google Scholar 

  65. Gélinas-Phaneuf N, Del Maestro RF (2013) Surgical expertise in neurosurgery: integrating theory into practice. Neurosurgery 73(Suppl 1):30–38

    Article  PubMed  Google Scholar 

  66. El Ahmadieh TY, Aoun SG, El Tecle NE et al (2013) A didactic and hands-on module enhances resident microsurgical knowledge and technical skill. Neurosurgery 73(Suppl 1):51–56

    Article  PubMed  Google Scholar 

  67. Lobel D, Elder J, Schirmer C et al (2013) A novel craniotomy simulator provides a validated method to enhance education in the management of traumatic brain injury. Neurosurgery 73:57–65

    PubMed  Google Scholar 

  68. Ray WZ, Ganju A, Harrop JS, Hoh DJ (2013) Developing an anterior cervical diskectomy and fusion simulator for neurosurgical resident training. Neurosurgery 73(Suppl 1):100–106

    Article  PubMed  Google Scholar 

  69. Harrop J, Rezai AR, Hoh DJ et al (2013) Neurosurgical training with a novel cervical spine simulator: posterior foraminotomy and laminectomy. Neurosurgery 73(Suppl 1):94–99

    Article  PubMed  Google Scholar 

  70. Guyatt G, Bombardier C, Tugwell P (1986) Measuring disease-specific quality of life in clinical trials. CMAJ 134:889–895

    CAS  PubMed Central  PubMed  Google Scholar 

  71. Downing SM (2003) Validity: on meaningful interpretation of assessment data. Med Educ 37:830–837

    Article  PubMed  Google Scholar 

  72. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (1994) Standards for educational and psychological testing. American Educational Research Association, Washington, DC

    Google Scholar 

  73. American Psychologoy Association (1966) Standards for educational and psychological tests and manuals. American Psychological Association, Washington, DC

    Google Scholar 

  74. Kane MT (2006) Validation. In: Brennan RL (ed) Educational measurement, 4th edn. Praeger, Westport

    Google Scholar 

  75. De Vet HCW, Terwee CB, Mokkink LB, Knol DL (2011) Measurement in medicine, 1st edn. Cambridge University Press, New York

    Book  Google Scholar 

  76. Hogle NJ, Liu Y, Ogden RT, Fowler DL (2014) Evaluation of surgical fellows’ laparoscopic performance using Global Operative Assessment of Laparoscopic Skills (GOALS). Surg Endosc. doi:10.1007/s00464-013-3324-6

    PubMed  Google Scholar 

  77. Van der Vleuten C (1996) The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ 1:41–67

    Article  Google Scholar 

Download references

Acknowledgments

Thanks to all the experts for their valuable contribution to this Delphi study: Dr. G. Cinalli, Italy; Dr. S. Constantini, Israel; Dr. P. Decq, France; Dr. S. deRibaupierre, Canada; Dr. C. DiRocco, Italy; Dr. Y. Ersahin, Turkey; Dr. J. Grotenhuis, the Netherlands; Dr. N. Gupta, United States of America; Dr. E. Hoving, the Netherlands; Dr. I. Pollack, United States of America; Dr. C. Sainte-Rose, France; Dr. S. Santoreneos, Australia; Dr. H. Schroeder, Germany; Dr. D. Thompson, United Kingdom; Dr. B. Warf, United States of America; Dr. J. Wellons, United States of America; and Dr. S. Zymberg, Brazil.

Author information

Authors and Affiliations

  1. Center for Image Guided Innovation and Therapeutic Intervention (CIGITI), The Hospital for Sick Children, Toronto, Ontario, Canada

    Gerben E. Breimer & James M. Drake

  2. Department of Neurosurgery, The Hospital for Sick Children, 555 University Avenue, Room 1504, Toronto, ON, M5G 1X8, Canada

    Gerben E. Breimer & James M. Drake

  3. Department of Neurosurgery, University Medical Center Groningen, Groningen, The Netherlands

    Gerben E. Breimer & Eelco W. Hoving

  4. Division of Clinical Neurological Sciences, Western University, London, Ontario, Canada

    Faizal A. Haji

  5. SickKids Learning Institute, The Hospital for Sick Children, Toronto, Ontario, Canada

    Faizal A. Haji

  6. The Wilson Centre for Research in Education, University of Toronto, Toronto, Ontario, Canada

    Faizal A. Haji

Authors
  1. Gerben E. Breimer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Faizal A. Haji
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eelco W. Hoving
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. James M. Drake
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to James M. Drake.

Appendix

Appendix

Table 5 Three initial lists for online survey
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Breimer, G.E., Haji, F.A., Hoving, E.W. et al. Development and content validation of performance assessments for endoscopic third ventriculostomy. Childs Nerv Syst 31, 1247–1259 (2015). https://doi.org/10.1007/s00381-015-2716-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00381-015-2716-4

Keywords

Search

Navigation