Skip to main content

Advertisement

SpringerLink
Log in

Construct and face validity of the educational computer-based environment (ECE) assessment scenarios for basic endoneurosurgery skills

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

In neurosurgery education, there is a paradigm shift from time-based training to criterion-based model for which competency and assessment becomes very critical. Even virtual reality simulators provide alternatives to improve education and assessment in neurosurgery programs and allow for several objective assessment measures, there are not many tools for assessing the overall performance of trainees. This study aims to develop and validate a tool for assessing the overall performance of participants in a simulation-based endoneurosurgery training environment.

Methods

A training program was developed in two levels: endoscopy practice and beginning surgical practice based on four scenarios. Then, three experiments were conducted with three corresponding groups of participants (Experiment 1, 45 (32 beginners, 13 experienced), Experiment 2, 53 (40 beginners, 13 experienced), and Experiment 3, 26 (14 novices, 12 intermediate) participants). The results analyzed to understand the common factors among the performance measurements of these experiments. Then, a factor capable of assessing the overall skill levels of surgical residents was extracted. Afterwards, the proposed measure was tested to estimate the experience levels of the participants. Finally, the level of realism of these educational scenarios was assessed.

Results

The factor formed by time, distance, and accuracy on simulated tasks provided an overall performance indicator. The prediction correctness was very high for the beginners than the one for experienced surgeons in Experiments 1 and 2. When non-dominant hand is used in a surgical procedure-based scenario, skill levels of surgeons can be better predicted. The results indicate that the scenarios in Experiments 1 and 2 can be used as an assessment tool for the beginners, and scenario-2 in Experiment 3 can be used as an assessment tool for intermediate and novice levels. It can be concluded that forming the balance between perceived action capacities and skills is critical for better designing and developing skill assessment surgical simulation tools.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Ghaderi I, Manji F, Park YS, Juul D, Ott M, Harris I, Farrell TM (2015) Technical skills assessment toolbox: a review using the unitary framework of validity. Ann Surg 261(2):251–262

    Article  PubMed  Google Scholar 

  2. Kavic MS (2006) Simulators: a new use for an old paradigm. J Soc Laparoendosc Surg 10(3):281

    Google Scholar 

  3. Mason WTM, Strike PW (2003) Short communication see one, do one, teach one—is this still how it works? A comparison of the medical and nursing professions in the teaching of practical procedures. Med Teach 25(6):664–666

    Article  CAS  PubMed  Google Scholar 

  4. Rodriguez-Paz J, Kennedy M, Salas E, Wu AW, Sexton JB, Hunt EA, Pronovost PJ (2009) Beyond “see one, do one, teach one”: toward a different training paradigm. Qual Saf Health Care 18(1):63–68

    CAS  PubMed  Google Scholar 

  5. Heros RC, Morcos JM (2006) Neurosurgical education. Clin Neurosurg 53:26–37

    PubMed  Google Scholar 

  6. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84(2):273–278

    Article  CAS  PubMed  Google Scholar 

  7. Aggarwal R, Mytton OT, Derbrew M, Hananel D, Heydenburg M, Issenberg B, MacAulay C, Mancini ME, Morimoto T, Soper N, Ziv A, Reznick R (2010) Training and simulation for patient safety. Qual Saf Health Care 19(Suppl 2):i34–i43. doi:10.1136/qshc.2009.038562

    Article  PubMed  Google Scholar 

  8. Anderson DD, Long S, Thomas GW, Putnam MD, Bechtold JE, Karam MD (2015) Objective structured assessments of technical skills (OSATS) does not assess the quality of the surgical result effectively. Clin. Orthop Relat Res 474:871–873.

    Google Scholar 

  9. Parker SL, McGirt MJ, Asher AL, Selden NR (2015) Quality improvement in neurological surgery graduate medical education. Neurosurg Clin N Am 26(2):231–238

    Article  PubMed  Google Scholar 

  10. Kirkman MA, Ahmed M, Albert AF, Wilson MH, Nandi D, Sevdalis N (2014) The use of simulation in neurosurgical education and training: a systematic review. J Neurosurg 121(2):228–246

    Article  PubMed  Google Scholar 

  11. Rivard JD, Vergis AS, Unger BJ, Hardy KM, Andrew CG, Gillman LM, Park J (2014) Construct validity of individual and summary performance metrics associated with a computer-based laparoscopic simulator. Surg Endosc 28(6):1921–1928

    Article  PubMed  Google Scholar 

  12. Cotin S, Delingette H, Ayache N (2000) A hybrid elastic model for real-time cutting, deformations, and force feedback for surgery training and simulation. Vis Comput 16(8):437–452

    Article  Google Scholar 

  13. Greco EF, Glenn R, Allan O (2010) Identifying and classifying problem areas in laparoscopic skills acquisition: can simulators help? Acad Med 85(10):S5–S8

    Article  PubMed  Google Scholar 

  14. Bogdanova R, Boulanger P, Zheng B (2016) Depth perception of surgeons in minimally ınvasive surgery. Surg Innov 23: 515–524.

    Article  PubMed  Google Scholar 

  15. Yamaguchi S, Konishi K, Yasunaga T, Yoshida D, Kinjo N, Kobayashi K et al (2007) Construct validity for eye-hand coordination skill on a virtual reality laparoscopic surgical simulator. Surg Endosc 21(12):2253–2257

    Article  PubMed  Google Scholar 

  16. Silvennoinen M, Mecklin JP, Saariluoma P, Antikainen T (2009) Expertise and skill in minimally invasive surgery. Scand J Surg 98(4):209–213

    Article  CAS  PubMed  Google Scholar 

  17. Pallant J (2005) SPSS survival manual: a step by step guide to data analysis using SPSS, 2nd edn. Allen & Unwin, Sydney

    Google Scholar 

  18. Csikszentmihalyi M (2000) Beyond boredom and anxiety. Experiencing flow in work and play. Jossey-Bass, San Francisco

    Google Scholar 

  19. Deci EL (1975) Intrinsic motivation. Plenum Press, New York

    Book  Google Scholar 

  20. Hunt JMcV (1965) Intrinsic motivation and its role in development. Neb Symp Mot 13:198–274

    Google Scholar 

  21. Nakamura J, Csikszentmihalyi M (2014) The concept of flow. Flow and the foundations of positive psychology. Springer, Dordrecht, pp 239–263

    Google Scholar 

  22. Ramos P, Montez J, Tripp A, Ng CK, Gill IS, Hung AJ (2014) Face, content, construct and concurrent validity of dry laboratory exercises for robotic training using a global assessment tool. BJU Int 113(5):836–842

    Article  PubMed  Google Scholar 

  23. Verdaasdonk EGG, Stassen LPS, Van Wijk RPJ, Dankelman J (2007) The influence of different training schedules on the learning of psychomotor skills for endoscopic surgery. Surg Endosc 21(2):214–219

    Article  CAS  PubMed  Google Scholar 

  24. Schreuder HW, van Dongen KW, Roeleveld SJ, Schijven MP, Broeders IA (2009) Face and construct validity of virtual reality simulation of laparoscopic gynecologic surgery. Am J Obstet Gynecol 200(5):540–541

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This study is conducted for improving the scenario designs of the educational materials which are developed for endoneurosurgery education project (ECE: Tubitak 1001, Project No: 112K287) purposes. The authors would like to thank the support of TÜBİTAK 1001 program for realizing this research. The researchers would also like to thank the ECE project team and the Hacettepe University Medical School for their valuable support throughout the research.

Author information

Authors and Affiliations

  1. Department of Software Engineering, Faculty of Engineering, Atilim University, Ankara, Turkey

    Nergiz Ercil Cagiltay

  2. Department of Psychology, Faculty of Arts and Sciences, Çankaya University, Ankara, Turkey

    Erol Ozcelik

  3. Department of Computer Engineering, Faculty of Engineering, Atilim University, Ankara, Turkey

    Gokhan Sengul

  4. Department of Neurosurgery, Faculty of Medicine, Hacettepe University, Ankara, Turkey

    Mustafa Berker

Authors
  1. Nergiz Ercil Cagiltay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Erol Ozcelik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gokhan Sengul
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mustafa Berker
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nergiz Ercil Cagiltay.

Ethics declarations

Disclosures

Nergiz Ercil Cagiltay, Erol Ozcelik, Gokhan Sengul, and Mustafa Berker have no conflicts of interests of financial ties to disclose.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cagiltay, N.E., Ozcelik, E., Sengul, G. et al. Construct and face validity of the educational computer-based environment (ECE) assessment scenarios for basic endoneurosurgery skills. Surg Endosc 31, 4485–4495 (2017). https://doi.org/10.1007/s00464-017-5502-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-017-5502-4

Keywords

Search

Navigation