Advertisement

Measuring the Impact of Educational Interventions: A Quantitative Approach

  • Jenepher A. MartinEmail author
Chapter
Part of the Innovation and Change in Professional Education book series (ICPE, volume 17)

Overview

This chapter will discuss impact evaluation, an important method of measuring the effectiveness of an educational intervention. This form of evaluation represents a subset of program evaluation and focuses on outcomes and consequential events related to an educational intervention. In doing so, it incorporates several different quantitative methods and is typically reserved for stable, long-standing educational programs/curricula. Many of these methods are also used as part of program evaluation as a whole and in surgical research. Readers are directed to Chaps. 23 (“Demystifying Program Evaluation for Surgical Education”, Battista et al.) and 30 (“Researching in Surgical Education: An Orientation”, Ajjawi and McIllhenny) for more information on these subjects. In addition to providing a working definition of impact evaluation, this chapter will help define key concepts related to its successful use as well as aid in delineating the most useful quantitative methods to employ.

Keywords

Program evaluation Impact evaluation Quantitative measurements Outcome measurements Quasi-experimental design Pre-/post-test design Interrupted time series Reliability Validity 

References

  1. 1.
    Patton, M. Q. (1997). Utilization-focused evaluation (4th ed.). Thousand Oaks: Sage Publications.Google Scholar
  2. 2.
    Owen, J. M. (2006). Program evaluation: Forms and approaches (3rd ed.). Crows Nest: Allen and Unwin.Google Scholar
  3. 3.
    Fudickar, A., et al. (2012). The effect of the WHO Surgical Safety Checklist on complication rate and communication. Deutsches Ärzteblatt International, 109(42), 695–701.Google Scholar
  4. 4.
    Evers, U., et al. (2013). ‘Get your life back’: Process and impact evaluation of an asthma social marketing campaign targeting older adults. BMC Public Health, 13, 759–768.CrossRefGoogle Scholar
  5. 5.
    Tavakol, M., & Sanders, J. (2014). Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part I. Medical Teacher, 36(9), 746–756.CrossRefGoogle Scholar
  6. 6.
    Papaconstantinou, H. T., et al. (2013). Implementation of a surgical safety checklist: Impact on surgical team perspectives. The Oschner Journal, 13, 299–309.Google Scholar
  7. 7.
    Seymour, N. E., et al. (2002). Virtual reality training improves operating room performance. Results of a randomized, double-blinded study. Annals of Surgery, 236(4), 458–464.CrossRefGoogle Scholar
  8. 8.
    Anastakis, D. J., et al. (1999). Assessment of technical skills transfer from the bench training model to the human model. American Journal of Surgery, 177(2), 167–170.CrossRefGoogle Scholar
  9. 9.
    Moulton, C. E., et al. (2006). Teaching surgical skills: What kind of practice makes perfect? A randomized, controlled trial. Annals of Surgery, 244(3), 400–409.Google Scholar
  10. 10.
    Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues for field settings (1st ed.). Chicago: Rand McNally.Google Scholar
  11. 11.
    Martling, A. L., et al. (2000). Effect of a surgical training programme on outcome of rectal cancer in the County of Stockholm. Lancet, 356(9224), 93–96.CrossRefGoogle Scholar
  12. 12.
    Rosser, J. C., et al. (2007). The impact of video games on training surgeons in the 21st century. Archives of Surgery, 142, 181–186.CrossRefGoogle Scholar
  13. 13.
    Tavakol, M., & Sanders, J. (2014). Quantitative and qualitative methods in medical education research: AMEE Guide No 90: Part II. Medical Teacher, 36(10), 838–848.CrossRefGoogle Scholar
  14. 14.
    Artino, A. R., et al. (2014). Developing questionnaires for educational research: AMEE Guide No.87. Medical Teacher, 36(6), 463–474.CrossRefGoogle Scholar
  15. 15.
    DeVellis, R. F. (2014). Scale development: Theory and applications (2nd ed.). Newbury Park: Sage Publications.Google Scholar
  16. 16.
    Dillman, D., et al. (2009). Internet, mail and mixed-mode surveys: The tailored design method (3rd ed.). Hoboken: Wiley.Google Scholar
  17. 17.
    American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.Google Scholar
  18. 18.
    American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.Google Scholar
  19. 19.
    Yule, S., et al. (2008). Surgeons’ non-technical skills in the operating room: Reliability testing of the NOTSS behaviour rating system. World Journal of Surgery, 32, 548–556.CrossRefGoogle Scholar
  20. 20.
    Regher, G., et al. (1998). Comparing psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine, 73(9), 993–997.CrossRefGoogle Scholar
  21. 21.
    Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119, 166.e7–166.e16.CrossRefGoogle Scholar
  22. 22.
    Hojat, M., et al. (2002). Physician empathy: Definition, components, measurement and relationship to gender and speciality. American Journal of Psychiatry, 159(90), 1563–1569.CrossRefGoogle Scholar
  23. 23.
    Cook, D. A., & Hatala, R. (2016). Validation of educational assessments: A primer for simulation and beyond. Advances in Simulation, 1, 31.CrossRefGoogle Scholar
  24. 24.
    Martin, J. A., et al. (1997). Objective structured assessment of technical skill (OSATS) for surgical residents. British Journal of Surgery, 84, 273–278.CrossRefGoogle Scholar
  25. 25.
    Reznick, R., et al. (1997). Testing technical skill via an innovative ‘bench station’ examination. American Journal of Surgery, 173(3), 226–230.CrossRefGoogle Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Medical Student ProgramsEastern Health Clinical SchoolBox HillAustralia
  2. 2.Faculty of Medicine Nursing and Health SciencesMonash UniversityClaytonAustralia
  3. 3.School of Medicine, Deakin UniversityMelbourneAustralia

Personalised recommendations