Abstract
As curricular reforms are implemented, there is often urgency among scholars to swiftly evaluate curricular outcomes and establish whether desired impacts have been realized. Consequently, many evaluative studies focus on summative program outcomes without accompanying evaluations of implementation. This runs the risk of Type III errors, whereby outcome evaluations rest on unverified assumptions about the appropriate implementation of prescribed curricular activities. Such errors challenge the usefulness of the evaluative studies, casting doubt on accumulated knowledge about curricular innovations, and posing problems for educational systems working to mobilize scarce resources. Unfortunately, however, there is long-standing inattention to the evaluation of implementation in health professions education (HPE). To address this, we propose an accessible framework that provides substantive guidance for evaluative research on implementation of curricular innovations. The Prescribed-Intended-Enacted-Sustainable (PIES) framework that is articulated in this paper, introduces new concepts to HPE—with a view to facilitating more nuanced examination of the evolution of curricula as they are implemented. Critically, the framework is theoretically grounded, integrating evaluation and implementation science as well as education theory. It outlines when, how, and why evaluators need to direct attention to curricular implementation, providing guidance on how programs can map out meaningful evaluative research agendas. Ultimately, this work is intended to support evaluators and educators, seeking to design evaluation studies that provide more faithful, useful representations of the intricacies of curricular change implementation.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Albarqouni, L., Glasziou, P., & Hoffmann, T. (2018). Completeness of the reporting of evidence-based practice educational interventions: A review. Medical Education, 52(2), 161–170. https://doi.org/10.1111/medu.13410
Alkin, M. C., & King, J. A. (2017). Definitions of evaluation use and misuse, evaluation influence, and factors affecting use. American Journal of Evaluation, 38(3), 434–450. https://doi.org/10.1177/1098214017717015
Baker, L., Shing, L. K., Wright, S., Mylopoulos, M., Kulasegaram, K., & Ng, S. (2019). Aligning and applying the paradigms and practices of education. Academic Medicine: Journal of the Association of American Medical Colleges, 94(7), 1060–1060.
Basch, C. E., Sliepcevich, E. M., Gold, R. S., Duncan, D. F., & Kolbe, L. J. (1985). Avoiding type III errors in health education program evaluations: A case study. Health Education Behaviour., 12(3), 315–331. https://doi.org/10.1177/109019818501200311
Biggs, J. (1985). The role of metalearning in study processes. British Journal of Educational Psychology, 55, 185–212.
Bismack, A. S., Arias, A. M., Davis, E. A., & Palincsar, A. S. (2014). Connecting curriculum materials and teachers: Elementary science teachers’ enactment of a reform-based curricular unit. Journal of Science Teacher Education, 25(4), 489–512. https://doi.org/10.1007/s10972-013-9372-x
Boyd, V. A., Whitehead, C. R., Thille, P., Ginsburg, S., Brydges, R., & Kuper, A. (2018). Competency-based medical education: The discourse of infallibility. Medical Education, 52(1), 45–57. https://doi.org/10.1111/medu.13467
Brown, C., & Zhang, D. (2017). Accounting for discrepancies in teachers’ attitudes towards evidence use and actual instances of evidence use in schools. Cambridge Journal of Education., 47(2), 277–295. https://doi.org/10.1080/0305764X.2016.1158784
Brydges, R., Tran, J., Goffi, A., Lee, C., & Miller, D. M. M. (2020). Resident learning trajectories in the workplace: A self-regulated learning analysis. Medical Education. https://doi.org/10.1111/medu.14288
Burkhauser, M. A., & Lesaux, N. K. (2017). Exercising a bounded autonomy: Novice and experienced teachers’ adaptations to curriculum materials in an age of accountability. Journal of Curriculum Studies, 49(3), 291–312. https://doi.org/10.1080/00220272.2015.1088065
Castro Superfine, A., Marshall, A. M., & Kelso, C. (2015). Fidelity of implementation: Bringing written curriculum materials into the equation. Curriculum Journal, 26(1), 164–191. https://doi.org/10.1080/09585176.2014.990910
Caverzagie, K. J., Nousiainen, M. T., Ferguson, P. C., et al. (2017). Overarching challenges to the implementation of competency-based medical education. Medical Teacher, 39(6), 588–593. https://doi.org/10.1080/0142159X.2017.1315075
Cianciolo, A. T., & Regehr, G. (2019). Learning theory and educational intervention: Producing meaningful evidence of impact through layered analysis. Academic Medicine, 94(6), 789–794. https://doi.org/10.1097/ACM.0000000000002591
Cofrancesco, J., Wright, S. M., Vohr, E., & Ziegelstein, R. C. (2017). Creating an “education shark tank” to encourage and support educational scholarship and innovation. Academic Medicine, 92(11), 1578–1582. https://doi.org/10.1097/ACM.0000000000001715
Cummings, A., Lund, S., Campling, N., May, C., Richardson, A., & Myall, M. (2017). Implementing communication and decision-making interventions directed at goals of care: a theory-led scoping review. BMJ Open, 7(10), https://doi.org/10.1136/bmjopen-2017-017056.
Daneman, D., & Benatar, S. (2019). Dynamic tensions following new pedagogy in undergraduate medical education. Academic Medicine. https://doi.org/10.1097/acm.0000000000002795
Dobson, D., & Cook, T. J. (1980). Avoiding type III error in program evaluation. Results from a field experiment. Evaluation and Program Planning., 3(4), 269–276. https://doi.org/10.1016/0149-7189(80)90042-7
Drake, C., & Sherin, M. G. (2006). Practicing change: Curriculum adaptation and teacher narrative in the context of mathematics education reform. Curriculum Inquiry, 36(2), 153–187. https://doi.org/10.1111/j.1467-873X.2006.00351.x
Gehrke, N. J., Knapp, M. S., & Sirotnik, K. A. (1992). In search of the school curriculum. Review of Research in Education, 18, 51–110.
Glasgow, R. E., Harden, S. M., Gaglio, B., et al. (2019). RE-AIM planning and evaluation framework: Adapting to new science and practice with a 20-year review. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2019.00064
Goldhaber, D., Holden, K. L., & Grout, C. (2019). Errors in administrative education data: A cautionary tale. Educational Researcher, 48(3), 179–182. https://doi.org/10.3102/0013189X19837598
Hafferty, F. W. (1998). Beyond curriculum reform: Confronting medicine’s hidden curriculum. Academic Medicine, 73, 403–407.
Harn, B., Parisi, D., & Stoolmiller, M. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools? Exceptional Children, 79(2), 181–193. https://doi.org/10.1177/001440291307900204
Hawe, P., Shiell, A., & Riley, T. (2004). Complex interventions: How “out of control” can a randomised controlled trial be? BMJ, 328(7455), 1561. https://doi.org/10.1136/bmj.328.7455.1561
Hawick, L., Cleland, J., & Kitto, S. (2017). Getting off the carousel: Exploring the wicked problem of curriculum reform. Perspectives Medical Education, 6(5), 337–343. https://doi.org/10.1007/s40037-017-0371-z
Hawick, L., Kitto, S., & Cleland, J. (2016). Curriculum reform: The more things change, the more they stay the same? Perspectives on Medical Education., 5(1), 5–7. https://doi.org/10.1007/s40037-016-0252-x
Hawkins, R. E., Welcher, C. M., Holmboe, E. S., et al. (2015). Implementation of competency-based medical education: Are we addressing the concerns and challenges? Medical Education, 49(11), 1086–1102. https://doi.org/10.1111/medu.12831
Hoffmann, T. C., Glasziou, P. P., Boutron, I., et al. (2014). Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. BMJ, 348(March), 1–12. https://doi.org/10.1136/bmj.g1687
Holmboe, E. S., Sherbino, J., Englander, R., Snell, L., & Frank, J. R. (2017). A call to action: The controversy of and rationale for competency-based medical education. Medical Teacher, 39(6), 574–581. https://doi.org/10.1080/0142159X.2017.1315067
Horsley, T., & Regehr, G. (2018a). When are two interventions the same ? Implications for reporting guidelines in education. Medical Education, 52, 141–143. https://doi.org/10.1111/medu.13496
Horsley, T., & Regehr, G. (2018b). When are two interventions the same? Implications for reporting guidelines in education. Medical Education, 52(2), 141–143. https://doi.org/10.1111/medu.13496
Hudson, J. N., Farmer, E. A., Weston, K. M., & Bushnell, J. A. (2015). Using a framework to implement large-scale innovation in medical education with the intent of achieving sustainability. BMC Medical Education. https://doi.org/10.1186/s12909-014-0282-1
Hulme, J. A., & Winstone, N. E. (2017). Do no harm Risk aversion versus risk management in the context of pedagogic frailty. Knowledge Management and E-Learning., 9(3), 261–274. https://doi.org/10.34105/j.kmel.2017.09.016
Irby, D. M., & Wilkerson, L. A. (2003). Educational innovations in academic medicine and environmental trends. Journal of General Internal Medicine, 18(5), 370–376. https://doi.org/10.1046/j.1525-1497.2003.21049.x
Jackson, N. (2004). Developing the concept of metalearning. Innovations in Education and Teaching International, 41(4), 391–403. https://doi.org/10.1080/1470329042000276995
Keiser, J. M., & Lambdin, D. V. (1996). The clock is ticking: Time constraint issues in mathematics teaching reform. Journal of Education Research, 90(1), 23–31. https://doi.org/10.1080/00220671.1996.9944440
Leonard, S. N., Fitzgerald, R. N., Riordan, G., et al. (2017). Using developmental evaluation as a design thinking tool for curriculum innovation in professional higher education curriculum innovation in professional higher education. Higher Education Research and Development, 2016, 4360. https://doi.org/10.1080/07294360.2015.1087386
Leong, Y. H., & Chick, H. L. (2011). Time pressure and instructional choices when teaching mathematics. Mathematics Education Research Journal, 23(3), 347–362. https://doi.org/10.1007/s13394-011-0019-y
Lucas, U., & Meyer, J. H. F. (2004). Supporting student awareness: Understanding student preconceptions of their subject matter within introductory courses. Innovations in Education and Teaching International, 41(4), 459–471. https://doi.org/10.1080/1470329042000277039
Maltenfort, M. (2015). Type I, type II, and occasionally type III. Journal of Spinal Disorders and Techniques, 28(5), 189. https://doi.org/10.1097/BSD.0000000000000285
Mansour, N. (2009). Science teachers’ beliefs and practices: Issues, implications and research agenda. International Journal of Environmental and Science Education., 4(1), 25–48.
Manzone, J., Regehr, G., Garbedian, S., & Brydges, R. (2019). Assigning medical students learning goals: Do they do it, and what happens when they don’t? Teaching and Learning in Medicine, 31(5), 528–535. https://doi.org/10.1080/10401334.2019.1600520
Martimianakis, M. A., & Hafferty, F. W. (2016). Exploring the interstitial space between the ideal and the practised: Humanism and the hidden curriculum of system reform. Medical Education, 50(3), 278–280. https://doi.org/10.1111/medu.12982
McLaughlin, M., & Mitra, D. (2001). Theory-based change and change-based theory: Going deeper going broader. Journal of Educational Change., 2(4), 301–323. https://doi.org/10.1023/A:1014616908334
Meinema, J. G., Buwalda, N., van Etten-Jamaludin, F., Visser, M. R., & van Nynke, D. (2019). Intervention descriptions in medical education: What can be improved? Academic medicine a systematic review and checklist. Academic Medicine. https://doi.org/10.1097/ACM.0000000000002428
Montague, S. (2019). Does your implementation fit your theory of change? Canadian Journal of Program Evaluation., 33(3), 316–335. https://doi.org/10.3138/cjpe.53008
Nelson, J., & Campbell, C. (2017). Evidence-informed practice in education: Meanings and applications. Educational Researcher, 59(2), 127–135. https://doi.org/10.1080/00131881.2017.1314115
Nousiainen, M. T., Caverzagie, K. J., Ferguson, P. C., & Frank, J. R. (2017). Implementing competency-based medical education: What changes in curricular structure and processes are needed? Medical Teacher, 39(6), 594–598. https://doi.org/10.1080/0142159X.2017.1315077
O’Sullivan, P. S. (2019). What questions guide investing in our faculty? Academic Medicine, 94, S11–S13. https://doi.org/10.1097/ACM.0000000000002910
Onyura, B. (2020). Useful to whom? Evaluation utilization theory and boundaries for program evaluation scope. Medical Education. https://doi.org/10.1111/medu.14281
Onyura, B., Baker, L., Cameron, B., Friesen, F., & Leslie, K. (2016). Evidence for curricular and instructional design approaches in undergraduate medical education: An umbrella review. Medical Teacher, 38(2), 150–161. https://doi.org/10.3109/0142159X.2015.1009019
Patterson, F., & Zibarras, L. D. (2017). Selecting for creativity and innovation potential: Implications for practice in healthcare education. Adv Heal Sci Educ., 22(2), 417–428. https://doi.org/10.1007/s10459-016-9731-4
Patton, M. Q. (1997). Utilization-Focused Evaluation: The New Century Text (3rd ed.). Thousand Oaks, CA: Sage.
Petty, E., & Golden, R. N. (2017). Embracing innovation in medical education. Wisconsin Medical Journal, 116(3), 179–180.
Phillips, A. C., Lewis, L. K., McEvoy, M. P., et al. (2016). Development and validation of the guideline for reporting evidence-based practice educational interventions and teaching (GREET). BMC Medical Education, 16(1), 1–10. https://doi.org/10.1186/s12909-016-0759-1
Pintó, R. (2005). Introducing curriculum innovations in science: Identifying teachers’ transformations and the design of related teacher education. Science & Education, 89(1), 1–12. https://doi.org/10.1002/sce.20039
Pleger, L., & Sager, F. (2016). Betterment, undermining, support and distortion : A heuristic model for the analysis of pressure on evaluators. Evaluation and Program Planning, 69, 166–172.
Proctor, E., Silmere, H., Raghavan, R., et al. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7.
Remillard, J. T. (2005). Examining key concepts in research on teachers’ use of mathematics curricula. Review of Educational Research, 75(2), 211–246. https://doi.org/10.3102/00346543075002211
Scanlon JW, Horst P, Nay JN, Schmidt RE, Waller JD. Evaluability assessment: Avoiding type III and IV errors. In: Evaluation Management: A Sourcebook of Readings. University of California Libraries; 1977:264–282.
Scheirer, M. A., & Dearing, J. W. (2011). An agenda for research on the sustainability of public health programs. American Journal of Public Health, 101(11), 2059–2067. https://doi.org/10.2105/AJPH.2011.300193
Scriven, M. (1972). Prose and cons about goal-free evalution. Journal Education Evaluation., 3(4), 1–7.
Shadle, S. E., Marker, A., & Earl, B. (2017). Faculty drivers and barriers: laying the groundwork for undergraduate STEM education reform in academic departments. International Journal of STEM Education. https://doi.org/10.1186/s40594-017-0062-7
Shediac-Rizkallah, M. C., & Bone, L. R. (1998). Planning far the sustainability of community-based health programs: Conceptual frameworks and future directions for research, practice and policy. Health Education Research, 13(1), 87–108. https://doi.org/10.1093/her/13.1.87
Sherin, M. G., & Drake, C. (2009). Curriculum strategy framework: Investigating patterns in teachers’ use of a reform-based elementary mathematics curriculum. Journal of Curriculum Studies, 41(4), 467–500. https://doi.org/10.1080/00220270802696115
Smagorinsky, P., Lakly, A., & Johnson, T. (2002). Acquiescence, accommodation, and resistance in learning to teach within a prescribed curriculum - Pr. English Education, 34(3), 187–213. https://doi.org/10.2307/40173127
Sridharan, S., & Nakaima, A. (2019). Till time (and poor planning) do us part: Programs as dynamic systems—Incorporating planning of sustainability into theories of change. Canadian Journal of Program Evaluation., 33(3), 375–394. https://doi.org/10.3138/cjpe.53055
Stevens, C. D. (2018). Repeal and replace? A note of caution for medical school curriculum reformers. Academic Medicine, 93(10), 1425–1427. https://doi.org/10.1097/ACM.0000000000002219
Tannenbaum, D. (2012). Rebuttal: will the Triple C curriculum produce better family physicians? Yes. Canadian Family Physician., 58(10), e538.
Van Melle, E., Frank, J. R., Holmboe, E. S., Dagnone, D., Stockley, D., & Sherbino, J. (2019). A core components framework for evaluating implementation of competency-based medical education programs. Academic Medicine, 94(7), 1002–1009. https://doi.org/10.1097/ACM.0000000000002743
Varpio, L., Bell, R., Hollingworth, G., et al. (2012). Is transferring an educational innovation actually a process of transformation. Advances in Health Sciences Education. https://doi.org/10.1007/s10459-011-9313-4
Wallace, C. S., & Kang, N. H. (2004). An investigation of experienced secondary science teachers’ beliefs about inquiry: An examination of competing belief sets. Journal of Research in Science Teaching, 41(9), 936–960. https://doi.org/10.1002/tea.20032
Whitehead, C. R. (2012). Will the triple C curriculum produce better family physicians? No. Canadian Family Physician., 58(10), 1071–1073.
Whitehead, C. R., Hodges, B. D., & Austin, Z. (2013). Captive on a carousel: Discourses of “new” in medical education 1910–2010. Adv Heal Sci Educ., 18(4), 755–768. https://doi.org/10.1007/s10459-012-9414-8
Zimmerman, B., & Schunk, D. H. (2004). Self-regulating intellectual processes and outcomes: A social cognitive perspective. In D. Y. Dai & R. J. Steinberg (Eds.), Motivation, Emotion and Cognition: Integrative Perspectives on Intellectual Functioning and Development (pp. 323–349). Lawrence Erlbaum.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Onyura, B., Lass, E., Lazor, J. et al. Vitalizing the evaluation of curricular implementation: a framework for attending to the “how and whys” of curriculum evolution. Adv in Health Sci Educ 27, 553–572 (2022). https://doi.org/10.1007/s10459-021-10083-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-021-10083-6