Abstract
Although the principles behind assessment for and as learning are well-established, there can be a struggle when reforming traditional assessment of learning to a program which encompasses assessment for and as learning. When introducing and reporting reforms, tensions in faculty may arise because of differing beliefs about the relationship between assessment and learning and the rules for the validity of assessments. Traditional systems of assessment of learning privilege objective, structured quantification of learners’ performances, and are done to the students. Newer systems of assessment promote assessment for learning, emphasise subjectivity, collate data from multiple sources, emphasise narrative-rich feedback to promote learner agency, and are done with the students. This contrast has implications for implementation and evaluative research. Research of assessment which is done to students typically asks, “what works”, whereas assessment that is done with the students focuses on more complex questions such as “what works, for whom, in which context, and why?” We applied such a critical realist perspective drawing on the interplay between structure and agency, and a systems approach to explore what theory says about introducing programmatic assessment in the context of pre-existing traditional approaches. Using a reflective technique, the internal conversation, we developed four factors that can assist educators considering major change to assessment practice in their own contexts. These include enabling positive learner agency and engagement; establishing argument-based validity frameworks; designing purposeful and eclectic evidence-based assessment tasks; and developing a shared narrative that promotes reflexivity in appreciating the complex relationships between assessment and learning.
Similar content being viewed by others
References
Adie, L. E., Willis, J., & Van der Kleij, F. M. (2018). Diverse perspectives on student agency in classroom assessment. The Australian Educational Researcher, 45(1), 1–12.
American Educational Research Association, American Psychological Association, & National Council on measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
Archer, M. (1982). Morphogenesis versus structuration: On combining structure and action. The British Journal of Sociology, 33(4), 455–483. https://doi.org/10.2307/589357
Archer, M. S. (2000). Being human: The problem of agency. Cambridge University Press.
Archer, M. (2002). Realism and the problem of agency. Alethia, 5(1), 11–20.
Archer, M. (2003). Structure, agency and the internal conversation. Cambridge University Press.
Archer, M. S. (2007). Making our way through the world: Human reflexivity and social mobility. Cambridge University Press.
Ashwin, P. (2008). Accounting for structure and agency in ‘close-up’research on teaching, learning and assessment in higher education. International Journal of Educational Research, 47(3), 151–158.
Baird, J. A., Andrich, D., Hopfenbeck, T. N., & Stobart, G. (2017). Assessment and learning: Fields apart? Assessment in Education: Principles, Policy and Practice, 24(3), 317–350.
Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy and Practice, 18(1), 5–25.
Bhaskar, R. (1978). A realist theory of science. UK Harvester Press.
Bhaskar, R., Danermark, B., & Price, L. (2018). Interdisciplinarity and wellbeing: A critical realist general theory of interdisciplinarity. Routledge.
Billett, S. (2001). Vocational educators: Understanding practice at work. In C. Velkde (Ed.), International perspectives on competence in the workplace: Research policy and practice. Dordecht: Kluwer Academic.
Bok, H. G., Jaarsma, D. A., Spruijt, A., Van Beukelen, P., Van der Vleuten, C. P., et al. (2016). Feedback-giving behaviour in performance evaluations during clinical clerkships. Medical Teacher, 38(1), 88–95. https://doi.org/10.3109/0142159X.2015.1017448
Case, J. M. (2015). A social realist perspective on student learning in higher education: The morphogenesis of agency. Higher Education Research and Development, 34(5), 841–852. https://doi.org/10.1080/07294360.2015.1011095
Cilliers, F. J., Schuwirth, L. W., Adendorff, H. J., Herman, N., & Van der Vleuten, C. P. (2010). The mechanism of impact of summative assessment on medical students’ learning. Advances in Health Sciences Education: Theory and Practice, 15(5), 695–715. https://doi.org/10.1007/s10459-010-9232-9
Clauser, B., Margolis, M., & Case, S. (2006). Testing for licensure and certification in the professions. Educational Measurement, 4, 701–731.
Cook, D. A., & Beckman, T. J. (2006). Current concepts in validity and reliability for psychometric instruments: Theory and application. The American Journal of Medicine, 119(2), 166 e7-166 e16. https://doi.org/10.1016/j.amjmed.2005.10.036
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281–302. https://doi.org/10.1037/h0040957
Dalkin, S. M., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M. (2015). What’s in a mechanism? Development of a key concept in realist evaluation. Implementation Science, 10(1), 49. https://doi.org/10.1186/s13012-015-0237-x
Dannefer, E. F. (2013). Beyond assessment of learning toward assessment for learning: Educating tomorrow’s physicians. Medical Teacher, 35(7), 560–563. https://doi.org/10.3109/0142159X.2013.787141
De Souza, D. E. (2013). Elaborating the Context-Mechanism-Outcome configuration (CMOc) in realist evaluation: A critical realist perspective. Evaluation, 19(2), 141–154.
Delandshere, G. (2001). Implicit theories, unexamined assumptions and the status quo of educational assessment. Assessment in Education: Principles, Policy and Practice, 8(2), 113–133.
Dijkstra, J., Galbraith, R., Hodges, B. D., McAvoy, P. A., McCrorie, P., et al. (2012). Expert validation of fit-for-purpose guidelines for designing programmes of assessment. BMC Medical Education, 12, 20. https://doi.org/10.1186/1472-6920-12-20
Downing, S. M. (2003). Validity: On meaningful interpretation of assessment data. Medical Education, 37(9), 830–837. https://doi.org/10.1046/j.1365-2923.2003.01594.x
Driessen, E., Van der Vleuten, C., Schuwirth, L., Van Tartwijk, J., & Vermunt, J. (2005). The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: A case study. Medical Education, 39(2), 214–220. https://doi.org/10.1111/j.1365-2929.2004.02059.x
Driessen, E. W., Van Tartwijk, J., Govaerts, M., Teunissen, P., & Van der Vleuten, C. P. (2012). The use of programmatic assessment in the clinical workplace: A Maastricht case report. Medical Teacher, 34(3), 226–231. https://doi.org/10.3109/0142159X.2012.652242
Driessen, E., Van Tartwijk, J., Van der Vleuten, C., & Wass, V. (2007). Portfolios in medical education: Why do they meet with mixed success? A systematic review (Review). Medical Education, 41(12), 1224–1233. https://doi.org/10.1111/j.1365-2923.2007.02944.x
Ellaway, R. H., Kehoe, A., & Illing, J. (2020). Critical realism and realist inquiry in medical education. Academic Medicine, 95(7), 984–988. https://doi.org/10.1097/ACM.0000000000003232
Epstein, R. M., & Hundert, E. M. (2002). Defining and assessing professional competence. JAMA, 287(2), 226–235. https://doi.org/10.1001/jama.287.2.226
Groff, R. (2004). Critical realism, post-positivism and the possibility of knowledge. Routledge.
Haggerty, M. (1918). Tests of applicants for admission to University of Minnesota Medical School. Journal of Educational Psychology, 9(5), 278.
Harden, R. M., & Gleeson, F. A. (1979). Assessment of clinical competence using an Objective Structured Clinical Examination (OSCE). Medical Education, 13(1), 41–54.
Harrison, C. J., Konings, K. D., Schuwirth, L., Wass, V., & Van der Vleuten, C. (2015). Barriers to the uptake and use of feedback in the context of summative assessment. Advances in Health Sciences Education, 20(1), 229–245. https://doi.org/10.1007/s10459-014-9524-6
Harrison, C. J., Konings, K. D., Schuwirth, L. W. T., Wass, V., & Van der Vleuten, C. P. M. (2017). Changing the culture of assessment: The dominance of the summative assessment paradigm. BMC Medical Education, 17(1), 73. https://doi.org/10.1186/s12909-017-0912-5
Heeneman, S., Oudkerk Pool, A., Schuwirth, L. W., Van der Vleuten, C. P., & Driessen, E. W. (2015). The impact of programmatic assessment on student learning: Theory versus practice. Medical Education, 49(5), 487–498. https://doi.org/10.1111/medu.12645
Hodges, B. (2006). Medical education and the maintenance of incompetence. Medical Teacher, 28(8), 690–696. https://doi.org/10.1080/01421590601102964
Hodges, B. (2013). Assessment in the post-psychometric era: Learning to love the subjective and collective. Medical Teacher, 35(7), 564–568. https://doi.org/10.3109/0142159X.2013.789134
Hutchings, M., Scammell, J., & Quinney, A. (2013). Praxis and reflexivity for interprofessional education: Towards an inclusive theoretical framework for learning. Journal of Interprofessional Care, 27(5), 358–366. https://doi.org/10.3109/13561820.2013.784729
Kahn, P. (2013). The informal curriculum: A case study on tutor reflexivity, corporate agency and medical professionalism. Teaching in Higher Education, 18(6), 631–642. https://doi.org/10.1080/13562517.2013.774356
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
Khanna, P., Roberts, C., & Lane, A. S. (2021). Designing health professional education curricula using systems thinking perspectives. BMC Medical Education, 21, 20. https://doi.org/10.1186/s12909-020-02442-5
Leibowitz, B., Van Schalkwyk, S., Ruiters, J., Farmer, J., & Adendorff, H. (2012). “It’s been a wonderful life”: Accounts of the interplay between structure and agency by “good” university teachers. Higher Education, 63(3), 353–365.
Marceau, M., Gallagher, F., Young, M., & St-Onge, C. (2018). Validity as a social imperative for assessment in health professions education: A concept analysis. Medical Education, 52(6), 641–653. https://doi.org/10.1111/medu.13574
Meeuwissen, S. N. E., Stalmeijer, R. E., & Govaerts, M. (2019). Multiple-role mentoring: Mentors’ conceptualisations, enactments and role conflicts. Medical Education, 53(6), 605–615. https://doi.org/10.1111/medu.13811
Messick, S. (1995). Validity of psychological-assessment—validation of inferences from persons responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066x.50.9.741
Newble, D. I., & Swanson, D. B. (1988). Psychometric characteristics of the objective structured clinical examination. Medical Education, 22(4), 325–334. https://doi.org/10.1111/j.1365-2923.1988.tb00761.x
Norcini, J., Anderson, M. B., Bollela, V., Burch, V., Costa, M. J., et al. (2018). 2018 Consensus framework for good assessment. Medical Teacher, 40(11), 1102–1109. https://doi.org/10.1080/0142159X.2018.1500016
Pawson, R., Tilley, N., & Tilley, N. (1997). Realistic evaluation. Sage.
Priestley, M., Edwards, R., Priestley, A., & Miller, K. (2012). Teacher agency in curriculum making: Agents of change and spaces for manoeuvre. Curriculum Inquiry, 42(2), 191–214. https://doi.org/10.1111/j.1467-873X.2012.00588.x
Priestley, M., Edwards, R., Priestley, A., & Miller, K. (2015). Teacher agency in curriculum making: Agents of change and spaces for manoeuvre. Curriculum Inquiry, 42(2), 191–214. https://doi.org/10.1111/j.1467-873X.2012.00588.x
Roberts, C., Shadbolt, N., Clark, T., & Simpson, P. (2014). The reliability and validity of a portfolio designed as a programmatic assessment of performance in an integrated clinical placement. BMC Medical Education, 14(1), 197.
Roberts, C., Kumar, K., & Finn, G. (2020). Navigating the qualitative manuscript writing process: some tips for authors and reviewers. BMC Med Educ, 20, 439. https://doi.org/10.1186/s12909-020-02370-4.
Schut, S., Driessen, E., Van Tartwijk, J., Van der Vleuten, C., & Heeneman, S. (2018). Stakes in the eye of the beholder: An international study of learners’ perceptions within programmatic assessment. Medical Education, 52(6), 654–663. https://doi.org/10.1111/medu.13532
Schut̄, S., Van Tartwijk, J., Driessen, E., Van der Vleuten, C., & Heeneman, S. (2020). Understanding the influence of teacher-learner relationships on learners’ assessment perception. Advances in Health Sciences Education: Theory and Practice, 25(2), 441–456. https://doi.org/10.1007/s10459-019-09935-z
Schut, S., Maggio, L. A., Heeneman, S., Van Tartwijk, J., Van der Vleuten, C., et al. (2021). Where the rubber meets the road: An integrative review of programmatic assessment in health care professions education. Perspectives on Medical Education, 10(1), 6–13. https://doi.org/10.1007/s40037-020-00625-w
Schuwirth, L., & Ash, J. (2013). Assessing tomorrow’s learners: in competency-based education only a radically different holistic method of assessment will work. Six things we could forget. Medical Teacher, 35(7), 555–559. https://doi.org/10.3109/0142159X.2013.787140
Schuwirth, L., Valentine, N., & Dilena, P. (2017). An application of programmatic assessment for learning (PAL) system for general practice training. GMS Journal for Medical Education. https://doi.org/10.3205/zma001133
Schuwirth, L. W., & Van der Vleuten, C. P. (2006). A plea for new psychometric models in educational assessment. Medical Education, 40(4), 296–300. https://doi.org/10.1111/j.1365-2929.2006.02405.x
Schuwirth, L. W., & Van der Vleuten, C. P. (2011). Programmatic assessment: From assessment of learning to assessment for learning. Medical Teacher, 33(6), 478–485. https://doi.org/10.3109/0142159X.2011.565828
Schuwirth, L. W., & Van der Vleuten, C. P. (2012). Programmatic assessment and Kane’s validity perspective. Medical Education, 46(1), 38–48. https://doi.org/10.1111/j.1365-2923.2011.04098.x
Schuwirth, L. W., & Van der Vleuten, C. P. (2019). Current assessment in medical education: Programmatic assessment. Journal of Applied Testing Technology, 20(S2), 2–10.
Schuwirth, L. W. T., & Van der Vleuten, C. P. M. (2020). A history of assessment in medical education. Advances in Health Sciences Education: Theory and Practice, 25(5), 1045–1056. https://doi.org/10.1007/s10459-020-10003-0
St-Onge, C., Young, M., Eva, K. W., & Hodges, B. (2017). Validity: One word with a plurality of meanings (journal article). Advances in Health Sciences Education: Theory and Practice, 22(4), 853–867. https://doi.org/10.1007/s10459-016-9716-3
Swanson, D. B., & Roberts, T. E. (2016). Trends in national licensing examinations in medicine. Medical Education, 50(1), 101–114. https://doi.org/10.1111/medu.12810
Ten Cate, O., & Regehr, G. (2019). The power of subjectivity in the assessment of medical trainees. Academic Medicine, 94(3), 333–337. https://doi.org/10.1097/ACM.0000000000002495
Torre, D. M., Schuwirth, L., & Van der Vleuten, C. (2020). Theoretical considerations on programmatic assessment. Medical Teacher, 42(2), 213–220. https://doi.org/10.1080/0142159X.2019.1672863
Torre, D., Rice, N. E., Ryan, A., Bok, H., Dawson, L.J., Bierer, B., Wilkinson, T. J., Tait, G. R., Laughlin, T., Veerapen, K. & Heeneman, S. (2021). Ottawa 2020 consensus statements for programmatic assessment 2: Implementation and practice. Medical Teacher. https://doi.org/10.1080/0142159X.2021.1956681
Valentine, N., Durning, S., Shanahan, E. M., & Schuwirth, L. (2021). Fairness in human judgement in assessment: A hermeneutic literature review and conceptual framework. Advances in Health Sciences Education: Theory and Practice, 26(2), 713–738. https://doi.org/10.1007/s10459-020-10002-1
Van der Vleuten, C., Lindemann, I., & Schmidt, L. (2018). Programmatic assessment: The process, rationale and evidence for modern evaluation approaches in medical education. Medical Journal of Australia, 209(9), 386–388. https://doi.org/10.5694/mja17.00926
Van der Vleuten, C. P., & Schuwirth, L. W. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309–317. https://doi.org/10.1111/j.1365-2929.2005.02094.x
Van der Vleuten, C. P., Schuwirth, L. W., Driessen, E. W., Dijkstra, J., Tigelaar, D., et al. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 34(3), 205–214. https://doi.org/10.3109/0142159X.2012.652239
Van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., Govaerts, M. J. B., & Heeneman, S. (2015). Twelve Tips for programmatic assessment. Medical Teacher, 37(7), 641–646. https://doi.org/10.3109/0142159X.2014.973388
Vosniadou, S. (2014). Examining cognitive development from a conceptual change point of view: The framework theory approach. European Journal of Developmental Psychology, 11(6), 645–661. https://doi.org/10.1080/17405629.2014.921153
Vosniadou, S., Ioannides, C., Dimitrakopoulou, A., & Papademetriou, E. (2001). Designing learning environments to promote conceptual change in science. Learning and Instruction, 11(4–5), 381–419. https://doi.org/10.1016/S0959-4752(00)00038-4
Watling, C. J., & Ginsburg, S. (2019). Assessment, feedback and the alchemy of learning. Medical Education, 53(1), 76–85. https://doi.org/10.1111/medu.13645
Webster-Wright, A. (2009). Reframing professional development through understanding authentic professional learning. Review of Educational Research, 79(2), 702–739. https://doi.org/10.3102/0034654308330970
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Roberts, C., Khanna, P., Lane, A.S. et al. Exploring complexities in the reform of assessment practice: a critical realist perspective. Adv in Health Sci Educ 26, 1641–1657 (2021). https://doi.org/10.1007/s10459-021-10065-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-021-10065-8