Skip to main content

Towards a program of assessment for health professionals: from training into practice

Abstract

Despite multifaceted attempts to “protect the public,” including the implementation of various assessment practices designed to identify individuals at all stages of training and practice who underperform, profound deficiencies in quality and safety continue to plague the healthcare system. The purpose of this reflections paper is to cast a critical lens on current assessment practices and to offer insights into ways in which they might be adapted to ensure alignment with modern conceptions of health professional education for the ultimate goal of improved healthcare. Three dominant themes will be addressed: (1) The need to redress unintended consequences of competency-based assessment; (2) The potential to design assessment systems that facilitate performance improvement; and (3) The importance of ensuring authentic linkage between assessment and practice. Several principles cut across each of these themes and represent the foundational goals we would put forward as signposts for decision making about the continued evolution of assessment practices in the health professions: (1) Increasing opportunities to promote learning rather than simply measuring performance; (2) Enabling integration across stages of training and practice; and (3) Reinforcing point-in-time assessments with continuous professional development in a way that enhances shared responsibility and accountability between practitioners, educational programs, and testing organizations. Many of the ideas generated represent suggestions for strategies to pilot test, for infrastructure to build, and for harmonization across groups to be enabled. These include novel strategies for OSCE station development, formative (diagnostic) assessment protocols tailored to shed light on the practices of individual clinicians, the use of continuous workplace-based assessment, and broadening the focus of high-stakes decision making beyond determining who passes and who fails. We conclude with reflections on systemic (i.e., cultural) barriers that may need to be overcome to move towards a more integrated, efficient, and effective system of assessment.

This is a preview of subscription content, access via your institution.

References

  1. ABA: American Board of Anesthesiology. (2014). MOCA minute. http://www.theaba.org/MOCA/MOCA-Minute. Last accessed November 2, 2015.

  2. AFMC: Association of Faculties of Medicine in Canada. (2010). The Future of Medical Education in Canada (FMEC): A collective vision for MD education. Retrieved from http://www.afmc.ca/fmec/pdf/collective_vision.pdf.

  3. Bernabeo, E., Hood, S., Iobst, W., Holmboe, E., & Caverzagie, K. (2013). Optimizing the implementation of practice improvement modules in training: Lessons from educators. Journal of Graduate Medical Education, 5(1), 74–80.

    Article  Google Scholar 

  4. Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. P. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

    Google Scholar 

  5. Bogo, M., Regehr, C., Logie, C., et al. (2011). Adapting objective structured clinical examinations to assess social work students’ performance and reflections. Journal of Social Work Education, 47, 5–18.

    Article  Google Scholar 

  6. Bordage, G., Meguerditchian, A. N., & Tamblyn, R. (2013). Avoidable adverse events: A content analysis of a national qualifying examination. Academic Medicine, 88, 1493–1498.

    Article  Google Scholar 

  7. Boud, D., & Molloy, E. (Eds.). (2013). Feedback in higher and professional education: Understanding it and doing it well. London: Routledge.

    Google Scholar 

  8. Butler, R. (1987). Task-involving and ego-involving properties of evaluation: Effects of different feedback conditions on motivational perceptions, interest, and performance. Journal of Educational Psychology, 79, 474–482.

    Article  Google Scholar 

  9. Cadieux, G., Tamblyn, R., Dauphinee, D., & Libman, M. (2007). Predictors of inappropriate antibiotic prescribing among primary care physicians. CMAJ, 177(8), 877–883.

    Article  Google Scholar 

  10. Choudhry, N. K., Fletcher, R. H., & Soumerai, S. B. (2005). Systematic review: The relationship between clinical experience and quality of health care. Annals of Internal Medicine, 142(4), 260–273.

    Article  Google Scholar 

  11. Cizek, G. J. (2012). Defining and distinguishing validity: Interpretations of score meaning and justification on test use. Psychological Methods, 17, 31–43.

    Article  Google Scholar 

  12. Colliver, J. A. (2002). Educational theory and medical education practice: A cautionary note for medical school faculty. Academic Medicine, 77(12), 1217–1220.

    Article  Google Scholar 

  13. Cook, D. A. (2014). When I say… validity. Medical Education, 48(10), 948–949.

    Article  Google Scholar 

  14. Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kane’s framework. Medical Education, 49, 560–575.

    Article  Google Scholar 

  15. Cote, L., & Bordage, G. (2012). Content and conceptual frameworks of preceptor feedback in response to residents’ educational needs. Academic Medicine, 87(9), 1274–1281.

    Article  Google Scholar 

  16. Cruess, R., & Cruess, S. (2014). Updating the Hippocratic Oath to include medicine’s social contract. Medical Education, 48(1), 95–100.

    Article  Google Scholar 

  17. Custers, E. (2010). Long-term retention of basic science knowledge: A review study. Advances in Health Sciences Education, 15(1), 109–128.

    Article  Google Scholar 

  18. Downing, S. M. (2003). Validity: On the meaningful interpretation of assessment data. Medical Education, 37, 830–837.

    Article  Google Scholar 

  19. Ellaway, R. H., Pusic, M. V., Galbraith, R. M., & Cameron, T. (2014). Developing the role of big data and analytics in health professional education. Medical Teacher, 36(3), 216–222.

    Article  Google Scholar 

  20. Ericsson, K. A. (2004). Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Academic Medicine, 79, S70–S81.

    Article  Google Scholar 

  21. Eva, K. W. (2002). The aging physician: Changes in cognitive processing and their impact on medical practice. Academic Medicine, 77, S1–S6.

    Article  Google Scholar 

  22. Eva, K. W. (2003). On the generality of specificity. Medical Education, 37, 587–588.

    Article  Google Scholar 

  23. Eva, K. W. (2009). Diagnostic error in medical education: Where wrongs can make rights. Advances in Health Sciences Education, 14, 71–81.

    Article  Google Scholar 

  24. Eva, K. W., Bordage, G., Campbell, C., Galbraith, R., Ginsburg, S., Holmboe, E., & Regehr, G. (2013). Medical Education Assessment Advisory Committee report to the Medical Council of Canada on Current Issues in Health Professional and Health Professional Trainee Assessment. Retrieved from http://mcc.ca/wp-content/uploads/Reports-MEAAC.pdf.

  25. Eva, K. W., & Cunnington, J. P. (2006). The difficulty with experience: Does practice increase susceptibility to premature closure? Journal of Continuing Education in the Health Professions, 26(3), 192–198.

    Article  Google Scholar 

  26. Eva, K. W., & Hodges, B. D. (2012). Scylla or Charbydis? Can we navigate between objectification and judgment in assessment? Medical Education, 46, 914–919.

    Article  Google Scholar 

  27. Eva, K. W., Munoz, J., Hanson, M. D., Walsh, A., & Wakefield, J. (2010). Which factors, personal or external, most influence students’ generation of learning goals? Academic Medicine, 85, S102–S105.

    Article  Google Scholar 

  28. Eva, K. W., & Regehr, G. (2013). Effective feedback for maintenance of competence: From data delivery to trusting dialogues. CMAJ, 185, 463–464.

    Article  Google Scholar 

  29. Eva, K. W., Regehr, G., & Gruppen, L. D. (2012). Blinded by ‘insight’: Self-assessment and its role in performance improvement. In B. D. Hodges & L. Lingard (Eds.), The question of competence: Reconsidering medical education in the twenty-first century (pp. 131–154). Ithaca, NY: Cornell University Press.

    Google Scholar 

  30. Farmer, E. A., & Page, G. (2005). A practical guide to assessing clinical decision-making skills using the key features approach. Medical Education, 39, 1188–1194.

    Article  Google Scholar 

  31. Frank, J. R., Snell, L. S., Cate, O. T., Holmboe, E. S., Carraccio, C., Swing, S. R., et al. (2010). Competency-based medical education: theory to practice. Medical Teacher, 32(8), 638–645.

    Article  Google Scholar 

  32. Galbraith, R. M., Clyman, S., & Melnick, D. E. (2011). Conceptual perspectives: Emerging changes in the assessment paradigm. In J. P. Hafler (Ed.), Extraordinary learning in the workplace (pp. 87–100). Berlin: Springer.

    Chapter  Google Scholar 

  33. Galbraith, R. M., Hawkins, R. E., & Holmboe, E. S. (2008). Making self-assessment more effective. Journal of Continuing Education in the Health Professions, 28(1), 20–24.

    Article  Google Scholar 

  34. Gierl, M. J., & Lai, H. (2013). Evaluating the quality of medical multiple-choice items created with automated processes. Medical Education, 47(7), 726–733.

    Article  Google Scholar 

  35. Gierl, M. J., Lai, H., & Turner, S. R. (2012). Using automatic item generation to create multiple-choice test items. Medical Education, 46(8), 757–765.

    Article  Google Scholar 

  36. Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2014). Seeing the ‘black box’ differently: Assessor cognition from three research perspectives. Medical Education, 48(11), 1055–1068.

    Article  Google Scholar 

  37. Ginsburg, S., Eva, K., & Regehr, G. (2013). Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Academic Medicine, 88(10), 1539–1544.

    Article  Google Scholar 

  38. Ginsburg, S., McIlroy, J., Oulanova, O., Eva, K., & Regehr, G. (2010). Toward authentic clinical evaluation: Pitfalls in the pursuit of competency. Academic Medicine, 85(5), 780–786.

    Article  Google Scholar 

  39. Ginsburg, S., Regehr, G., & Lingard, L. (2004). Basing the evaluation of professionalism on observable behaviours: A cautionary tale. Academic Medicine, 79(10, Suppl), S1–S4.

    Article  Google Scholar 

  40. Goldszmidt, M., Minda, J. P., & Bordage, G. (2013). What physicians reason about during clinical encounters: Time to be more explicit. Academic Medicine, 88(3), 390–394.

    Article  Google Scholar 

  41. Guadagnoli, M., Morin, M. P., & Dubrowski, A. (2012). The application of the challenge point framework in medical education. Medical Education, 46(5), 447–453.

    Article  Google Scholar 

  42. Harrison, C. J., Könings, K. D., Schuwirth, L., Wass, V., & van der Vleuten, C. (2015). Barriers to the uptake and use of feedback in the context of summative assessment. Advances in Health Sciences Education, 20(1), 229–245.

    Article  Google Scholar 

  43. Hatala, R., Marr, S., Cuncic, C., & Bacchus, C. M. (2011). Modification of an OSCE format to enhance patient continuity in a high-stakes assessment of clinical performance. BMC Medical Education, 11, 23.

    Article  Google Scholar 

  44. Hawkins, et al. (under review). The ABMS MOC Part III examination: Value, concerns and alternative formats.

  45. Hays, R., & Gay, S. (2011). Reflection or ‘pre-reflection’: What are we actually measuring in reflective practice? Medical Education, 45(2), 116–118.

    Article  Google Scholar 

  46. Hodges, B. (2003). OSCE! variations on a theme by Harden. Medical Education, 37(12), 1134–1140.

    Article  Google Scholar 

  47. Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682.

    Article  Google Scholar 

  48. James, J. T. (2013). A new, evidence-based estimate of patient harms associated with hospital care. Journal of Patient Safety, 9(3), 122–128.

    Article  Google Scholar 

  49. Jarvis-Selinger, S., Pratt, D. D., & Regehr, G. (2012). Competency is not enough: integrating identity formation into the medical education discourse. Academic Medicine, 87(9), 1185–1190.

    Article  Google Scholar 

  50. Kane, M. T. (1992). An argument-based approach to validation. Psychological Bulletin, 112, 527–535.

    Article  Google Scholar 

  51. Karpicke, J. D., & Roediger, H. L, I. I. I. (2008). The critical importance of retrieval for learning. Science, 319, 966–968.

    Article  Google Scholar 

  52. Kennedy, T. J., Regehr, G., Baker, G. R., & Lingard, L. A. (2009). ‘It’s a cultural expectation…’ The pressure on medical trainees to work independently in clinical practice. Medical Education, 43(7), 645–653.

    Article  Google Scholar 

  53. Klass, D. A. (2007). Performance-based conception of competence is changing the regulation of physicians’ professional behavior. Academic Medicine, 82(6), 529–535.

    Article  Google Scholar 

  54. Kluger, A. N., & van Dijk, D. (2010). Feedback, the various tasks of the doctor, and the feedforward alternative. Medical Education, 44, 1166–1174.

    Article  Google Scholar 

  55. Kogan, J. R., Conforti, L., Bernabeo, E., Iobst, W., & Holmboe, E. S. (2011). Opening the black box of postgraduate trainee assessment in the clinical setting via observation: A conceptual model. Medical Education, 45, 1048–1060.

    Article  Google Scholar 

  56. Kogan, J. R., & Holmboe, E. (2013). Realizing the promise and importance of performance-based assessment. Teaching and Learning in Medicine, 25(Suppl 1), S68–S74.

    Article  Google Scholar 

  57. Kogan, J. R., Holmboe, E. S., & Hauer, K. R. (2009). Tools for direct observation and assessment of clinical skills of medical trainees: A systematic review. JAMA, 302, 1316–1326.

    Article  Google Scholar 

  58. Kohn, L. T., Corrigan, J. M., & Donaldson, M. S. (Eds.). (1999). To err is human: building a safer health system. Washington, DC: National Academy Press, Institute of Medicine.

    Google Scholar 

  59. Kornell, N., & Son, L. K. (2009). Learners’ choices and beliefs about self-testing. Memory, 17, 493–501.

    Article  Google Scholar 

  60. Kromann, C. B., Bohnstedt, C., Jensen, M. L., & Ringsted, C. (2010). The testing effect on skills learning might last 6 months. Advances in Health Sciences Education, 15(3), 395–401.

    Article  Google Scholar 

  61. Krumholz, et al. (under review). Recommendations to the American Board of Internal Medicine (ABIM): A vision for certification in internal medicine in 2020.

  62. Larsen, D. P., Butler, A. C., & Roediger, H. L, 3rd. (2008). Test-enhanced learning in medical education. Medical Education, 42(10), 959–966.

    Article  Google Scholar 

  63. MacRae, H. M., Cohen, R., Regehr, G., Reznick, R., & Burnstein, M. (1997). A new assessment tool: the patient assessment and management examination. Surgery, 122(2), 335–343.

    Article  Google Scholar 

  64. Mann, K., Gordon, J., & MacLeod, A. (2009). Reflection and reflective practice in health professions education: A systematic review. Advances in Health Sciences Education, 14(4), 595–621.

    Article  Google Scholar 

  65. Mann, K. V., van der Vleuten, C., Eva, K., Armson, H., Chesluk, B., Dornan, T., et al. (2011). Tensions in informed self-assessment: How the desire for feedback and reticence to collect and use it conflict. Academic Medicine, 86, 1120–1127.

    Article  Google Scholar 

  66. Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52, 1187–1197.

    Article  Google Scholar 

  67. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–104). New York: American Council on Education and Macmillan.

    Google Scholar 

  68. Morcke, A. M., Dornan, T., & Elka, B. (2013). Outcome (competency) based education: an exploration of its origins, theoretical basis and empirical evidence. Advances in Health Sciences Education, 18, 851–863.

    Article  Google Scholar 

  69. Mutabdzic, D., Mylopoulos, M., Murnaghan, M. L., Patel, P., Zilbert, N., Seemann, N., et al. (2015). Coaching surgeons: Is culture limiting our ability to improve? Annals of Surgery, 262(2), 213–216.

    Article  Google Scholar 

  70. Mylopoulos, M., & Regehr, G. (2011). Putting the expert together again. Medical Education, 45(9), 920–926.

    Article  Google Scholar 

  71. Mylopoulos, M., & Scardamalia, M. (2008). Doctors’ perspectives on their innovations in daily practice: implications for knowledge building in health care. Medical Education, 42(10), 975–981.

    Article  Google Scholar 

  72. Neve, H., & Hanks, S. (2016). When I say … capability. Medical Education, 50 (in press).

  73. Newble, D. I., & Jaeger, K. (1983). The effect of assessments and examinations on the learning of medical students. Medical Education, 17(3), 165–171.

    Article  Google Scholar 

  74. Newell, K. M., Liu, Y., & Mayer-Kress, G. (2001). Time scales in motor learning and development. Psychological Review, 108, 57–82.

    Article  Google Scholar 

  75. Norcini, J. J. (2005). Current perspectives in assessment: The assessment of performance at work. Medical Education, 39(9), 880–889.

    Article  Google Scholar 

  76. Norcini, J., Anderson, B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., et al. (2011). Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 conference. Medical Teacher, 33(3), 206–214.

    Article  Google Scholar 

  77. Norcini, J. J., Blank, L. L., Duffy, F. D., & Fortna, G. S. (2003). The mini-CEX: A method for assessing clinical skills. Annals of Internal Medicine, 138(6), 476–481.

    Article  Google Scholar 

  78. Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical Teacher, 29(9), 855–871.

    Article  Google Scholar 

  79. Norman, G., Dore, K., & Grierson, L. (2012). The minimal relationship between simulation fidelity and transfer of learning. Medical Education, 46(7), 636–647.

    Article  Google Scholar 

  80. Norman, G., Neville, A., Blake, J. M., & Mueller, B. (2010). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Medical Teacher, 32(6), 496–499.

    Article  Google Scholar 

  81. Norman, G. R., Norcini, J., & Bordage, G. (2014). Competency-based education: Milestones or millstones. Journal of Graduate Medical Education, 6(1), 1–6.

    Article  Google Scholar 

  82. Page, G., & Bordage, G. (1995). The medical council of Canada’s key feature project: A more valid written exam. of clinical decision-making skills. Academic Medicine, 70, 104–110.

    Article  Google Scholar 

  83. Pugh, D., Hamstra, S. J., Wood, T. J., Humphrey-Murto, S., Touchie, C., Yudkowsky, R., Bordage, G. (2014). A procedural skills OSCE: Assessing technical and non-technical skills of internal medicine residents. Advances in health sciences education. Retrieved from http://link.springer.com/article/10.1007/s10459-014-9512-x?sa_campaign=email/event/articleAuthor/onlineFirst.

  84. Razack, S., Hodges, B., Steinert, Y., & Maguire, M. (2015). Seeking inclusion in an exclusive process: Discourses of medical school student selection. Medical Education, 49, 36–47.

    Article  Google Scholar 

  85. RCPSC: Royal College of Physicians and Surgeons of Canada. (2011). Assessment strategies within the revised maintenance of certification program, draft recommendations.

  86. Regehr, G. (1994). Chickens and children do not an expert make. Academic Medicine, 69, 970–971.

    Article  Google Scholar 

  87. Regehr, G., Eva, K., Ginsburg, S., Halwani, Y., & Sidhu, R. (2011). Future of medical education in Canada postgraduate project environmental scan. Paper 13. Assessment in postgraduate medical education: Trends and issues in assessment in the workplace. Retrieved from http://www.afmc.ca/pdf/fmec/13_Regehr_Assessment.pdf.

  88. Rohrer, D., & Pashler, H. (2010). Recent research on human learning challenges conventional instructional strategies. Educational Research, 38, 406–412.

    Article  Google Scholar 

  89. Sargeant, J., Eva, K. W., Armson, H., Chesluk, B., Dornan, T., Holmboe, E., et al. (2011). Features of assessment learners use to make informed self-assessments of clinical performance. Medical Education, 45, 636–647.

    Article  Google Scholar 

  90. Schön, D. (1983). The reflective practitioner: How professionals think in action. London: Temple Smith.

    Google Scholar 

  91. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78, 153–189.

    Article  Google Scholar 

  92. Swanson, D., & Roberts, T. (2016). Trends in national licensing examinations. Medical Education, 50(1) (in press).

  93. Tamblyn, R., Abrahamowicz, M., Dauphinee, D., et al. (2007). Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA, 298(9), 993–1001.

    Article  Google Scholar 

  94. Teunissen, P. W., & Westerman, M. (2011). Opportunity or threat: The ambiguity of the consequences of transitions in medical education. Medical Education, 45(1), 51–59.

    Article  Google Scholar 

  95. van der Vleuten, C. (1996). The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education, 1, 41–67.

    Article  Google Scholar 

  96. van der Vleuten, C. P., & Schuwirth, L. W. (2005). Assessing professional competence: From methods to programmes. Medical Education, 39(3), 309–317.

    Article  Google Scholar 

  97. van Tartwijk, J., & Driessen, E. W. (2009). Portfolios for assessment and learning: AMEE Guide no. 45. Medical Teacher, 31(9), 790–801.

    Article  Google Scholar 

  98. Watling, C., Driessen, E., van der Vleuten, C. P., & Lingard, L. (2014). Learning culture and feedback: An international study of medical athletes and musicians. Medical Education, 48(7), 713–723.

    Article  Google Scholar 

  99. Wenghofer, E., Klass, D., Abrahamowicz, M., et al. (2009). Doctor scores on national qualifying examinations predict quality of care in future practice. Medical Education, 43(12), 1166–1173.

    Article  Google Scholar 

  100. Williams, R. G., Klamen, D. L., Markwell, S. J., Cianciolo, A. T., Colliver, J. A., & Verhulst, S. J. (2014). Variations in senior medical student diagnostic justification ability. Academic Medicine, 89(5), 790–798.

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the Medical Council of Canada (MCC) through the work of the authors as members of the Medical Education Assessment Advisory Committee. The focus was not constrained to MCC practices, however, and the content of the paper does not necessarily reflect MCC policy.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Kevin W. Eva.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Eva, K.W., Bordage, G., Campbell, C. et al. Towards a program of assessment for health professionals: from training into practice. Adv in Health Sci Educ 21, 897–913 (2016). https://doi.org/10.1007/s10459-015-9653-6

Download citation

Keywords

  • Assessment
  • Health professional education
  • Competency-based education
  • Continuing professional development