Skip to main content
SpringerLink
Log in

Considerations that will determine if competency-based assessment is a sustainable innovation

  • Reflections
  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Educational assessment for the health professions has seen a major attempt to introduce competency based frameworks. As high level policy developments, the changes were intended to improve outcomes by supporting learning and skills development. However, we argue that previous experiences with major innovations in assessment offer an important road map for developing and refining assessment innovations, including careful piloting and analyses of their measurement qualities and impacts. Based on the literature, numerous assessment workshops, personal interactions with potential users, and our 40 years of experience in implementing assessment change, we lament the lack of a coordinated approach to clarify and improve measurement qualities and functionality of competency based assessment (CBA). To address this worrisome situation, we offer two roadmaps to guide CBA’s further development. Initially, reframe and address CBA as a measurement development opportunity. Secondly, using a roadmap adapted from the management literature on sustainable innovation, the medical assessment community needs to initiate an integrated plan to implement CBA as a sustainable innovation within existing educational programs and self-regulatory enterprises. Further examples of down-stream opportunities to refocus CBA at the implementation level within faculties and within the regulatory framework of the profession are offered. In closing, we challenge the broader assessment community in medicine to step forward and own the challenge and opportunities to reframe CBA as an innovation to improve the quality of the clinical educational experience. The goal is to optimize assessment in health education and ultimately improve the public’s health.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • American Board of Emergency Medicine (ABEM). (2018). History. Retrieved March 25, 2018 from https://www.abme.org/public/general-information/history.

  • American Educational Research Association (AERA), the American Psychological Association (APA), and the National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing. Washington DC: AERA.

    Google Scholar 

  • Bismil, R., Dudek, N. L., & Wood, T. J. (2014). In-training evaluations: Developing an automated screening tool to measure report quality. Medical Education, 48, 724–732.

    Article  Google Scholar 

  • Boulet, J. R., Ben-David, M. F., Ziv, A., Burdick, W. P., Curtis, M., Peitzman, S., et al. (1998). Using standardized patients to assess the interpersonal skills of physicians. Academic Medicine, 73(10 Suppl), S94–S96.

    Article  Google Scholar 

  • Boulet, J., Smee, S., Dillon, G. F., & Gimpel, J. R. (2009). The use of standardized patients assessments for certification and licensure decisions. Simulation in Healthcare: Journal of the Society for Simulation in Healthcare, 4(1), 35–42.

    Article  Google Scholar 

  • Carriccio, C., Englander, R., Van Melle, E., ten Cate, O., Lochlear, J., Chan, M. K., et al. (2016). Advancing competency-based medical education. A charter for clinician educators. Academic Medicine, 91(5), 645–649.

    Article  Google Scholar 

  • Chen, F. M., Baucher, H., & Burstin, H. (2014). A call for outcomes research in medical education. Academic Medicine, 79(10), 955–960.

    Article  Google Scholar 

  • Christensen, C. M. (2000a). The innovator’s Dilemma. When new technologies cause great firms to fail (pp. vii–xv). New York: Harper Business.

    Google Scholar 

  • Christensen, C. M. (2000b). The Innovator’s Dilemma. When new technologies cause great firms to fail (pp. 225–229). New York: Harper Business.

    Google Scholar 

  • Christensen, C. M., Raynor, M. E., & Van Bever, D. (2013). Chapter 2: How can we beat our most powerful competitors? In C. Christensen & M. Raynor (Eds.), The innovator’s solution: Creating and sustaining successful growth (pp. 31–55). Boston: Harvard Business Review Press.

    Google Scholar 

  • Clauser, B. E., Margolis, M. J., & Swanson, D. B. (2018). Chapter 2: Issues of validity and reliability for assessments in medical education. In E. S. Holmboe, R. E. Hawkins, & S. Durning (Eds.), Practical guide to the evaluation of clinical competence (2nd ed., pp. 22–36). Philadelphia, PA: Elsevier Inc.

    Google Scholar 

  • Cook, D. A., Kuper, A., Hatala, R., & Ginsburg, S. (2016). When assessment data are words: Validity evidence for qualitative educational assessments. Academic Medicine, 91(10), 1359–1369.

    Article  Google Scholar 

  • Dath, D., & Iobst, W. (2010). The importance of faculty development in the transition to competency-based medical education. Medical Teacher, 32(8), 683–686.

    Article  Google Scholar 

  • Dauphinee, W. D. (2002). Licensure and certification. In G. Norman, C. Van der Vleuten, & D. Newble (Eds.), International handbook of research in medical education (pp. 835–882). Dordrecht: Kluwer Academic Publishers.

    Chapter  Google Scholar 

  • Dauphinee, W. D., & Reznick, R. K. (2011). A framework for designing, implementing, and sustaining a national simulation network. Simulation in healthcare. Journal of the Society for Simulation in Healthcare, 6(2), 94–100.

    Article  Google Scholar 

  • Dudek, N. L., Marks, M. B., Wood, T. J., Dojeiji, S., Bandiera, G., Hatala, R., et al. (2012). Quality evaluation reports—Can a faculty development program make a difference? Medical Teacher, 34(11), e725–e731.

    Article  Google Scholar 

  • Eva, K. W., Bordage, G., Campbell, C., Galbraith, R., Ginsburg, S., Holmboe, E., et al. (2016). Towards a program of assessment for health professionals: From training into practice. Advances in Health Professions Education, 21(4), 893–913.

    Google Scholar 

  • Govaerts, M. (2015). Work-based assessment and the assessment for learning: Threats to validity. Journal of Graduate Medical Education, 7(2), 265–267.

    Article  Google Scholar 

  • Govaerts, M., & van der Veuten, C. P. M. (2013). Validity in work-based assessment: Expanding our horizons. Medical Education, 47, 1164–1173.

    Article  Google Scholar 

  • Hauer, K. E., ten Cate, O., Boscardin, C. K., Iobst, W., Holmboe, E. S., Chesluk, B., et al. (2016). Ensuring resident competence; A narrative review of the literature on group decision making to inform the work of Clinical Competence Committees. Journal of Graduate Medical Education, 8(2), 156–164.

    Article  Google Scholar 

  • Hawkins, R. E., Welcher, C. M., Holmboe, E. S., Kirk, L. M., Norcini, J. J., Simona, K. B., et al. (2015). Implementation of competency-based medical education: Are we addressing the concerns and challenges? Medical Education, 49(11), 1086–1102.

    Article  Google Scholar 

  • Holmboe, E. S., ten Cate, O., Durning, S. J., & Hawkins, R. E. (2017). Chapter 1: Assessment challenges in the era of outcomes-based medical education. In E. S. Holmboe, R. E. Hawkins, & S. Durning (Eds.), Practical guide to the evaluation of clinical competence (pp. 1–19). Philadelphia PA: Elsevier Inc.

    Google Scholar 

  • Holmboe, E. S., Ward, D. S., Reznick, R. K., Katsufrakis, P. J., Leslie, K. M., Patel, V., et al. (2011). Faculty developments in assessment: The missing link in competency-based medical education. Academic Medicine, 86(4), 460–467.

    Article  Google Scholar 

  • Holmboe, E. S., Yamazaki, K., Edgar, L., Conforti, L., Yaghmour, N., Miller, R. S., et al. (2015). Reflections on the first 2 years of milestone implementation. Journal of Graduate Medical Education, 7(3), 506–511.

    Article  Google Scholar 

  • Iobst, W. F., & Holmboe, E. S. (2015). Building the continuum of competency-based medical education. Perspectives in Medical Education, 4(4), 165–167.

    Article  Google Scholar 

  • Kane, M. (1992). The assessment of professional competence. Evaluation in the Health Professions, 15(2), 163–182.

    Article  Google Scholar 

  • Klamen, D. L., Williams, R. G., Roberts, N., & Cianciolo, A. T. (2016). Competencies, milestones, and EPAs—Are those who ignore the past condemned to repeat it? Medical Teacher, 38(9), 1–7.

    Article  Google Scholar 

  • Lee, S. S. (1974). Health insurance in Canada—An overview and commentary. NEJM, 290(13), 713–716.

    Article  Google Scholar 

  • Lurie, S. J., Mooney, C. J., & Lyness, J. M. (2011). Commentary: Pitfalls in assessment of competency-based educational objectives. Academic Medicine, 86(4), 412–414.

    Article  Google Scholar 

  • Maatsch, J. L., Krome, R. L., Sprafka, S., & Maclean, C. B. (1976). The emergency medicine specialty certification examination (EMSCE). Annals of Emergency Medicine, 5(7), 529–535.

    Google Scholar 

  • Munger, B. S., Krome, R. L., Maatsch, J. C., & Podgorny, G. (1982). The certification examination in emergency medicine: An update. Annals of Emergency Medicine, 11(2), 91–96.

    Article  Google Scholar 

  • Norcini, J., & Burch, V. (2007). Workplace-based assessment as an educational tool: AMEE Guide No. 31. Medical Teacher, 29, 855–871.

    Article  Google Scholar 

  • Norman, G., Norcini, J., & Bordage, G. (2014). Competency-based education: Milestones or millstones? Journal of Graduate Medical Education, 6(1), 1–6.

    Article  Google Scholar 

  • Regehr, G., Eva, K., Ginsburg, S., Halwani, Y., & Sidhu, R. (2011). Assessment on postgraduate medical education: Trends and issues in assessment in the workplace. Members of the FMEC PC consortium. Retrieved March 27, 2018 from https://www.afmc.ca/pdf/fmec/_Regehr_Assessment.pdf.

  • Reznick, R. K., Blackmore, D., Cohen, R., Baumber, J. S., Rothman, A., Smee, S., et al. (1993). An objective structured clinical examination for the licentiate of the Medical Council of Canada. Academic Medicine, 68(10 suppl), S4–S6.

    Article  Google Scholar 

  • Solon, J. A., Sheps, C. A., & Lee, S. S. (1960). Delineating patterns of medical care. American Journal of Public Health, 50(8), 1105–1113.

    Article  Google Scholar 

  • Swanson, D. B., Clauser, B. E., & Case, S. M. (1999). Clinical skills assessment with standardized patients in high stakes tests: A framework. Advances in Health Sciences Education Theory and Practice, 4, 67–106.

    Article  Google Scholar 

  • Tamblyn, R. (1998). Use of standardized patients in the assessment of medical practice. CMAJ, 158(2), 205–207.

    Google Scholar 

  • Tamblyn, R., Abrahamowicz, M., Dauphinee, W. D., Hanley, J. A., Norcini, J., Girard, N., et al. (2002). Association between licensure examination scores and practice in primary care. JAMA, 288(23), 3019–3026.

    Article  Google Scholar 

  • Ten Cate, O. (2013). Nuts and bolts of entrustable professional activities. Journal of Graduate Medical Education, 5(1), 157–158.

    Article  Google Scholar 

  • Ten Cate, O., & Scheele, F. (2007). Viewpoint: Competency-based postgraduate training: Can we bridge the gap between theory and clinical practice? Academic Medicine, 82(6), 542–547.

    Article  Google Scholar 

  • Van der Vleuten, C. P. M., Schuwirth, L. W. T., Driessen, E. W., & Govaerts, M. J. B. (2015). Twelve tips for programmatic assessment. Medical Teacher, 37, 641–646.

    Article  Google Scholar 

  • Van der Vleuten, C. P. M., Schuwirth, L. W. T., Drissen, E. W., Djkstra, J., Tigelaar, D., Baartman, L. K., et al. (2012). A model for programmatic assessment fit for purpose. Medical Teacher, 35, 204–214.

    Google Scholar 

  • Van Loon, K. A., Teunissen, P. W., Driessen, E. W., & Scheele, F. (2016). The role of generic competencies in the entrustment of professional competency-based curriculum assessment. Journal of Graduate Medical Education, 8(4), 546–552.

    Article  Google Scholar 

  • Warm, E. J. (2016). Working backward from the patient. Journal of Graduate Medical Education, 8(1), 101–103.

    Article  Google Scholar 

  • Williams, R. G., Mellinger, J. G., Dunnington, G. L., & Klamen, D. L. (2015). Placing constraints on the use of ACGME milestones. A commentary on limitations of global performance ratings. Academic Medicine, 90(4), 404–407.

    Article  Google Scholar 

  • Wong, B. M., & Holmboe, E. S. (2016). Transforming the academic faculty perspective in graduate medical education to better align educational and clinical outcomes. Academic Medicine, 91(4), 473–479.

    Article  Google Scholar 

Download references

Acknowledgements

The authors recognize the informal concerns and insights expressed by international workshop attendees and offered by post-graduate offices and program directors in Canada (WDD) and in the USA (JN and JB).

Author information

Authors and Affiliations

  1. Foundation for the Advancement of International Medical Education and Research, 3624 Market Street, Fourth Floor, Philadelphia, PA, 19104, USA

    W. Dale Dauphinee, John R. Boulet & John J. Norcini

  2. McGill University, 1140 Pine Avenue West, Montreal, QC, H3A 1A3, Canada

    W. Dale Dauphinee

  3. Saint Andrews, NB, Canada

    W. Dale Dauphinee

Authors
  1. W. Dale Dauphinee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John R. Boulet
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John J. Norcini
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

WDD revised the current version based on the feedback from reviewers and editors. WDD added management frameworks. WDD, JB, and JN selected established assessment frameworks that were useful in their work as innovators. JB and JN edited the text and clarified the content for the final version.

Corresponding author

Correspondence to W. Dale Dauphinee.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethics approval

None. No primary data was generated and no candidate based research was conducted.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dauphinee, W.D., Boulet, J.R. & Norcini, J.J. Considerations that will determine if competency-based assessment is a sustainable innovation. Adv in Health Sci Educ 24, 413–421 (2019). https://doi.org/10.1007/s10459-018-9833-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-018-9833-2

Keywords

Search

Navigation