Skip to main content

Advertisement

SpringerLink
Log in

The key-features approach to assess clinical decisions: validity evidence to date

  • Review
  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

The key-features (KFs) approach to assessment was initially proposed during the First Cambridge Conference on Medical Education in 1984 as a more efficient and effective means of assessing clinical decision-making skills. Over three decades later, we conducted a comprehensive, systematic review of the validity evidence gathered since then. The evidence was compiled according to the Standards for Educational and Psychological Testing’s five sources of validity evidence, namely, Content, Response process, Internal structure, Relations to other variables, and Consequences, to which we added two other types related to Cost-feasibility and Acceptability. Of the 457 publications that referred to the KFs approach between 1984 and October 2017, 164 are cited here; the remaining 293 were either redundant or the authors simply mentioned the KFs concept in relation to their work. While one set of articles reported meeting the validity standards, another set examined KFs test development choices and score interpretation. The accumulated validity evidence for the KFs approach since its inception supports the decision-making construct measured and its use to assess clinical decision-making skills at all levels of training and practice and with various types of exam formats. Recognizing that gathering validity evidence is an ongoing process, areas with limited evidence, such as item factor analyses or consequences of testing, are identified as well as new topics needing further clarification, such as the use of the KFs approach for formative assessment and its place within a program of assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ali, K., & Bordage, G. (1995). Validity of key features for a family medicine pilot exam at the college of physicians and surgeons Pakistan. Journal of the College of Physicians and Surgeons of Pakistan, 5, 256–260.

    Google Scholar 

  • Allen, T. (2005). A comparison of the performance of an oral certification examination of clinical reasoning skills in emergency medicine with the performance of similar North American examinations. Master’s thesis, Université Laval.

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing (pp. 11–21). Washington: American Education Research Association.

    Google Scholar 

  • Amini, M., Kojuri, J., Karimian Lofti, F., Moghadami, M., Dehghani, M. R., Azarpyra, N., et al. (2011a). Talents for Future: Report of the Second National Medical Science Olympiad in Islamic Republic of Iran. Iranian Red Crescent Medical Journal, 13, 377–381.

    Google Scholar 

  • Amini, M., Moghadami, M., Kojuri, J., Abbasi, H., et al. (2011b). An innovative method to assess clinical reasoning skills: Clinical reasoning tests in the second national medical science Olympiad in Iran. BMC Research Notes, 4, 418–425.

    Google Scholar 

  • Ashoorion, V., Liaghatdar, M. J., & Adibi, P. (2012). What variables can influence clinical reasoning? Journal of Research in Medical Sciences, 17, 1170–1175.

    Google Scholar 

  • Bateman, J., Allen, M. E., Samani, D., Kidd, J., & Davis, D. (2013). Virtual patients design: exploring what works and why. A grounded theory study. Medical Education, 47, 595–606.

    Google Scholar 

  • Benedict, N., Smithburger, P., Donihi, A. C., Empey, P., Kobulinsky, L., Seybert, A., et al. (2017). Blended simulation progress testing for assessment of practice readiness. American Journal of Pharmaceutical Education, 81, 1–13.

    Google Scholar 

  • Bernabeo, E. C., Holmboe, E., Ross, K., Chesluk, B., & Ginsburg, S. (2013). The utility of vignettes to stimulate reflection on professionalism: Theory and practice. Advances in Health Sciences Education, 18, 463–484.

    Google Scholar 

  • Bloch, R. D., & Burgi, H. (2002). The Swiss catalogue of learning objectives. Medical Teacher, 24, 144–150.

    Google Scholar 

  • Bordage, G., Brailovsky, C., Carretier, H., & Page, G. (1995a). Content validation of key features on a national examination of clinical decision-making skills. Academic Medicine, 70, 276–281.

    Google Scholar 

  • Bordage, G., Brailovsky, C. A., Cohen, T., & Page, G. (1996). Maintaining and enhancing key decision-making skills from graduation into practice: an exploratory study. In A. J. J. A. Scherpbier, C. P. M. van der Vleuten, & J. J. Rethans (Eds.), Advances in Medical Education (pp. 128–130). Dordrecht: Kluwer Academic.

    Google Scholar 

  • Bordage, G., Carretier, H., Bertrand, R., & Page, G. (1995b). Comparing times and performance of french- and english-speaking candidates taking a national examination of clinical decision-making skills. Academic Medicine, 70, 359–365.

    Google Scholar 

  • Bordage, G., Meguerditchian, A., & Tamblyn, R. (2013). Practice indicators of suboptimal care and avoidable adverse events: A content analysis of a national qualifying examination. Academic Medicine, 88, 1493–1498.

    Google Scholar 

  • Bordage, G., & Page, G. (1987). An alternative to PMPs: The “key feature concept”. In I. R. Hart & R. Harden (Eds.), Further developments in assessing clinical competence (pp. 59–75). Ottawa: Can-Heal Publications.

    Google Scholar 

  • Bosner, S., Pickert, J., & Stibane, T. (2015). Teaching differential diagnosis in primary care using an inverted classroom approach: student satisfaction and gain in skills and knowledge. BMC Medical Education, 15, 1–7.

    Google Scholar 

  • Brailovsky, C., Bordage, G., & Page, G. (1998). Components of variance on a Key-Feature (Q4) Paper of the medical council of canada’s exam. In Proceedings of the 8th Ottawa Conference on Medical Education (pp. 169–175). Philadelphia.

  • Breckwoldt, J., Ludwig, J. R., Plener, J., Schröder, T., Gruber, H., & Peters, H. (2016). Differences in procedural knowledge after a “spaced” and a “massed” version of an intensive course in emergency medicine, investigating a very short spacing interval. BMC Medical Education, 16, 249.

    Google Scholar 

  • Bronander, K. A., Lang, V., Nixon, J., Harrell, H. E., Kovach, R., Hingle, S., et al. (2015). How we developed and piloted an electronic key features examination for the internal medicine clerkship Based on a US National Curriculum. Medical Teacher, 37, 807–812.

    Google Scholar 

  • Buisson, S., & Lévesque-Cardinal, S. (2000). Nouvel examen professionnel de l’O.I.I.Q.—volet pratique : une ttawace préparatoire au ttawa de l’Outaouais. Pédagogie collégiale, 14, 41–42.

    Google Scholar 

  • Burrows, P., Khan, A., Trafford, P., & Whiteman, J. (2012). The induction and refresher scheme simulated surgery. Education in Primary Care, 23, 335–341.

    Google Scholar 

  • Carr, M. M., Hewitt, J., Scardamalia, M., & Reznick, R. K. (2002). Internet-based otolaryngology case discussions for medical students. Journal of Otolaryngology, 31, 197–201.

    Google Scholar 

  • Carriere, B., Gagnon, R., Charlin, B., Downing, S., & Bordage, G. (2009). Assessing clinical reasoning in pediatric emergency medicine: Validity evidence for a script concordance test. Annals of Emergency Medicine, 53, 647–652.

    Google Scholar 

  • Childs, R. A., Dunn, J. L., van Barneveld, C., & Jaciw, A. P. (2007). Does It Matter if You “Kill” the Patient or Order Too Many Tests? Scoring Alternatives for a Test of Clinical Reasoning Skill. International Journal of Testing, 7, 127–139.

    Google Scholar 

  • Childs, R. A., Dunn, J. L., van Barneveld, C., Jaciw, A. P., & McIlroy, J. H. (2003). Differential weighting of errors on a test of clinical reasoning skills. Academic Medicine, 78, S62–S64.

    Google Scholar 

  • Cilliers, F. J., Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2012). Modelling the pre-assessment learning effects of assessment: Evidence in the validity chain. Medical Education, 46, 1087–1098.

    Google Scholar 

  • Clark, M. (2010). Continuing competence: Overcoming our blind spots. Instep newsletter. Victoria: College of Occupational Therapists of British Columbia.

    Google Scholar 

  • Consorti, F., Della Rocca, C., Familiari, G., Gallo, P., Riggio, O., Sperandeo, F., et al. (2015). Verso una Laurea professionalizzante. Certificazione delle Competenze professionali. Medicina e Chirurgia., 65, 2931–2941.

    Google Scholar 

  • Daniels, V. J., Bordage, G., Gierl, M. J., & Yudkowsky, R. (2014). Effect of clinically discriminating, evidence-based checklist items on the reliability of scores from an Internal Medicine residency OSCE. Advances in Health Sciences Education, 19, 497–506.

    Google Scholar 

  • Dauphinee W.D., Boulais A.P., Smee S.M., Rothman A.I., Reznick R., & Blackmore D. (1998). Examination Results of the Licentiate of the Medical Council of Canada: Trends, Issues, and Future Considerations. In: Proceedings of the 8th Ottawa Conference on Medical Education (pp. 92–98). Philadelphia.

  • De Champlain, A. F. (2015). Best-fit model of exploratory and confirmatory factor analysis of the 2010 Medical Council of Canada Qualifying Examination Part I clinical decision-making cases. Journal of Educational Evaluation for Health Professions., 12, 11. https://doi.org/10.3352/jeehp.2015.12.11.

    Article  Google Scholar 

  • De Champlain, A. F., Melnick, D., Scoles, P., et al. (2003). Assessing medical students’ clinical sciences knowledge in France: A collaboration between the NBME and a consortium of french medical schools. Academic Medicine, 78, 509–517.

    Google Scholar 

  • de la Calzada, C. S. (2015). A framework for clinical reasoning in adult cardiology. Advances in Medical Education Practice, 6, 489–495.

    Google Scholar 

  • Desjardins, I., Touchie, C., Pugh, D., Wood, T. J., & Humphrey-Murto, S. (2014). The impact of cueing on written examinations of clinical decision making: a case study. Medical Education, 48, 255–261.

    Google Scholar 

  • Dory, V., Gagnon, R., & Charlin, B. (2010). Is case-specificity content-specificity? An analysis of data from extended-matching questions. Advances in Health Sciences Education, 15, 55–63.

    Google Scholar 

  • Doucet, M. D., Purdy, R. A., Kaufman, D. M., & Langille, D. B. (1998). Comparison of problem-based learning and lecture format in continuing medical education on headache diagnosis and management. Medical Education, 32, 590–596.

    Google Scholar 

  • Downing, S. M. (2004). Reliability: On the reproducibility of assessment data. Medical Education, 38, 1006–1012.

    Google Scholar 

  • Downing, S. M. (2009). What is good item discrimination? In S. M. Downing & R. Yudkowsky (Eds.), Assessment in Health Professions Education (p. 108). New York: Routledge.

    Google Scholar 

  • Downing, S. M., & Haladyna, T. M. (2009). Validity and its threats. In S. M. Downing & R. Yudkowsky (Eds.), Assessment in Health Professions Education (p. 33). New York: Routledge.

    Google Scholar 

  • Downing, S. M., & Yudkowsky, R. (Eds.). (2009). Assessment in health professions education. New York: Routledge.

    Google Scholar 

  • Duncan-Hewitt, W., Jungnickel, P., & Evans, R. L. (2007). Development of an office of teaching, learning, and assessment in a pharmacy school. American Journal of Pharmaceutical Education, 71(35), 1–8.

    Google Scholar 

  • Elstein, A. S., Shulman, L. S., & Sprafka, S. A. (1978). Medical problem solving. Cambridge: Harvard University Press.

    Google Scholar 

  • Eva, K., Bordage, G., Campbell, C., Galbraith, R., Ginsburg, S., Holmboe, E., et al. (2016). Towards a program of assessment for health professionals: From training into practice. Advances in Health Sciences Education, 21, 897–913.

    Google Scholar 

  • Eva, K. W., & Regehr, G. (2013). Effective feedback for maintenance of competence: From data delivery to trusting dialogues. Canadian Medical Association Journal, 185, 463–464.

    Google Scholar 

  • Eva, K., & Wood, T. (2003). Can the strength of candidates be discriminated based on ability to circumvent the biasing effect of prose? Implications for evaluation and education. Academic Medicine, 78, S78–S81.

    Google Scholar 

  • Eva, K. W., Wood, T., Riddle, J., Touchie, C., & Bordage, G. (2010). How clinical features are presented matters to weaker diagnosticians. Medical Education, 44, 775–785.

    Google Scholar 

  • Farmer, E. A., & Hinchy, J. (2005). Assessing general practice decision-making skills—The Key features approach. Australian Family Physician, 34, 1059–1061.

    Google Scholar 

  • Farmer, E. A., & Page, G. A. (2005). Practical guide to assessing clinical decision-making skills using the key features approach. Medical Education, 39, 1188–1194.

    Google Scholar 

  • Fischer, M. R., Kopp, V., Holzer, M., Ruderich, F., & Junger, J. (2005). A modified electronic key feature examination for undergraduate medical students: validation threats and opportunities. Medical Teacher, 27, 450–455.

    Google Scholar 

  • Flinn, F., & Armstrong, C. (2011). Junior doctors’ extended work hours and the effects on performance: The Irish Case. International Journal of Quality Health Care, 2011(23), 210–217.

    Google Scholar 

  • Fowell, S. L., & Jolly, B. (2000). Combining marks, scores and grades. Reviewing common practices reveals some bad habits. Medical Education, 34, 785–786.

    Google Scholar 

  • Fuller, R., Homer, M., Pell, G., & Hallam, J. (2017). Managing extremes of assessor judgment within the OSCE. Medical Teacher, 39, 58–66.

    Google Scholar 

  • Gauthier, G., & Lajoie, S. P. (2014). Do expert clinical teachers have a shared understanding of what constitute a competent reasoning performance in case-based teaching? Instructional Science, 42, 579–594.

    Google Scholar 

  • Geisler, P., Hummel, C., & Piebes, S. (2014). Evaluating evidence-informed clinical reasoning proficiency in oral practical examinations. Athletic Training Education Journal, 9, 43–48.

    Google Scholar 

  • Gerhard-Szep, S., Guentsch, A., Pospiech, P., et al. (2016). Assessment formats in dental medicine: An overview. GMS Journal for Medical Education, 33, 1–43.

    Google Scholar 

  • Gierl, M. J., Latifi, S., Lai, H., Boulais, A.-P., & DeChamplain, A. (2014). Automated essay scoring and the future of educational assessment in medical education. Medical Education, 48, 950–962.

    Google Scholar 

  • Goldszmidt, M., Minda, J. P., & Bordage, G. (2013). Developing a unified list of physicians’ reasoning tasks during Clinical Encounters: Time to be more explicit. Academic Medicine, 88, 390–394.

    Google Scholar 

  • Goulet, F., Jacques, A., Gagnon, R., Charlin, B., & Shabah, A. (2010). Poorly performing physicians: does the script concordance test detect bad clinical reasoning? Journal of Continuing Education in the Health Professions, 30, 161–166.

    Google Scholar 

  • Haladyna, T. M. (2004). Chapter 7. Item generation: Key features—Steps in developing key features problems. In T. M. Haladyna (Ed.), Developing and validating multiple-choice test items (pp. 165–170). Mahwah: Lawrence-Erlbaum Associates.

    Google Scholar 

  • Hatala, R., & Norman, G. R. (2002). Adapting the key features examination for a clinical clerkship. Medical Education, 36, 160–165.

    Google Scholar 

  • Heid, J., Bauch, M., Brass, K., Hess, F., Junger, J., Haag, M., et al. (2006). Development and usage of a secure assessment software system for the medical education. GMS Medizinische Informatik, Biometrie und Epidemiologie, 10, 1–6.

    Google Scholar 

  • Heist, B. S., Kishida, N., Deshpande, G., Hamaguchi, S., & Kobayashi, H. (2016). Virtual patients to explore and develop clinical case summary statement skills amongst Japanese resident physicians: A mixed methods study. BMC Medical Education, 16, 39–46.

    Google Scholar 

  • Heitzmann, N., Fischer, F., Kühne-Eversmann, L., & Fischer, M. R. (2015). Enhancing diagnostic competence with self-explanation prompts and adaptable feedback. Medical Education, 49, 993–1003.

    Google Scholar 

  • Hrynchak, P., Takahashi, S. G., & Nayer, N. (2014). Key-feature questions for assessment of clinical reasoning: A literature review. Medical Education, 48, 870–883.

    Google Scholar 

  • Hurtz, G. M., Chinn, R. N., Barnhill, G. C., & Hertz, N. R. (2012). Measuring clinical decision making: Do key features problems measure higher level cognitive processes? Evaluation in the Health Professions, 35, 396–415.

    Google Scholar 

  • Huwendiek, S., Reichert, F., Duncker, C., de Leng, B. A., van der Vleuten, C. P. M., Muijtjens, A. M. M., et al. (2017). Electronic Assessment of Clinical Reasoning in Clerkships: A Mixed-Methods Comparison of Long-menu Key-Feature Problems with Context-Rich Single Best Answer Questions. Medical Teacher, 39, 476–485.

    Google Scholar 

  • Jacques, A. (2006). Maintaining competence: A professional challenge. Bulletin of the Kuwait Institute for Medical Specializations, 5, 74–79.

    Google Scholar 

  • Jacques, A., Sindon, A., Bourque, A., Bordage, G., & Ferland, J. J. (1995). Structured oral interview One way to identify family physicians’ educational needs. Canadian Family Physician, 41, 1346–1352.

    Google Scholar 

  • Jost, M., Brüstle, P., Giesler, M., Rijntjes, M., & Brich, J. (2017). Effects of additional team–based learning on students’ clinical reasoning skills: a pilot study. BMC Research Notes, 10, 282. https://doi.org/10.1186/s13104-017-2614-9.

    Article  Google Scholar 

  • Kawasumi, Y. (2009) The association between physician competence at licensure and the quality of asthma management and patient morbidity. Master’s thesis. McGill University. Montreal.

  • Keegan, D. A., Scott, I., Sylvester, M., Tan, A., Horrey, K., & Weston, W. (2017). Shared Canadian Curriculum in Family Medicine (SHARC-FM). Creating a national consensus on relevant and practical training for medical students. Canadian Family Physician, 63, e223–e231.

    Google Scholar 

  • Kopp, V., Moltner, A., & Fisher, M. R. (2006). Key feature problems for the assessment of procedural knowledge: a practical guide. GMS Zeitschrift für Medizinische Ausbildung, 23, 1–6.

    Google Scholar 

  • Kopp, V., Stark, R., & Fischer, M. R. (2008). Fostering diagnostic knowledge through computer supported, case-based worked examples: effects of erroneous examples and feedback. Medical Education, 42, 823–829.

    Google Scholar 

  • Kopp, V., Stark, R., Heitzmann, N., & Fischer, M. R. (2009a). Self-regulated learning with case-based worked examples: effects of errors. Evaluation and Research in Education, 22, 107–119.

    Google Scholar 

  • Kopp, V., Stark, R., Kuhne-Eversmann, L., & Fischer, M. R. (2009b). Do worked examples foster medical students’ diagnostic knowledge of hyperthyroidism? Medical Education, 43, 1210–1217.

    Google Scholar 

  • Korenstein, D., Thomas, D. C., Foldes, C., Ross, J., Halm, E., & McGinn, T. (2003). An evidence-based domestic violence education program for internal medicine residents. Teaching and Learning in Medicine, 15, 262–266.

    Google Scholar 

  • Krautter, M., Junger, J., Koehl-Hackert, N., Nagelmann, L., & Nikendei, C. (2012). Evaluation of a structured, longitudinal training program for the preparation for the second state exam (M2)—A quantitative analysis. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen., 106, 110–115.

    Google Scholar 

  • Kumta, S. M., Tsang, P. L., Hung, L. K., & Cheng, J. C. Y. (2003). Fostering critical thinking skills through a web-based tutorial programme for final year medical students—A randomized controlled study. Journal of Educational Multimedia and Hypermedia, 12, 267–273.

    Google Scholar 

  • Kwa, S. K., Amin, S. M., & Ng, A. C. (2007). Avoiding common errors in key feature problems. Malaysian Family Physician, 2, 18–21.

    Google Scholar 

  • Ladyshewsky, R. K. (2002). A quasi-experimental study of the differences in performance and clinical reasoning using individual learning versus reciprocal peer coaching. Physiotherapy Theory and Practice, 18, 17–31.

    Google Scholar 

  • Ladyshewsky, R., Baker, R., Jones, M., & Nelson, L. (2000). Evaluating clinical performance in physical therapy with simulated patients. Journal of Physical Therapy Education, 14, 31–37.

    Google Scholar 

  • Lang, V. (2015). Validity Evidence for a Key Features Examination in the Internal Medicine Clerkship. Master’s thesis. University of Illinois at Chicago.

  • LaRochelle, J., Durning, S. J., Boulet, J. R., van der Vleuten, C. P. M., van Merrienboer, J., & Donkers, J. (2016). Beyond standard checklist assessment: Question sequence may impact student performance. Perspectives in Medical Education, 5, 95–102.

    Google Scholar 

  • Latifi, S., Gierl, M. J., Boulais, A. P., & DeChamplain, A. (2016). Using automated scoring to evaluate written responses in English and French on a high-stakes clinical competency examination. Evaluation in the Health Professions, 39, 100–113.

    Google Scholar 

  • Laughlin, T., Wetmore, S., Allen, T., Brailovsky, C., Crichton, T., Bethune, C., et al. (2012). Defining competency-based evaluation objectives in family medicine: Communication skills. Canadian Family Physician, 58, e217–e224.

    Google Scholar 

  • Lawrence, K., Allen, T., Brailovsky, C., Crichton, T., Bethune, C., Donoff, M., et al. (2011). Carpentier MP, Visser S. Defining competency-based evaluation objectives in family medicine Key-feature approach. Canadian Family Physician, 57, e373–e380.

    Google Scholar 

  • Lee, R. P., Venkatesh, B., & Morley, P. (2009). Evidence-based evolution of the high stakes postgraduate intensive care examination in Australia and New Zealand. Anaesthesia and Intensive Care, 37, 525–531.

    Google Scholar 

  • Lehmann, R., Thiessen, C., Frick, B., Bosse, H. M., Nikendei, C., Hoffmann, G. F., et al. (2015). Improving pediatric basic life support performance through blended learning with web-based virtual patients: Randomized controlled trial. Journal of Medical Internet Research, 17, e162.

    Google Scholar 

  • Leung, F. H., Herold, J., & Iglar, K. (2016). Family medicine mandatory assessment of progress: Results of a pilot administration of a family medicine competency-based in-training examination. Canadian Family Physician, 62, e263–e267.

    Google Scholar 

  • Lintzeris, N., Ritter, A., Dunlop, A., & Muhleisen, P. (2002). Training primary health care professionals to provide buprenorphine and LAAM treatment. Substance Abuse, 23, 245–254.

    Google Scholar 

  • Litchfield, R. E., Oakland, M. J., & Anderson, J. A. (2000). Improving dietetics education with interactive communication technology. Journal of the American Dietetic Association, 100, 1191–1194.

    Google Scholar 

  • Litchfield, R. E., Oakland, M. J., & Anderson, J. (2002). Promoting and evaluating competence in on-line dietetics education. Journal of the American Dietetic Association, 102, 1455–1458.

    Google Scholar 

  • Maiburg, B. H., Rethans, J. J., Schuwirth, L. W., Mathus-Vliegen, L. M., & van Ree, J. W. (2003). Controlled trial of effect of computer-based nutrition course on knowledge and practice of general practitioner trainees. American Journal of Clinical Nutrician, 77, 1019S–1024S.

    Google Scholar 

  • Mandin, H., & Dauphinee, W. D. (2000). Conceptual guidelines for developing and maintaining curriculum and examination objectives: the experience of the Medical Council of Canada. Academic Medicine, 75, 1031–1037.

    Google Scholar 

  • Manns, P. J., & Darrah, J. (2012). A structured process to develop scenarios for use in evaluation of an evidence-based approach in clinical decision making. Advances in Medical Education Practice, 3, 113–119.

    Google Scholar 

  • Matthes, J., Look, A., Kahne, A. K., Tekian, A., & Herzig, S. (2008). The semi-structured triple jump—a new assessment tool reflects qualifications of tutors in a PBL course on basic pharmacology. Archives of Pharmacology, 377, 55–63.

    Google Scholar 

  • McGuire, C. H., Solomon, L. M., & Forman, P. M. (1976). Clinical simulations: Selected problems in patient management (2nd ed.). New York, NY: Appleton-CenturyCrofts.

    Google Scholar 

  • Medical Council of Canada. (2012) Guidelines for the Development of Key Feature Problems and Test Cases. Ottawa: Medical Council of Canada. August, 2012. http://mcc.ca/wp-content/uploads/CDM-Guidelines.pdf; http://mcc.ca/wp-content/uploads/Lignes-directrices-PDC.pdf Accessed January 19, 2017.

  • Medical Council of Canada. (2016) 2016 MCCQE Part I Annual Technical Report. http ://mcc.ca/wp-content/uploads/MCCQE-Part-I-Annual-Technical-Report-2016-EN.pdf Accessed June 26, 2017.

  • Medical Council of Canada. (2017a). Online Demo MCCEE Sample Test. Ottawa: Medical Council of Canada.

  • Medical Council of Canada. (2017b). Test Committee Resources. Ottawa: Medical Council of Canada. http://mcc.ca/about/test-committee-resources/Accessed January 23, 2017.

  • Medical Council of Canada. (2017c). Clinical decision making – Exam tips. Ottawa: Medical Council of Canada. https://www.youtube.com/watch?v=ln6X_sVenWc Accessed April 1, 2017.

  • Medical Council of Canada. (2017d). Self-Administered Examination (SAE). Ottawa: Medical Council of Canada. http://mcc.ca/examinations/self-administered-exam/Accessed April 1, 2017.

  • Medical Council of Canada. (2017e). MCC Part-I Examination Demo. Ottawa: Medical Council of Canada. http://mcc.ca/examinations/mccqe-part-i/exam-preparation-resources/Accessed June 19, 2017.

  • Medical Council of Canada. (2017f). When and how you will receive your MCCQE Part I results. http://mcc.ca/examinations/mccqe-part-i/result/Accessed June 19, 2017.

  • Medical Council of Canada. (2017 g). Scoring Aggregator. http://mcc.ca/scoring-aggregator/Accessed June 20, 2017.

  • Medical Council of Canada. (2017 h). Scoring. http://mcc.ca/examinations/mccqe-part-i/scoring/Accessed June 27, 2017.

  • Miller, G. E. (1966). New Thoughts on Old Examinations. Federation Bull., 53, 390–396.

    Google Scholar 

  • Miller, F., Jacques, A., Brailovsky, C., Sindon, A., & Bordage, G. (1997). When to recommend compulsory versus optional CME programs? A study to establish criteria. Academic Medicine, 72, 760–764.

    Google Scholar 

  • Monnier, P., Bedard, M. J., Gagnon, R., & Charlin, B. (2011). The relationship between script concordance test scores in an obstetrics-gynecology rotation and global performance assessments in the curriculum. International Journal of Medical Education, 2, 3–6.

    Google Scholar 

  • National Board of Osteopathic Medical Examiners (NBOME). COMLEX-USA, Level 3. Assessment of Competencies for Osteopathic Medical Licensure. https://www.nbome.org/exams-assessments/comlex-usa/comlex-usa-level-3/Accessed September 08, 2017.

  • Naude, A., Wium, A. M., & duPlessis, S. (2011). Re-engineering the curriculum at a rural institution: Reflection on the process of development. South African Journal of Health Education, 25, 760–783.

    Google Scholar 

  • Nikendei, C., Mennin, S., Weyrich, P., Kraus, B., Zipfel, S., Schrauth, M., et al. (2009). Effects of a supplementary final year curriculum on students’ clinical reasoning skills as assessed by key-feature examination. Medical Teacher, 31, e438–e442.

    Google Scholar 

  • Norcini, J. J., Anderson, B., Bollela, V., Burch, V., Costa, M. J., Duvivier, R., et al. (2011). Criteria for good assessment: consensus statement and recommendations from the ttawa 2010 conference. Medical Teacher, 33, 206–214.

    Google Scholar 

  • Norcini, J. J., & Guille, R. (2002). Chapter 25: Combining tests and setting standards. In G. R. Norman, C. P. M. van der Vleuten, & D. I. Newble (Eds.), International handbook of research in medical education (pp. 811–834). Berlin: Springer.

    Google Scholar 

  • Norcini, J.J., Swanson, D.S., Grosso, L.J., & Webster, G.D. (1983) A comparison of several methods for scoring Patient Management Problems. In Proceedings of the 22 nd research in medical education conference. Association of American Medical Colleges.

  • Norman, G., Alan, N., Blake, J. M., & Mueller, B. (2010). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Medical Teacher, 32, 496–499.

    Google Scholar 

  • Norman, G., Bordage, G., Curry, L., Dauphinee, D., Jolly, B., Newble, D., et al. (1984). Review of recent innovations in assessment. In R. Wakeford (Ed.), Directions in Clinical Assessment. Report of the Cambridge Conference on the Assessment of Clinical Competence (pp. 9–27). Cambridge: Office of the Regius Professor of Physic. Cambridge University School of Clinical Medicine.

    Google Scholar 

  • Norman, G., Bordage, G., Page, G., & Keane, D. (2006). How specific is case specificity? Medical Education, 40, 618–623.

    Google Scholar 

  • Norman, G. R., & Feightner, J. W. (1981). A comparison of behavior on simulated patients and patient management problems. Medical Education, 15, 26–32.

    Google Scholar 

  • Norman, G.R., Tugwell, P., Jacoby, L.L., & Muzzin, L.J. (1983). The Generalizability of Measures of Clinical Problem Solving. In Proceedings of the 22nd Conference on Research in Medical Education (pp. 110–114). Washington: Association of American Medical Colleges.

  • Page, G. (2008). An Exploratory Review of Content, Format and Performance Differences on the MCQ and CDM Components of the 2008 QE Part 1 Examination (p. 17). Ottawa: Report to the Medical Council of Canada.

    Google Scholar 

  • Page, G., & Bordage, G. (1995). The Medical Council of Canada’s Key Feature Project: A more valid written examination of clinical decision-making skills. Academic Medicine, 70, 104–110.

    Google Scholar 

  • Page, G., & Bordage, G. (2004). Better test score reliability with multi-question key feature cases: refining our view of case specificity. In Presented at the Asian Pacific Medical Education Conference. Singapore.

  • Page, G., Bordage, G., & Alen, T. (1995). Developing key-feature problems and examinations to assess clinical decision-making skills. Academic Medicine, 70, 194–201.

    Google Scholar 

  • Page, G., Bordage, G., Harasym, P., Bowmer, I., & Swanson, D. (1990b). A revision of the Medical Council of Canada’s qualifying examlnatlon: Pilot test results. In R. Zwierstra, W. Bender, W. Hiemstra, & R. Scherpbier (Eds.), Teaching and assessing clinical competence (pp. 403–407). Groningen: Boekwerk Publ.

    Google Scholar 

  • Page, G., Boulais, A.P., Blackmore, D., & Dauphinee, D. (2000a). Justifying the use of short answer questions in the KF Problems of the MCCC’s qualifying exam. In Presented at the 9th Ottawa Conference. Cape Town.

  • Page, G., Broudo, D., Blackmore, D., Schulzer, M., Bordage, G., (1990a). Cueing as a factor in written examinations of clinical decision-making skills. In Proceedings of the international conference on current developments in assessing clinical competence (pp. 184–191), Ottawa.

  • Page, G., Farmer, L., Spike, N., & McDonald, E. (2000b). The use of short-answer questions in the key features problems on the Royal Australian College of General Practitioners Fellowship Examination. In Presented at the 9th Ottawa Conference. Cape Town.

  • Pinnock, R., & Jones, A. (2008). An undergraduate paediatric curriculum based on clinical presentations and ‘key features’. Journal of Paediatrics and Child Health, 44, 661–664.

    Google Scholar 

  • Rademakers, J., Ten Cate, T. H. J., & Bar, P. R. (2005). Progress testing with short answer questions. Medical Teacher, 27, 578–582.

    Google Scholar 

  • Raupach, T., Muenscher, C., Anders, S., Steinbach, R., Pukrop, T., Hege, I., et al. (2009). Web-based collaborative training of clinical reasoning: A randomized trial. Medical Teacher, 31, e431–e437.

    Google Scholar 

  • Reinert, A., Berlin, A., Swan-Sein, A., Nowygrod, R., & Fingeret, A. (2014). Validity and reliability of a novel written examination to assess knowledge and clinical decision-making skills of medical students on the surgery clerkship. American Journal of Surgery, 207, 236–242.

    Google Scholar 

  • Reiter, H., Eva, K. W., Rosenfeld, J., & Norman, G. R. (2007). Multiple mini-interviews predict clerkship and licensing examination performance. Medical Education, 41, 378–384.

    Google Scholar 

  • Renaud, J. S., Ratté, F., Theriault, J. F., Roy, A. M., & Cote, L. (2016). Questions de planification clinique: un nouvel outil pour évaluer la capacité des étudiants en médecine à identifier les éléments-clés discriminants d’un diagnostic différentiel. Pédagogie Médicale, 17, 65–75.

    Google Scholar 

  • Reznick, R. K., Smee, S., Baumber, J. S., et al. (1993). Guidelines for estimating the real cost of an objective structured clinical examination. Academic Medicine, 68, 513–517.

    Google Scholar 

  • Rotthoff, T., Baehring, T., Dicken, H. D., Fahron, U., Fischer, M. R., Adler, M., et al. (2006a). Case-based computerized examinations for medical students—objective, implementation and experiences. GMS Medizinische Informatik, Biometrie und Epidemiologie, 2, 11.

    Google Scholar 

  • Rotthoff, T., Baehring, T., Dicken, H. D., Fahron, U., Richter, B., Fischer, M. R., et al. (2006b). Comparison between long-menu and open-ended questions in computerized medical assessments. a randomized controlled trial. BMC Medical Education, 6, 50.

    Google Scholar 

  • Schaper, E., Tipold, A., & Ehlers, J. P. (2013). Use of key feature questions in summative assessment of veterinary medicine students. Irish Veterinary Journal, 66, 3.

    Google Scholar 

  • Schenowitz, G. (2000). Le projet STEP ou l’évaluation des besoins de formation revisitée. Acta Endoscopica, 30, 265–268.

    Google Scholar 

  • Schmidmaier, R., Eiber, S., Ebersbach, R., Schiller, M., Hege, I., Holzer, M., et al. (2013). Learning the facts in medical school is not enough: which factors predict successful application of procedural knowledge in a laboratory setting? BMC Medical Education, 13, 28–37.

    Google Scholar 

  • Schreiner, U., Haefner, A., Gologan, R., & Obertacke, U. (2011). Effective teaching modifies medical student attitudes toward pain symptoms. European Journal of Trauma Emergency Surgery., 37, 655–659.

    Google Scholar 

  • Schubach, F., Goos, M., Fabry, G., Vach, W., & Boeker, M. (2017). Virtual patients in the acquisition of clinical reasoning skills: does presentation mode matter? A quasi-randomized controlled trial. BMC Medical Education, 17(1), 165.

    Google Scholar 

  • Schubert, S., Ortwein, H., Dumitsch, A., Schwantes, U., Wilhelm, O., & Kiessling, C. (2008). A situational judgement test of professional behaviour: development and validation. Medical Teacher, 30, 528–533.

    Google Scholar 

  • Schuwirth, L.W.T. (1998a). An approach to the assessment of medical problem solving: Computerised case-based testing. Doctoral dissertation. University of Maastricht.

  • Schuwirth, L. W. T., Blackmore, D. E., Mom, E., van den Wildenberg, F., Stoffers, H. E. J. H., & van der Vleuten, C. P. M. (1999a). How to write short cases for assessing problem-solving skills. Medical Teacher, 21, 144–150.

    Google Scholar 

  • Schuwirth, L. W. T., Gorter, S., van der Heijde, D., Rethans, J. J., Brauer, J., Houben, H., et al. (2005). The role of a computerised case-based testing procedure in practice performance assessment. Advances in Health Sciences Education, 10, 145–155.

    Google Scholar 

  • Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2003). The use of clinical simulations in assessment. Medical Education, 37(Suppl. 1), 65–71.

    Google Scholar 

  • Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2010). Cost-affective assessment. In K. Walsh (Ed.), Cost Effectiveness in Medical Education (pp. 94–100). Oxford: Radcliffe.

    Google Scholar 

  • Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2011a). General overview of the theories used in assessment: AMEE Guide No. 57. Medical Teacher, 33, 783–797.

    Google Scholar 

  • Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2011b). Conceptualising surgical education assessment. In H. Fry & R. Kneebone (Eds.), Surgical education: Theorising an emerging domain (pp. 75–90). Berlin: Springer.

    Google Scholar 

  • Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2012). Programmatic assessment and Kane’s validity perspective. Medical Education, 46, 38–48.

    Google Scholar 

  • Schuwirth, L. W. T., van der Vleuten, C. P. M., De Kock, C. A., Peperkamp, A. G. W., & Donkers, H. H. L. M. (1996a). Computerized case-based testing: a modern method to assess clinical decision making. Medical Teacher, 18, 294–299.

    Google Scholar 

  • Schuwirth, L. W. T., van der Vleuten, C. P. M., & Donkers, H. H. L. M. (1996b). A closer look at cueing effects in multiple-choice questions. Medical Education, 30, 44–49.

    Google Scholar 

  • Schuwirth, L. W. T., van der Vleuten, C. P. M., Stoffers, H. E. J. H., & Peperkamp, A. G. W. (1996c). Computerized long-menu questions as an alternative to open-ended questions in computerized assessment. Medical Education, 30, 50–55.

    Google Scholar 

  • Schuwirth, L.W.T., Verheggen, M.M., Boshuizen, H.P.A., van der Waarten, Th.H.A.M., & van der Vleuten, C.P.M. (1998b). Validation of key-feature assessment using think-aloud protocols. In Proceedings of the 8th Ottawa Conference on Medical Education (p. 655), Philadelphia.

  • Schuwirth, L. W. T., Verheggen, M. M., van der Vleuten, C. P. M., Boshuizen, H. P. A., & Dinant, G. J. (2001). Do short cases elicit different thinking processes than factual knowledge questions do? Medical Education, 35, 348–356.

    Google Scholar 

  • Schuwirth, L. W. T., Verhoeven, B. H., Scherpbier, A. J. J. A., Mom, E. M. A., Cohen-Schotanus, J., Van Rossum, H. J. M., et al. (1999b). An inter- and intra-university comparison with short case-based testing. Advances in Health Sciences Education, 4, 233–244.

    Google Scholar 

  • Shiozawa, T., Butz, B., Herlan, S., Kramer, A., & Hirt, B. (2017). Interactive anatomical and surgical live stream lectures improve students’ academic performance in applied clinical anatomy. Anatomical Sciences Education, 10, 46–52.

    Google Scholar 

  • Skakun, E. N., Maguire, T. O., & Cook, D. A. (1994). Strategy Choices in Multiple-choice Items. Academic Medicine, 69, S7–S9.

    Google Scholar 

  • Smith, S. R., Dollase, R. H., & Boss, J. A. (2003). Assessing students’ performances in a competency-based curriculum. Academic Medicine, 78, 97–107.

    Google Scholar 

  • Smith, S., Kogan, J. R., Berman, N. B., Dell, M. S., Brock, D. M., & Robins, L. S. (2016). The Development and Preliminary Validation of a Rubric to Assess Medical Students’ Written Summary Statements in Virtual Patient Cases. Academic Medicine, 91, 94–100.

    Google Scholar 

  • Spike, N. A., & Hays, R. B. (1999). Analysis by training status of performance in the certification examination for Australian family doctors. Medical Education, 33, 612–615.

    Google Scholar 

  • Stark, R., Kopp, V., & Fischer, M. R. (2011). Case-based learning with worked examples in complex domains: Two experimental studies in undergraduate medical education. Learning and Instruction, 21, 22–33.

    Google Scholar 

  • Sturmberg, J. P., Crowe, P., & Hughes, C. (2003). Computer-assisted instruction: guiding learning through a key features approach in a community-based medical course. Medical Teacher, 25, 332–335.

    Google Scholar 

  • Sturmberg, J. P., & Martin, C. M. (2016). Diagnosis—The limiting focus of taxonomy. Journal of Evaluation in Clinical Practice, 22, 103–111.

    Google Scholar 

  • Tamblyn, R., Abrahamowicz, M., Bartlett, G., Winslade, N., Jacques, A., Klass, D., Wenghofer, E., Smee, S., Dauphinee, D., Blackmore, D., Bartman, I., Buckeridge, D., & Hanley, J. (2009). The Quebec-ontario follow-up study of the association between scores achieved on the MCCQE Part II examination and performance in clinical practice. Report to the Medical Council of Canada.

  • Tamblyn, R., Abrahamowicz, M., Dauphinee, D., Wenghofer, E., Jacques, A., Klass, D., et al. (2007). Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA, 298, 993–1001.

    Google Scholar 

  • Tamblyn, R., Abrahamowicz, M., Dauphinee, D., Wenghofer, E., Jacques, A., Klass, D., et al. (2010). Influence of physicians’ management and communication ability on patients’ persistence with antihypertensive medication. Archives of Internal Medicine, 170, 1064–1072.

    Google Scholar 

  • Touchie C., & Streefkerk C. for the Blueprint Project Team. (2014). Blueprint Project – Qualifying Examinations Blueprint and Content Specifications. Ottawa, Ontario: Medical Council of Canada. September 2014. http://mcc.ca/wp-content/uploads/Blueprint-Report.pdf Accessed April 12, 2017.

  • Trudel, J. L., Bordage, G., & Downing, S. M. (2008). Reliability and validity of key feature cases for the self-assessment of colon and rectal surgeons. Annals of Surgery, 248, 252–258.

    Google Scholar 

  • van der Vleuten, C. P. M. (1996). The assessment of professional competence: developments, research and practical implications. Advances in Health Sciences Education, 1, 41–67.

    Google Scholar 

  • van der Vleuten, C. P. M., Norman, G. R., & Schuwirth, L. W. T. (2008). Assessing clinical reasoning. In J. Higgs, M. Jones, S. Loftus, & N. Christensen (Eds.), Clinical reasoning in the health professions (pp. 413–421). Philadelphia: Elsevier.

    Google Scholar 

  • van der Vleuten, C. P. M., & Schuwirth, L. W. T. (2005). Assessing professional competence: from methods to programmes. Medical Education, 39, 309–317.

    Google Scholar 

  • van der Vleuten, C. P. M., Schuwirth, L. W. T., Scheele, F., Driessen, E. W., & Hodges, B. (2010). The assessment of professional competence: building blocks for theory development. Best Practice & Research in Clinical Obstetrics and Gynaecology, 24, 703–719.

    Google Scholar 

  • Vaughan, B., & Morrison, T. (2015). Assessment in the final year clinical practicum of an Australian osteopathy program. International Journal of Osteopathic Medicine, 18, 278–286.

    Google Scholar 

  • Wearne, S. (2008). The RACGP Fellowship examination: 10 tips for answering key feature problems. Australian Family Physician, 37, 559–561.

    Google Scholar 

  • Wenghofer, E., Klass, D., Abrahamowicz, M., Dauphinee, D., Jacques, A., Smee, S., et al. (2009). Doctor scores on national qualifying examinations predict quality of care in future practice. Medical Education, 43, 1166–1173.

    Google Scholar 

  • Wetmore, S., Laughlin, T., Lawrence, K., Donoff, M., Allen, T., Brailovsky, C., et al. (2012). Defining competency-based evaluation objectives in family medicine—Procedure skills. Canadian Family Physician, 58, 775–780.

    Google Scholar 

  • Williams, R., Klamen, D., Markwell, S. J., Cianciolo, A. T., Colliver, J. A., & Verhulst, S. J. (2014). Variations in senior medical student diagnostic justification ability. Academic Medicine, 89, 790–798.

    Google Scholar 

  • Winslade, N. (2000). Assessment of Medical Assistant’s Knowledge of Authorized Pharmaceuticals. Master’s thesis, University of Maastricht.

  • Winslade, N. (2001). A System to Assess the Achievement of Doctor of Pharmacy Students. American Journal of Pharmaceutical Education, 65, 363–392.

    Google Scholar 

  • Yousefichaijan, P., Jafari, F., Kahbazi, M., Rafiei, M., & Pakniyat, A. G. (2016). The effect of short-term workshop on improving clinical reasoning skill of medical students. Medical Journal of the Islamic Repubublic of Iran, 30, 396.

    Google Scholar 

  • Zamani, S., Amini, M., Masoumi, S. Z., Delavari, S., Namaki, M. J., & Kojuri, J. (2017). The comparison of the key feature of clinical reasoning and multiple-choice examinations in clinical decision makings ability. Biomedical Research, 28, 1115–1119.

    Google Scholar 

  • Zieky, M. J., & Perie, M. (2006). A primer on setting cut scores on tests of educational achievement (p. 20). Princeton, NJ: Educational Testing Service Inc.

    Google Scholar 

Download references

Acknowledgements

We are grateful to Hrynchak, Takahashi and Nayer for sharing their 2014 references and to Maureen Clark, librarian at the University of Illinois at Chicago for her exceptional skills with database searches and Kimberly Hu for her unflagging assistance in retrieving the papers and preparing the references. We are also grateful for the judicious and helpful comments from the external reviewers.

Funding

This study was supported in part by a travel grant from the Medical Council of Canada for which we are grateful. The views expressed here are not necessarily those of the MCC.

Author information

Authors and Affiliations

  1. Department of Medical Education, College of Medicine, University of Illinois at Chicago, Chicago, USA

    G. Bordage

  2. Department of Medicine, Faculty of Medicine, University of British Columbia, Vancouver, Canada

    G. Page

Authors
  1. G. Bordage
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. G. Page
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to G. Bordage.

Ethics declarations

Conflict of interests

The authors wish to indicate that they were among those who created the Key-Features concept and have led various research and development projects related to the Key-Features approach reported here. To the best of their ability, the authors remained as impartial as possible. The authors alone are responsible for the content and writing of the article.

Appendix: Uses of the Key-Features’ approach

Appendix: Uses of the Key-Features’ approach

See Tables 3, 4, and 5.

Table 3 Uses of the Key-Features approach for assessment according to clinical disciplines
Full size table
Table 4 Uses of the Key-Features approach as stimulus material or outcome measures in research studies
Full size table
Table 5 Uses of the Key-Features approach for defining learning objectives or instruction plan
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bordage, G., Page, G. The key-features approach to assess clinical decisions: validity evidence to date. Adv in Health Sci Educ 23, 1005–1036 (2018). https://doi.org/10.1007/s10459-018-9830-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-018-9830-5

Keywords

Search

Navigation