To the Editor,

All McMaster medical students undertake two weeks of anesthesia training during their clerkship. We believe that the rotation’s “traditional” multiple-choice exam tested the students’ factual knowledge rather than an understanding of principles. Our goal was to design an exam that better assessed the application of knowledge to a clinical situation.

The traditional examination had flaws typical of many multiple-choice question (MCQ) exams including logical cues (spelling and grammatical errors that allow examinees to eliminate incorrect answers because they do not make grammatical or logical sense), unfocused items (the stem fails to pose a direct question), clang associations (the language used in the question is repeated within the correct response option), convergence strategy (the correct option has the most in common with the other options), and the “except” format (which can interfere with knowledge assessment because of misinterpretation of the question).1

We formed a group that included the Anesthesia Undergraduate Program Director, three anesthesia residents, and one medical student. All members completed the “Mac Health Primer on Writing or Editing Quality MCQ Items” and two completed courses offered by the National Board of Medical Examiners.

Our theoretical framework to revise the MCQ exam was based on principles highlighted in Miller’s pyramid of clinical competence, which divides professional authenticity into four categories: knows, knows how, shows, and does.2 Our goal was to assess knows how. We eliminated outdated topics and used a Modified Essay Questions format to create a new question bank, in which a question was preceded by a clinical vignette.2 Out of the 130 questions in the previous bank, 100 were edited and the remaining 30 were deemed non-salvageable; 72 new questions were created. Of the 100 edited questions, 40% contained logical cues, 11% contained unfocused items, 14% showed clang associations, 10% showed convergence strategy, and 37% utilized the except format (Figure).

Figure
figure 1

A) Example of a question edited where the “except” flaw was eliminated and a clinical stem was added. B) Example of a question created to assess knowledge application (in clinical practice). C) Example of a question with a “logical cue” flaw. D) Example of a question with an “unfocused items” flaw. E) Example of a question with a “clang association” flaw. F) Example of a question with a “convergence strategy” flaw

Full size image

Criterion-based validity refers to “how well the instrument under consideration compares to certain gold standard measurements.”3 Predictive validity is a subset of criterion validity and can be thought of as the “degree to which a measurement instrument accurately predicts theoretically expected outcomes”. We believe that the results from the modified exam have increased construct validity with regard to their reflection of the student’s assimilation and understanding of the information provided during their anesthesia clerkship rotation. We share our experience with the hope of helping academic centres to achieve the goal of assessing the category knows how.

Our new exam addresses the design flaws common to many MCQs. We hope that these efforts can serve as a stimulus for other undergraduate anesthesia programs to reconsider their MCQ examinations. Future plans include a comparison of the new exam scores with old exam scores and assessment of how they correlate with clinical performance of anesthesia trainees.