Abstract
Multiple-choice-questions are common in medical examinations, but guessing biases assessment results. Confidence-based-testing (CBT) integrates indicated confidence levels. It has been suggested that correctness of and confidence in an answer together indicate knowledge levels thus determining the quality of a resulting decision. We used a CBT approach to investigate whether decision quality improves during undergraduate medical education. 3rd- and 5th-year students attended formative multiple-choice exams on pharmacological issues. Students were asked to indicate their confidence in a given answer. Correctness of answers was scored binary (1-correct; 0-wrong) and confidence levels were transformed to an ordinal scale (guess: 0; rather unsure: 0.33; rather sure: 0.66; very sure: 1). 5th-year students gave more correct answers (73 ± 16 vs. 49 ± 13 %, p < 0.05) and were on average more confident regarding the correctness of their answers (0.61 ± 0.18 vs. 0.46 ± 0.13, p < 0.05). Correlation of these parameters was stronger for 5th-year students (r = 0.81 vs. r = 0.52), but agreement of confidence and correctness (‘centration’) was lower. By combining the Bland-and-Altman approach with categories of decision-quality we found that 5th-year students were more likely to be ‘well-informed’ (41 vs. 5 %), while more 3rd-students were ‘uninformed’ (24 vs. 76 %). Despite a good correlation of exam results and confidence in given answers increased knowledge might be accompanied by a more critical view at the own abilities. Combining the statistical Bland-and-Altman analysis with a theoretical approach to decision-quality, more advanced students are expected to apply correct beliefs, while their younger fellows are rather at risk to hesitate or to act amiss.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Albano, M. G., Cavallo, F., Hoogenboom, R., Magni, F., Majoor, G., Manenti, F., et al. (1996). An international comparison of knowledge levels of medical students: The Maastricht Progress Test. Medical Education, 30(4), 239–245.
Anderson, J. (2004). Medical teacher 25th anniversary series multiple-choice questions revisited. Medical Teacher, 26, 110–113.
Bland, J. M., & Altman, D. G. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, 1(8476), 307–310.
Bland, J. M., & Altman, D. G. (2012). Agreed statistics: Measurement method comparison. Anesthesiology, 116, 182–185.
Bruno, J. (2005). Method and system for knowledge assessment and learning incorporating feedbacks. United States Patent: 6921268.
Bush, M. E. (2006). Quality assurance of multiple-choice tests. Quality Assurance in Education, 14, 398–404.
Confucius & Legge, J. (1971). Confucian analects, the great learning, and the doctrine of the mean. New York: Dover Publications.
Egger, S. S., Drewe, J., & Schlienger, R. G. (2003). Potential drug–drug interactions in the medication of medical patients at hospital discharge. European Journal of Clinical Pharmacology, 58, 773–778.
Friedman, C., Gatti, G., Elstein, A., Franz, T., Murphy, G., & Wolf, F. (2001). Are clinicians correct when they believe they are correct? Implications for medical decision support. Studies in Health Technology and Informatics, 1, 454–458.
Gardner-Medwin, A.R., Gahan, M. (2003). Formative and summative confidence-based assessment. Presented at the 7th International Computer-Aided Assessment Conference, Loughborough.
Henmon, V. A. C. (1911). The relation of the time of a judgment to its accuracy. Psychological Review, 18, 186–201.
Hunt, D. P. (2003). The concept of knowledge and how to measure it. Journal of Intellectual Capital, 4, 100–113.
Keers, R. N., Williams, S. D., Cooke, J., & Ashcroft, D. M. (2013). Prevalence and nature of medication administration errors in health care settings: A systematic review of direct observational evidence. Annals of Pharmacotherapy, 47, 237–256.
Kolbitsch, J., Ebner, M., Nagler, W. & Scerbakov, N. (2008). Can confidence assessment enhance traditional multiple choice testing? (Paper presented at the Interactive Computer aided Learning (ICL) International Conference, Villach).
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77, 1121.
Leclercq, D. (1983). Confidence marking, its use in testing. In Postlethwaite, Choppin (Eds.), Evaluation in education (pp. 161–287). Oxford: Pergamon.
Matthes, J., Johannsen, W., & Herzig, S. (2013). Change of medical students’ self-appraisal of pharmacological knowledge and skills over time. Naunyn-Schmiedeberg´s. Archives of Pharmacology, 386(Suppl 1), S52.
Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9), 63–67.
Osterberg, K., Kölbel, S. & Brauns, K. (2006). Der Progress Test Medizin: Erfahrungen an der Charité Berlin. GMS Zeitschrift für Medizinische Ausbildung 23(3), Doc46.
Rippey, R. M., & Voytovich, A. E. (1985). Anomalous responses on confidence-scored tests. Evaluation and the Health Professions, 8, 109–119.
Sveiby, K. E. (1997). The new organizational wealth: Managing and measuring knowledge-based assets. California, SA: Berrett-Koehler Publishers.
Tobaiqy, M., McLay, J., & Ross, S. (2007). Foundation year 1 doctors and clinical pharmacology and therapeutics teaching. A retrospective view in light of experience. British Journal of Clinical Pharmacology, 64, 363–372.
Ward, M., Gruppen, L., & Regehr, G. (2002). Measuring self-assessment: Current state of the art. Advances in Health Sciences Education, 7, 63–80.
Zimmerman, D. W., & Williams, R. H. (2003). A new look at the influence of guessing on the reliability of multiple-choice tests. Applied Psychological Measurement, 27, 357–371.
Acknowledgments
The authors would like to thank Dr. med. Dipl.-Math. Hartmut Stützer for his advice on the statistical analysis of the data.
Conflict of interest
Daniela Kampmeyer is working since January 2013 for the editorial staff of AMBOSS, an e-learning platform dealing with medical multiple choice questions, MIAMED enterprise, Cologne, Germany. The other authors declare that they have no competing interests.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kampmeyer, D., Matthes, J. & Herzig, S. Lucky guess or knowledge: a cross-sectional study using the Bland and Altman analysis to compare confidence-based testing of pharmacological knowledge in 3rd and 5th year medical students. Adv in Health Sci Educ 20, 431–440 (2015). https://doi.org/10.1007/s10459-014-9537-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-014-9537-1