Abstract
Educators have long lamented the tendency of students to engage in rote memorization in preparation for tests rather than engaging in deep learning where they attempt to gain meaning from their studies. Rote memorization driven by objective exams has been termed a steering effect. Progress testing (PT), in which a comprehensive examination sampling all of medicine is administered repeatedly throughout the entire curriculum, was developed with the stated aim of breaking the steering effect of examinations and of promoting deep learning. PT is an approach historically linked to problem-based learning (PBL) although there is a growing recognition of its applicability more broadly. The purpose of this article is to summarize the salient features of PT drawn from the literature, provide a critical review of these features based upon the same literature and psychometric considerations drawn from the Standards for Educational and Psychological Testing and provide considerations of what should be part of best practices in applying PT from an evidence-based and a psychometric perspective.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The Angoff Method asks judges to visualized a hypothetical borderline pass group and then asks them to identify the percentage who would answer the item correctly. Verhoeven specifically instructed the judges not to correct for chance success, so as best we can tell, the percentage was based on the number of items that the borderline group should answer correctly out of the total of 250 T/F items.
References
Albanese, M. A. (1986). The correction for guessing: A further analysis of Angoff and Schrader. Journal of Educational Measurement, 23(3), 225–235.
Albanese, M. (2008). Benchmarking progress tests for cross-institutional comparisons: Which road taken makes a difference and all roads have bumps. Medical Education, 42, 4–7.
Arnold, L., & Willoughby, T. L. (1990). The quarterly profile examination. Academic Medicine, 65, 515–516.
Blake, J. M., Norman, G. R., Keane, D. R., Mueller, C. B., Cunnington, J., & Didyk, N. (1996). Introducing progress testing in McMaster University’s problem-based medical curriculum: Psychometric properties and effect on learning. Academic Medicine, 71(9), 1002–1007.
Cross, L. H., & Frary, R. B. (1977). An empirical test of Lord’s theoretical results regarding formula scoring of multiple choice tests. Journal of Educational Measurement, 14(4), 313–321.
Joint Committee on Standards for Educational and Psychological Testing of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington DC: American Educational Research Association.
Kolen, M. J., (1988). Traditional equating methodology. Educational Measurement: Issues and Practice 115.
Kolen, M. J., & Brennan, R. L. (1995). Test equating. New York: Springer.
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134.
McHarg, J., Bradley, P., Chamberlain, S., Ricketts, C., Searle, J., & McLachlan, J. C. (2005). Assessment of progress tests. Medical Education, 39, 221–227.
Muijtjens, A. M. M., Schuwirth, L. W. T., Coen-Schotanus, J., Thoben, A. J. N. M., & van der Vleuten, C. P. M. (2008). Benchmarking by cross-institutional comparison of student achievement in a progress test. Medical Education, 42, 82–88.
Muijtjens, A. M. M., van Mameren, H., Hoogenboom, R. J. I., Evers, J. L. H., & van der Vleuten, C. P. M. (1999). The effect of a ‘don’t know’ option on test scores: number-right and formula scoring compared. Medical Education, 33, 267–275.
Norman, G. R., Neville, A., Blake, J. M., & Mueller, B. (2010a). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Medical Teacher, 32, 496–499.
Norman, G., Neville, A., Blake, J. M., & Mueller, B. (2010b). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Medical Teacher, 32(6), 496–499.
Personal communication with Louis Arnold, August 31, 2007.
Rademakers, J., Ten Cate, T. J., & Bär, P. R. (2005). Progress testing with short answer questions. Medical Teacher, 27(7), 578–582.
Swanson, D. B., Holtzman, K. Z., & Bulter, A. (2010). Cumulative achievement testing: Progress testing in reverse. Medical Teacher, 32(6), 516–520.
Traub, R. E., Hambleton, R. K., & Singh, B. (1969). Effects of promised reward and threatened penalty on performance of a multiple-choice vocabulary test. Educational and Psychological Measurement, 29, 847–861.
Van der Vleuten, C. P., Schuwirth, L. W., Muijtjens, A. M., Thoben, A. J., Cohen-Schotanus, J., & van Boven, C. P. (2004). Cross institutional collaboration in assessment: A case on progress testing. Medical Teacher, 26(8), 719–725.
Van der Vleuten, C. P. M., Verwijnen, G. M., & Wijnen, H. F. W. (1996). Fifteen years of experience with progress testing in a problem-based learning curriculum. Medical Teacher, 18, 102–109.
Verhoeven, B. H., Snellen-Balendong, H. A., Hay, I. T., Boon, J. M., van der Linde, M. J., Blitz-Lindeque, J. J., et al. (2005). The versatility of progress testing assessed in an international context: A start for benchmarking global standardization? Medical Teacher, 27(6), 514–520.
Verhoeven, B. H., Van der Steeg, A. F. W., Scherpbier, A. J. J. A., Muijtjens, A. M. M., Verwijnen, G. M., & van der Vleuten, C. P. M. (1999). Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judges. Medical Education, 33, 832–837.
Willoughby, T. L., Dimond, E. G., & Smull, N. W. (1977). Correlation of quarterly profile examination and national board of medical examiner scores. Educational and Psychological Measurement, 37, 445–449.
Willoughby, T. L., & Hutcheson, S. J. (1978). Edumetric validity of the quarterly profile examination. Educational and Psychological Measurement, 38, 1057–1061.
Woolliscroft, J. O., Swanson, D. B., Case, S. M., & Ripkey, D. R. (1995). Monitoring the effectiveness of the clinical curriculum: Use of a cross-clerkship exam to assess development of diagnostic skills. In A. I. Rothman & R. Cohen (Eds.), Proceedings of the sixth Ottawa conference on medical education (pp. 476–478). Toronto: University of Toronto Bookstore Custom Publishing.
Author information
Authors and Affiliations
Corresponding author
Additional information
Susan M. Case was formerly with the National Conference of Bar Examiners and before that the National Board of Medical Examiners.
Rights and permissions
About this article
Cite this article
Albanese, M., Case, S.M. Progress testing: critical analysis and suggested practices. Adv in Health Sci Educ 21, 221–234 (2016). https://doi.org/10.1007/s10459-015-9587-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-015-9587-z