Progress testing: critical analysis and suggested practices

Abstract

Educators have long lamented the tendency of students to engage in rote memorization in preparation for tests rather than engaging in deep learning where they attempt to gain meaning from their studies. Rote memorization driven by objective exams has been termed a steering effect. Progress testing (PT), in which a comprehensive examination sampling all of medicine is administered repeatedly throughout the entire curriculum, was developed with the stated aim of breaking the steering effect of examinations and of promoting deep learning. PT is an approach historically linked to problem-based learning (PBL) although there is a growing recognition of its applicability more broadly. The purpose of this article is to summarize the salient features of PT drawn from the literature, provide a critical review of these features based upon the same literature and psychometric considerations drawn from the Standards for Educational and Psychological Testing and provide considerations of what should be part of best practices in applying PT from an evidence-based and a psychometric perspective.

This is a preview of subscription content, log in to check access.

Notes

  1. 1.

    The Angoff Method asks judges to visualized a hypothetical borderline pass group and then asks them to identify the percentage who would answer the item correctly. Verhoeven specifically instructed the judges not to correct for chance success, so as best we can tell, the percentage was based on the number of items that the borderline group should answer correctly out of the total of 250 T/F items.

References

  1. Albanese, M. A. (1986). The correction for guessing: A further analysis of Angoff and Schrader. Journal of Educational Measurement, 23(3), 225–235.

    Article  Google Scholar 

  2. Albanese, M. (2008). Benchmarking progress tests for cross-institutional comparisons: Which road taken makes a difference and all roads have bumps. Medical Education, 42, 4–7.

    Article  Google Scholar 

  3. Arnold, L., & Willoughby, T. L. (1990). The quarterly profile examination. Academic Medicine, 65, 515–516.

    Article  Google Scholar 

  4. Blake, J. M., Norman, G. R., Keane, D. R., Mueller, C. B., Cunnington, J., & Didyk, N. (1996). Introducing progress testing in McMaster University’s problem-based medical curriculum: Psychometric properties and effect on learning. Academic Medicine, 71(9), 1002–1007.

    Article  Google Scholar 

  5. Cross, L. H., & Frary, R. B. (1977). An empirical test of Lord’s theoretical results regarding formula scoring of multiple choice tests. Journal of Educational Measurement, 14(4), 313–321.

    Article  Google Scholar 

  6. Joint Committee on Standards for Educational and Psychological Testing of the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington DC: American Educational Research Association.

    Google Scholar 

  7. Kolen, M. J., (1988). Traditional equating methodology. Educational Measurement: Issues and Practice 115.

  8. Kolen, M. J., & Brennan, R. L. (1995). Test equating. New York: Springer.

    Google Scholar 

  9. Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134.

    Article  Google Scholar 

  10. McHarg, J., Bradley, P., Chamberlain, S., Ricketts, C., Searle, J., & McLachlan, J. C. (2005). Assessment of progress tests. Medical Education, 39, 221–227.

    Article  Google Scholar 

  11. Muijtjens, A. M. M., Schuwirth, L. W. T., Coen-Schotanus, J., Thoben, A. J. N. M., & van der Vleuten, C. P. M. (2008). Benchmarking by cross-institutional comparison of student achievement in a progress test. Medical Education, 42, 82–88.

    Article  Google Scholar 

  12. Muijtjens, A. M. M., van Mameren, H., Hoogenboom, R. J. I., Evers, J. L. H., & van der Vleuten, C. P. M. (1999). The effect of a ‘don’t know’ option on test scores: number-right and formula scoring compared. Medical Education, 33, 267–275.

    Article  Google Scholar 

  13. Norman, G. R., Neville, A., Blake, J. M., & Mueller, B. (2010a). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Medical Teacher, 32, 496–499.

    Article  Google Scholar 

  14. Norman, G., Neville, A., Blake, J. M., & Mueller, B. (2010b). Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Medical Teacher, 32(6), 496–499.

    Article  Google Scholar 

  15. Personal communication with Louis Arnold, August 31, 2007.

  16. Rademakers, J., Ten Cate, T. J., & Bär, P. R. (2005). Progress testing with short answer questions. Medical Teacher, 27(7), 578–582.

    Article  Google Scholar 

  17. Swanson, D. B., Holtzman, K. Z., & Bulter, A. (2010). Cumulative achievement testing: Progress testing in reverse. Medical Teacher, 32(6), 516–520.

    Article  Google Scholar 

  18. Traub, R. E., Hambleton, R. K., & Singh, B. (1969). Effects of promised reward and threatened penalty on performance of a multiple-choice vocabulary test. Educational and Psychological Measurement, 29, 847–861.

    Article  Google Scholar 

  19. Van der Vleuten, C. P., Schuwirth, L. W., Muijtjens, A. M., Thoben, A. J., Cohen-Schotanus, J., & van Boven, C. P. (2004). Cross institutional collaboration in assessment: A case on progress testing. Medical Teacher, 26(8), 719–725.

    Article  Google Scholar 

  20. Van der Vleuten, C. P. M., Verwijnen, G. M., & Wijnen, H. F. W. (1996). Fifteen years of experience with progress testing in a problem-based learning curriculum. Medical Teacher, 18, 102–109.

    Article  Google Scholar 

  21. Verhoeven, B. H., Snellen-Balendong, H. A., Hay, I. T., Boon, J. M., van der Linde, M. J., Blitz-Lindeque, J. J., et al. (2005). The versatility of progress testing assessed in an international context: A start for benchmarking global standardization? Medical Teacher, 27(6), 514–520.

    Article  Google Scholar 

  22. Verhoeven, B. H., Van der Steeg, A. F. W., Scherpbier, A. J. J. A., Muijtjens, A. M. M., Verwijnen, G. M., & van der Vleuten, C. P. M. (1999). Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judges. Medical Education, 33, 832–837.

    Article  Google Scholar 

  23. Willoughby, T. L., Dimond, E. G., & Smull, N. W. (1977). Correlation of quarterly profile examination and national board of medical examiner scores. Educational and Psychological Measurement, 37, 445–449.

    Article  Google Scholar 

  24. Willoughby, T. L., & Hutcheson, S. J. (1978). Edumetric validity of the quarterly profile examination. Educational and Psychological Measurement, 38, 1057–1061.

    Article  Google Scholar 

  25. Woolliscroft, J. O., Swanson, D. B., Case, S. M., & Ripkey, D. R. (1995). Monitoring the effectiveness of the clinical curriculum: Use of a cross-clerkship exam to assess development of diagnostic skills. In A. I. Rothman & R. Cohen (Eds.), Proceedings of the sixth Ottawa conference on medical education (pp. 476–478). Toronto: University of Toronto Bookstore Custom Publishing.

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Mark Albanese.

Additional information

Susan M. Case was formerly with the National Conference of Bar Examiners and before that the National Board of Medical Examiners.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Albanese, M., Case, S.M. Progress testing: critical analysis and suggested practices. Adv in Health Sci Educ 21, 221–234 (2016). https://doi.org/10.1007/s10459-015-9587-z

Download citation

Keywords

  • Progress testing
  • Longitudinal assessment
  • Formative assessment
  • High stakes assessment