Abstract
The appeal of a shorter testing time makes a computer adaptive testing approach highly desirable for use in multiple assessment and learning contexts. However, for those who have been tasked with designing, configuring, and deploying adaptive tests for operational use at scale, preparing an adaptive test is anything but simple. The process often involves a complex interplay among psychometricians, content experts, and technologists who operate with different vocabularies and subject matter expertise. This paper presents the authors’ experience of developing smart platforms for designing, configuring, and deploying adaptive assessments. We present six design principles for the development of such systems. Through an example of an authoring system now used in production, we discuss smart feedback loops built into the system and how they support efficient iteration. We conclude that it is not possible to overstate the importance of process transition support when launching smart platforms, and that a thoughtful and integrated user experience that allows each of the people in the process to work in a context most accessible to them given their background and experience yields substantial business value.
This is a preview of subscription content, access via your institution.
References
ACT. (2020a). Echo-Adapt.
ACT. (2020b). Echo-Adapt: User Guide 1.58.
Bock, R.D., & Mislevy, R.J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6 (4), 431–444. https://doi.org/10.1177/014662168200600405. http://journals.sagepub.com/doi/10.1177/014662168200600405.
Chang, H.H., & van der Linden, W.J. (2003). Optimal stratification of item pools in α-stratified computerized adaptive testing. Applied Psychological Measurement, 27 (4), 262–274 . https://doi.org/10.1177/0146621603027004002. http://journals.sagepub.com/doi/10.1177/0146621603027004002.
Dermeval, D., Paiva, R., Bittencourt, I., Vassileva, J., & Borges, D. (2018). Authoring tools for designing intelligent tutoring systems: a systematic review of the literature. Int J Artif Intell Educ, 336–384. https://doi.org/10.1007/s40593-017-0157-9. https://link.springer.com/article/10.1007/s40593-017-0157-9.
Hiatt, J.M. (2006). ADKAR: a model for change in business, government, and our community. Prosci Learning Center Publications.
I.M.S. Global. (2015). IMS question & test interoperability (QTI) specification. http://www.imsglobal.org/question/index.html.
I.M.S. Global. (2020). About the IMS Global Learning ConsortiumAP. https://www.imsglobal.org/aboutims.html.
Jiang, B. (2020). RSCAT: Shadow-test approach to computerized adaptive testing. https://cran.r-project.org/package=RSCAT, r package version 1.1.0.
Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, A. (2020). A systematic review of automatic question generation for educational purposes. Int J Artif Intell Educ, 121–204. https://doi.org/10.1007/s40593-019-00186-y.
Luo, X. (2020). Automated test assembly with mixed-integer programming: The effects of modeling approaches and solvers. Journal of Educational Measurement, https://doi.org/10.1111/jedm.12262. https://onlinelibrary.wiley.com/doi/abs/10.1111/jedm.12262.
Pandarova, I., Schmidt, T., Hartig, J., Boubekki, A., Jones, R., & Brefeld, U. (2019). Predicting the difficulty of exercise items for dynamic difficulty adaptation in adaptive language tutoring. Int J Artif Intell Educ, 342–367. https://doi.org/10.1007/s40593-019-00180-4.
Smith, J.C., & Taskin, Z.C. (2007). A tutorial guide to mixed-integer programming models and solution techniques.
Sottilare, R., Baker, R., Graesser, A., & Lester, J. (2018). Special issue on the generalized intelligent framework for tutoring (gift): Creating a stable and flexible platform for innovations in aied research. Int J Artif Intell Educ, 139–151. https://doi.org/10.1007/s40593-017-0149-9.
Sympson, J.B., & Hetter, R.D. (1985). Controlling item-exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the military testing association, San Diego CA: Navy personnel research and development center.
van der Linden, W.J. (2005). Linear models for optimal test design. New York: Springer. https://doi.org/10.1007/0-387-29054-0. http://link.springer.com/10.1007/0-387-29054-0.
van der Linden, W.J. (2009). Elements of adaptive testing, (pp. 31–55). New York: Springer. chap constrained adaptive testing with shadow tests.
van der Linden, W.J., & Veldkamp, B.P. (2007). Conditional item-exposure control in adaptive testing using item-ineligibility probabilities. Journal of Educational and Behavioral Statistics, 32(4), 398–418. https://doi.org/10.3102/1076998606298044. http://journals.sagepub.com/doi/10.3102/1076998606298044.
Acknowledgements
The authors are grateful to Dr. Sara Vispoel for her thoughtful review and comments during the preparation of this paper.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interests
The authors declare that they have no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article belongs to the Topical Collection: Creating and improving adaptive learning: Smart authoring tools and processes
Guest Editors: Stephen B. Gilbert, Andrew M. Olney and Kelly Rivers
Rights and permissions
About this article
Cite this article
Barrett, M.D., Jiang, B. & Feagler, B.E. A Smart Authoring System for Designing, Configuring, and Deploying Adaptive Assessments at Scale. Int J Artif Intell Educ 32, 28–47 (2022). https://doi.org/10.1007/s40593-021-00258-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40593-021-00258-y