A Data-Based Simulation Study of Reliability for an Adaptive Assessment Based on Knowledge Space Theory
- 154 Downloads
A large-scale simulation study of the assessment effectiveness of a particular instantiation of knowledge space theory is described. In this study, data from more than 700,000 actual assessments in mathematics using the ALEKS (Assessment and LEarning in Knowledge Spaces) software were used to determine response probabilities for the same number of simulated assessments, for the purpose of examining reliability. Several measures of reliability were examined, borrowed from existing psychometric approaches, with an eye toward developing measures for evaluating reliability for adaptive assessments. The results are compared to analogous results for assessments having mathematics content overlapping that of the ALEKS assessment, and some consequences and future directions are discussed.
KeywordsKnowledge space theory Simulation Reliability Adaptive assessment
We are grateful to the associate editor and four anonymous reviewers for many helpful comments on a previous draft of this paper.
- ACT. (2012). ACT Compass Internet Version Reference Manual. http://act-stage.adobecqms.net/content/dam/act/unsecured/documents/CompassReferenceManual.pdf. Accessed January 29, 2019.
- ACT. (2013–2014). ACT Explore Technical Manual. https://www.act.org/content/dam/act/unsecured/documents/Explore-TechManual.pdf. Accessed April 9, 2017.
- American Institutes for Research. (2013–2014). State of Delaware End-of-Course (EOC) Technical Report. Evidence of Reliability and Validity (Vol. 4).Google Scholar
- Chaiklin, S. (2003). The zone of proximal development in Vygotsky’s analysis of learning and instruction. In A. Kozulin, B. Gindis, V.S. Ageyev, S.M. Miller (Eds.) Vygotsky’s educational theory and practice in clutural context (pp. 39–64). New York: Cambridge University Press.Google Scholar
- Falmagne, J.-Cl., Albert, D., Doble, C. W., Eppstein, D., Hu, X. (Eds.). (2013). Knowledge spaces: application in education. Berlin: Springer.Google Scholar
- Hambleton, R.K., Swaminathan, H., Rogers, H.J. (1991). Fundamentals of item response theory. Newbury Park: Sage Press.Google Scholar
- Hockemeyer, C. (2017). The collection of computer science bibliographies: bibliography on knowledge spaces. http://liinwww.ira.uka.de/bibliography/Ai/knowledge.spaces.html. Accessed February 14, 2017.
- Lynch, D., & Howlin, C.P. (2014). Real world usage of an adaptive testing algorithm to uncover latent knowledge. In 7th international conference of education, research and innovation (ICERI2014 Proceedings). Seville.Google Scholar
- Matayoshi, J., Granziol, U., Doble, C., Uzun, H., Cosyn, E. (2018). Forgetting curves and testing effect in an adaptive learning and assessment system. In Proceedings of the 11th international conference on educational data mining (pp. 607–612).Google Scholar
- Texas Education Agency. (2015–2016). Technical Digest. http://tea.texas.gov/Student_Testing_and_Accountability/Testing/Student_Assessment_Overview/Technical_Digest_2015-2016. Accessed April 12, 2017.
- Thissen, D. (2000). Reliability and measurement precision. In Computerized adaptive testing: a primer, 2nd edn (pp. 159–184). London: Lawrence Erlbaum Associates.Google Scholar
- Vie, J.-J., Popineau, F., Bruillard, E., Bourda, Y. (2017). A review of recent advances in adaptive assessment. In A. Peña-Ayala (Ed.) Learning analytics: fundaments, applications, and trends (pp. 113–142): Springer International Publishing.Google Scholar
- Vygotsky, L.S. (1978). Mind and society: the development of higher mental processes. Cambridge: Harvard University Press.Google Scholar
- Woodruff, D., Traynor, A., Cui, Z., Fang, Y. (2013). A comparison of three methods for computing scale score conditional standard errors of measurement. ACT Research Report Series No. 7.Google Scholar
- Zhu, Q., & Lowe, P.A. (2018). Split-half reliability. In Frey, B.B. (Ed.) The SAGE encyclopedia of educational research, measurement, and evaluation. Thousand Oaks: SAGE Publications.Google Scholar