Abstract
Mode comparability between paper and online versions of a test cannot be simply assumed. This paper presents the research designs, statistical analyses, and major findings from a series of special studies intended to ensure score comparability for a high-stakes testing program, including an online timing study and two mode comparability studies as well as a general framework that guided the design of these studies. The framework views score comparability as a matter of degree and the evaluation of score comparability as a matter of score validation. The high-stakes uses of the test scores required stringent score comparability which was obtained by applying test-equating methodologies under a randomly equivalent groups design. Meanwhile, score equivalency and construct equivalency were examined through statistical analyses of test results and responses to survey questions. The comparability framework and results from these studies may provide guidance for other testing programs transitioning from paper to online or when evaluating score comparability in general.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
G. Camilli, L.A. Shepard, Methods for Identifying Biased Test Items (SAGE Publications, Thousand Oaks, CA, 1994)
M.T. Kane, in Educational Measurement, 4th edn., ed. by R. L. Brennan. Validation (American Council on Education and Praeger, Westport, CT, 2006), pp. 17–64
N.M. Kingston, Comparability of computer- and paper-administered multiple-choice tests for k-12 populations: a synthesis. Appl. Meas. Educ. 22, 22–37 (2009)
M.J. Kolen, Threats to score comparability with applications to performance assessments and computerized adaptive tests. Educ. Assess. 6(2), 73–96 (1999)
M. Kolen, R. Brennan, Test Equating, Scaling, and Linking: Methods and Practices, 3rd edn. (Springer, New York, NY, 2014)
H.V. Leeson, The mode effect: a literature review of human and technological issues in computerized testing. Int. J. Test. 6(1), 1–24 (2006)
S. Lottridge, A. Nicewander, M. Schulz, H. Mitzel, Comparability of paper-based and computer-based tests: a review of the methodology. Paper submitted to the CCSSO Technical Issues in Large Scale Assessment Comparability Research Group, 2008. Retrieved fromhttps://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnxwYXBlcn ZlcnN1c3NjcmVlbnxneDoxNTU0NmE0NDY0NTQ4MzA4
N. Mantel, W. Haenszel, Statistical aspects of the analysis of data from retrospective studies of disease. J. Natl. Cancer Inst. 22, 719–748 (1959)
J. Mazzeo, A.L. Harvey, The Equivalence of Scores from Automated and Conventional Educational and Psychological Tests: A Review of the Literature (College Board Report No. 88–8; ETS RR No. 88–21) (College Entrance Examination Board, New York, NY, 1988)
A.D. Mead, F. Drasgow, Equivalence of computerized and paper-and-pencil cognitive ability tests: a meta-analysis. Psychol. Bull. 114, 449–458 (1993)
A. Mroch, D. Li, T. Thompson, A framework for evaluating score comparability. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL, 2015
P. Mutler, in Cognitive Aspects of Electronic Next Processing, ed. by H. van Oostendorp, S. de Mul. Interface design and optimization of reading of continuous text (Ablex, Norwood, NJ, 1996), pp. 161–180
C.G. Parshall, J.A. Spray, J.C. Kalohn, T. Davey, Practical Considerations in Computer-Based Testing (Springer-Verlag, New York, NY, 2002)
M. Pommerich, Developing computerized versions of paper-and-pencil tests: mode effects for passage-based tests. J. Technol. Learn. Assess. 2(6) (2004.) Available from http://ejournals.bc.edu/ojs/index.php/jtla/article/view/1666/1508
U. Schroeders, O. Wilhelm, Equivalence of reading and listening comprehension across test media. Educ. Psychol. Meas. 71(5), 849–869 (2011)
T. Wang, M.J. Kolen, Evaluating comparability in computerized adaptive testing: issues, criteria and an example. J. Educ. Meas. 38, 19–49 (2001)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Li, D., Yi, Q., Harris, D.J. (2017). Mode Comparability Studies for a High-Stakes Testing Program. In: van der Ark, L.A., Wiberg, M., Culpepper, S.A., Douglas, J.A., Wang, WC. (eds) Quantitative Psychology. IMPS 2016. Springer Proceedings in Mathematics & Statistics, vol 196. Springer, Cham. https://doi.org/10.1007/978-3-319-56294-0_31
Download citation
DOI: https://doi.org/10.1007/978-3-319-56294-0_31
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-56293-3
Online ISBN: 978-3-319-56294-0
eBook Packages: Mathematics and StatisticsMathematics and Statistics (R0)