Abstract
Current modeling of response times on test items has been strongly influenced by the paradigm of experimental reaction-time research in psychology. For instance, some of the models have a parameter structure that was chosen to represent a speed-accuracy tradeoff, while others equate speed directly with response time. Also, several response-time models seem to be unclear as to the level of parametrization they represent. A hierarchical framework for modeling speed and accuracy on test items is presented as an alternative to these models. The framework allows a “plug-and-play approach” with alternative choices of models for the response and response-time distributions as well as the distributions of their parameters. Bayesian treatment of the framework with Markov chain Monte Carlo (MCMC) computation facilitates the approach. Use of the framework is illustrated for the choice of a normal-ogive response model, a lognormal model for the response times, and multivariate normal models for their parameters with Gibbs sampling from the joint posterior distribution.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Albert, J.H. (1992). Bayesian estimation of normal-ogive item response curves using Gibbs sampling. Journal of Educational and Behavioral Statistics, 17, 261–269.
Béguin, A.A., & Glas, C.A.W. (2001). MCMC estimation and some fit analysis of multidimensional IRT models. Psychometrika, 66, 541–562.
Carlin, B.P., & Louis, T.A. (2000). Bayes and Empirical Bayes Methods for Data Analysis. Boca Raton, FL: Chapman & Hall.
Douglas, J., Kosorok, M., & Chewning, B. (1999). A latent variable model for multivariate psychometric response times. Psychometrika, 64, 69–82.
Dubey, S.D. (1969). A new derivation of the logistic distribution. Naval Research Logistics Quarterly, 16, 37–40.
Fox, J.P., & Glas, C.A.W. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 271–288.
Glas, C.A.W., & van der Linden, W.J. (2006). Modeling variability in item parameters in item response models. Psychometrika. Submitted.
Jansen, M.G.H. (1986). A Bayesian version of Rasch’s multiplicative Poisson model for the number of errors on achievement tests. Journal of Educational Statistics, 11, 51–65.
Jansen, M.G.H. (1997a). Rasch model for speed tests and some extensions with applications to incomplete designs. Journal of Educational and Behavioral Statistics, 22, 125–140.
Jansen, M.G.H. (1997b). Rasch’s model for reading speed with manifest exploratory variables. Psychometrika, 62, 393–409.
Jansen, M.G.H., & Duijn, M.A.J. (1992). Extensions of Rasch’s multiplicative Poisson model. Psychometrika, 57, 405–414.
Johnson, V.E., & Albert, J.H. (1999). Ordinal Data Modeling. New York: Springer-Verlag.
Luce, R.D. (1986). Response times: Their Roles in Inferring Elementary Mental Organization. Oxford, UK: Oxford University Press.
Maris, E. (1993). Additive and multiplicative models for gamma distributed variables, and their application as psychometric models for response times. Psychometrika, 58, 445–469.
Oosterloo, S.J. (1975). Modellen voor Reaktie-tijden [Models for Reaction Times]. Unpublished master’s thesis, Faculty of Psychology, University of Groningen, The Netherlands.
Patz, R.J., & Junker, B.W. (1999). Applications and extensions of MCMC in IRT: Mulitple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 34, 342–366.
Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: The University of Chicago Press. (Original published in 1960).
Roskam, E.E. (1987). Toward a psychometric theory of intelligence. In E.E. Roskam & R. Suck (Eds.), Progress in Mathematical Psychology (pp. 151–171). Amsterdam: North-Holland.
Roskam, E.E. (1997). Models for speed and time-limit tests. In W.J. van der Linden & R.K. Hambleton (Eds.), Handbook of Modern Item Response Theory (pp. 187–208). New York: Springer.
Rouder, J.N., Sun, D., Speckman, P.L., Lu, J., & Zhou, D. (2003). A hierarchical Bayesian statistical framework for response time distributions. Psychometrika, 68, 589–606.
Scheiblechner, H. (1979). Specific objective stochastic latency mechanisms. Journal of Mathematical Psychology, 19, 18–38.
Scheiblechner, H. (1985). Psychometric models for speed-test construction: The linear exponential model. In S.E. Embretson (Ed.), Test design: Developments in psychology and education (pp. 219–244). New York: Academic Press.
Schnipke, D.L., & Scrams, D.J. (1997). Modeling response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213–232.
Schnipke, D.L., & Scrams, D.J. (1999). Representing response time information in item banks (LSAC Computerized Testing Report No. 97-09). Newtown, PA: Law School Admission Council.
Schnipke, D.L., & Scrams, D.J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In C.N. Mills, M. Potenza, J.J. Fremer & W. Ward (Eds.), Computer-Based Testing: Building the Foundation for Future Assessments (pp. 237–266). Hillsdale, NJ: Lawrence Erlbaum Associates.
Swanson, D.B., Featherman, C.M., Case, S.M., Luecht, R.M., & Nungester, R. (1999, March). Relationship of response latency to test design, examinee proficiency and item difficulty in computer-based test administration. Paper presented at the Annual Meeting of the National Council on Measurement in Education, Chicago, IL.
Swanson, D.B., Case, S.E., Ripkey, D.R., Clauser, B.E., & Holtman, M.C. (2001). Relationships among item characteristics, examinee characteristics, and response times on USMLE, Step 1. Academic Medicine, 76, 114–116.
Tatsuoka, K.K., & Tatsuoka, M.M. (1980). A model for incorporating response-time data in scoring achievement tests. In D.J. Weiss (Ed.), Proceedings of the 1979 Computerized Adaptive Testing Conference (pp. 236–256). Minneapolis, MN: University of Minnesota, Department of Psychology, Psychometric Methods Program.
Thissen, D. (1983). Timed testing: An approach using item response theory. In D.J. Weiss (Ed.), New Horizons in Testing: Latent Trait Test Theory and Computerized Adaptive Testing. New York: Academic Press.
Townsend, J.T., & Ashby, F.G. (1983). Stochastic Modeling of Elementary Psychological Processes. Cambridge, UK: Cambridge University Press.
van Breukelen, G.J.P. (2005). Psychometric modeling of response speed and accuracy with mixed and conditional regression. Psychometrika, 70, 359–376.
van der Linden, W.J. (2005). Linear Models for Optimal Test Design. New York: Springer-Verlag.
van der Linden, W.J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181–204.
van der Linden, W.J. (2007a). Conceptual Issues in Response-Time Modeling. Submitted.
van der Linden, W.J. (2007b). Using response times for item selection in adaptive tests. Journal of Educational and Behavioral Statistics, 32.
van der Linden, W.J., & Guo, F. (2006). Two Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing. Manuscript submitted for publication.
van der Linden, W.J., & Hambleton, R.K. (1997). Handbook of Modern Item Response Theory. New York: Springer-Verlag.
van der Linden, W.J., Breithaupt, K., Chuah, S.C., & Zhang, Y. (2007). Detecting differential speededness in multistage testing. Journal of Educational Measurement, 44, in press.
van der Linden, W.J., Klein Entink, R.H., & Fox, J.-P. (2006). IRT Parameter Estimation with Response Times as Collateral Information. Manuscript submitted for publication.
van der Linden, W.J., Scrams, D.J., & Schnipke, D.L. (1999). Using response-time constraints to control for speededness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210.
Verhelst, N.D., Verstralen, H.H.F.M., & Jansen, M.G. (1997). A logistic model for time-limit tests. In W.J. van der Linden & R.K. Hambleton (Eds.), Handbook of Modern Item Response Theory (pp. 169–185). New York: Springer-Verlag.
Author information
Authors and Affiliations
Corresponding author
Additional information
This study received funding from the Law School Admission Council (LSAC). The opinions and conclusions contained in this paper are those of the author and do not necessarily reflect the policy and position of LSAC. The author is indebted to the American Institute of Certified Public Accountants for the data set in the empirical example and to Rinke H. Klein Entink for his computational assistance
Rights and permissions
About this article
Cite this article
van der Linden, W.J. A Hierarchical Framework for Modeling Speed and Accuracy on Test Items. Psychometrika 72, 287–308 (2007). https://doi.org/10.1007/s11336-006-1478-z
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-006-1478-z