Skip to main content

Advertisement

SpringerLink
Log in

Modeling Differences Between Response Times of Correct and Incorrect Responses

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

While standard joint models for response time and accuracy commonly assume the relationship between response time and accuracy to be fully explained by the latent variables of the model, this assumption of conditional independence is often violated in practice. If such violations are present, taking these residual dependencies between response time and accuracy into account may both improve the fit of the model to the data and improve our understanding of the response processes that led to the observed responses. In this paper, we propose a framework for the joint modeling of response time and accuracy data that allows for differences in the processes leading to correct and incorrect responses. Extensions of the standard hierarchical model (van der Linden in Psychometrika 72:287–308, 2007. https://doi.org/10.1007/s11336-006-1478-z) are considered that allow some or all item parameters in the measurement model of speed to differ depending on whether a correct or an incorrect response was obtained. The framework also allows one to consider models that include two speed latent variables, which explain the patterns observed in the responses times of correct and of incorrect responses, respectively. Model selection procedures are proposed and evaluated based on a simulation study, and a simulation study investigating parameter recovery is presented. An application of the modeling framework to empirical data from international large-scale assessment is considered to illustrate the relevance of modeling possible differences between the processes leading to correct and incorrect responses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. Since these models only consider observed RT, the latent variable in the measurement model for RT only captures response speed: The degree to which a person displays the tendency to provide responses quickly rather than slowly. This tendency should not be equated with cognitive speed (however defined exactly).

  2. For clarification it may be relevant to point out that each of these approaches models conditional dependence between RA and RT, which may arise due to many sources and need not be reducible or even linked to a speed–accuracy trade-off (Bolsinova, Tijmstra, Molenaar, & De Boeck, 2017b). Hence, none of these approaches specifically attempt to model the speed–accuracy trade-off.

  3. Note that van der Linden and Glas  (2010) considered a constrained version of the one-factor model in which the factor loadings of all items are equal to 1, and the variance of the speed latent variable is freely estimated.

  4. Since we do not include any strong prior information in the estimation of the parameters, the posterior means would be very close to the maximum likelihood estimates, meaning that the difference between both the AIC and mAIC and the BIC and mBIC can be expected to be minimal.

  5. For each item the responses were divided in bins based on RTs with 5 s per bin. The proportions of correct responses were computed per bin. The lower bound of the first bin in which the proportion of correct responses was above zero was used a threshold for disengaged responses. In total, 8.35% of the observed responses were flagged as disengaged.

  6. This measure is aimed at capturing the possible differences between the RTs of correct and incorrect response in how strongly they are correlated among each other, and hence whether depending on RA the RTs show stronger or weaker structural patterns.

References

  • Albert, J. (1992). Bayesian estimation of Normal Ogive item response curves using Gibbs sampling. Journal of Educational Statistics, 17, 251–269. https://doi.org/10.2307/1165149.

    Article  Google Scholar 

  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 395–479). Reading: Addison-Wesley.

    Google Scholar 

  • Bolsinova, M., De Boeck, P., & Tijmstra, J. (2017). Modelling conditional dependence between response time and accuracy. Psychometrika, 82(4), 1126–1148.

    Article  Google Scholar 

  • Bolsinova, M., & Tijmstra, J. (2016). Posterior predictive checks for conditional independence between response time and accuracy. Journal of Educational and Behavioral Statistics, 41(2), 123–145.

    Article  Google Scholar 

  • Bolsinova, M., & Tijmstra, J. (2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71(1), 13–38.

    Article  Google Scholar 

  • Bolsinova, M., Tijmstra, J., & Molenaar, D. (2017a). Response moderation models for conditional dependence between response time and response accuracy. British Journal of Mathematical and Statistical Psychology, 70(2), 257–279.

    Article  Google Scholar 

  • Bolsinova, M., Tijmstra, J., Molenaar, D., & De Boeck, P. (2017b). Conditional dependence between response time and accuracy: An overview of its possible sources and directions for distinguishing between them. Frontiers in Psychology, 8, 202. https://doi.org/10.3389/fpsyg.2017.00202.

  • Gelman, A., Meng, X., & Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 4, 733–807.

    Google Scholar 

  • Goldhammer, F., Martens, T., Christoph, G., & Lüdtke, O. (2016). Test-taking engagement in PIAAC. OECD Education Working Papers, (133). https://doi.org/10.1787/5jlzfl6fhxs2-en

  • Hoff, P. D. (2009). A first course in Bayesian statistical methods. New York: Spinger. https://doi.org/10.1007/978-0-387-92407-6.

    Book  Google Scholar 

  • Klein Entink, R., Fox, J., & van der Linden, W. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74, 21–48.

    Article  Google Scholar 

  • Klein Entink, R., van der Linden, W. J., & Fox, J.-P. (2009). A Box–Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62(3), 621–640.

    Article  Google Scholar 

  • Lord, F., & Novick, M. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

    Google Scholar 

  • Meng, X. B., Tao, J., & Chang, H. H. (2015). A conditional joint modeling approach for locally dependent item responses and response times. Journal of Educational Measurement, 52, 1–27. https://doi.org/10.1111/jedm.12060.

    Article  Google Scholar 

  • Meng, X.-L. (1994). Posterior predictive p-values. The Annals of Statistics, 22(3), 1142–1160.

    Article  Google Scholar 

  • Molenaar, D., Tuerlinckx, F., & van der Maas, H. (2015a). A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times. Multivariate Behavioural Research, 50, 56–74. https://doi.org/10.1080/00273171.2014.962684.

    Article  Google Scholar 

  • Molenaar, D., Tuerlinckx, F., & van der Maas, H. (2015b). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68, 197–219. https://doi.org/10.1111/bmsp.12042.

    Article  PubMed  Google Scholar 

  • OECD. (2012). Literacy, numeracy and problem solving in technology-rich environments. Paris: OECD Publishing. https://doi.org/10.1787/9789264128859-en.

  • Partchev, I., & De Boeck, P. (2012). Can fast and slow intelligence be differentiated? Intelligence, 40, 23–32. https://doi.org/10.1016/j.intell.2011.11.002.

    Article  Google Scholar 

  • Ranger, J., & Ortner, T. (2012). The case of dependency of responses and response times: A modeling approach based on standard latent trait models. Psychological Test and Assessment Modeling, 54, 128–148.

    Google Scholar 

  • Rouder, J. N., Sun, D., Speckman, P. L., Lu, J., & Zhou, D. (2003). A hierarchical bayesian statistical framework for response time distributions. Psychometrika, 68(4), 589–606.

    Article  Google Scholar 

  • Scherer, R., Greiff, S., & Hautamäki, J. (2015). Exploring the relation between time on task and ability in complex problem solving. Intelligence, 48, 37–50.

    Article  Google Scholar 

  • Semmes, R., Davison, M. L., & Close, C. (2011). Modeling individual differences in numerical reasoning speed as a random effect of response time limits. Applied Psychological Measurement, 35(6), 433–446.

    Article  Google Scholar 

  • Tanner, M., & Wong, W. (1987). The calculation of posterior distributions by data augmentation. Journal of the American Statistical Association, 82, 528–540. https://doi.org/10.2307/2289457.

    Article  Google Scholar 

  • van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioural Statistics, 31, 181–204. https://doi.org/10.3102/10769986031002181.

    Article  Google Scholar 

  • van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308. https://doi.org/10.1007/s11336-006-1478-z.

    Article  Google Scholar 

  • van der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46(3), 247–272.

    Article  Google Scholar 

  • van der Linden, W. J., & Glas, C. A. W. (2010). Statistical tests of conditional independence between responses and/or response times on test items. Psychometrika, 75(1), 120–139. https://doi.org/10.1007/s11336-009-9129-9.

    Article  Google Scholar 

  • van der Linden, W. J., Klein Entink, R. H., & Fox, J. P. (2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34, 327–347. https://doi.org/10.1177/0146621609349800.

    Article  Google Scholar 

  • van der Maas, H. L., & Jansen, B. R. (2003). What response times tell of children’s behavior on the balance scale task. Journal of Experimental Child Psychology, 85(2), 141–177.

    Article  Google Scholar 

  • van der Maas, H. L., & Wagenmakers, E.-J. (2005). A psychometric analysis of chess expertise. The American journal of psychology, 118, 29–60.

    PubMed  Google Scholar 

  • Van de Schoot, R., Lugtig, P., & Hox, J. (2012). A checklist for testing measurement invariance. European Journal of Developmental Psychology, 9(4), 486–492.

    Article  Google Scholar 

  • Wang, C., Chang, H. H., & Douglas, J. A. (2013a). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66(1), 144–168.

    Article  Google Scholar 

  • Wang, C., Fan, Z., Chang, H. H., & Douglas, J. A. (2013b). A semiparametric model for jointly analyzing response times and accuracy in computerized testing. Journal of Educational and Behavioral Statistics, 38(4), 381–417.

    Article  Google Scholar 

  • Zhan, P., Jiao, H., & Liao, D. (2017). Cognitive diagnosis modelling incorporating item response times. British Journal of Mathematical and Statistical Psychology, 71, 262–286.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. ACTNext, 500 ACT dr., Iowa City, IA, 52243, USA

    Maria Bolsinova

  2. Tilburg University, Tilburg, The Netherlands

    Jesper Tijmstra

Authors
  1. Maria Bolsinova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jesper Tijmstra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Bolsinova.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 147 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bolsinova, M., Tijmstra, J. Modeling Differences Between Response Times of Correct and Incorrect Responses. Psychometrika 84, 1018–1046 (2019). https://doi.org/10.1007/s11336-019-09682-5

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-019-09682-5

Keywords

Search

Navigation