Skip to main content
SpringerLink
Log in

Response Mixture Modeling: Accounting for Heterogeneity in Item Characteristics across Response Times

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

In item response theory modeling of responses and response times, it is commonly assumed that the item responses have the same characteristics across the response times. However, heterogeneity might arise in the data if subjects resort to different response processes when solving the test items. These differences may be within-subject effects, that is, a subject might use a certain process on some of the items and a different process with different item characteristics on the other items. If the probability of using one process over the other process depends on the subject’s response time, within-subject heterogeneity of the item characteristics across the response times arises. In this paper, the method of response mixture modeling is presented to account for such heterogeneity. Contrary to traditional mixture modeling where the full response vectors are classified, response mixture modeling involves classification of the individual elements in the response vector. In a simulation study, the response mixture model is shown to be viable in terms of parameter recovery. In addition, the response mixture model is applied to a real dataset to illustrate its use in investigating within-subject heterogeneity in the item characteristics across response times.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. \(\tilde{\pi }\) is calculated using the formula given above, \(\tilde{\uppi } =\frac{1}{\hbox {N}\times \hbox {n}}\mathop \sum \nolimits _{p=1}^N \mathop \sum \nolimits _{i=1}^n \pi _{pi} \), with \(\uppi _{\mathrm{pi}}\) calculated using Eq. 5 in which item parameters \(\upzeta _{0},\upzeta _{1},\uplambda _{i}\), and \(\upsigma _{i}\) are substituted by their posterior means, and nuisance parameters \(\tau _{p}\) are numerically integrated out using a normal distribution for \(\uptau _{p}\).

References

  • Bacci, S., Pandolfi, S., & Pennoni, F. (2014). A comparison of some criteria for states selection in the latent Markov model for longitudinal data. Advances in Data Analysis and Classification, 8(2), 125–145.

    Article  Google Scholar 

  • Bartholomew, D. J., Knott, M., & Moustaki, I. (2011). Latent variable models and factor analysis: An unified approach (Vol. 904). Hoboken: Wiley.

    Book  Google Scholar 

  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In E. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores (chap) (pp. 17–20). Reading, MA: Addison Wesley.

    Google Scholar 

  • Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459.

    Article  Google Scholar 

  • Bolsinova, M., & Maris, G. (2016). A test for conditional independence between response time and accuracy. British Journal of Mathematical and Statistical Psychology, 69(1), 62–79.

    Article  Google Scholar 

  • Bolsinova, M., & Tijmstra, J. (2016). Posterior predictive checks for conditional independence between response time and accuracy. Journal of Educational and Behavioral Statistics, 41(2), 123–145.

    Article  Google Scholar 

  • Bolsinova, M., de Boeck, P., & Tijmstra, J. (in press). Modelling conditional dependence between response time and accuracy. Psychometrika.

  • Bolsinova, M., Tijmstra, J., Molenaar, D., & De Boeck, P. (2017). Conditional dependence between response time and accuracy: An overview of its possible sources and directions for distinguishing between them. Frontiers in Psychology, 8, 202.

    Article  PubMed  PubMed Central  Google Scholar 

  • Carpenter, P. A., Just, M. A., & Shell, P. (1990). What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices Test. Psychological Review, 97, 404–431.

    Article  PubMed  Google Scholar 

  • Celeux, G., Forbes, F., Robert, C. P., & Titterington, D. M. (2006). Deviance information criteria for missing data models. Bayesian Analysis, 1(4), 651–673.

    Article  Google Scholar 

  • Cohen, J. (1983). The cost of dichotomization. Applied Psychological Measurement, 7, 249–253.

    Article  Google Scholar 

  • De Boeck, P., Chen, H., & Davison, M. (2017). Spontaneous and imposed speed of cognitive test responses. British Journal of Mathematical and Statistical Psychology, 70(2), 225–237.

    Article  PubMed  Google Scholar 

  • De Boeck, P., & Partchev, I. (2012). IRTrees: Tree-based item response models of the GLMM family. Journal of Statistical Software, 48(1), 1–28.

    Google Scholar 

  • DiTrapani, J., Jeon, M., De Boeck, P., & Partchev, I. (2016). Attempting to differentiate fast and slow intelligence: Using generalized item response trees to examine the role of speed on intelligence tests. Intelligence, 56, 82–92.

    Article  Google Scholar 

  • Dolan, C. V., Colom, R., Abad, F. J., Wicherts, J. M., Hessen, D. J., & van de Sluis, S. (2006). Multi- group covariance and mean structure modeling of the relationship between the WAIS-III common factors and sex and educational attainment in Spain. Intelligence, 34, 193–210.

    Article  Google Scholar 

  • Dolan, C. V., & van der Maas, H. L. (1998). Fitting multivariage normal finite mixtures subject to structural equation modeling. Psychometrika, 63(3), 227–253.

    Article  Google Scholar 

  • Ericsson, K. A., & Staszewski, J. J. (1989). Skilled memory and expertise: Mechanisms of exceptional performance. In D. Klahr & K. Kotovsky (Eds.), Complex information processing: The impact of Herbert A. Simon. NY: Hillsdale: Erlbaum.

    Google Scholar 

  • Ferrando, P. J., & Lorenzo-Seva, U. (2007a). An item response theory model for incorporating response time data in binary personality items. Applied Psychological Measurement, 31, 525–543.

    Article  Google Scholar 

  • Ferrando, P. J., & Lorenzo-Seva, U. (2007b). A measurement model for Likert responses that incorporates response time. Multivariate Behavioral Research, 42, 675–706.

    Article  Google Scholar 

  • Fox, J. P., Klein Entink, R., & Linden, W. (2007). Modeling of responses and response times with the package cirt. Journal of Statistical Software, 20(7), 1–14.

    Article  Google Scholar 

  • Grabner, R. H., Ansari, D., Koschutnig, K., Reishofer, G., Ebner, F., & Neuper, C. (2009). To retrieve or to calculate? Left angular gyrus mediates the retrieval of arithmetic facts during problem solving. Neuropsychologia, 47(2), 604–608.

    Article  PubMed  Google Scholar 

  • Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7, 457–511.

    Article  Google Scholar 

  • Gilks, W. R., Richardson, S., & Spiegelhalter, D. J. (1996). Introducing markov chain monte carlo. In: Markov chain Monte Carlo in practice (pp. 1–19). US: Springer.

  • Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights From a computer-based large-scale assessment. Journal of Educational Psychology, 106, 608–626.

    Article  Google Scholar 

  • Hall, D. B. (2000). Zero-inflated Poisson and binomial regression with random effects: A case study. Biometrics, 56(4), 1030–1039.

    Article  PubMed  Google Scholar 

  • Jansen, B. R., & Van der Maas, H. L. (2001). Evidence for the phase transition from Rule I to Rule II on the balance scale task. Developmental Review, 21(4), 450–494.

    Article  Google Scholar 

  • Jeon, M., & De Boeck, P. (2016). A generalized item response tree model for psychological assessments. Behavior Research Methods, 48(3), 1070–1085.

    Article  PubMed  Google Scholar 

  • Larson, G. E., & Alderton, D. L. (1990). Reaction time variability and intelligence: A “worst performance” analysis of individual differences. Intelligence, 14(3), 309–325.

    Article  Google Scholar 

  • Lubke, G. H., & Muthén, B. (2005). Investigating population heterogeneity with factor mixture models. Psychological Methods, 10(1), 21.

    Article  PubMed  Google Scholar 

  • Luce, R. D. (1986). Response times. New York: Oxford University Press.

    Google Scholar 

  • MacCallum, R. C., Zhang, S., Preacher, K. J., & Rucker, D. D. (2002). On the practice of dichotomization of quantitative variables. Psychological Methods, 7, 19–40.

    Article  PubMed  Google Scholar 

  • McLeod, L., Lewis, C., & Thissen, D. (2003). A Bayesian method for the detection of item preknowledge in computerized adaptive testing. Applied Psychological Measurement, 27(2), 121–137.

    Article  Google Scholar 

  • Mellenbergh, G. J. (1994). Generalized linear item response theory. Psychological Bulletin, 115, 300–300.

    Article  Google Scholar 

  • Min, Y., & Agresti, A. (2005). Random effect models for repeated measures of zero-inflated count data. Statistical Modelling, 5(1), 1–19.

    Article  Google Scholar 

  • Molenaar, D., Oberski, D., Vermunt, J., & De Boeck, P. (2016). Hidden Markov item response theory models for responses and response times. Multivariate Behavioral Research, 51(5), 606–626.

    Article  PubMed  Google Scholar 

  • Molenaar, D., Tuerlinckx, F., & van der Maas, H. L. J. (2015). A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times. Multivariate Behavioral Research, 50(1), 56–74.

    Article  PubMed  Google Scholar 

  • Partchev, I., & De Boeck, P. (2012). Can fast and slow intelligence be differentiated? Intelligence, 40(1), 23–32.

    Article  Google Scholar 

  • Plummer, M. (2003). JAGS: A program for analysis of Bayesian graphical models using Gibbs sampling. In Proceedings of the 3rd international workshop on distributed statistical computing (DSC 2003) (pp. 20–22).

  • Rabbitt, P. (1979). How old and young subjects monitor and control responses for accuracy and speed. British Journal of Psychology, 70, 305–311.

    Article  Google Scholar 

  • Ranger, J., & Ortner, T. M. (2011). Assessing personality traits through response latencies using item response theory. Educational and Psychological Measurement, 71(2), 389–406.

    Article  Google Scholar 

  • Rummel, R. J. (1970). Applied factor analysis. Evanston: Northwestern University Press.

    Google Scholar 

  • Schnipke, D. L., & Scrams, D. J. (1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34, 213–232.

    Article  Google Scholar 

  • Shiffrin, R. M., & Schneider, W. (1977). Controlled and automatic human information processing: II. Perceptual learning, automatic attending and a general theory. Psychological Review, 84, 127.

    Article  Google Scholar 

  • Spiegelhalter, D. J., Thomas, A., Best, N. G., & Gilks, W. R. (1995). BUGS: Bayesian inference using Gibbs sampling, Version 0.50. Cambridge: MRC Biostatistics Unit.

    Google Scholar 

  • Spiegelhalter, D. J., Best, N. G.-, Carlin, B. P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit (with discussion). Journal of the Royal Statistical Society Series B, 64, 583–640.

    Article  Google Scholar 

  • Stan Development Team. (2015). Stan modeling language users guide and reference manual. Version 2(9)

  • Siegler, R. S. (1976). Three aspects of cognitive development. Cognitive Psychology, 8, 481–520.

    Article  Google Scholar 

  • Thissen, D. (1983). Timed testing: An approach using item response testing. In D. J. Weiss (Ed.), New horizons in testing: Latent trait theory and computerized adaptive testing (pp. 179–203). New York: Academic Press.

    Google Scholar 

  • Thomas, A., Hara, B. O., Ligges, U., & Sturtz, S. (2006). Making BUGS open. R News, 6, 12–17.

    Google Scholar 

  • Titterington, D. M., Smith, A. F. M., & Makov, U. E. (1985). Statistical analysis of finite mixture distributions. Chicester: Wiley.

    Google Scholar 

  • Tuerlinckx, F., & De Boeck, P. (2005). Two interpretations of the discrimination parameter. Psychometrika, 70(4), 629–650.

    Article  Google Scholar 

  • Van Harreveld, F., Wagenmakers, E. J., & Van Der Maas, H. L. (2007). The effects of time pressure on chess skill: An investigation into fast and slow processes underlying expert performance. Psychological Research, 71, 591–597.

    Article  PubMed  Google Scholar 

  • Van der Linden, W. J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308.

    Article  Google Scholar 

  • Van der Linden, W. J. (2009). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46(3), 247–272.

    Article  Google Scholar 

  • Van der Linden, W. J., Breithaupt, K., Chuah, S. C., & Zhang, Y. (2007). Detecting differential speededness in multistage testing. Journal of Educational Measurement, 44(2), 117–130.

    Article  Google Scholar 

  • Van der Linden, W. J., & Glas, C. A. (2010). Statistical tests of conditional independence between responses and/or response times on test items. Psychometrika, 75, 120–139.

    Article  Google Scholar 

  • Van der Linden, W. J., & Guo, F. (2008). Bayesian procedures for identifying aberrant response-time patterns in adaptive testing. Psychometrika, 73(3), 365–384.

    Article  Google Scholar 

  • Van der Linden, W. J., Klein Entink, R. H., & Fox, J. P. (2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34(5), 327–347.

    Article  Google Scholar 

  • van der Linden, W. J., & van Krimpen-Stoop, E. M. (2003). Using response times to detect aberrant responses in computerized adaptive testing. Psychometrika, 68(2), 251–265.

    Article  Google Scholar 

  • van der Maas, H. L., Molenaar, D., Maris, G., Kievit, R. A., & Borsboom, D. (2011). Cognitive psychology meets psychometric theory: On the relation between process models for decision making and latent variable models for individual differences. Psychological review, 118(2), 339–356.

    Article  PubMed  Google Scholar 

  • Wang, C., & Xu, G. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68, 456–477.

    Article  PubMed  Google Scholar 

  • Wang, C., Xu, G., & Shang, Z. (2016). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika. https://doi.org/10.1007/s11336-016-9525-x.

  • Watanabe, S. (2010). Asymptotic equivalence of Bayes cross validation and widely applicable Information criterion in singular learning theory. Journal of Machine Learning Research, 11, 3571–3594.

    Google Scholar 

  • Yung, Y. F. (1997). Finite mixtures in confirmatory factor-analysis models. Psychometrika, 62(3), 297–330.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Psychological Methods, Department of Psychology, University of Amsterdam, PO box 15906, 1001 NK , Amsterdam, The Netherlands

    Dylan Molenaar

  2. Ohio State University, Columbus, OH, USA

    Paul de Boeck

Authors
  1. Dylan Molenaar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul de Boeck
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dylan Molenaar.

Additional information

The research by Dylan Molenaar was made possible by a grant from the Netherlands Organization for Scientific Research (NWO VENI- 451-15-008). We are grateful to three anonymous reviewers whose comments led to substantial improvements of this paper.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 6 KB)

Appendix: OpenBUS Syntax to Fit the Full Response Mixture Model

Appendix: OpenBUS Syntax to Fit the Full Response Mixture Model

figure a

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Molenaar, D., de Boeck, P. Response Mixture Modeling: Accounting for Heterogeneity in Item Characteristics across Response Times. Psychometrika 83, 279–297 (2018). https://doi.org/10.1007/s11336-017-9602-9

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-017-9602-9

Keywords

Search

Navigation