Psychometrika

, Volume 73, Issue 3, pp 365–384 | Cite as

Bayesian Procedures for Identifying Aberrant Response-Time Patterns in Adaptive Testing

Theory and Methods

Abstract

In order to identify aberrant response-time patterns on educational and psychological tests, it is important to be able to separate the speed at which the test taker operates from the time the items require. A lognormal model for response times with this feature was used to derive a Bayesian procedure for detecting aberrant response times. Besides, a combination of the response-time model with a regular response model in an hierarchical framework was used in an alternative procedure for the detection of aberrant response times, in which collateral information on the test takers’ speed is derived from their response vectors. The procedures are illustrated using a data set for the Graduate Management Admission Test® (GMAT®). In addition, a power study was conducted using simulated cheating behavior on an adaptive test.

Keywords

adaptive testing Bayesian predictive checks cheating collateral information hierarchical modeling response times 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Albert, J.H. (1992). Bayesian estimation of normal-ogive item response curves using Gibbs sampling. Journal of Educational and Behavioral Statistics, 17, 261–269. CrossRefGoogle Scholar
  2. Bradlow, E.T., Weiss, R.E., & Cho, M. (1998). Bayesian detection of outliers in computerized adaptive tests. Journal of the American Statistical Association, 93, 910–919. CrossRefGoogle Scholar
  3. Casella, G., & Berger, R.L. (2002). Statistical inference (2nd ed.). Pacific Grove: Duxbury. Google Scholar
  4. Chang, H.-H., & Stout, W. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37–52. CrossRefGoogle Scholar
  5. Fisher, R.A. (1925). Statistical methods for research workers. Edinburgh: Oliver & Boyd. Google Scholar
  6. Gelman, A., Carlin, J.B, Stern, H., & Rubin, D.B. (1995). Bayesian data analysis. London: Chapman & Hall. Google Scholar
  7. Glas, C.A.W., & Meijer, R.R. (2003). A Bayesian approach to person fit analysis in item response theory models. Applied Psychological Measurement, 27, 217–233. CrossRefGoogle Scholar
  8. Johnson, V.E., & Albert, J.H. (1999). Ordinal data modeling. New York: Springer. Google Scholar
  9. Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading: Addison-Wesley. Google Scholar
  10. Meijer, R.R., & Sijtsma, K. (1995). Detection of aberrant item response patterns: A review of recent developments. Applied Measurement in Education, 8, 261–272. CrossRefGoogle Scholar
  11. Meijer, R.R., & Sijtsma, K. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107–135. CrossRefGoogle Scholar
  12. Miller, G.A. (1956). The magic number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81–97. CrossRefPubMedGoogle Scholar
  13. Owen, R.J. (1969). A Bayesian approach to tailored testing (Research Report 69-92). Princeton, NJ, Educational Testing Service. Google Scholar
  14. Owen, R.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70, 351–356. CrossRefGoogle Scholar
  15. van der Linden, W.J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31, 181–204. CrossRefGoogle Scholar
  16. van der Linden, W.J. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308. CrossRefGoogle Scholar
  17. van der Linden, W.J. (2008). Using response times for item selection in adaptive tests. Journal of Educational and Behavioral Statistics, 33. In press. Google Scholar
  18. van der Linden, W.J., & van Krimpen-Stoop, E.M.L.A. (2003). Using response times to detect aberrant response patterns in computerized adaptive testing. Psychometrika, 68, 251–265. CrossRefGoogle Scholar
  19. van der Linden, W.J., Scrams, D.J., & Schnipke, D.L. (1999). Using response-time constraints to control for speededness in computerized adaptive testing. Applied Psychological Measurement, 23, 195–210. CrossRefGoogle Scholar
  20. van Krimpen-Stoop, E.M.L.A., & Meijer, R.R. (2001). CUSUM-based person fit statistics for adaptive testing. Journal of Educational and Behavioral Statistics, 26, 199–218. CrossRefGoogle Scholar

Copyright information

© The Psychometric Society 2008

Authors and Affiliations

  1. 1.Department of Research Methodology, Measurement, and Data AnalysisUniversity of TwenteEnschedeThe Netherlands
  2. 2.Graduate Management Admission CouncilMcLeanUSA

Personalised recommendations