Bayesian model checking in cognitive diagnostic models

Abstract

Checking that models adequately present data is an essential component of applied statistical inference. Psychometricans increasingly use complex models to analyze test taker’s responses. The appeal of using complex cognitive diagnostic models (CDMs) is undeniable, as psychometricians can fit and build models that represent complex cognitive processes in the test while simultaneous controlling observation errors. With a trend toward diagnosing fine-grained skills that are responsible for test performance, both new methods and extensions of existing methods of assessing person-fit in CDMs are required. Posterior predictive method (PP) is the most commonly used method in evaluating the effectiveness of person fit statistics in detecting aberrant response patterns in CDMs. In addition, a less known Bayesian model checking method, prior predictive posterior simulation method (PPPS), will also be used to investigate the effectiveness of chosen person-fit statistics. Three person-fit statistics, log-likelihood statistic ($l_{\text{z}}$), un-weighted between-set index (UB), and response conformity index (RCI) are chosen in this study.