Addressing model uncertainty in item response theory person scores through model averaging

Abstract

Item banks are often created in large-scale research and testing settings in the social sciences to predict individuals’ latent trait scores. A common procedure is to fit multiple candidate item response theory (IRT) models to a calibration sample and select a single best-fitting IRT model. The parameter estimates from this model are then used to obtain trait scores for subsequent respondents. However, this model selection procedure ignores model uncertainty stemming from the fact that the model ranking in the calibration phase is subject to sampling variability. Consequently, the standard errors of trait scores obtained from subsequent respondents do not reflect such uncertainty. Ignoring such sources of uncertainty contributes to the current replication crisis in the social sciences. In this article, we propose and demonstrate an alternative procedure to account for model uncertainty in this context—model averaging of IRT trait scores and their standard errors. We outline the general procedure step-by-step and provide software to aid researchers in implementation, both for large-scale research settings with item banks and for smaller research settings involving IRT scoring. We then demonstrate the procedure with a simulated item-banking illustration, comparing model selection and model averaging within sample in terms of predictive coverage. We conclude by discussing ways that model averaging and IRT scoring can be used and investigated in future research.

Appendix: modelavgIRT R function

1.1 modelavgIRT R function description

This function reads in person scores (e.g., EAP scores) and their standard errors from the validation sample, and information criteria values (BIC, AIC) from the calibration sample from each of a set of candidate IRT models and outputs model-averaged person scores and standard errors (see manuscript equations 4 and 5).

1.2 modelavgIRT R function input

personscores:

A data set consisting of person scores obtained from each candidate model in the validation sample, with rows denoting person and columns denoting model

personSEs:

A data set consisting of person score standard errors obtained from each candidate model in the validation sample, with rows denoting person and columns denoting model

selectionindex:

List of information criteria values (BIC, AIC) for each model, in the order of the columns of personscores and personSEs

rescale:

Logical; if set to TRUE (default), prior to averaging, each models’ person scores will be rescaled to have mean of 0 and a variance of 1 and standard errors will be rescaled proportionally

1.3 modelavgIRT R function Code

modelavgIRT <- function(personscores,personSEs,selectionindex,rescale=TRUE) {

##rescale personscores to have mean 0 and var 1

#rescale personSEs proportionally

if(rescale==TRUE){

for(i in seq(ncol(personscores))){

personscores[,i] <- (personscores[,i] - mean(personscores[,i]))/sd(personscores[,i])

personSEs[,i] <- personSEs[,i]/sd(personscores[,i])

}

}

##compute weights

weights <- c(rep(NA,length(selectionindex)))

for(i in seq(length(selectionindex))){

weights[i] <- sum(exp(-

.5*selectionindex[1:length(selectionindex)]+.5*selectionindex[i]))^(-1)

}

##compute averaged person scores

avg.personscore <- matrix(NA,nrow(personscores),1)

for(i in seq(nrow(personscores))){

avg.personscore[i,] <- sum(weights*personscores[i,])

}

##compute averaged person SEs

avg.personSE <- matrix(NA,nrow(personSEs),1)

for(i in seq(nrow(personSEs))){

avg.personSE[i,] <- sum(weights*sqrt(personSEs[i,]^2+(personscores[i,]-

avg.personscore[i,])^2))

}

output <- list(weights,avg.personscore,avg.personSE)

names(output) <- c(“weights”,”Average person score”,”Average person SE”)

return(output)

}