Multidimensional item response theory models for testlet-based doubly bounded data

Abstract

A testlet-based visual analogue scale (VAS) is a doubly bounded scaling approach (e.g., from 0% to 100% or from 0 to 1) composed of multiple adjectives, nouns, or sentences (statements/items) within testlets for measuring individuals’ attitudes, opinions, or career interests. While testlet-based VASs have many advantages over Likert scales, such as reducing response style effects, the development of proper statistical models for analyzing testlet-based VAS data lags behind. This paper proposes a novel beta copula model and a competing logit-normal model based on the item response theory framework, assessed by Bayesian parameter estimation, model comparison, and goodness-of-fit statistics. An empirical career interest dataset based on a testlet-based VAS design was analyzed using the proposed models. Simulation studies were conducted to assess the two models’ parameter recovery. The results show that the beta copula model had superior fit in the empirical data analysis, and also exhibited good parameter recovery in the simulation studies, suggesting that it is a promising statistical approach to testlet-based doubly bounded responses.

Appendices

Appendix 1

Fig. 16

Outfit statistics under the beta copula model for 30 career interest items

Fig. 17

Infit statistics under the beta copula model for 30 career interest items

Appendix 2

Fig. 18

Average absolute biases of item standard error estimators for logit normal model. Note. ${{\varvec{R}}}_{\theta }$ denotes correlation of latent traits; Σ_x denotes correlation of items

Fig. 19

Average root-mean-square errors of item standard error estimators for logit normal model. Note. ${{\varvec{R}}}_{\theta }$ denotes correlation of latent traits; Σ_x denotes correlation of items

Fig. 20

Average absolute biases of item standard error estimators for beta copula model. Note. ${{\varvec{R}}}_{\theta }$ denotes correlation of latent traits; R_x denotes correlation of items

Fig. 21

Average root-mean-square errors of item standard error estimators for beta copula model. Note. ${{\varvec{R}}}_{\theta }$ denotes correlation of latent traits; R_x denotes correlation of items

Appendix 3

Fig. 22

Average absolute biases and root-mean-square errors of item standard error estimators for logit normal model given a fixed set of realistic correlation matrices from the empirical career interest data analyses. Note. ${{\varvec{R}}}_{\theta }$ denotes correlation of latent traits; Σ_x denotes covariance matrix of items

Fig. 23

Average absolute biases and root-mean-square errors of item standard error estimators for beta copula model given a fixed set of realistic correlation matrices from the empirical career interest data analyses. Note. ${{\varvec{R}}}_{\theta }$ denotes correlation of latent traits; R_x denotes correlation of items

Appendix 4

Fig. 24

Average absolute biases and root-mean-square errors of item standard error estimators for logit normal model for three testlets with two, three, and four items, respectively. Note. ${{\varvec{R}}}_{\theta }$ denotes correlation of latent traits; Σ_x denotes covariance matrix of items

Fig. 25

Average absolute biases and root-mean-square errors of item standard error estimators for beta copula model for three testlets with two, three, and four items, respectively. Note. ${{\varvec{R}}}_{\theta }$ denotes correlation of latent traits; R_x denotes correlation of items

Appendix 5

Fig. 26

Average nonconvergence rates of item parameters and person parameters for logit normal model by using the Heidelberger Welch’s convergence diagnostic

Fig. 27

Average nonconvergence rates of item parameters and person parameters for beta copula model by using the Heidelberger Welch’s convergence diagnostic