Hierarchical Bayesian Modeling for Test Theory Without an Answer Key

Abstract

Cultural Consensus Theory (CCT) models have been applied extensively across research domains in the social and behavioral sciences in order to explore shared knowledge and beliefs. CCT models operate on response data, in which the answer key is latent. The current paper develops methods to enhance the application of these models by developing the appropriate specifications for hierarchical Bayesian inference. A primary contribution is the methodology for integrating the use of covariates into CCT models. More specifically, both person- and item-related parameters are introduced as random effects that can respectively account for patterns of inter-individual and inter-item variability.

Appendices

Appendix A. Proof of Identifiability for the General Condorcet Model

Let the parameters be (D,G,Z) of the GCM with parameter spaces, respectively (0,1)^N, (0,1)^N, {0,1}^M. Let Ω be the parameter space of (D,G,Z). Let Y be an N×M matrix of 1s and 0s. Let S be the space of all Y. Let h:Ω→Π, where Π is the space of all probability distribution over Y. Let p(y∣(D,G,Z)) be a particular probability density function of Π.

Definition

In this context a model is identified in case h is one-to-one, meaning that two different sets of parameters necessarily produce different probability density functions over Y.

Observation 1

The model is not identified if we allow all items to be false or if we allow all items to be true. Let Z be all 1s and Z′ be all 0s, then so long as

$$ \forall i \quad D_i + (1-D_i)g_i = \bigl(1-D'_i\bigr)g'_i, $$

the model gives identical probabilities.

Observation 2

If we exclude these two extremes from the space of possible Z′s, the model is identified. We need to show that

$$\begin{aligned} & \forall\mathbf{Y} \in\mathbf{S}: \\ &\quad p({\mathbf{Y}} \mid\mathbf{D}, \mathbf{G}, \mathbf{Z}) = p\bigl({\mathbf {Y}} \mid \mathbf{D}', \mathbf{G}', \mathbf{Z}' \bigr) \\ &\quad \Rightarrow\mathbf{D} = \mathbf{D}',\quad \mathbf{G} = \mathbf{G}',\quad \mathbf{Z} = \mathbf{Z}'. \end{aligned}$$

Suppose Z=Z′. Then there are $Z_{k}=1=Z'_{k}$ and $Z_{l}=0=Z'_{l}$. From these we have for all informants i: $D_{i} + (1-D_{i})g_{i} = D'_{i} + (1-D'_{i})g'_{i}$ and $(1-D_{i})g_{i} = (1-D'_{i})g'$. From these we have $D_{i} = D'_{i}$ and $G_{i}=G'_{i}$. So if the model is not identified it must be that Z≠Z′. Pick k (without loss of generality since primed and unprimed can be swaped) with Z _k =1, $Z'_{k}=0$. For all informants i we have

$$ D_i + (1-D_i)g_i = \bigl(1-D'_i\bigr)g'_i. $$

(A.1)

Next pick j such that $Z'_{j}=1$, this is possible because we have eliminated the case where Z′ can all be zeros. If Z _j =1 then for all i,

$$ D_i + (1-D_i)g_i = D'_i+ \bigl(1-D'_i \bigr)g'_i, $$

and coupled with Equation (A.1), this is not possible. On the other hand if Z _j =0, then for all informants i, we have

$$ (1-D_i)g_i = D'_i+ \bigl(1-D'_i\bigr)g'_i, $$

and coupled with Equation (A.1), this is not possible. Thus the model is identified.

Appendix B. Posterior Distributions of the HGCMs

The posterior distribution given the data for the HGCM with Gaussian population distributions can be derived the following way: For notational convenience, all of the person- and item-specific parameters are respectively collected into corresponding vectors (i.e., θ, g, δ, and Z). First, the conditional posterior for the model in which the probability variables have population distributions assigned on the transformed variable scale is derived as

$$\begin{aligned} &\operatorname{Pr}\bigl(\boldsymbol{\theta}, \boldsymbol{\beta}_{\theta}, \sigma^2_{\theta}, \boldsymbol{g}, \boldsymbol{ \beta}_g, \sigma^2_g, \boldsymbol{\delta}, \boldsymbol{\beta}_{\delta}, \sigma^2_{\delta}, \mathbf{Z}, \pi \mid \mathbf{Y}\bigr) \\ &\quad\propto \prod_{i=1}^I \prod _{k=1}^K \bigl( Z_k D_{ik} + (1 - D_{ik})g_i \bigr)^{(Y_{ik} \equiv1)} \\ &\qquad{}\times\bigl(-Z_k D_{ik} + D_{ik} +(1 - D_{ik}) (1-g_i) \bigr)^{(Y_{ik} \equiv0)} \\ &\qquad{} \times\prod_{i=1}^I \mathrm{Normal}_2 \left (\left [ \begin{array}{c} \mathrm{logit} (\theta_{i}) \\ \mathrm{logit} (g_i) \end{array} \right ] \biggm| \left[ \begin{array}{c} \boldsymbol{\beta}_{l (\theta)} \\ \boldsymbol{\beta}_{l(g)} \end{array} \right] , \boldsymbol{\Sigma}_{l(\theta g)} \right ) \\ &\qquad{}\times\prod_{k=1}^K \mathrm{Normal}\bigl(\mathrm{logit}(\delta_{k}) \mid \boldsymbol{ \beta}_{l(\delta)}, \sigma^2_{\delta}\bigr) \prod _{k=1}^K \mbox {Bernoulli}(Z_k \mid\pi) \\ &\qquad{}\times\mathrm{Normal}_{J+1}(\boldsymbol{\beta}_{l(\theta)} \mid0, 10 \mathbf{I}_{J+1}) \mathrm{Normal}_{J+1}(\boldsymbol{ \beta}_{l(g)} \mid0, 1000 \mathbf{I}_{J+1}) \\ &\qquad{}\times\mathrm{Normal}_{H+1}(\boldsymbol{\beta}_{l(\delta)} \mid0, 10 \mathbf{I}_{H+1}) \mbox{Uniform}( \pi\mid0,1) \\ &\qquad{}\times \mbox{Inverse-Wishart}(\mathbf{I}_{2}, 3) \mbox{Inverse-Gamma}\bigl(\sigma ^2_{\delta} \mid0.01, 0.01 \bigr). \end{aligned}$$

After the proportionality sign, the first double product describes the likelihood of the parameters given the data, based on Equations (3). It is followed by the products of the population densities of the person-specific parameters, as specified in Equations (8). The next line describes the population densities of the item-specific parameters as in Equations (9) and (10). Finally, the last two lines multiple all the above with the prior densities as chosen in Equations (17), (19), (21) and (20).

The only modification for the prior settings of the HGCM with the beta population distributions concerns the variance parameters. As θ _i and g _i are sampled univariately, priors have to be set for their ‘precision’ parameters (that determine their population variance). As is typically done for precision parameters, a moderately diffuse Gamma distribution can be chosen, where

$$\begin{aligned} \tau_\theta&\sim\mbox{Gamma}(1, 0.1). \end{aligned}$$

(B.1)

Then τ _g as well as τ _δ for item difficulty are set similarly.

Note that the $\mathrm{HGCM}_{\mathcal{B}}$ is different only in terms of the population distributions for the item-and person-specific parameters. As discussed earlier, the beta distribution is parameterized in terms of regression coefficients and a precision parameter, as specified in Equations (16) and (B.1), and the posterior is written as

$$\begin{aligned} &\operatorname{Pr}(\boldsymbol{\theta}, \boldsymbol{\beta}_{\theta}, \tau_{\theta}, \boldsymbol{g}, \boldsymbol{\beta}_g, \tau_g, \boldsymbol{\delta}, \boldsymbol{\beta}_{\delta}, \tau_{\delta}, \mathbf{Z}, \pi \mid\mathbf{Y})\\ &\quad\propto \prod _{i=1}^I \prod _{k=1}^K \bigl( Z_k D_{ik} + (1 - D_{ik})g_i \bigr)^{(Y_{ik} \equiv1)} \\ &\qquad{}\times\bigl(-Z_k D_{ik} + D_{ik}+(1 - D_{ik}) (1-g_i) \bigr)^{(Y_{ik} \equiv0)} \\ &\qquad{} \times\prod_{i=1}^I \mathrm{Beta} ( \theta_{i} \mid\boldsymbol {\beta}_{\theta} , \tau_{\theta} ) \\ &\qquad{} \times\prod_{i=1}^I \mathrm{Beta} (g_{i} \mid\boldsymbol{\beta }_{g} , \tau_{g} ) \\ &\qquad{} \times\prod_{k=1}^K \mathrm{Beta} ( \delta_{k} \mid \boldsymbol{\beta }_{\delta}, \tau_{\delta}) \prod_{k=1}^K \mathrm{Bernoulli}(Z_k \mid\pi) \\ &\qquad{}\times\mathrm{Normal}_{J+1}(\boldsymbol{\beta}_{\theta} \mid0, 10 \mathbf{I}_{J+1}) \mathrm{Normal}_{J+1}(\boldsymbol{ \beta}_{g} \mid0, 1000 \mathbf{I}_{J+1}) \\ &\qquad{}\times\mathrm{Normal}_{H+1}(\boldsymbol{\beta}_{\delta} \mid0, 10 \mathbf{I}_{H+1}) \mbox{Uniform}( \pi\mid0,1) \\ &\qquad{} \times\mbox{Gamma}(\tau_\theta \mid1, 0.1) \mbox{Gamma}( \tau_g \mid1, 0.1)\mbox{Gamma}(\tau_\delta \mid1, 0.1). \end{aligned}$$

Appendix C. JAGS Code for the HGCMs

3.1 C.1 Normal Population Distributions

model{

   for (i in 1:n){

      for (k in 1:m){

        D[i, k] <- (theta[i]*(1-delta[k]))/

                    (theta[i]*(1-delta[k])+(1-theta[i])

                    *delta[k])}

        pY[i,k] <- g[i] - D[i,k]*g[i] + D[i,k]*z[k]

        Y[i,k]  ~ dbern(pY[i,k])}}




   for (i in 1:n){




          mean_theta[i]    <- Xtheta[i,]%*%coeff_theta

          mean_g[i]        <- Xg[i,]%*%coeff_g




          mean_thetag[i,1] <- mean_theta[i]

          mean_thetag[i,2] <- mean_g[i]




          logit_thetag[i,1:2] ~ dmnorm(mean_thetag[i,1:2],

                                precM_thetag)




          theta[i] <- ilogit(logit_thetag[i,1])

          g[i]     <- ilogit(logit_thetag[i,2])




   }




   for (k in 1:m){

          Z[k]           ~ dbern(PI)

          delta[k]       <- ilogit(logit_delta[k])

          mean_delta[k]  <- Xdelta[k,]%*%coeff_delta

          logit_delta[k] ~ dnorm(mean_delta[k],prec_delta)}




   for (cov in 1:nrofthetacov){

        coeff_theta[cov] ~ dnorm(0, 0.01)}




   for (cov in 1:nrofgcov){

        coeff_g[cov] ~ dnorm(0, 0.01)}




   cov_thetag[1:2,1:2] <- inverse(precM_thetag[1:2, 1:2])

   var_theta           <- cov_thetag[1,1]

   var_g               <- cov_thetag[2,2]

   corr_thetag         <- cov_thetag[1,2]/

                          (sqrt(var_theta)*sqrt(var_g))




   precM_thetag[1:2, 1:2] ~ dwish(ID[1:2,1:2], 3)

    ID[1,1] <- 1

    ID[2,2] <- 1

    ID[1,2] <- 0

    ID[2,1] <- 0




   for (cov in 1:nrofdeltacov){

        coeff_delta[cov] ~ dnorm(0, 0.01)}




   prec_delta ~ dgamma(0.01, 0.01)

   var_delta  <- 1/prec_delta




   PI ~ dunif(0, 1)}

3.2 C.2 Beta Population Distributions

model{

   for (i in 1:n){

      for (k in 1:m){

        D[i, k] <- (theta[i]*(1-delta[k]))/

                   (theta[i]*(1-delta[k])+(1-theta[i])

                    *delta[k])}

        pY[i,k] <- g[i] - D[i,k]*g[i] + D[i,k]*z[k]

        Y[i,k]  ~ dbern(pY[i,k])}}




   for (i in 1:n){




        mean_theta[i] <- ilogit(Xtheta[i,]%*%coeff_theta)

        mean_g[i]     <- ilogit(Xg[i,]%*%coeff_g)




        theta[i] ~ dbeta(mean_theta[i]*prec_theta,

                        (1-mean_theta[i])*prec_theta)

        g[i]     ~ dbeta(mean_g[i]*prec_g,

                        (1-mean_g[i])*prec_g)}




   for (k in 1:m){

        Z[k]          ~ dbern(PI)

        mean_delta[k] <- ilogit(Xdelta[k,]%*%coeff_delta)

        delta[k]      ~ dbeta(mean_delta[k]*prec_delta,

                              (1-mean_delta[k])*prec_delta)}




   for (cov in 1:nrofthetacov){

        coeff_theta[cov] ~ dnorm(0, 0.01)}




   for (cov in 1:nrofgcov){

        coeff_g[cov] ~ dnorm(0, 0.01)}




   for (cov in 1:nrofdeltacov){

        coeff_delta[cov] ~ dnorm(0, 0.01)}




   prec_theta_root ~ dgamma(1, 0.1)

   prec_theta      <- pow(prec_theta_root,2)




   prec_g_root ~ dgamma(1, 0.1)

   prec_g      <- pow(prec_g_root,2)




   prec_delta_root ~ dgamma(1, 0.1)

   prec_delta      <- pow(prec_delta_root,2)




   PI ~ dunif(0, 1)}