Bayesian variable selection for latent class analysis using a collapsed Gibbs sampler

Abstract

Latent class analysis is used to perform model based clustering for multivariate categorical responses. Selection of the variables most relevant for clustering is an important task which can affect the quality of clustering considerably. This work considers a Bayesian approach for selecting the number of clusters and the best clustering variables. The main idea is to reformulate the problem of group and variable selection as a probabilistically driven search over a large discrete space using Markov chain Monte Carlo (MCMC) methods. Both selection tasks are carried out simultaneously using an MCMC approach based on a collapsed Gibbs sampling method, whereby several model parameters are integrated from the model, substantially improving computational performance. Post-hoc procedures for parameter and uncertainty estimation are outlined. The approach is tested on simulated and real data .

Appendix: Comparison with reversible jump MCMC

In this section we investigate how the performance of a collapsed Gibbs sampler compares with an RJMCMC based on using all model parameters. We divide this investigation into two tasks: selecting (a) the number of classes, and (b) which variables to include. We implement the approach of Pan and Huang (2013) using already available software to investigate the efficacy of RJMCMC for the former task, and outline our own approach to perform the latter for the case where the observed data is binary only. We find that the approach performs reasonably well when selecting the number of classes, although its performance is somewhat slower than that of the collapsed sampler. The implemented approach performs poorly when performing variable selection.

1.1 Number of classes

To identify the number of groups in a dataset using RJMCMC methods, we apply software¹ implementing the approach of Pan and Huang (2013). We applied the software to the binary and non-binary Dean and Raftery datasets described in Sect. 2.2, running the sampler for 100,000 iterations in both cases. All prior settings were set by default. In both cases, the non-informative variables were removed, since the sole task was to identify the correct number of classes.

As the software for this approach was implemented as a C++ programme, it can be thought of as broadly comparable to our own collapsed sampler, which is implemented in C; for the binary and non-binary datasets, the software took roughly 25 and 90 mins to run respectively, based on the same hardware specifications described previously. In both cases, this was markedly longer than the collapsed sampler took, despite the fact that the model was exploring the group space only, and the dimension of the data had been reduced.

The results from the samplers are shown in Table 14. In the case of the binary data, the correct number of groups is chosen as the most likely candidate, although, with a lower posterior probability in comparison to the collapsed sampler. In the case of the non-binary data, $G = 2$ is incorrectly is chosen as the most likely candidate, with some level of uncertainty surrounding which model is the most suitable.

Table 14 Posterior probability for number of groups in the binary and non-binary Dean and Raftery datasets using the reversible jump approach of (Pan and Huang 2013)

1.2 Variable selection

Recall that for the variable inclusion/exclusion step, a variable ${m^*}$ is selected at random from $1, \ldots , M.$ An inclusion or exclusion move is then proposed, based on the current status of the variable. In what follows, we assume that the state space has $G$ groups, and that the data is binary, so that $X_{nm} \in \{0, 1\}$, for all $n = 1, \ldots , N$ and $m = 1, \ldots , M.$

1.2.1 Inclusion step

Suppose we select a variable ${m^*},$ which is currently excluded from the model. For the inclusion step, dropping the variable index, we propose the following move:

• 1. Generate $u_1, \ldots , u_{G-1} \sim \hbox {Uniform}(-\epsilon , \epsilon ),$ and set $u_{G} = - \sum ^{G-1}_{i=1} u_i.$

• 2. Set

  $$\begin{aligned} \log \left( \frac{\theta _{1}}{1 - \theta _{1}}\right) = \log \left( \frac{\beta }{1 - \beta }\right) + u_1, \end{aligned}$$

  which is equivalent to setting

  $$\begin{aligned} \theta _1 = \frac{\beta e^{u_1}}{1 + \beta (e^{u_1} -1 )}. \end{aligned}$$

  Similarly, for $g = 2, \ldots , G,$ set:

  $$\begin{aligned} \theta _g = \frac{\beta e^{u_g}}{1 + \beta (e^{u_g} -1 )}. \end{aligned}$$

Then the proposed move is accepted with probability $\alpha$, where

$$\begin{aligned} \alpha&= \min \left( 1, \frac{ p_{\mathrm {cl}}(\mathbf {X}, \mathbf {Z}|\tilde{{\varvec{\theta }}},\tilde{{\varvec{\nu }}},{\varvec{\tau }},G) }{ p_{\mathrm {cl}}(\mathbf {X}, \mathbf {Z}|{\varvec{\theta }},{\varvec{\nu }},{\varvec{\tau }},G) } \frac{p_{\mathrm {n}}(\mathbf {X}|\tilde{{\varvec{\rho }}},\tilde{{\varvec{\nu }}})}{p_{\mathrm {n}}(\mathbf {X}|{{\varvec{\rho }}},{{\varvec{\nu }}})} \right. \\&\, \times \frac{ \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {n}}} p(\tilde{{\varvec{\rho }}}_m | \beta ) }{ \prod _{m \in {\varvec{\nu }}_{\mathrm {n}}} p({\varvec{\rho }}_m | \beta ) } \frac{ \prod _{g=1}^G \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {cl}}} p(\tilde{{\varvec{\theta }}}_{gm} | \beta ) }{ \prod _{g=1}^G \prod _{m \in {\varvec{\nu }}_{\mathrm {cl}}} p({\varvec{\theta }}_{gm} | \beta ) } \\&\, \times \left. \frac{ \prod _{g=1}^G \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {cl}}} p(\tilde{{\varvec{\theta }}}_{gm} | \beta ) }{ \prod _{g=1}^G \prod _{m \in {\varvec{\nu }}_{\mathrm {cl}}} p({\varvec{\theta }}_{gm} | \beta ) } |{\mathcal J}| \times p(\xi \rightarrow \xi ^*) \right) , \end{aligned}$$

where

$$\begin{aligned}&\frac{ p_{\mathrm {cl}}(\mathbf {X}, \mathbf {Z}|\tilde{{\varvec{\theta }}},\tilde{{\varvec{\nu }}},{\varvec{\tau }},G) }{ p_{\mathrm {cl}}(\mathbf {X}, \mathbf {Z}|{\varvec{\theta }},{\varvec{\nu }},{\varvec{\tau }},G) } = \prod ^G_{g=1} \theta _{gm}^{S_{g{m^*}}}(1 - \theta _{g{m^*}})^{S^C_{g{m^*}}}, \\&\frac{p_{\mathrm {n}}(\mathbf {X}|\tilde{{\varvec{\rho }}},\tilde{{\varvec{\nu }}})}{p_{\mathrm {n}}(\mathbf {X}|{{\varvec{\rho }}},{{\varvec{\nu }}})} = \frac{1}{\rho _{m^*}^{N_{m^*}}(1 - \rho _{m^*})^{N - N_{m^*}}},\\&\frac{ \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {n}}} p(\tilde{{\varvec{\rho }}}_m | \beta ) }{ \prod _{m \in {\varvec{\nu }}_{\mathrm {n}}} p({\varvec{\rho }}_m | \beta ) } = \frac{\Gamma (\beta )^2}{\Gamma ({2\beta })} \times \frac{1}{ \rho _{m^*}^{\beta -1}(1 - \rho _{m^*}^{\beta -1})},\\&\!\frac{ \prod _{g\!=\!1}^G \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {cl}}} p(\tilde{{\varvec{\theta }}}_{gm} | \beta ) }{ \prod _{g=1}^G \prod _{m \in {\varvec{\nu }}_{\mathrm {cl}}} p({\varvec{\theta }}_{gm} | \beta ) } \!=\! G \frac{\Gamma (2 \beta )}{\Gamma (\beta )^2} \prod ^G_{g=1} \theta _{gm^*}^{\beta \!-\!1} (1 \!-\! \theta _{gm^*})^{\beta \!-\!1},\\ \end{aligned}$$

and we define $S_{g{m^*}} = \sum ^N_{n=1} X_{nm^*}Z_{ng},$ $S^C_{g{m^*}} = \sum ^N_{n=1} (1 - X_{nm^*})Z_{ng},$ and $N_{m^*} = \sum ^N_{n=1} X_{nm^*}.$ Here we use $p(\xi \rightarrow \xi ^*) = 1/M$ to denote the probability of the proposed move. Finally, the Jacobian ${\mathcal J}$ is defined as ${\mathcal J}_{1g} = \frac{\partial \theta _{gm^*}}{\partial \rho _{m^*}},$ and ${\mathcal J}_{kg} = \frac{\partial \theta _{gm^*}}{\partial u_{k-1}},$ for $g = 1, \dots , G \hbox { and } k = 2, \dots , G.$

1.2.2 Exclusion step

If the variable ${m^*},$ is currently included in the model, we propose the exclusion step,

$$\begin{aligned} \rho = \frac{\left( \prod ^G_{g=1} \theta _g \right) ^{1/G} }{ \left( \prod ^G_{g=1} \theta _g \right) ^{1/G} + \left( \prod ^G_{g=1} (1 - \theta _g) \right) ^{1/G} }, \end{aligned}$$

where again we have dropped the variable index. Using this expression, we then obtain

$$\begin{aligned} u_g = \left( 1 - \frac{1}{G} \right) \log \left( \frac{ \theta _g }{ 1 - \theta _g } \right) - \frac{1}{G} \sum _{j \ne g} \log \left( \frac{ \theta _j }{ 1 - \theta _j } \right) , \end{aligned}$$

for $g = 1, \ldots , G-1$, demonstrating the required bijection between ${\varvec{\theta }}_{m^*}$ and $(\rho _{m^*}, \mathbf {u})$. The proposed move is again accepted with probability $\alpha$, where

$$\begin{aligned} \alpha&= \min \left( 1, \frac{ p_{\mathrm {cl}}(\mathbf {X}, \mathbf {Z}|\tilde{{\varvec{\theta }}},\tilde{{\varvec{\nu }}},{\varvec{\tau }},G) }{ p_{\mathrm {cl}}(\mathbf {X}, \mathbf {Z}|{\varvec{\theta }},{\varvec{\nu }},{\varvec{\tau }},G) } \frac{p_{\mathrm {n}}(\mathbf {X}|\tilde{{\varvec{\rho }}},\tilde{{\varvec{\nu }}})}{p_{\mathrm {n}}(\mathbf {X}|{{\varvec{\rho }}},{{\varvec{\nu }}})} \right. \\&\quad \times \frac{ \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {n}}} p(\tilde{{\varvec{\rho }}}_m | \beta ) }{ \prod _{m \in {\varvec{\nu }}_{\mathrm {n}}} p({\varvec{\rho }}_m | \beta ) } \frac{ \prod _{g=1}^G \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {cl}}} p(\tilde{{\varvec{\theta }}}_{gm} | \beta ) }{ \prod _{g=1}^G \prod _{m \in {\varvec{\nu }}_{\mathrm {cl}}} p({\varvec{\theta }}_{gm} | \beta ) } \\&\quad \times \left. \frac{ \Gamma (\beta )^2 }{ G\Gamma (2 \beta ) } \times \frac{1 }{ \prod ^G_{g=1} \theta _{gm^*}^{\beta -1} (1 - \theta _{gm^*})^{\beta -1} } |{\mathcal J}| \times p(\xi \rightarrow \xi ^*) \right) \end{aligned}$$

where the calculations are inverted, so that

$$\begin{aligned}&\frac{ p_{\mathrm {cl}}(\mathbf {X}, \mathbf {Z}|\tilde{{\varvec{\theta }}},\tilde{{\varvec{\nu }}},{\varvec{\tau }},G) }{ p_{\mathrm {cl}}(\mathbf {X}, \mathbf {Z}|{\varvec{\theta }},{\varvec{\nu }},{\varvec{\tau }},G) } = \frac{1}{ \prod ^G_{g=1} \theta _{gm}^{S_{g{m^*}}}(1 - \theta _{g{m^*}})^{S^C_{g{m^*}}} }, \\&\frac{p_{\mathrm {n}}(\mathbf {X}|\tilde{{\varvec{\rho }}},\tilde{{\varvec{\nu }}})}{p_{\mathrm {n}}(\mathbf {X}|{{\varvec{\rho }}},{{\varvec{\nu }}})} = \rho _{m^*}^{N_{m^*}} (1 - \rho _{m^*})^{N - N_{m^*} },\\&\frac{ \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {n}}} p(\tilde{{\varvec{\rho }}}_m | \beta ) }{ \prod _{m \in {\varvec{\nu }}_{\mathrm {n}}} p({\varvec{\rho }}_m | \beta ) } = \frac{ \Gamma ({2\beta }) }{ \Gamma (\beta )^2 } { \rho _{m^*}^{\beta -1}(1 - \rho _{m^*}^{\beta -1})},\\&\frac{ \prod _{g=1}^G \prod _{m \in \tilde{{\varvec{\nu }}}_{\mathrm {cl}}} p(\tilde{{\varvec{\theta }}}_{gm} | \beta ) }{\prod _{g=1}^G \prod _{m \in {\varvec{\nu }}_{\mathrm {cl}}} p({\varvec{\theta }}_{gm} | \beta )} = \frac{ \Gamma (\beta )^2 }{ G\Gamma (2 \beta )} \\&\qquad \qquad \qquad \qquad \qquad \qquad \qquad \times \frac{1 }{ \prod ^G_{g=1} \theta _{gm^*}^{\beta -1} (1 - \theta _{gm^*})^{\beta -1} }.\\ \end{aligned}$$

The probability of the proposed move remains, $p(\xi \rightarrow \xi ^*) = 1/M$.

1.2.3 Dean and Raftery data application

We apply this approach to the binary Dean and Raftery dataset described previously in Sect. 2.2. Here, we fix the number of groups to the true value $G = 2$, so that the model search is based on variable selection only. The sampler was run for 50,000 iterations, with $\epsilon = 1$, which resulted in an acceptance probability for the inclusion/exclusion move of $\alpha \approx 0.12.$

The posterior probability for variable inclusion from the sampler are shown in Table 15. None of the informative variables are selected as frequently as for the collapsed sampler, with the model only finding weak evidence for variable 1, and failing to distinguish between the other variables.

Table 15 Posterior probability for variable inclusion in the binary Dean and Raftery data using RJMCMC