Variable Selection for Mixed Data Clustering: Application in Human Population Genomics

Abstract

Model-based clustering of human population genomic data, composed of 1,318 individuals arisen from western Central Africa and 160,470 markers, is considered. This challenging analysis leads us to develop a new methodology for variable selection in clustering. To explain the differences between subpopulations and to increase the accuracy of the estimates, variable selection is done simultaneously to clustering. We proposed two approaches for selecting variables when clustering is managed by the latent class model (i.e., mixture considering independence within components). The first method simultaneously performs model selection and parameter inference. It optimizes the Bayesian Information Criterion with a modified version of the standard expectation–maximization algorithm. The second method performs model selection without requiring parameter inference by maximizing the Maximum Integrated Complete-data Likelihood criterion. Although the application considers categorical data, the proposed methods are introduced in the general context of mixed data (data composed of different types of features). As the first step, the interest of both proposed methods is shown on simulated and several benchmark real data. Then, we apply the clustering method to the human population genomic data which permits to detect the most discriminative genetic markers. The proposed method implemented in the R package VarSelLCM is available on CRAN.

Appendices

Appendix A: Details on the Closed Form of the Integrated Complete-Data Log-likelihood

To compute the integrated complete-data log-likelihood, we give the value p(x_∙j|g, ω_j, z) for any type of data (continuous, integer, and categorical).

• If variable j is continuous, then:

  $$ p(\textbf{x}_{\bullet j}|g,\omega_{j},\textbf{z})=\left\{ \begin{array}{ll} \pi^{-n/2} \left( \frac{b_{j}^{a_{j}/2}d_{j}^{1/2}}{{\Gamma}(a_{j}/2)}\right)^{g} {\prod}_{k = 1}^{g} \frac{{\Gamma}(A_{kj}/2)}{B_{kj}^{A_{kj}}D_{kj}^{1/2}} & \text{if } \omega_{j} = 1 \\ \pi^{-n/2} \frac{b_{j}^{a_{j}}d_{j}^{1/2}}{{\Gamma}(a_{j}/2)}\frac{{\Gamma}(A_{j}/2)}{B_{j}^{A_{j}}D_{j}^{1/2}} & \text{if } \omega_{j} = 0 \end{array}\right. , $$

  where, A_j = n + a_j, ${B_{j}^{2}}={b_{j}^{2}} + {\sum }_{i = 1}^{n} (x_{ij} - \bar {\text {x}}_{j})^{2} + \frac {(c_{j} - \bar {\text {x}}_{j})^{2}}{d_{j}^{-1} + n^{-1}}$, D_j = n + d_j, $\bar {\text {x}}_{j}=\frac {1}{n}{\sum }_{i = 1}^{n} x_{ij}$, A_kj = n_k + a_j, $B_{kj}^{2}={b_{j}^{2}} + {\sum }_{i = 1}^{n} z_{ik} (x_{ij} - \bar {\text {x}}_{jk})^{2} + \frac {(c_{j} - \bar {\text {x}}_{jk})^{2}}{d_{j}^{-1} + n_{k}^{-1}}$, D_kj = n_k + d_j, $\bar {\text {x}}_{jk}=\frac {1}{n_{k}}{\sum }_{i = 1}^{n} z_{ik}x_{ij}$ and $n_{k}={\sum }_{i = 1}^{n}z_{ik}$.

• If variable j is integer, then:

  $$ p(\textbf{x}_{\bullet j}|g,\omega_{j},\textbf{z})=\left\{ \begin{array}{ll} \frac{1}{{\prod}_{i = 1}^{n} {\Gamma}(x_{ij}+ 1)} \left( \frac{b_{j}^{a_{j}}}{{\Gamma}(a_{j})}\right)^{g} {\prod}_{k = 1}^{g} {\Gamma}(A_{kj})B_{kj}^{-A_{kj}} & \text{if } \omega_{j} = 1 \\ \frac{1}{{\prod}_{i = 1}^{n} {\Gamma}(x_{ij}+ 1)} \frac{b_{j}^{a_{j}}}{{\Gamma}(a_{j})} {\Gamma}(A_{j})B_{j}^{-A_{j}} & \text{if } \omega_{j} = 0 \end{array}\right. , $$

  where, $A_{j}={\sum }_{i = 1}^{n} x_{ij}+a_{j}$, $B_{j}={b_{j}^{2}} + n$, $A_{kj}={\sum }_{i = 1}^{n} z_{ik}x_{ij}+a_{j}$ and $B_{j}={b_{j}^{2}} + {\sum }_{i = 1}^{n}z_{ik}$.

• If variable j is categorical with m_j levels, then:

  $$ p(\textbf{x}_{\bullet j}|g,\omega_{j},\textbf{z})=\left\{ \begin{array}{ll} \left( \frac{{\Gamma}\left( m_{j} a\right)}{{\Gamma}(a)^{m_{j}}} \right)^{g} \prod \limits_{k = 1}^{g} \frac{{\prod}_{h = 1}^{m_{j}}{\Gamma}\left( {\sum}_{i = 1}^{n} z_{ik}{1}_{\{x_{ij}=h\}} + a_{j}\right)}{{\Gamma}\left( {\sum}_{i = 1}^{n} z_{ik} + m_{j} a_{j}\right)} & \text{if } \omega_{j} = 1 \\ \frac{{\Gamma}\left( m_{j} a\right)}{{\Gamma}(a)^{m_{j}}} \frac{{\prod}_{h = 1}^{m_{j}}{\Gamma}\left( {\sum}_{i = 1}^{n} {1}_{\{x_{ij}=h\}} + a_{j}\right)}{{\Gamma}\left( n + m_{j} a_{j}\right)}& \text{ if } \omega_{j} = 0 \end{array}\right. . $$

Appendix B: EM Algorithm To Optimize the BIC Criterion for Data with Missing Values

The EM algorithm starts at a initial point (m^[0], θ^[0]) with m^[0] = (g, ω^[0]) randomly sampled and its iteration [r] is composed of two steps:

E step:

Computation of the fuzzy partition

$$ t_{ik}^{[r]}:=\frac{\tau_{k}^{[r-1]} {\prod}_{j \in \mathbf{O}_{i}} f_{kj}(x_{ij} | \boldsymbol{\alpha}_{kj}^{[r-1]})}{{\sum}_{\ell = 1}^{g} \tau_{\ell}^{[r-1]} {\prod}_{j \in \mathbf{O}_{i}} f_{\ell j}(x_{ij} | \boldsymbol{\alpha}_{\ell j}^{[r-1]})}, $$

M step:

Maximization of the expectation of the penalized complete-data log-likelihood over (ω, θ), hence m^[r] = (g, ω^[r]) with

$$ \omega_{j}^{[r]}=\left\{ \begin{array}{rl} 1 & \text{if } {\Delta}_{j}^{[r]} > 0 \\ 0 & \text{otherwise} \end{array}\right., \tau_{k}^{[r]}=\frac{n_{k}^{[r]}}{n} \text{ and } \boldsymbol{\alpha}^{[r]}_{jk}=\left\{ \begin{array}{rl} \boldsymbol{\alpha}^{\star [r]}_{kj} & \text{if } \omega_{j}^{[r]}= 1 \\ \tilde{\boldsymbol{\alpha}}_{kj} & \text{otherwise} \end{array}\right., $$

where ${\Delta }_{j}={\sum }_{k = 1}^{g} {\sum }_{\{i: j\in \mathbf {O}_{i} \}} t_{ik}^{[r]} \left (\ln f_{kj}(x_{ij} | \boldsymbol {\alpha }^{\star [r]}_{kj})- \ln f_{1j}(x_{ij} | \tilde {\boldsymbol {\alpha }}_{1j})\right ) - (g-1)\nu _{j} c$, where $\tilde {\boldsymbol {\alpha }}_{1j}=\text {arg max}_{\boldsymbol {\alpha }_{1j}} {\sum }_{\{i: j\in \mathbf {O}_{i} \}} \ln f_{1j}(x_{ij} | \boldsymbol {\alpha }_{1j})$ and where $\boldsymbol {\alpha }^{\star [r]}_{kj}=\text {arg max}_{\boldsymbol {\alpha }_{1j}} {\sum }_{\{i: j\in \mathbf {O}_{i} \}}t_{ik}^{[r]} \ln f_{1j}(x_{ij} | \boldsymbol {\alpha }_{1j})$.

Appendix C: Details on the Closed Form of the Integrated Complete-Data Log-Likelihood for Data with Missing Values

To compute the integrated complete-data log-likelihood, for data with missing values, we give the value p(x_∙j|g, ω_j, z) for any type of data (continuous, integer, and categorical) containing missing values.

• If variable j is continuous, then:

  $$ p(\textbf{x}_{\bullet j}|g,\omega_{j},\textbf{z})=\left\{ \begin{array}{ll} \pi^{-n_{j}/2} \left( \frac{b_{j}^{a_{j}/2}d_{j}^{1/2}}{{\Gamma}(a_{j}/2)}\right)^{g} {\prod}_{k = 1}^{g} \frac{{\Gamma}(A_{kj}/2)}{B_{kj}^{A_{kj}}D_{kj}^{1/2}} & \text{if } \omega_{j} = 1 \\ \pi^{-n_{j}/2} \frac{b_{j}^{a_{j}}d_{j}^{1/2}}{{\Gamma}(a_{j}/2)}\frac{{\Gamma}(A_{j}/2)}{B_{j}^{A_{j}}D_{j}^{1/2}} & \text{if } \omega_{j} = 0 \end{array}\right. , $$

  where, , A_j = n_j + a_j, ${B_{j}^{2}}={b_{j}^{2}} + {\sum }_{\{i: j\in \mathbf {O}_{i} \}} (x_{ij} - \bar {\text {x}}_{j})^{2} + \frac {(c_{j} - \bar {\text {x}}_{j})^{2}}{d_{j}^{-1} + n_{j}^{-1}}$, D_j = n_j + d_j, $\bar {\text {x}}_{j}=\frac {1}{n_{j}}{\sum }_{\{i: j\in \mathbf {O}_{i} \}} x_{ij}$, A_kj = n_jk + a_j, $B_{kj}^{2}={b_{j}^{2}} + {\sum }_{\{i: j\in \mathbf {O}_{i} \}} z_{ik} (x_{ij} - \bar {\text {x}}_{jk})^{2} + \frac {(c_{j} - \bar {\text {x}}_{jk})^{2}}{d_{j}^{-1} + n_{jk}^{-1}}$, D_kj = n_jk + d_j, $\bar {\text {x}}_{jk}=\frac {1}{n_{jk}}{\sum }_{\{i: j\in \mathbf {O}_{i} \}} z_{ik}x_{ij}$ and $n_{jk}={\sum }_{\{i: j\in \mathbf {O}_{i} \}}z_{ik}$.

• If variable j is integer, then:

  $$ p(\textbf{x}_{\bullet j}|g,\omega_{j},\textbf{z}) = \left\{ \begin{array}{ll} \frac{1}{{\prod}_{\{i: j\in \mathbf{O}_{i} \}} {\Gamma}(x_{ij}+ 1)} \left( \frac{b_{j}^{a_{j}}}{{\Gamma}(a_{j})}\right)^{g} {\prod}_{k = 1}^{g} {\Gamma}(A_{kj})B_{kj}^{-A_{kj}} & \text{if } \omega_{j} = 1 \\ \frac{1}{{\prod}_{\{i: j\in \mathbf{O}_{i} \}} {\Gamma}(x_{ij}+ 1)} \frac{b_{j}^{a_{j}}}{{\Gamma}(a_{j})} {\Gamma}(A_{j})B_{j}^{-A_{j}} & \text{if } \omega_{j} = 0 \end{array}\right. , $$

  where, $A_{j}={\sum }_{\{i: j\in \mathbf {O}_{i} \}}+a_{j}$, $B_{j}={b_{j}^{2}} + n_{j}$, $A_{kj}={\sum }_{\{i: j\in \mathbf {O}_{i} \}} z_{ik}x_{ij}+a_{j}$, $B_{j}={b_{j}^{2}} + {\sum }_{\{i: j\in \mathbf {O}_{i} \}}z_{ik}$ and .

• If variable j is categorical with m_j levels, then: