Likelihood-Based Approach to Gene Set Enrichment Analysis with a Finite Mixture Model

Abstract

In this paper, we study a parametric modeling approach to gene set enrichment analysis. Existing methods have largely relied on nonparametric approaches employing, e.g., categorization, permutation or resampling-based significance analysis methods. These methods have proven useful yet might not be powerful. By formulating the enrichment analysis into a model comparison problem, we adopt the likelihood ratio-based testing approach to assess significance of enrichment. Through simulation studies and application to gene expression data, we will illustrate the competitive performance of the proposed method.

Appendix

1.1 EM Algorithm for Estimating the Finite Mixture Model

We begin with the finite mixture model in (1) given $(\hat{\theta}_{0},\hat{\mu}_{0},\hat{\sigma}^{2}_{0})$ and K. Define indicators w _ik ∈{0,1} following a multinomial distribution, Pr(w _ik =1)=θ _k , $\sum_{k=0}^{K} w_{ik}=1$, and conditionally we assume z _i |w _ik =1∼f _k . The complete data likelihood function for (z _i ,w _ik ) can be written as

$$\prod_{i=1}^m \bigl\{\hat{ \theta}_0f_0\bigl(z_i;\hat{ \mu}_0,\hat{\sigma}_0^2\bigr)\bigr \}^{w_{i0}} \prod_{k=1}^K\bigl\{ \theta_kf_k\bigl(z_i;\mu_k, \hat{\sigma}_0^2\bigr)\bigr\}^{w_{ik}}. $$

In the E-step, the conditional probabilities can be checked to be

$$T_{0,i} = \frac{\hat{\theta}_0f_0(z_i;\hat{\mu}_0, \hat{\sigma}_0^2)}{\hat{\theta_0}f_0(z_i;\hat{\mu}_0, \hat{\sigma}_0^2)+\sum_{k=1}^K\theta_kf_k(z_i;\mu_k,\hat{\sigma}_0^2)}, $$

$$T_{k,i} = \frac{\theta_kf_k(z_i;\mu_k,\hat{\sigma}_0^2)}{\hat{\theta}_0 f_0(z_i;\hat{\mu}_0, \hat{\sigma}_0^2) +\sum_{j=1}^K \theta_jf_j(z_i;\mu_j,\hat{\sigma}_0^2)}. $$

In the M-step, the conditional expected log likelihood can be checked to be proportional to

which can be easily verified to be maximized by

$$\hat{\theta}_k = (1-\hat{\theta}_0)\frac{\sum_{i=1}^m T_{k,i}}{\sum_{j=1}^K \sum_{i=1}^mT_{j,i}}, \quad\quad \hat{\mu}_k = \frac{\sum_{i=1}^m T_{k,i}z_i}{\sum_{i=1}^m T_{k,i}}, \quad k\ge 1. $$

Given only $(\hat{\mu}_{0},\hat{\sigma}^{2}_{0})$ with θ ₀ also being a parameter, we have

We can easily check that

$$\hat{\theta}_k = \frac{1}{m}\sum _{i=1}^m T_{k,i}, \quad k\ge 0, \quad\quad \hat{ \mu}_k = \frac{\sum_{i=1}^m T_{k,i}z_i}{\sum_{i=1}^m T_{k,i}}, \quad k>0. $$

1.2 EM Algorithm for Estimating the Gene Set Model

The complete data likelihood function for a gene set A given $(\hat{\theta}_{0},\hat{\mu}_{0},\hat{\sigma}_{0}^{2},\hat{\mu}_{k})$ is

$$\prod_{i\in A} \prod_{k=0}^K \bigl\{\nu_kf_k\bigl(z_i;\hat{ \mu}_k,\hat{\sigma}^2_0\bigr)\bigr \}^{w_{ik}}. $$

The conditional expected log likelihood can easily be checked to be

$$\sum_{i\in A} \sum_{k=0}^KT_{k,i} \log\nu_k, \quad T_{k,i}=\frac{\nu_{k}f_0(z_i;\hat{\mu}_k, \hat{\sigma}_0^2)}{ \sum_{0=1}^K \nu_jf_j(z_i;\hat{\mu}_j, \hat{\sigma}_0^2)}, \quad k\ge 0. $$

We can easily verify that

$$\hat{\nu}_{k} = \frac{\sum_{i\in A} T_{k,i}}{m_A}, \quad k\ge 0. $$

1.3 EM Algorithm for Estimating the Model Under no Enrichment

The complete data likelihood can be written as

where $\nu_{0}+\sum^{K}_{k=1}\nu_{lk}=1$, l=1,2. The conditional expected log likelihood can be easily checked to be

$$\sum_{i\in A}\Biggl\{ T_{0,i}\log \nu_0+\sum_{k=1}^KT_{k,i} \log\nu_{1k}\Biggr\} + \sum_{j\in A^c}\Biggl\{ T_{0,j}\log\nu_0+\sum_{k=1}^KT_{k,j} \log\nu_{2k}\Biggr\}, $$

where

To maximize the conditional log likelihood, we use the Lagrange multiplier method

Setting the gradient vector ∇Q=0 yields the following equations:

From the first three equations we can obtain

When plugging these into the last two equations, we obtain

$$\hat{\nu}_{0} = \frac{\sum_{i\in A}T_{0,i}+\sum_{j\in A^c}T_{0,j}}{m}, $$

and

$$\hat{\nu}_{1k} = (1-\hat{\nu}_{0})\frac{ \sum_{i\in A} T_{k,i}}{\sum_{l=1}^K \sum_{i\in A} T_{l,i}}, \quad \quad \hat{\nu}_{2k} = (1-\hat{\nu}_{0})\frac{\sum_{j\in A^c} T_{k,j}}{\sum_{l=1}^K \sum_{j\in A^c} T_{l,j}}. $$