D-trace estimation of a precision matrix using adaptive Lasso penalties

Abstract

The accurate estimation of a precision matrix plays a crucial role in the current age of high-dimensional data explosion. To deal with this problem, one of the prominent and commonly used techniques is the $\ell _1$ norm (Lasso) penalization for a given loss function. This approach guarantees the sparsity of the precision matrix estimate for properly selected penalty parameters. However, the $\ell _1$ norm penalization often fails to control the bias of obtained estimator because of its overestimation behavior. In this paper, we introduce two adaptive extensions of the recently proposed $\ell _1$ norm penalized D-trace loss minimization method. They aim at reducing the produced bias in the estimator. Extensive numerical results, using both simulated and real datasets, show the advantage of our proposed estimators.

Appendices

Appendix A: Numerical results

See Tables 5, 6, 7, 8, 9, 10, 11 and 12.

Table 5 Average KL losses (with standard deviations) over 100 replications

Table 6 Average Frobenius norm losses (with standard deviations) over 100 replications

Table 7 Average operator norm losses (with standard deviations) over 100 replications

Table 8 Average matrix $\ell _1$ norm losses (with standard deviations) over 100 replication

Table 9 Average specificity (with standard deviations) over 100 replications

Table 10 Average sensitivity (with standard deviations) over 100 replications

Table 11 Average MCC (with standard deviations) over 100 replications

Table 12 Average accuracy (with standard deviations) over 100 replications

Appendix B: Algorithms

In this section, we describe in detail the steps of the algorithm for obtaining the estimators DT, ADT and WADT based on the alternating direction method. First, we introduce matrices $\Omega _0$ and $\Omega _1$. Next, we consider the following optimization problem instead of problem (3):

$$\begin{aligned} \begin{array}{ll} &{}\widehat{\Omega }_{\text {DT}}=\arg \min \limits _{\Omega _1\succ \epsilon I}\ \dfrac{1}{2}\text {trace}(\Omega ^2 S)-\text {trace}(\Omega )+\tau ||\Omega _0||_{1,\text {off}},\\ &{} \quad \text {subject to} \ \ \ \ \{\Omega ,\Omega \}=\{\Omega _0,\Omega _1\} \end{array} \end{aligned}$$

(21)

Note that the problems (3) and (21) are equivalent. The Lagrangian of the problem (21) has the following form:

$$\begin{aligned} L(\Omega ,\Omega _0,\Omega _1,\Lambda _0, \Lambda _1)= & {} \dfrac{1}{2}\text {trace}(\Omega ^2 S)-\text {trace}(\Omega )+\tau ||\Omega _0||_{1,\text {off}}+h(\Omega _1\succeq \epsilon I) \nonumber \\&+\,\text {trace}(\Lambda _0(\Omega -\Omega _0))+\text {trace}(\Lambda _1(\Omega -\Omega _1)) \nonumber \\&+\,\dfrac{\rho }{2}||\Omega -\Omega _0||_2^2+\dfrac{\rho }{2}||\Omega -\Omega _1||_2^2, \end{aligned}$$

(22)

where $\rho$, $\Lambda _0$, $\Lambda _1$ are the multipliers and $h(\Omega _1\succeq \epsilon I)$ is an indicator function, which returns 0 if the statement $\Omega _1\succeq \epsilon I$ is true and $\infty$, otherwise. For simplicity, we take $\rho =1$. Assume that $(\Omega ^t,\Omega ^t_0,\Omega ^t_1,\Lambda _0^t,\Lambda _1^t)$ is the solution at step t, for $t=0,1,2,\ldots$. The solution is updated according to the following:

$$\begin{aligned} \Omega ^{t+1}=\arg \min _{\Omega =\Omega ^T}L(\Omega ,\Omega ^t_0,\Omega ^t_1,\Lambda _0^t,\Lambda _1^t), \end{aligned}$$

(23)

$$\begin{aligned} \{\Omega ^{t+1}_0,\Omega ^{t+1}_1\}=\underset{\Omega _0=\Omega _0^T, \Omega _1\succeq \epsilon I}{\text {argmin}}L(\Omega ^{t+1},\Omega _0,\Omega _1,\Lambda _0^t,\Lambda _1^t), \end{aligned}$$

(24)

$$\begin{aligned} \{\Lambda ^{t+1}_0,\Lambda ^{t+1}_1\}=\{\Lambda ^{t}_0,\Lambda ^{t}_1\}+\{\Omega ^{t+1}-\Omega ^{t+1}_0,\Omega ^{t+1}-\Omega ^{t+1}_1\}. \end{aligned}$$

(25)

From the Eq. (23) we have the following:

$$\begin{aligned} \Omega ^{t+1}=\underset{\Omega =\Omega ^T}{\text {argmin}}\dfrac{1}{2} \text {trace}(\Omega ^2 (S+2I))-\text {trace}(\Omega (I+\Omega _0^t+\Omega _1^t-\Lambda _0^t-\Lambda _1^t)). \end{aligned}$$

(26)

First, for any $p\times p$ symmetric matrix $Z\succ 0$ and any $p\times p$ symmetric matrix Y we define a matrix G(Z, Y). Assuming that $Z=UVU^T$ is the eigendecomposition of matrix Z and $v_1\ge \cdots \ge v_p$ are its eigenvalues, we define

$$\begin{aligned} G(Z,Y)=U\{(U^T Y U)\circ C \}U^T, \end{aligned}$$

(27)

where $C_{i,j}=\dfrac{2}{v_i+v_j}$ for $1\le i,j\le p$ and $\circ$ denotes the Hadamard product of matrices. Zhang and Zou (2014) proved that we can write the solution of the problem (26) as $\Omega ^{t+1}=G(S+2I;I+\Omega _0^t+\Omega _1^t-\Lambda _0^t-\Lambda _1^t )$. For more details we refer to Theorem 1 in Zhang and Zou (2014).

From the first part of the Eq. (24) it follows that

$$\begin{aligned} \Omega _0^{t+1}=\underset{\Omega _0=\Omega _0^T}{\text {argmin}}\dfrac{1}{2} \text {trace}(\Omega _0^2)-\text {trace}(\Omega _0(\Omega ^{t+1}+\Lambda _0^t))+\tau ||\Omega _0||_{1,\text {off}}. \end{aligned}$$

(28)

We rewrite the problem (28) in the following form:

$$\begin{aligned} \Omega _0^{t+1}=\underset{\Omega _0=\Omega _0^T}{\text {argmin}}\dfrac{1}{2} \text {trace}(\Omega _0^2)-\text {trace}(\Omega _0 A)+\tau ||\Omega _0||_{1,\text {off}}, \end{aligned}$$

(29)

where $A=(\Omega ^{t+1}+\Lambda _0^t)$. It is easy to check that the solution of the problem (29) is given as $\Omega _0^{t+1}=T(A, \tau )$, where operator T is defined in (6). As mentioned earlier, (29) is a crucial step of the algorithm, which leads to the soft-thresholding operator T. Thus, by substituting the operator T with the operator AT or WAT, we can obtain the our proposed estimators ADT or WADT, respectively.

From the second part of the Eq. (24) it follows that

$$\begin{aligned} \Omega _1^{t+1}=\underset{\Omega _1\succeq \epsilon I}{\text {argmin}}\dfrac{1}{2} \text {trace}(\Omega _1^2)-\text {trace}(\Omega _1(\Omega ^{t+1}+\Lambda _1^t)). \end{aligned}$$

(30)

The solution of the problem (30) is given as

$$\begin{aligned} \Omega _1^{t+1}=[\Omega ^{t+1}+\Lambda _1^t]_+, \end{aligned}$$

(31)

where for any symmetric matrix Z with an eigendecomosition $Z=U_Z \text {diag}(\alpha _1,\ldots ,\alpha _p)U_Z^T$ the operator $[Z]_+$ is defined as $[Z]_+=U_Z \text {diag}(\max \{\alpha _1,\epsilon \},\ldots ,\max \{\alpha _p,\epsilon \})U_Z^T$.

After having all the steps of the alternating direction method provided above, we describe Algorithm 1.

It is important to note that we can significantly reduce the computational time of Algorithm 1 by discarding the constraint $\Omega \succeq \epsilon I$ in the initial optimization problem (DT, WADT or ADT). This enables us to omit the step $\Theta _1^{t+1}=[\Theta ^{t+1}+\Lambda _1^t]_+$ from 2(c), which is the most computationally expensive part of the algorithm. We can call the optimization problem without the constraint $\Omega \succeq \epsilon I$ the secondary problem, defined as:

$$\begin{aligned} \tilde{\Omega }=\arg \min _{\Omega ^T = \Omega }\ \dfrac{1}{2}\text {trace}(\Omega ^2 S)-\text {trace}(\Omega )+\tau \text {PEN}(\Omega ), \end{aligned}$$

(32)

where $\text {PEN}(\Omega )$ term is defined according to the estimation method (DT, ADT or WADT). Following Zhang and Zou (2014), we also present the simplified version of Algorithm 1.

In other words, if $\tilde{\Omega }\succeq \epsilon I$, we have $\hat{\Omega }=\tilde{\Omega }$, otherwise we use Algorithm 1 to find $\hat{\Omega }$ considering $\tilde{\Omega }$ as the initial value of $\hat{\Omega }$. It is clear that Algorithm 2 is not self-contained and the implementation of Algorithm 1 may be required for some iterations. However, the introduction of Algorithm 2 may save the computational time considerably.

For both algorithms we consider convergence if the following two conditions are satisfied:

$$\begin{aligned} \dfrac{||\Theta ^{t+1}-\Theta ^{t}||_2}{\max (1,||\Theta ^{t}||_2,||\Theta ^{t+1}||_2)}<10^{-7}, \ \ \ \ \dfrac{||\Theta ^{t+1}_0-\Theta ^{t}_0||_2}{\max (1,||\Theta ^{t}_0||_2,||\Theta ^{t+1}_0||_2)}<10^{-7}. \end{aligned}$$

Finally, in the algorithm we use $\epsilon =10^{-8}$.

For more details, we refer to Zhang and Zou (2014).