High-dimensional disjoint factor analysis with its EM algorithm version

Abstract

Vichi (Advances in Data Analysis and Classification, 11:563–591, 2017) proposed disjoint factor analysis (DFA), which is a factor analysis procedure subject to the constraint that variables are mutually disjoint. That is, in the DFA solution, each variable loads only a single factor among multiple ones. It implies that the variables are clustered into exclusive groups. Such variable clustering is considered useful for high-dimensional data with variables much more than observations. However, the feasibility of DFA for high-dimensional data has not been considered in Vichi (2017). Thus, one purpose of this paper is to show the feasibility and usefulness of DFA for high-dimensional data. Another purpose is to propose a new computational procedure for DFA, in which an EM algorithm is used. This procedure is called EM-DFA in particular, which can serve the same original purpose as in Vichi (2017) but more efficiently. Numerical studies demonstrate that both DFA and EM-DFA can cluster variables fairly well, with EM-DFA more computationally efficient.

Appendix

Here, we consider minimizing loss function (7) over b_k with the other parameter kept fixed. For the sake of simplicity, let us omit the subscript k from the symbols in (7). Then, it is simplified as f(b) = log|bb′ + ${{\varvec{\Psi}}}$|+ trS(bb′ + ${{\varvec{\Psi}}}$)⁻¹. Using W = ${{\varvec{\Psi}}}$^−1/2S${{\varvec{\Psi}}}$^−1/2 and

$$\phi \left( {\mathbf{b}} \right) = { 1} + {\mathbf{b^{\prime}}}{{\varvec{\Psi}}}^{{ - {1}}} {\mathbf{b}},$$

(20)

the loss function can be rewritten as

$$\begin{aligned} f\left( {\mathbf{b}} \right) & \, = \log |{{\varvec{\Psi}}}| + \log \phi \left( {\mathbf{b}} \right) + {\text{tr}}{\mathbf{S}}{{\varvec{\Psi}}}^{ - 1} - \frac{1}{\phi \left( {\mathbf{b}} \right)}{\mathbf{b^{\prime}}}{{\varvec{\Psi}}}^{ - 1} {\mathbf{S}}{{\varvec{\Psi}}}^{ - 1} {\mathbf{b}} \\ & = {\text{log}}|{\mathbf{S}}| - {\text{log}}|{\mathbf{W}}| + {\text{ log}}\phi \left( {\mathbf{b}} \right) + {\text{tr}}{\mathbf{W}} \\ &- \frac{1}{\phi \left( {\mathbf{b}} \right)}{\mathbf{b^{\prime}}}{{\varvec{\Psi}}}^{{ - {1}/{2}}} {\mathbf{W}}{{\varvec{\Psi}}}^{{ - {1}/{2}}} {\mathbf{b}}, \\ \end{aligned}$$

(21)

where we have used |bb′ + Ψ|= ϕ(b)|Ψ| (Seber, 2008, p. 312), (bb′ + Ψ)⁻¹ = Ψ⁻¹ − ϕ(b)⁻¹Ψ⁻¹bb′Ψ⁻¹ (Seber, 2008, p. 309), and |Ψ| =|S|/|W|.

Using Σ = bb′ + Ψ, (21) is also expressed as f(b) = log|Σ|+ trSΣ⁻¹. This minimizer must satisfy ∂f(b)/∂b = (Σ⁻¹ − Σ⁻¹SΣ⁻¹)b = 0_m, or equivalently,

$${\mathbf{b}} = {\mathbf{S \Sigma}}^{{ - {1}}} {\mathbf{b}}.$$

(22)

Multiplying both sides by b′${{\varvec{\Psi}}}$⁻¹b leads to

$$({\mathbf{b^{\prime}}}{{\varvec{\Psi}}}^{{ - {1}}} {\mathbf{b}}){\mathbf{b}} = {\mathbf{S \Sigma}}^{{ - {1}}} {\mathbf{bb^{\prime}}}{{\varvec{\Psi}}}^{{ - {1}}} {\mathbf{b}} = {\mathbf{S \Sigma}}^{{ - {1}}} ({\mathbf{\Sigma}} -{{\varvec{\Psi}}}){{\varvec{\Psi}}}^{{ - {1}}} {\mathbf{b}} = {\mathbf{S}}{{\varvec{\Psi}}}^{{ - {1}}} {\mathbf{b}} - {\mathbf{S \Sigma}}^{{ - {1}}} {\mathbf{b}} = {\mathbf{S}}{{\varvec{\Psi}}}^{{ - {1}}} {\mathbf{b}} - {\mathbf{b}},$$

(23)

where bb′ = Σ − ${{\varvec{\Psi}}}$ and (22) have been used. We can use (20) to rewrite (23) as S${{\varvec{\Psi}}}$⁻¹b = ϕ(b)b and premultiply both sides by ${{\varvec{\Psi}}}$^−1/2 to have ${{\varvec{\Psi}}}$^−1/2S${{\varvec{\Psi}}}$^−1/2${{\varvec{\Psi}}}$^−1/2b = ϕ(b) ${{\varvec{\Psi}}}$^−1/2b, i.e.,

$${\mathbf{W}}({{\varvec{\Psi}}}^{{ - {1}/{2}}} {\mathbf{b}}) \, = \phi \left( {\mathbf{b}} \right)({{\varvec{\Psi}}}^{{ - {1}/{2}}} {\mathbf{b}}) \, .$$

(24)

This is an eigen equation showing that an eigenvalue of W is expressed as (20), when b is the optimal.

Using (20) and (24), we can rewrite the final term on the right side of (21) as ϕ(b)⁻¹b′${{\varvec{\Psi}}}$^−1/2W${{\varvec{\Psi}}}$^−1/2b = ϕ(b)⁻¹b′${{\varvec{\Psi}}}$^−1/2{ϕ(b)( ${{\varvec{\Psi}}}$^−1/2b)} = b′${{\varvec{\Psi}}}$⁻¹b = ϕ(b) − 1. Thus, (21) is rewritten as

$$\begin{aligned} f\left( {\mathbf{b}} \right) & \, ={\text{log}}|{\mathbf{S}}| - {\text{log}}|{\mathbf{W}}| + {\text{ log}}\phi \left( {\mathbf{b}} \right) + {\text{tr}}{\mathbf{W}} - \left\{ {\phi \left( {\mathbf{b}} \right) - {1}} \right\} \\ & = {\text{ log}}|{\mathbf{S}}| - {\text{log}}|{\mathbf{W}}| + {\text{ tr}}{\mathbf{W}} - \left\{ {\phi \left( {\mathbf{b}} \right) - {\text{log}}\phi \left( {\mathbf{b}} \right) - {1}} \right\}. \\ \end{aligned}$$

(25)

This shows that minimizing (21) over b amounts to maximizing g = ϕ(b) − logϕ(b) − 1. Here, dg/dϕ(b) = 1 − 1/ϕ(b) shows a larger ϕ(b) leading to a greater g, and b is subject to (20) being an eigenvalue of W. Thus, the maximization is attained by selecting b so that (20) is the largest eigenvalue of W: ρ_max(W) = 1 + b′${{\varvec{\Psi}}}$⁻¹b. This holds for b = Ψ^1/2u{ρ_max(W) − 1}^1/2, i.e., (7).