In-Network Principal Component Analysis with Diffusion Strategies

Abstract

Principal component analysis (PCA) is a very well-known statistical analysis technique. In its conventional formulation, it requires the eigen-decomposition of the sample covariance matrix. Due to its high-computational complexity and large memory requirements, the estimation of the covariance matrix and its eigen-decomposition do not scale up when dealing with big data, such as in large-scale networks. Numerous studies have been conducted to overcome this issue, often by partitioning the unknown matrix. In this paper, we propose a novel framework for estimating the principal axes, iteratively and in a distributed in-network scheme, without the need to estimate the covariance matrix. To this end, a coupling is operated between criteria for iterative PCA and several strategies for in-network processing. The investigated strategies can be grouped in two classes, noncooperative and cooperative such as information diffusion and consensus strategies. Theoretical results on the performance of these strategies are provided, as well as a convergence analysis. The performance of the proposed approach for in-network PCA is illustrated on diverse applications, such as image processing and time series in wireless sensor networks, with a comparison to state-of-the-art techniques.

Appendix 1: Theoretical Results on the Performance of Diffusion Strategies

In this appendix, we study the theoretical performance of the diffusion strategies for in-network PCA in terms of convergence and recursive error.

1.1 General Diffusion Model

We introduce a general diffusion model so that ATC and CTA strategies are special cases of this model. We consider the case of extracting the first principal axis. The case of multiple axes can be easily derived as in the previous section. The general diffusion model is:

$$\begin{aligned} \varvec{\phi }_{k,t}&=\sum_{l\in{\mathcal{V}}_{k}} a_{1,kl} \, \varvec{w}_{l,t-1},\\ \varvec{\psi }_{k,t}&=\varvec{\phi }_{k,t}+\eta_{k,t} \, \sum_{l\in{\mathcal{V}}_k}c_{lk}\nabla_{\varvec{w}} J_l(\varvec{\phi }_{k,t}),\\ \varvec{w}_{k,t}&=\sum_{l\in{\mathcal{V}}_{k}} a_{2,kl} \, \varvec{\psi }_{l,t}, \end{aligned}$$

where $a_{1,kl}$, $c_{lk}$ and $a_{2,kl}$ are nonnegative coefficients. Let $\varvec{A}_1$, $\varvec{C}$ and $\varvec{A}_2$ be the matrices of these coefficients, respectively, so that the k-th column of $\varvec{A}_1$ and $\varvec{A}_2$ consist respectively of $\{ a_{1,kl}, l=1\ldots ,N\}$ and $\{ a_{2,kl}, l=1\ldots ,N\}$, and the k-th row of $\varvec{C}$ is formed of $\{ c_{kl}, l=1\ldots ,N\}$. Then these matrices satisfy the following conditions:

$$\varvec{A}_1^\top {{1\!\!1}}_N={{1\!\!1}}_N, \quad \varvec{C}{{1\!\!1}}_N={{1\!\!1}}_N, \quad \varvec{A}_2^\top {{1\!\!1}}_N={{1\!\!1}}_N,$$

where ${{1\!\!1}}_N$ is the $(N\times 1)$ vector whose entries are equal to one. In other words, $\varvec{A}_1$ and $\varvec{A}_2$ are left stochastic matrices and $\varvec{C}$ is a right stochastic matrix. Let $\varvec{I}_N$ be the $(N \times N)$ identity matrix. Cooperation strategies are derived according to the choice of matrices $\varvec{A}_1$, $\varvec{C}$ and $\varvec{A}_2$. For example, if $\varvec{A}_1=\varvec{I}_N$ and $\varvec{A}_2=\varvec{A}$, we get the ATC strategy. If $\varvec{A}_1=\varvec{A}$ and $\varvec{A}_2=\varvec{I}_N$, we have the CTA strategy. The choice $\varvec{C}=\varvec{I}_N$ is the case of no information exchange with the exception of the aggregation step. Furthermore, if $\varvec{A}_1=\varvec{A}_2=\varvec{C}=\varvec{I}_N$, we have the noncooperative strategy. Table 2 shows these different cases.

Table 2 Different choices of $\varvec{A}_1$, $\varvec{C}$ and $\varvec{A}_2$ corresponding to different strategies

1.2 Recursive Error

Our goal is to examine the convergence of the estimated $\varvec{w}_{k,t}$ obtained from the diffusion strategies toward the optimal solution of the centralized strategy, denoted $\varvec{w}_*$. For this, we introduce the following error vectors:

$$\begin{aligned} \widetilde{\varvec{\phi }}_{k,t}&= \varvec{w}_*-\varvec{\phi }_{k,t} \\ \widetilde{\varvec{\psi }}_{k,t}&= \varvec{w}_*-\varvec{\psi }_{k,t} \\ \widetilde{\varvec{w}}_{k,t}&= \varvec{w}_*-\varvec{w}_{k,t} \end{aligned}$$

Each of these error vectors measures the residue on the principal axis $\varvec{w}_*$. Before studying the convergence, we recall that:

$$y_{\varvec{w},k}=\varvec{w}^\top \varvec{x}_k =\varvec{x}_k^\top \varvec{w}.$$

(12)

Multiplying on the left both side by $\varvec{x}_k$ and averaging, we get:

$$\begin{aligned}{\mathbb{E}}(\varvec{x}_k y_{\varvec{w},k})&={\mathbb{E}}(\varvec{x}_k\varvec{x}_k^\top )\varvec{w}, \\ \varvec{m}_{w,k}&=\varvec{\mathfrak{C}}\varvec{w},\end{aligned}$$

(13)

where $\varvec{m}_{w,k}={\mathbb{E}}(\varvec{x}_k y_{\varvec{w},k})$. We deduce that, at the optimum,

$$\varvec{w}_*=\varvec{\mathfrak{C}}^{-1}\varvec{m}_{w,k}$$

(14)

is the solution of a linear system of equations and that this solution can be calculated by each node directly from $\varvec{\mathfrak{C}}$ and $\varvec{m}_{w,k}$. It is helpful to reinterpret this observation as the solution to a problem of minimizing the mean square reconstruction error. Consider then the cost function:

$$J_k^{\text{exp}}(\varvec{w})= \tfrac{1}{4} {\mathbb{E}} \Vert \varvec{x}_k - y_{\varvec{w},k} \varvec{w}\Vert ^2.$$

(15)

By developing this expression, we get:

$$J_k^{\text{exp}}(\varvec{w})=\tfrac{1}{4} \Big [{\mathbb{E}} \Vert \varvec{x}_k \Vert ^2 -\varvec{m}_{w,k}^\top \varvec{w}-\varvec{w}^\top \varvec{m}_{w,k}+\varvec{w}^\top {\mathbb{E}}(y_{\varvec{w},k}^2)\varvec{w}\Big ].$$

Taking the gradient of $J_k^{\text{exp}}(\varvec{w})$ with respect to $\varvec{w}$ gives:

$$\nabla_{\varvec{w}} J_k^{\text{exp}}(\varvec{w})= {\mathbb{E}}(y_{\varvec{w},k}^2) \varvec{w}-\varvec{m}_{w,k}.$$

By replacing the cost function in the general model, we obtain:

$$\varvec{\phi }_{k,t}=\sum_{l\in{\mathcal{V}}_{k}} a_{1,kl} \, \varvec{w}_{l,t-1},$$

(16)

$$\varvec{\psi }_{k,t}=\varvec{\phi }_{k,t}+\eta_{k,t} \, \sum_{l\in{\mathcal{V}}_k}c_{lk}(\varvec{m}_{w,k}- {\mathbb{E}}(y_{\varvec{w},k}^2) \varvec{\phi }_{k,t}),$$

(17)

$$\varvec{w}_{k,t}=\sum_{l\in{\mathcal{V}}_{k}} a_{2,kl} \, \varvec{\psi }_{l,t},$$

(18)

Using Eqs. (12) and (13), and the fact that $\varvec{w}_*$ is the eigenvector of the covariance matrix $\varvec{\mathfrak{C}}$, we have:

$$\varvec{m}_{w,k}=\varvec{\mathfrak{C}}\varvec{w}_*=\lambda \varvec{w}_*,$$

and

$${\mathbb{E}}(y_{\varvec{w},k}^2) = {\mathbb{E}}(\varvec{w}_*^\top \varvec{x}_k\varvec{x}_k^\top \varvec{w}_*) = \varvec{w}_*^\top \varvec{\mathfrak{C}}\varvec{w}_* = \lambda \Vert \varvec{w}_*\Vert ^2 =\lambda,$$

since $\varvec{w}_*$ is a unit-norm vector. Subtracting $\varvec{w}_*$ from both sides of the Eqs. (16)–(18), we get:

$$\widetilde{\varvec{\phi }}_{k,t}=\sum_{l\in{\mathcal{V}}_{k}} a_{1,kl} \, \widetilde{\varvec{w}}_{l,t-1},$$

(19)

$$\widetilde{\varvec{\psi }}_{k,t}=\Big ( \varvec{I}_p - \eta_{k,t} \, \sum_{l\in{\mathcal{V}}_k}c_{lk}\lambda \Big ) \widetilde{\varvec{\phi }}_{k,t},$$

(20)

$$\widetilde{ \varvec{w}}_{k,t}=\sum_{l\in{\mathcal{V}}_{k}} a_{2,kl} \, \widetilde{\varvec{\psi }}_{l,t}.$$

(21)

We can describe these relations by gathering information through the network in blocks of vectors and matrices. We collect the error vectors across all nodes in the $(N\times 1)$ block vectors, whose individual entries are of size $(p \times 1)$ each:

$$\widetilde{\psi _t}=\left[\begin{array}{l} \widetilde{\varvec{\psi }}_{1,t} \\ \widetilde{\varvec{\psi }}_{2,t} \\ \vdots \\ \widetilde{\varvec{\psi }}_{N,t} \\ \end{array}\right], \quad \widetilde{\phi _t}=\left[\begin{array}{l} \widetilde{\varvec{\phi }}_{1,t} \\ \widetilde{\varvec{\phi }}_{2,t} \\ \vdots \\ \widetilde{\varvec{\phi }}_{N,t} \\ \end{array}\right], \quad \widetilde{\varvec{w}_t}=\left[\begin{array}{l} \widetilde{\varvec{w}}_{1,t} \\ \widetilde{\varvec{w}}_{2,t} \\ \vdots \\ \widetilde{\varvec{w}}_{N,t} \\ \end{array}\right].$$

These block vectors represent the error of the network at the iteration t. We also introduce the $(N\times N)$ diagonal matrices:

$$\begin{aligned} \varvec{{\mathcal{M}}}&= {\text{diag}}\left\{ \eta_{1,t}, \eta_{2,t}, \ldots , \eta_{N,t} \right\} \\ \varvec{\mathcal{L}}&= {\text{diag}} \left\{ \sum_{l\in{\mathcal{V}}_1}c_{l1}\lambda , \sum_{l\in{\mathcal{V}}_2}c_{l2}\lambda , \ldots , \sum_{l\in{\mathcal{V}}_N}c_{lN}\lambda \right\} \end{aligned}$$

In the case where $\varvec{C}=\varvec{I}_N$, we have $\varvec{\mathcal{L}}=\lambda \varvec{I}_N$. Furthermore, we introduce the Kronecker product:

$$\varvec{\mathcal{A}}_1=\varvec{A}_1\otimes \varvec{I}_p,\quad \varvec{\mathcal{A}}_2=\varvec{A}_2\otimes \varvec{I}_p$$

The matrix $\varvec{\mathcal{A}}_1$ is a $(N\times N)$ block matrix, whose (l, k) input block is $a_{1,lk}\varvec{I}_p$. Similarly for $\varvec{\mathcal{A}}_2$. Using these definitions, we can write the Eqs. (19)–(21) as follows:

$$\widetilde{\varvec{\phi }}_{t}=\varvec{\mathcal{A}}_1^\top \, \widetilde{\varvec{w}}_{t-1},$$

(22)

$$\widetilde{\varvec{\psi }}_{t}=\Big ( \varvec{I}_N - \varvec{\mathcal{M}}\varvec{\mathcal{L}}\Big ) \widetilde{\varvec{\phi }}_{t},$$

(23)

$$\widetilde{ \varvec{w}}_{t}=\varvec{\mathcal{A}}_2^\top \, \widetilde{\varvec{\psi }}_{t}.$$

(24)

Therefore, the network error $\widetilde{ \varvec{w}}_{t}$ evolves according to the following dynamics:

$$\widetilde{ \varvec{w}}_{t}=\varvec{\mathcal{A}}_2^\top (\varvec{I}_N - \varvec{\mathcal{M}}\varvec{\mathcal{L}}) \varvec{\mathcal{A}}_1^\top \widetilde{ \varvec{w}}_{t-1}.$$

(25)

If each node minimizes its own cost function $J_k^{\text{exp}}(\varvec{w})$ using the noncooperative strategy, then the vector error of the N nodes evolves according to the following dynamics:

$$\widetilde{ \varvec{w}}_{t}= (\varvec{I}_N - \lambda \varvec{\mathcal{M}}) \widetilde{ \varvec{w}}_{t-1},$$

where, since $\varvec{A}_1=\varvec{A}_2=\varvec{C}=\varvec{I}_N$ in this case, the matrices $\varvec{\mathcal{A}}_1$ and $\varvec{\mathcal{A}}_2$ do not appear and $\varvec{\mathcal{L}}$ boils down to $\lambda \varvec{I}_N$.

1.3 Convergence

The expression of the recursive error vector (25) includes block vectors and matrices. To examine the stability and convergence properties of this error, we investigate results on standard block vectors [30]. Indeed, the error vector $\widetilde{ \varvec{w}}_{t}$ in Eq. (25) converges to zero if, and only if, the matrix $\varvec{\mathcal{A}}_2^\top (\varvec{I}_N - \varvec{\mathcal{M}}\varvec{\mathcal{L}}) \varvec{\mathcal{A}}_1^\top$ is a stable matrix, i.e., all eigenvalues are strictly inside the unit disc [31]. Since $\varvec{A}_1$ and $\varvec{A}_2$ are left stochastic matrices [32], $\varvec{\mathcal{A}}_2^\top (\varvec{I}_N - \varvec{\mathcal{M}}\varvec{\mathcal{L}}) \varvec{\mathcal{A}}_1^\top$ is stable if $(\varvec{I}_N - \varvec{\mathcal{M}}\varvec{\mathcal{L}})$ is stable. This stability is checked by the following condition:

$$\eta_{k,t}<\frac{2}{\sum_{l\in{\mathcal{V}}_k} c_{lk} \lambda}$$

(26)

We note that this stability does not depend on the combination of matrices $\varvec{A}_1$ and $\varvec{A}_2$, but only on the matrix $\varvec{C}$. If there is no information exchange between the nodes except for the aggregation step, i.e., $\varvec{C}=\varvec{I}_N$, this condition becomes:

$$\eta_{k,t}<\frac{2}{ \lambda }$$

(27)

Similarly in the case of noncooperative strategy, i.e., $\varvec{A}_1=\varvec{A}_2=\varvec{C}=\varvec{I}_N$, we have the same condition.