Skip to main content

The APC Algorithm of Solving Large-Scale Linear Systems: A Generalized Analysis

  • Conference paper
  • First Online:
Communications and Networking (ChinaCom 2022)

Abstract

A new algorithm called accelerated projection-based consensus (APC) has recently emerged as a promising approach to solve large-scale systems of linear equations in a distributed fashion. The algorithm uses a federated computational architecture, and attracts increasing research interest; however, it’s performance analysis is still incomplete, e.g., the error performance under noisy condition has not yet been investigated. In this paper, we focus on providing a generalized analysis by the use of the linear system theory, such that the error performance of the APC algorithm for solving linear systems in presence of additive noise can be clarified. We specifically provide a closed-form expression of the error of solution attained by the APC algorithm. Numerical results demonstrate the error performance of the APC algorithm, validating the presented analysis.

This work is sponsored in part by the National Natural Science Foundation of China (grant no. 61971058, 61801048, 61631004, 62071063) and Beijing Natural Science Foundation (grant no. L202014, L192002).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    The Jordan block \({\textbf {J}}_{n_l} (\xi _l)\) is an \(n_l\)-by-\(n_l\) upper triangular matrix in which \(\xi _l\) appears \(n_l\) times on the main diagonal; if \(n_l > 1\), there are \(n_l-1\) elements 1 in the superdiagonal; all other elements are 0 [14].

References

  1. Azizan-Ruhi, N., Lahouti, F., Avestimehrm, A., Hassibi, B.: Distributed solution of large-scale linear systems via accelerated projection based consensus. IEEE Trans. Sig. Process. 67(14), 3806–3817 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  2. Alaviani, S., Elia, N.: A distributed algorithm for solving linear algebraic equations over random networks. IEEE Trans. Autom. Control 66(5), 2399–2406 (2021)

    Article  MathSciNet  MATH  Google Scholar 

  3. Zivojevic, D., et al.: Distributed weighted least-squares and Gaussian belief propagation: an integrated approach. In: Proceedings of 2021 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), pp. 432–437 (2021)

    Google Scholar 

  4. Wang, X., Mou, S., Sun, D.: Improvement of a distributed algorithm for solving linear equations. IEEE Trans. Ind. Electron. 64(4), 3113–3117 (2017)

    Article  Google Scholar 

  5. Zuo, C., Deng, H., Zhang, J., Qi, Y.: Distributed channel estimation algorithm for mmWave massive MIMO communication systems. In: Proceedings of 2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall), pp. 1–6 (2021)

    Google Scholar 

  6. Jia, M., Wang, Y., Shen, C., Hug, G.: Privacy-preserving distributed probabilistic load flow. IEEE Trans. Power Syst. 36(2), 1616–1627 (2021)

    Article  Google Scholar 

  7. Chakrabarti, K., Gupta, N., Chopra, N.: Iterative pre-conditioning to expedite the gradient-descent method. In: Proceedings of 2020 American Control Conference (ACC), pp. 3977–3982 (2020)

    Google Scholar 

  8. Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: challenges, methods, and future directions. IEEE Sig. Process. Mag. 37(3), 50–60 (2020)

    Article  Google Scholar 

  9. Chakrabarti, K., Gupta, N., Chopra, N.: Robustness of iteratively pre-conditioned gradient-descent method: the case of distributed linear regression problem. In: Proceedings of 2021 American Control Conference (ACC), pp. 2248–2253 (2021)

    Google Scholar 

  10. Dutta, S., Cadambe, V., Grover, P.: Short-Dot: computing large linear transforms distributedly using coded short dot product. In: Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS), pp. 2092–2100 (2016)

    Google Scholar 

  11. Dutta, S., Cadambe, V., Grover, P.: Short-Dot: computing large linear transforms distributedly using coded short dot products. IEEE Trans. Inf. Theory 65(10), 6171–6193 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  12. Dutta, S., et al.: On the optimal recovery threshold of coded matrix multiplication. IEEE Trans. Inf. Theory 66(1), 278–301 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  13. Zhao, Z., Lu, S., Hong, M., Palomar, D.: Distributed optimization for generalized phase retrieval over networks. In: Proceedings of 2018 52nd Asilomar Conference on Signals, Systems and Computers, pp. 48–52 (2018)

    Google Scholar 

  14. Horn, R., Johnson, C.: Matrix Analysis. Cambridge University Press, 2nd ed., Cambridge, USA (2013)

    Google Scholar 

  15. Chen, C.: Linear System Theory and Design. Oxford University Press, 3rd ed., Oxford, USA (1999)

    Google Scholar 

  16. Meyer, C.: Matrix Analysis and Applied Linear Algebra. Cambridge University Press, Cambridge (2000)

    Google Scholar 

Download references

Acknowledgment

This work is sponsored in part by the National Natural Science Foundation of China (grant no. 61801048, 61971058, 62071063) and Beijing Natural Science Foundation (grant no. L202014, L192002).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yuan Qi .

Editor information

Editors and Affiliations

Appendix: Proofs of Main Results

Appendix: Proofs of Main Results

Proof of Lemma 3: A computation reveals that

$$\begin{aligned} |\xi _{i, \pm }|= \sqrt{(\gamma -1)(\eta -1)} = \alpha ,\end{aligned}$$
(37)

for all \(i=1, \cdots , s\). Then, it’s a simple consequence of Lemma A.2 that \(\rho ({\textbf {G}}) = \alpha <1\) [1], i.e., the largest magnitude eigenvalue of \({\textbf {G}}\) is less than 1. This implies that \(\lim _{t \rightarrow \infty } {\textbf {G}}^t = {\textbf {O}}_{(M+1) s}\) and \(\sum _{l=0}^{\infty } {\textbf {G}}^{l} = ({\textbf {I}}-{\textbf {G}})^{-1}\) [16].                                           \(\blacksquare \)

Proof of Lemma 4: All we need to do is to show that the geometric multiplicity of eigenvalue \(1-\gamma \) is \((M-1) s\) [14, 16]. We begin by noting that

$$\begin{aligned} {\textbf {G}} - (1-\gamma ) {\textbf {I}}= & {} \left[ \begin{array}{cc} {\textbf {O}}_{M s} &{} \gamma \left[ \begin{array}{c} {\textbf {P}}_{1}^\bot \\ \vdots \\ {\textbf {P}}_{M}^\bot \\ \end{array} \right] \\ \frac{\eta (1-\gamma )}{M} \left[ \begin{array}{ccc} {\textbf {I}}_{s} &{} \cdots &{} {\textbf {I}}_{s} \end{array} \right] &{} {\textbf {B}} - (1-\gamma ) {\textbf {I}}_{s} \\ \end{array} \right] . \end{aligned}$$

Applying elementary row and column operations [14], we can transform \({\textbf {G}} - (1-\gamma ) {\textbf {I}}\) into a simple form

$$\begin{aligned} {\textbf {G}}_T = \left[ \begin{array}{cc} {\textbf {O}}_{M s} &{} \gamma \left[ \begin{array}{c} {\textbf {P}}_{1}^\bot \\ \vdots \\ {\textbf {P}}_{M}^\bot \\ \end{array} \right] \\ \frac{\eta (1-\gamma )}{M} \left[ \begin{array}{cccc} {\textbf {O}}_{s} &{} \cdots &{} {\textbf {O}}_{s} &{} {\textbf {I}}_{s} \end{array} \right] &{} {\textbf {O}}_{s} \\ \end{array} \right] . \end{aligned}$$

Since elementary operations do not change the rank of a matrix [14], it follows that

$$\begin{aligned} rank \left( {\textbf {G}} - (1-\gamma ) {\textbf {I}} \right) = rank \left( {\textbf {G}}_T\right) \le 2\,s.\end{aligned}$$
(38)

The geometric multiplicity of eigenvalue \(1-\gamma \) of \({\textbf {G}}\) is equal to \((M+1) s - rank \left( {\textbf {G}} - (1-\gamma ) {\textbf {I}} \right) \) that is not less than \((M-1) s\) according to (38). Furthermore, the geometric multiplicity should not be larger than algebraic multiplicity, i.e., \((M-1) s\), for \(1-\gamma \), and thus can only be \((M-1) s\) [14, Theorem 1.4.10]. This implies that the number of Jordan blocks of \({\textbf {G}}\) corresponding to \(1-\gamma \), is \((M-1) s\) [14]. Finally, because the geometric and algebraic multiplicities of \(1-\gamma \) are equal, every Jordan block corresponding to \(1-\gamma \) is 1-by-1 [14].                                                                                                    \(\blacksquare \)

Proof of Lemma 5: First note that \({\textbf {X}}\) is Hermitian and \({\textbf {X}}\) is unitarily diagonalizable [14, Theorem 2.5.6]. Then, applying [14, Theorem 2.5.3] yields the assertion (a).

To prove the assertion (b), let us verify whether \({\textbf {v}}_G(\xi )\) satisfies the eigenvalue-eigenvector equation \(({\textbf {G}} - \xi {\textbf {I}}) {\textbf {v}}_G(\xi ) = {\textbf {0}}_{(M+1) s}\), where

$$\begin{aligned}{} & {} {\textbf {G}} - \xi {\textbf {I}}_{(M+1) s} = \left[ \begin{array}{cc} (1-\gamma - \xi ) {\textbf {I}}_{M s} &{} \gamma \left[ \begin{array}{c} {\textbf {P}}_{1}^\bot \\ \vdots \\ {\textbf {P}}_{M}^\bot \\ \end{array} \right] \\ \frac{\eta (1-\gamma )}{M} \left[ \begin{array}{ccc} {\textbf {I}}_{s} &{} \cdots &{} {\textbf {I}}_{s} \end{array} \right] &{} {\textbf {B}} - \xi {\textbf {I}}_{s} \\ \end{array} \right] . \end{aligned}$$

On the one hand, it is easy to check that

$$\begin{aligned} \left[ \begin{array}{cc} (1-\gamma - \xi ) {\textbf {I}}_{M L} &{} \gamma \left[ \begin{array}{c} {\textbf {P}}_{1}^\bot \\ \vdots \\ {\textbf {P}}_{M}^\bot \\ \end{array} \right] \\ \end{array} \right] {\textbf {v}}_G(\xi ) = {\textbf {0}}_{M s}. \end{aligned}$$
(39)

On the other hand, since \({\textbf {X}} {\textbf {v}}_i = \theta _i {\textbf {v}}_i\) and \(\frac{\sum _{\ell = 1}^M {\textbf {P}}_{\ell }^\bot }{M} = {\textbf {I}}_{s} - {\textbf {X}}\), we have \(\frac{\sum _{\ell = 1}^M {\textbf {P}}_{\ell }^\bot }{M} {\textbf {v}}_i = (1 - \theta _i) {\textbf {v}}_i\) and \(\left( {\textbf {B}} - \xi {\textbf {I}}_{s}\right) {\textbf {v}}_i = (- \eta \gamma \theta _i +1 - \eta + \eta \gamma - \xi ) {\textbf {v}}_i\). As (21) ensures that \(-\frac{\eta \gamma (1-\gamma )}{1-\gamma - \xi } (1 - \theta ) - \eta \gamma \theta + 1 - \eta + \eta \gamma - \xi = 0\) if \(\xi \ne 1-\gamma \), it yields

$$\begin{aligned}{} & {} \left[ \begin{array}{cc} \frac{\eta (1-\gamma )}{M} \left[ \begin{array}{ccc} {\textbf {I}}_{s} &{} \cdots &{} {\textbf {I}}_{s} \end{array} \right] &{} {\textbf {B}} - \xi {\textbf {I}}_{s} \\ \end{array} \right] {\textbf {v}}_G(\xi ) \nonumber \\ {}{} & {} \quad = \left[ - \frac{\eta \gamma (1-\gamma )}{(1-\gamma - \xi )} \frac{\sum _{\ell = 1}^M {\textbf {P}}_{\ell }^\bot }{M} + \left( {\textbf {B}} - \xi {\textbf {I}}_{s}\right) \right] {\textbf {v}}_{i} = {\textbf {0}}_{s}. \quad \quad \end{aligned}$$
(40)

Finally, (39) and (40) together imply that \(({\textbf {G}} - \xi {\textbf {I}}) {\textbf {v}}_G(\xi ) = {\textbf {0}}_{(M+1) s}\) which completes the proof of the assertion (b).                                                                            \(\blacksquare \)

Proof of Lemma 6: It follows from (22) that \(\xi _{1, +} = \xi _{1, -}\) and \(\xi _{s, +} = \xi _{s, -}\) since \(\theta _s = \theta _{min}\) and \(\theta _1 = \theta _{max}\), while \(\xi _{i, +} \ne \xi _{i, -}\) if \(\theta _{min}< \theta _i < \theta _{max}\). One of the consequences of Lemma 5 is that the number of linearly dependent eigenvectors associated with every eigenvalue \(\xi \in \{\xi _{1, \pm },\cdots ,\xi _{s, \pm }\}\) is not larger than 2. Therefore, the geometric multiplicity of \(\xi \) is 1, and the Jordan block of \({\textbf {G}}\) with eigenvalue \(\xi \) is 2-by-2, when \(\xi = \xi _{1, +} = \xi _{1, -}\) or \(\xi = \xi _{s, +} = \xi _{s, -}\). If \(\xi = \xi _{i, +}\) or \(\xi = \xi _{i, -}\) with \(\xi _{i, +} \ne \xi _{i, -}\), then the geometric multiplicity of \(\xi \) is 2 such that the Jordan block with eigenvalue \(\xi \) is 1-by-1.                                                   \(\blacksquare \)

Proof of Lemma 7: It can easily be verified by using the property of direct sum together with [16, (7.10.7)].                                                                           \(\blacksquare \)

Proof of Theorem 2: Combining (25) and (26) yields (31), and then multiplying on both sides of (31) by \({\textbf {I}}_{(M+1) s}-{\textbf {G}}\) produces (33). By the Rayleigh quotient theorem [16] and Corollary 1, we obtain

$$\begin{aligned} \left\| \boldsymbol{\epsilon } \right\| _2 \le \left\| {\textbf {J}}^t \right\| _2 \left\| \left( {\textbf {I}}_{(M+1) s}-{\textbf {G}}\right) {\textbf {d}}(0) - \tilde{{\textbf {w}}}_d \right\| _2,\end{aligned}$$
(41)

where \(\left\| {\textbf {J}}^t \right\| _2 = O \left( \alpha ^{t} \right) \) [15].

The remaining part of the proof is to derive an upper bound on \(\left\| \left( {\textbf {I}}_{(M+1) s}-{\textbf {G}}\right) {\textbf {d}}(0) - \tilde{{\textbf {w}}}_d \right\| _2\). First, it is clear that

$$\begin{aligned} {\textbf {d}}(0) = \left[ \begin{array}{c} -{\textbf {P}}_1^\bot {\textbf {x}}^* + {\textbf {A}}_1^H \left( {\textbf {A}}_1 {\textbf {A}}_1^H \right) ^{-1} \tilde{w}_1 \\ \vdots \\ -{\textbf {P}}_M^\bot {\textbf {x}}^* + {\textbf {A}}_M^H \left( {\textbf {A}}_M {\textbf {A}}_M^H \right) ^{-1} \tilde{w}_M \\ - \frac{1}{M} \sum _{\ell =1}^M {\textbf {P}}_\ell ^\bot {\textbf {x}}^* + \frac{1}{M} \sum _{\ell =1}^M {\textbf {A}}_\ell ^H \left( {\textbf {A}}_\ell {\textbf {A}}_\ell ^H \right) ^{-1} \tilde{w}_\ell \\ \end{array} \right] , \end{aligned}$$
(42)

according to the initialization of the APC algorithm as in Line 5 of Algorithm 1 together with (19), and thus

$$\begin{aligned} \left( {\textbf {I}}_{(M+1) s}-{\textbf {G}}\right) {\textbf {d}}(0) - \tilde{{\textbf {w}}}_d = \left[ \begin{array}{c} - \gamma {\textbf {P}}_1^\bot \left( {\textbf {x}}^* + \bar{{\textbf {e}}}(0) \right) \\ \vdots \\ - \gamma {\textbf {P}}_M^\bot \left( {\textbf {x}}^* + \bar{{\textbf {e}}}(0) \right) \\ - \eta \gamma {\textbf {X}} \bar{{\textbf {e}}}(0) \\ \end{array} \right] . \end{aligned}$$

This yields

$$\begin{aligned}{} & {} \left\| \left( {\textbf {I}}_{(M+1) s}-{\textbf {G}}\right) {\textbf {d}}(0) - \tilde{{\textbf {w}}}_d \right\| _2^2 \le 2 \gamma ^2 M \left( 1 - \theta _{min} \right) \left\| {\textbf {x}}^* \right\| _2^2 \\ {}{} & {} \quad \quad + \left( 2 \gamma ^2 M \left( 1 - \theta _{min} \right) + \eta ^2 \gamma ^2 \theta _{max}^2 \right) \left\| \bar{{\textbf {e}}}(0) \right\| _2^2. \end{aligned}$$

Then, since \(\left\| \bar{{\textbf {e}}}(0) \right\| _2^2 \le \frac{\sum _{\ell =1}^M \left\| {\textbf {e}}_\ell (0) \right\| _2^2}{M}\), it follows from Lemma 1 and (42) that

$$\begin{aligned} \left\| \bar{{\textbf {e}}}(0) \right\| _2^2\le & {} \frac{M \left( {\textbf {x}}^* \right) ^H \left( {\textbf {I}} - {\textbf {X}}\right) {\textbf {x}}^* + \sum _{\ell = 1}^M \left( {\textbf {A}}_\ell {\textbf {A}}_\ell ^H \right) ^{-2} \tilde{w}_\ell }{M} \\\le & {} \left( 1 - \theta _{min} \right) \left\| {\textbf {x}}^* \right\| ^2_2 + \frac{\sum _{\ell = 1}^M \left( {\textbf {A}}_\ell {\textbf {A}}_\ell ^H \right) ^{-2} \tilde{w}_\ell }{M}, \end{aligned}$$

and

$$\begin{aligned}{} & {} \left\| \left( {\textbf {I}}_{(M+1) s}-{\textbf {G}}\right) {\textbf {d}}(0) - \tilde{{\textbf {w}}}_d \right\| _2 \le \left( 2 \gamma ^2 M \left( 1 - \theta _{min} \right) + \eta ^2 \gamma ^2 \theta _{max}^2 \right) \\{} & {} \quad \quad \times \left( \left( 2 - \theta _{min} \right) \left\| {\textbf {x}}^* \right\| ^2_2 + \frac{\sum _{\ell = 1}^M \left( {\textbf {A}}_\ell {\textbf {A}}_\ell ^H \right) ^{-2} \tilde{w}_\ell }{M} \right) . \end{aligned}$$

Substituting this inequality into (41) gives \(\Vert \boldsymbol{\epsilon } \Vert _2 = O \left( \alpha ^{t} \right) \).                                    \(\blacksquare \)

Proof of Theorem 3: Observe that

$$\begin{aligned}{} & {} \left[ \begin{array}{cccc} \frac{1}{M}{} {\textbf {I}}_{s}&\frac{1}{M}{} {\textbf {I}}_{s}&\cdots&{\textbf {I}}_{s}\end{array} \right] {\textbf {d}}(t) = 2 \bar{{\textbf {e}}}(t), \\ {}{} & {} \left[ \begin{array}{cccc}\frac{1}{M}{} {\textbf {I}}_{s}&\frac{1}{M}{} {\textbf {I}}_{s}&\cdots&{\textbf {I}}_{s}\end{array} \right] {\textbf {G}} {\textbf {d}}(t) = \left( 2{\textbf {I}}_{s} - \gamma (1+\eta ){\textbf {X}} \right) \bar{{\textbf {e}}}(t). \end{aligned}$$

A calculation also shows that \(\left[ \begin{array}{cccc}\frac{1}{M}{} {\textbf {I}}_{s}&\frac{1}{M}{} {\textbf {I}}_{s}&\cdots&{\textbf {I}}_{s}\end{array} \right] \tilde{{\textbf {w}}}_d = \frac{\gamma }{M} \boldsymbol{A}^H \boldsymbol{\Xi } \tilde{{\textbf {w}}}\). Combining these results, we get

$$\begin{aligned} \quad \gamma (1+\eta ){\textbf {X}} \bar{{\textbf {e}}}(t) = \frac{\gamma }{M} \boldsymbol{A}^H \boldsymbol{\Xi } \tilde{{\textbf {w}}} + \left[ \begin{array}{cccc}\frac{1}{M}{} {\textbf {I}}_{s}&\frac{1}{M}{} {\textbf {I}}_{s}&\cdots&{\textbf {I}}_{s}\end{array} \right] \boldsymbol{\epsilon }, \end{aligned}$$

which can be rewritten as (34). From Theorem 1, it follows that \(\Vert \boldsymbol{\bar{\epsilon }} \Vert _2 = O \left( \alpha ^{t}\right) \). Finally, based on the definition in (12), Theorem 3 can be verified.                                   \(\blacksquare \)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zhang, J., Xue, Y., Qi, Y., Wang, J. (2023). The APC Algorithm of Solving Large-Scale Linear Systems: A Generalized Analysis. In: Gao, F., Wu, J., Li, Y., Gao, H. (eds) Communications and Networking. ChinaCom 2022. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 500. Springer, Cham. https://doi.org/10.1007/978-3-031-34790-0_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-34790-0_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-34789-4

  • Online ISBN: 978-3-031-34790-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics