Skip to main content
Book cover

Splitting Methods in Communication, Imaging, Science, and Engineering pp 461–497Cite as

Decentralized Learning for Wireless Communications and Networking

Part of the Scientific Computation book series (SCIENTCOMP)

Abstract

This chapter deals with decentralized learning algorithms for in-network processing of graph-valued data. A generic learning problem is formulated and recast into a separable form, which is iteratively minimized using the alternating-direction method of multipliers (ADMM) so as to gain the desired degree of parallelization. Without exchanging elements from the distributed training sets and keeping inter-node communications at affordable levels, the local (per-node) learners consent to the desired quantity inferred globally, meaning the one obtained if the entire training data set were centrally available. Impact of the decentralized learning framework to contemporary wireless communications and networking tasks is illustrated through case studies including target tracking using wireless sensor networks, unveiling Internet traffic anomalies, power system state estimation, as well as spectrum cartography for wireless cognitive radio networks.

Keywords

  • Basis Expansion Model
  • Random Geometric Graph
  • Well Linear Unbiased Estimation
  • Optimal Primal Solution
  • Minimal Sufficient Statistic

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-41589-5_14
  • Chapter length: 37 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   169.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-41589-5
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   219.99
Price excludes VAT (USA)
Hardcover Book
USD   219.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 14.1
Fig. 14.2
Fig. 14.3
Fig. 14.4
Fig. 14.5
Fig. 14.6
Fig. 14.7
Fig. 14.8
Fig. 14.9

References

  1. Abur, A., Gomez-Exposito, A.: Power System State Estimation: Theory and Implementation. Marcel Dekker, New York, NY (2004)

    CrossRef  Google Scholar 

  2. Albaladejo, C., Sanchez, P., Iborra, A., Soto, F., Lopez, J. A., Torres, R.: Wireless sensor networks for oceanographic monitoring: A systematic review. Sensors. 10, 6948–6968 (2010)

    CrossRef  Google Scholar 

  3. Anderson, B. D., Moore, J. B.: Optimal Filtering. Prentice Hall, Englewood Cliffs, NJ (1979)

    MATH  Google Scholar 

  4. Barbarossa, S., Scutari, G.: Decentralized maximum likelihood estimation for sensor networks composed of nonlinearly coupled dynamical systems. IEEE Trans. Signal Process. 55 3456–3470 (2007)

    CrossRef  MathSciNet  Google Scholar 

  5. Bazerque, J. A., Giannakis, G. B.: Distributed spectrum sensing for cognitive radio networks by exploiting sparsity. IEEE Trans. Signal Process. 58, 1847–1862 (2010)

    CrossRef  MathSciNet  Google Scholar 

  6. Bazerque, J. A., Mateos, G., Giannakis, G. B.: Group Lasso on splines for spectrum cartography. IEEE Trans. Signal Process. 59, 4648–4663 (2011)

    CrossRef  MathSciNet  Google Scholar 

  7. Bertsekas, D. P., Tsitsiklis, J. N.: Parallel and distributed computation: Numerical methods. 2nd Edition, Athena Scientific, Boston (1997)

    MATH  Google Scholar 

  8. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge Univ. Press, UK (2004)

    CrossRef  MATH  Google Scholar 

  9. Boyd, S., Parikh, N., Chu, E., Peleato, B., Eckstein, J.: Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1–122 (2011)

    CrossRef  MATH  Google Scholar 

  10. Boyer, T. P., Antonov, J. I., Garcia, H. E., Johnson, D. R., Locarnini, R. A., Mishonov, A. V., Pitcher, M. T., Baranova, O. K., Smolyar, I. V.: World Ocean Database. NOAA Atlas NESDIS. 60, 190 (2005)

    Google Scholar 

  11. Candes, E. J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? Journal of the ACM. 58, 1–37 (2011)

    CrossRef  MATH  MathSciNet  Google Scholar 

  12. Candes, E. J., Tao, T.: Decoding by linear programming. IEEE Trans. Info. Theory. 51, 4203–4215 (2005)

    CrossRef  MATH  MathSciNet  Google Scholar 

  13. Cattivelli, F. S., Lopes, C. G., Sayed, A. H.: Diffusion recursive least-squares for distributed estimation over adaptive networks. IEEE Trans. Signal Process. 56, 1865–1877 (2008)

    CrossRef  MathSciNet  Google Scholar 

  14. Chandrasekaran, V., Sanghavi, S., Parrilo, P. R., Willsky, A. S.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21, 572–596 (2011)

    CrossRef  MATH  MathSciNet  Google Scholar 

  15. Chang, T., Hong, M., Wang, X.: Multiagent distributed large-scale optimization by inexact consensus alternating direction method of multipliers. In: Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (2014)

    Google Scholar 

  16. Chen, A. and Ozdaglar, A.: A fast distributed proximal-gradient method. In: Proceedings of Allerton Conference on Communication, Control, and Computing (2012)

    CrossRef  Google Scholar 

  17. Dall’Anese, E., Kim, S. J., Giannakis, G. B.: Channel gain map tracking via distributed kriging. IEEE Trans. Vehicular Tech. 60, 1205–1211 (2011)

    CrossRef  Google Scholar 

  18. Dall’Anese, E., Zhu, H., Giannakis, G. B.: Distributed optimal power flow for smart microgrids. IEEE Trans. on Smart Grid, 4, 1464–1475 (2013)

    CrossRef  Google Scholar 

  19. Davis, D., Yin, W.: Convergence rate analysis of several splitting schemes. In: R. Glowinski, S. Osher, W. Yin (eds.) Splitting Methods in Communication and Imaging, Science and Engineering. Springer (2016)

    Google Scholar 

  20. Deng, W., Lai, M., Peng, Z., Yin, W.: Parallel multi-block ADMM with o(1∕k) convergence. arXiv preprint arXiv:1312.3040 (2013)

    Google Scholar 

  21. Deng, W., Yin, W.: On the global and linear convergence of the generalized alternating direction method of multipliers. Manuscript, Journal of Scientific Computing, 66 (3):889–916 (2106)

    Google Scholar 

  22. Dimakis, A., Kar, S., Moura, J. M. F., Rabbat, M., Scaglione, A.: Gossip algorithms for distributed signal processing. Proc. of the IEEE. 89, 1847–1864 (2010)

    CrossRef  Google Scholar 

  23. Donoho, D. L.: Compressed sensing. IEEE Trans. Info. Theory. 52, 1289–1306 (2006)

    CrossRef  MATH  MathSciNet  Google Scholar 

  24. Duchi, J., Agarwal, A., Wainwright, M.: Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Trans. on Autom. Control, 57 592–606 (2012)

    CrossRef  MathSciNet  Google Scholar 

  25. Duda, R. O., Hart, P. E., Stork, D. G.: Pattern Classification. 2nd edition, Wiley, NY (2002)

    MATH  Google Scholar 

  26. Eckstein, J., Bertsekas, D.: On the Douglas-Rachford splitting method and the proximal point algorithm for maximal monotone operators. Mathematical Programming 55, 293–318 (1992)

    CrossRef  MATH  MathSciNet  Google Scholar 

  27. Farahmand, S., Roumeliotis, S. I., Giannakis, G. B.: Set-membership constrained particle filter: Distributed adaptation for sensor networks. IEEE Trans. Signal Process. 59, 4122–4138 (2011)

    CrossRef  MathSciNet  Google Scholar 

  28. Fazel, M.: Matrix rank minimization with applications. Ph.D. dissertation, Electrical Eng. Dept., Stanford University (2002)

    Google Scholar 

  29. Forero, P., Cano, A., Giannakis, G. B.: Consensus-based distributed support vector machines. Journal of Machine Learning Research. 11, 1663–1707 (2010)

    MATH  MathSciNet  Google Scholar 

  30. Forero, P., Cano, A., Giannakis, G. B.: Distributed clustering using wireless sensor networks. IEEE Journal of Selected Topics in Signal Processing. 5, 707–724 (2011)

    CrossRef  Google Scholar 

  31. Gabay, D., Mercier, B.: A dual algorithm for the solution of nonlinear variational problems via finite-element approximations. Comp. Math. Appl. 2, 17–40 (1976)

    CrossRef  MATH  Google Scholar 

  32. Giannakis, G. B., Kekatos, V., Gatsis, N., Kim, S.-J., Zhu, H., Wollenberg, B. F.: Monitoring and Optimization for Power Grids: A Signal Processing Perspective. IEEE Signal Processing Magazine, 30, 107–128 (2013)

    CrossRef  Google Scholar 

  33. Glowinski, R., Marrocco, A.: Sur l’approximation, par éléments finis d’orde un, et la résolution par pénalisation-dualité d’une classe de problèmes de Dirichlet non-linéaires. Rev. Francaise d‘Aut. Inf. Rech. Oper. 2, 41–76 (1975)

    Google Scholar 

  34. He, B., Yuan, X.: On the O(1∕t) convergence rate of the alternating direction method. SIAM Journal on Numerical Analysis 50, 700–709 (2012)

    CrossRef  MATH  MathSciNet  Google Scholar 

  35. He, B., Yuan, X.: On non-ergodic convergence rate of Douglas-Rachford alternating direction method of multipliers. Manuscript (2012)

    Google Scholar 

  36. Hlinka, O., Hlawatsch, F., Djuric, P. M.: Distributed particle filtering in agent networks. IEEE Signal Process. Mag. 30, 61–81 (2013)

    CrossRef  Google Scholar 

  37. Jakovetic, D., Xavier, J., Moura, J.: Fast distributed gradient methods. Manuscript

    Google Scholar 

  38. Kay, S.: Fundamentals of statistical signal processing: Estimation theory. Prentice-Hall, Englewood Cliffs (1993)

    MATH  Google Scholar 

  39. Kim, S.-J., Dall’Anese, E., Bazerque, J. A., Rajawat, K., Giannakis, G. B.: Advances in spectrum sensing and cross-layer design for cognitive radio networks. Elsevier E-Reference Signal Processing (2012)

    Google Scholar 

  40. Kushner, H. J., Yin, G. G.: Stochastic approximation and recursive algorithms and applications. 2nd Edition, Springer, Berlin, Germany (2003)

    MATH  Google Scholar 

  41. Lakhina, A., Crovella, M., Diot, C.: Diagnosing network-wide traffic anomalies. Proc. ACM SIGCOMM. Portland, OR (2004)

    CrossRef  Google Scholar 

  42. Ling, Q., Ribeiro, A.: Decentralized dynamic optimization through the alternating direction method of multipliers. IEEE Transactions on Signal Processing 62, 1185–1197 (2014)

    CrossRef  MathSciNet  Google Scholar 

  43. Ling, Q., Ribeiro, A.: Decentralized linearized alternating direction method of multipliers. In: Proceedings of IEEE Intl. Conf. on Acoustics, Speech, and Signal Processing (2014)

    CrossRef  Google Scholar 

  44. Lloyd, S. P.: Least-squares quantization in PCM. IEEE Trans. on Info. Theory. 28, 129–137 (1982)

    CrossRef  MATH  MathSciNet  Google Scholar 

  45. Lopes, C. G., Sayed, A. H.: Incremental adaptive strategies over distributed networks,” IEEE Trans. Signal Process. 55, 4064–4077 (2007)

    CrossRef  MathSciNet  Google Scholar 

  46. Lopes, C. G., Sayed, A. H.: Diffusion least-mean squares over adaptive networks: Formulation and performance analysis. IEEE Trans. Signal Process. 56, 3122–3136 (2008)

    CrossRef  MathSciNet  Google Scholar 

  47. Lu, Y., Roychowdhury, V., Vandenberghe, L.: Distributed parallel support vector machines in strongly connected networks. IEEE Tran. on Neural Networks. 19, 1167–1178 (2008)

    CrossRef  Google Scholar 

  48. Mardani, M., Mateos, G., Giannakis, G. B.: Decentralized sparsity-regularized rank minimization: Algorithms and applications. IEEE Trans. Signal Process. 61, 5374–5388 (2013)

    CrossRef  Google Scholar 

  49. Mardani, M., Mateos, G., Giannakis, G. B.: Dynamic Anomalography: Tracking Network Anomalies via Sparsity and Low Rank. IEEE Journal of Selected Topics in Signal Process. 7, 50–66 (2013)

    CrossRef  Google Scholar 

  50. Mardani, M., Mateos, G., Giannakis, G. B.: Recovery of low-rank plus compressed sparse matrices with application to unveiling traffic anomalies. IEEE Trans. Info. Theory. 59, 5186–5205 (2013)

    CrossRef  MathSciNet  Google Scholar 

  51. Mateos, G., Bazerque, J. A., Giannakis, G. B.: Distributed sparse linear regression. IEEE Trans. Signal Process. 58, 5262–5276 (2010)

    CrossRef  MathSciNet  Google Scholar 

  52. Mateos, G., Giannakis, G. B.: Distributed recursive least-squares: Stability and performance analysis. IEEE Trans. Signal Process. 60, 3740–3754 (2012)

    CrossRef  MathSciNet  Google Scholar 

  53. Mateos, G., Rajawat, K.: Dynamic network cartography. IEEE Signal Process. Mag. 30, 29–143 (2013)

    CrossRef  Google Scholar 

  54. Mateos, G., Schizas, I. D., Giannakis, G. B.: Distributed recursive least-squares for consensus-based in-network adaptive estimation. IEEE Trans. Signal Process. 57, 4583–4588 (2009)

    CrossRef  MathSciNet  Google Scholar 

  55. Mateos, G., Schizas, I. D., Giannakis, G. B.: Performance analysis of the consensus-based distributed LMS algorithm. EURASIP J. Advances Signal Process. Article ID 981030, 1–19 (2009)

    MATH  Google Scholar 

  56. Navia-Vazquez, A., Gutierrez-Gonzalez, D., Parrado-Hernandez, E., Navarro-Abellan, J. J.: Distributed support vector machines. IEEE Tran. on Neural Networks. 17, 1091–1097 (2006)

    CrossRef  Google Scholar 

  57. Nedic, A., Ozdaglar, A.: Distributed subgradient methods for multiagent optimization. IEEE Transactions on Automatic Control, 54, 48–61 (2009)

    CrossRef  MATH  MathSciNet  Google Scholar 

  58. Nowak, R. D.: Distributed EM algorithms for density estimation and clustering in sensor networks. IEEE Trans. on Signal Processing. 51, 2245–2253 (2003)

    CrossRef  Google Scholar 

  59. Olfati-Saber, R.: Distributed Kalman filter with embedded consensus filters. Proc. 44th IEEE Conf. Decision and Control. Seville, Spain (2005)

    Google Scholar 

  60. Rabbat, M., Nowak, R.: Quantized incremental algorithms for distributed optimization. IEEE Journal on Selected Areas in Communications, 23, 798–808 (2005)

    CrossRef  Google Scholar 

  61. Rabbat, M., Nowak, R., Bucklew, J.: Generalized consensus computation in networked systems with erasure links. In: Proceedings of IEEE International Workshop on Signal Processing Advances for Wireless Communications (2005)

    CrossRef  Google Scholar 

  62. Ram, S., Nedic, A., Veeravalli, V.: Distributed stochastic subgradient projection algorithms for convex optimization. Journal of Optimization Theory and Applications, 147, 516–545 (2010)

    CrossRef  MATH  MathSciNet  Google Scholar 

  63. Ribeiro, A., Schizas, I. D, Roumeliotis, S. I., Giannakis, G. B.: Kalman filtering in wireless sensor networks: Incorporating communication cost in state estimation problems. IEEE Control Syst. Mag. 30, 66–86 (2010)

    CrossRef  MathSciNet  Google Scholar 

  64. Ripley, B. D.: Spatial Statistics. Wiley, Hoboken, New Jersey (1981)

    CrossRef  MATH  Google Scholar 

  65. Roughan, M.: A case study of the accuracy of SNMP measurements. Journal of Electrical and Computer Engineering. Article ID 812979 (2010)

    Google Scholar 

  66. Saligrama, V., Alanyali, M., Savas, O.: Distributed detection in sensor networks with packet losses and finite capacity links. IEEE Trans. on Signal Processing. 54, 4118–4132 (2006)

    CrossRef  Google Scholar 

  67. Schizas, I. D., G. B. Giannakis, G. B.: Consensus-Based Distributed Estimation of Random Signals with Wireless Sensor Networks, Proc. of 40th Asilomar Conf. on Signals, Systems, and Computers, 530–534, Pacific Grove, CA (2006)

    Google Scholar 

  68. Schizas, I. D., Giannakis, G. B., Roumeliotis, S. I., Ribeiro, A.: Consensus in ad hoc WSNs with noisy links - Part II: Distributed estimation and smoothing of random signals. IEEE Trans. Signal Process. 56, 1650–1666 (2008)

    CrossRef  MathSciNet  Google Scholar 

  69. Schizas, I. D., Mateos, G., Giannakis, G. B.: Distributed LMS for consensus-based in-network adaptive processing. IEEE Trans. Signal Process. 57, 2365–2381 (2009)

    CrossRef  MathSciNet  Google Scholar 

  70. Schizas, I. D., Ribeiro, A., Giannakis, G. B.: Consensus in ad hoc WSNs with noisy links - Part I: Distributed estimation of deterministic signals. IEEE Trans. Signal Process. 56, 350–364 (2008)

    CrossRef  MathSciNet  Google Scholar 

  71. Schölkopf, B., Smola, A.: Learning with Kernels. Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Boston (2002)

    MATH  Google Scholar 

  72. Shi, W., Ling, Q., Wu, G., Yin, W.: EXTRA: An exact first-order algorithm for decentralized consensus optimization. SIAM Journal on Optimization, 25 (2), 944–966 (2015)

    CrossRef  MATH  MathSciNet  Google Scholar 

  73. Shi, W., Ling, Q., Yuan, K., Wu, G., Yin, W.: On the linear convergence of the ADMM in decentralized consensus optimization. IEEE Transactions on Signal Processing (2014)

    Google Scholar 

  74. Solo, V., Kong., X.: Adaptive signal processing algorithms: Stability and performance. Prentice-Hall, Englewood Cliffs (1995)

    Google Scholar 

  75. Srebro, N., Shraibman, A.: Rank, trace-norm and max-norm. Proc. 44th Learning Theory. Springer, 545–560 (2005)

    Google Scholar 

  76. Stoica, P., Moses, R.: Spectral Analysis of Signals. Prentice Hall (2005)

    Google Scholar 

  77. Tsianos, K. and Rabbat, M.: Distributed dual averaging for convex optimization under communication delays. Proc. of American Control Conference (2012)

    Google Scholar 

  78. Wang, H., Banerjee, A.: Online alternating direction method. arXiv preprint arXiv:1306.3721 (2013)

    Google Scholar 

  79. Wei, E., Ozdaglar, A.: Distributed alternating direction method of multipliers. Proc. Decision and Control Conference (2012)

    CrossRef  Google Scholar 

  80. Wen, Z., Goldfarb, D., Yin W.: Alternating direction augmented Lagrangian methods for semidefinite programming. Math. Prog. Comp. 2, 203–230 (2010)

    CrossRef  MATH  MathSciNet  Google Scholar 

  81. Wolfe, J., Haghighi, A., Klein, D.: Fully distributed EM for very large datasets. Proc. 25th Intl. Conference on Machine Learning. Helsinki, Finland (2008)

    Google Scholar 

  82. Wood A. J., Wollenberg, B. F.: Power Generation, Operation, and Control. Wiley & Sons, New York, NY (1984)

    Google Scholar 

  83. Xiao, L., Boyd, S.: Fast linear iterations for distributed averaging. Syst. Control Lett. 53, 65–78(2004)

    CrossRef  MATH  MathSciNet  Google Scholar 

  84. Xiao, L., Boyd, S., Kim, S. J.: Distributed average consensus with least-mean-square deviation. Journal of Parallel and Distributed Computing. 67, 33–46 (2007)

    CrossRef  MATH  Google Scholar 

  85. Yuan, K., Ling, Q., Yin, W.: On the convergence of decentralized gradient descent. SIAM Journal Optimization, 26 (3): 1835–1854 (2016)

    CrossRef  MATH  MathSciNet  Google Scholar 

  86. Zhang, Y., Ge, Z., Greenberg, A., Roughan, M.: Network anomography. Proc. ACM SIGCOM Conf. on Internet Measurements. Berkeley, CA (2005)

    CrossRef  Google Scholar 

  87. Zhu, H., Giannakis, G. B.: Power System Nonlinear State Estimation using Distributed Semidefinite Programming. IEEE Journal of Special Topics in Signal Processing, 8, 1039–1050 (2014)

    CrossRef  Google Scholar 

  88. Zhu, H., Cano, A., Giannakis, G. B.: Distributed consensus-based demodulation: algorithms and error analysis. IEEE Trans. on Wireless Comms. 9, 2044–2054 (2010)

    CrossRef  Google Scholar 

  89. Zhu, H., Giannakis, G. B., Cano, A.: Distributed in-network channel decoding. IEEE Trans. on Signal Processing. 57, 3970–3983 (2009)

    CrossRef  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Minnesota, 117 Pleasant Str., Minneapolis, MN, 55455, USA

    Georgios B. Giannakis

  2. University of Science and Technology of China, 443 Huangshan Road, Hefei, Anhui, 230027, China

    Qing Ling

  3. University of Rochester, 413 Hopeman Engineering Building, Rochester, NY, 14627, USA

    Gonzalo Mateos

  4. University of Texas at Arlington, 416 Yates Street, Arlington, TX, 76010, USA

    Ioannis D. Schizas

  5. University of Illinois at Urbana-Champaign, 4058 ECE Building, 306 N. Wright Street, Urbana, IL, 61801, USA

    Hao Zhu

Authors
  1. Georgios B. Giannakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qing Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gonzalo Mateos
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ioannis D. Schizas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hao Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Georgios B. Giannakis .

Editor information

Editors and Affiliations

  1. Department of Mathematics, University of Houston, Houston, Texas, USA

    Roland Glowinski

  2. Department of Mathematics, UCLA, Los Angeles, California, USA

    Stanley J. Osher

  3. Department of Mathematics, UCLA, Los Angeles, California, USA

    Wotao Yin

Rights and permissions

Reprints and Permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Giannakis, G.B., Ling, Q., Mateos, G., Schizas, I.D., Zhu, H. (2016). Decentralized Learning for Wireless Communications and Networking. In: Glowinski, R., Osher, S., Yin, W. (eds) Splitting Methods in Communication, Imaging, Science, and Engineering. Scientific Computation. Springer, Cham. https://doi.org/10.1007/978-3-319-41589-5_14

Download citation