Skip to main content
SpringerLink
Log in

Multibranch multilevel federated learning for a better feature extraction and a plug-and-play dynamic-adjusting double flow personalization approach

  • Published:
Applied Intelligence Aims and scope Submit manuscript

A Correction to this article was published on 24 January 2023

This article has been updated

Abstract

Federated learning (FL) is an emerging technique used to preserve the privacy of users’ data in training by a distributed machine learning approach. Previously, a client-edge-cloud hierarchical federated learning (HierFL) system was proposed to reduce the long communication latency between the client and the cloud. However, HierFL performs very poorly at extracting better features from the data in the client, which is also a common drawback of traditional FL architectures. HierFL also constrains the its hierarchy to server-edge-client three levels which is very restrictive. This paper proposes that the specifically designed FL architecture can naturally have an excellent effect on feature extraction. Specifically, this paper proposes a multibranch multilevel federated learning (MBMLFL) framework to perform a better job at feature extraction, and each branch and level has its own self specific effect. The proposed framework is also friendlier as well as further enhances the privacy. By extending FedAvg, we design an M2DCFedAvg algorithm for the framework to optimize its objective function distributedly, and by conducting numerous experiments and analyses, we propose a general epoch selection principle for all the FL methods: the \({1^{T}_{d}}\)-n principle. Experiments with various data distributions are performed on MBMLFL to comprehensively research its characteristics. Moreover, to complete the personalization ability of our framework, we propose a plug-and-play dynamic-adjusting double flow personalization approach (DADFPA) for our MBMLFL method, which will further enhance the ability of the MBMLFL, such that it comprehensively and adaptively satisfies the generalization and personalization needs of clients. Experiments show that the MBMLFL and DADFPA can improve their baselines by an average of 5.26% and 2.24% points, which demonstrates their excellent performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Algorithm 2
Fig. 4
Fig. 5

Similar content being viewed by others

Change history

References

  1. McMahan B, Moore E, Ramage D, Hampson S, Arcas BAy (2017) .. In: Proc 20th Int Conf Artif Intell Stat, Fort Lauderdale, vol 54, pp 1273–1282

  2. Liu L, Zhang J, Song S, Letaief KB (2020) Client-edge-cloud hierarchical federated learning. In: ICC 2020-2020 IEEE international conference on communications (ICC), IEEE, pp 1–6

  3. Sattler F, Wiedemann S, Müller K-R, Samek W (2019) Robust and communication-efficient federated learning from non-iid data. IEEE Trans Neural Netw Learn Syst 31(9):3400–3413

    Article  Google Scholar 

  4. Zhang L, Luo Y, Bai Y, Du B, Duan L-Y (2021) Federated learning for non-iid data via unified feature learning and optimization objective alignment. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4420–4428

  5. Kairouz P, McMahan HB, Avent B, Bellet A, Bennis M, Bhagoji AN, Bonawitz K, Charles Z, Cormode G, Cummings R et al Advances and open problems in federated learning. arXiv:1912.04977

  6. Hsieh K, Phanishayee A, Mutlu O, Gibbons PB (2020) The non-IID data quagmire of decentralized machine learning. In: Proc 37th Int Conf Mach Learn, Virtual Event, vol 119 of Proceedings of Machine Learning Research, pp 4387–4398

  7. Sattler F, Wiedemann S, Müller K-R, Samek W (2019) Robust and communication-efficient federated learning from non-iid data. IEEE Trans Neural Networks Learn Syst 31(9):3400– 3413

    Article  Google Scholar 

  8. Kulkarni V, Kulkarni M, Pant A (2020) Survey of personalization techniques for federated learning. In: 2020 fourth world Conf Smart Trends Syst Secur, Sustain, IEEE, pp 794–797

  9. Konečnỳ J, McMahan B, Ramage D Federated optimization: Distributed optimization beyond the datacenter. arXiv:1511.03575

  10. Konečnỳ J, McMahan HB, Yu FX, Richtárik P, Suresh AT, Bacon D Federated learning: Strategies for improving communication efficiency. arXiv:1610.05492

  11. Eichner H, Koren T, McMahan B, Srebro N, Talwar K (2019) Semi-cyclic stochastic gradient descent. In: Int Conf Mach Learn, pp 1764–1773

  12. Wang H, Kaplan Z, Niu D, Li B (2020) Optimizing federated learning on non-iid data with reinforcement learning. In: IEEE Conf Comput, Commun, IEEE, pp 1698–1707

  13. Huang Y, Chu L, Zhou Z, Wang L, Liu J, Pei J, Zhang Y (2021) Personalized cross-silo federated learning on non-iid data. In: Proc AAAI Conf Artif Intell, vol 35, pp 7865–7873

  14. Du Y, Yang S, Huang K (2020) High-Dimensional Stochastic gradient quantization for Communication-Efficient edge learning. IEEE Trans Signal Process 68:2128–2142

    Article  MathSciNet  MATH  Google Scholar 

  15. Lin Y, Han S, Mao H, Wang Y, Dally B (2018) Deep gradient compression: reducing the communication bandwidth for distributed training. In: 6Th Int Conf Learn Represent

  16. Bernstein J, Wang Y-X, Azizzadenesheli K, Anandkumar A (2018) SIGNSGD: Compressed optimisation for non-convex problems. In: Proc 35th Int Conf Mach Learn Sweden, vol 80, pp 559–568

  17. Sun J, Chen T, Giannakis GB, Yang Q, Yang Z Lazily aggregated quantized gradient innovation for communication-efficient federated learning. IEEE Trans Pattern Anal Mach Intell

  18. Stich SU, Cordonnier J-B, Jaggi M (2018) Sparsified SGD with memory. In: Bengio S, Wallach HM, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds) Adv Neural Inf Process Syst 31 Annu Conf Neural Inf Process Syst, pp 4452–4463

  19. Alistarh D, Hoefler T, Johansson M, Konstantinov N, Khirirat S, Renggli C (2018) The convergence of sparsified gradient methods. In: Adv Neural Inf Process Syst 31 Annu Conf Neural Inf Process Syst, pp 5977–5987

  20. Wang S, Tuor T, Salonidis T, Leung KK, Makaya C, He T, Chan K (2019) Adaptive federated learning in resource constrained edge computing systems. IEEE J Sel Areas Commun 37:1205– 1221

    Article  Google Scholar 

  21. Kamp M, Adilova L, Sicking J, Hüger F, Schlicht P, Wirtz T, Wrobel S (2018) Efficient decentralized deep learning by dynamic model averaging. In: Mach Learn Knowl Discov Databases - Eur Conf, vol 11051

  22. Aono Y, Hayashi T, Wang L, Moriai S (2017) Others, Privacy-preserving deep learning via additively homomorphic encryption. IEEE Trans Inf Forensics Secur 13(5):1333–1345

    Google Scholar 

  23. Bonawitz K, Ivanov V, Kreuter B, Marcedone A, McMahan HB, Patel S, Ramage D, Segal A, Seth K (2017) Practical secure aggregation for privacy-preserving machine learning. In: Proc 2017 ACM SIGSAC Conf Comput Commun Secur, pp 1175– 1191

  24. Riazi MS, Weinert C, Tkachenko O, Songhori EM, Schneider T, Koushanfar F (2018) Chameleon: A hybrid secure computation framework for machine learning applications. In: Proc 2018 Asia Conf Comput Commun Secur, pp 707–721

  25. Dwork C, McSherry F, Nissim K, Smith A (2006) Calibrating noise to sensitivity in private data analysis. In: Theory cryptogr Conf, Springer, pp 265–284

  26. Iyengar R, Near JP, Song D, Thakkar O, Thakurta A, Wang L (2019) Towards practical differentially private convex optimization. In: IEEE symp Secur Priv, IEEE, pp 299–316

  27. Wu X, Li F, Kumar A, Chaudhuri K, Jha S, Naughton J (2017) Bolt-on differential privacy for scalable stochastic gradient descent-based analytics. In: Proc 2017 ACM Int Conf Manag Data, pp 1307–1322

  28. Kulkarni V, Kulkarni M, Pant A (2020) Survey of personalization techniques for federated learning. In: Fourth world Conf Smart Trends Syst Secur Sustain, IEEE, pp 794–797

  29. Mansour Y, Mohri M, Ro J, Suresh AT Three approaches for personalization with applications to federated learning. arXiv:2002.10619

  30. Li T, Sanjabi M, Beirami A, Smith V (2020) Fair resource allocation in federated learning. In: 8Th int Conf Learn, Represent, OpenReview.net

  31. Fallah A, Mokhtari A, Ozdaglar AE (2020) Personalized federated learning with theoretical guarantees: a Model-Agnostic Meta-Learning approach. In: Larochelle H, Ranzato M, Hadsell R, Balcan M-F, Lin H-T (eds) Adv Neural Inf Process Syst

  32. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  33. Lehmann J, Isele R, Jakob M, Jentzsch A, Kontokostas D, Mendes PN, Hellmann S, Morsey M, Van Kleef P, Auer S et al (2015) Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia. Semant Web 6(2):167–195

    Article  Google Scholar 

  34. Krizhevsky A (2009) Learning multiple layers of features from tiny images. Technical report, University of Toronto, (http://www.cs.toronto.edu/kriz/learning-features-2009-TR.pdf)

  35. Vinyals O, Blundell C, Lillicrap T, Wierstra D, et al. (2016) Matching networks for one shot learning. Adv Neural Inform Process Syst 29:3630–3638

    Google Scholar 

  36. Simonyan K, Zisserman A (2015) Very deep convolutional networks for Large-Scale image recognition. In: Bengio Y, LeCun Y (eds) 3rd Int Conf Learn Represent

Download references

Acknowledgments

This work is funded by Natural Key Research and Development Program of China under Grant No. 2018YFA0704604.

Author information

Authors and Affiliations

  1. School of Mechanical and Power Engineering, East China University of Science and Technology, Shanghai, 200237, China

    Maoye Ren & Xinhai Yu

Authors
  1. Maoye Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinhai Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinhai Yu.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

The original online version of this article was revised: There are extra yellow squares appearing on this image that should not be included in the Figure 2 image.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ren, M., Yu, X. Multibranch multilevel federated learning for a better feature extraction and a plug-and-play dynamic-adjusting double flow personalization approach. Appl Intell 53, 13956–13971 (2023). https://doi.org/10.1007/s10489-022-04193-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04193-w

Keywords

Search

Navigation