Skip to main content
SpringerLink
Log in

Improving Gossip Learning via Limited Model Merging

  • Conference paper
  • First Online:
Advances in Computational Collective Intelligence (ICCCI 2023)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1864))

Included in the following conference series:

Abstract

Decentralized machine learning provides a unique opportunity to create data-driven applications without the need for large investments in centralized infrastructure. In our previous works, we introduced gossip learning for this purpose: models perform random walks in the network, and the nodes train the received models on the locally available data. We also proposed various improvements, like model sub-sampling, merging, and token-based flow control. Gossip learning is robust to failures, and does not require synchronization. Efficiency in terms of network bandwidth is also a major concern in the case of decentralized learning algorithms, especially when they are deployed in a network of IoT devices or smartphones. Here, we improve the model merging method to allow gossip learning to benefit more from token-based flow control. We experimentally evaluate our solution over several classification problems in simulations using an availability trace based on real-world smartphone measurements. Our results indicate that the improved variant significantly outperforms previously proposed solutions.

This work was supported by the European Union project RRF-2.3.1-21-2022-00004 within the framework of the Artificial Intelligence National Laboratory and project TKP2021-NVA-09, implemented with the support provided by the Ministry of Innovation and Technology of Hungary from the National Research, Development and Innovation Fund, financed under the TKP2021-NVA funding scheme.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/ormandi/Gossip-Learning-Framework/tree/privacy.

References

  1. Anguita, D., Ghio, A., Oneto, L., Parra, X., Reyes-Ortiz, J.L.: A public domain dataset for human activity recognition using smartphones. In: Esann, vol. 3, p. 3 (2013)

    Google Scholar 

  2. Bache, K., Lichman, M.: UCI machine learning repository (2013)

    Google Scholar 

  3. Belal, Y., Bellet, A., Mokhtar, S.B., Nitu, V.: PEPPER: empowering user-centric recommender systems over gossip learning. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 6(3), 1–27 (2022)

    Article  Google Scholar 

  4. Berta, Á., Bilicki, V., Jelasity, M.: Defining and understanding smartphone churn over the internet: a measurement study. In: Proceedings of the 14th IEEE International Conference on Peer-to-Peer Computing (P2P 2014). IEEE (2014). https://doi.org/10.1109/P2P.2014.6934317

  5. Bishop, C.M., Nasrabadi, N.M.: Pattern Recognition and Machine Learning, vol. 4. Springer, Heidelberg (2006)

    Google Scholar 

  6. Bottou, L.: Stochastic gradient descent tricks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 421–436. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_25

    Chapter  Google Scholar 

  7. Danner, G., Hegedűs, I., Jelasity, M.: Decentralized machine learning using compressed push-pull averaging. In: Proceedings of the 1st International Workshop on Distributed Infrastructure for Common Good, pp. 31–36 (2020)

    Google Scholar 

  8. Danner, G., Jelasity, M.: Token account algorithms: the best of the proactive and reactive worlds. In: Proceedings of The 38th International Conference on Distributed Computing Systems (ICDCS 2018), pp. 885–895. IEEE Computer Society (2018). https://doi.org/10.1109/ICDCS.2018.00090

  9. Giaretta, L., Girdzijauskas, Š.: Gossip learning: off the beaten path. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 1117–1124. IEEE (2019)

    Google Scholar 

  10. Guo, J., Zuo, Y., Wen, C.K., Jin, S.: User-centric online gossip training for autoencoder-based CSI feedback. IEEE J. Sel. Top. Signal Process. 16(3), 559–572 (2022)

    Article  Google Scholar 

  11. Hegedűs, I., Danner, G., Jelasity, M.: Decentralized learning works: an empirical comparison of gossip learning and federated learning. J. Parallel Distrib. Comput. 148, 109–124 (2021)

    Article  Google Scholar 

  12. Konečnỳ, J., McMahan, H.B., Yu, F.X., Richtárik, P., Suresh, A.T., Bacon, D.: Federated learning: strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016)

  13. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf

  14. Li, Y., Chen, C., Liu, N., Huang, H., Zheng, Z., Yan, Q.: A blockchain-based decentralized federated learning framework with committee consensus. IEEE Netw. 35(1), 234–241 (2020)

    Article  Google Scholar 

  15. McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial Intelligence and Statistics, pp. 1273–1282. PMLR (2017)

    Google Scholar 

  16. Montresor, A., Jelasity, M.: PeerSim: a scalable P2P simulator. In: Proceedings of the 9th IEEE International Conference on Peer-to-Peer Computing (P2P 2009), pp. 99–100. IEEE, Seattle (2009). https://doi.org/10.1109/P2P.2009.5284506. Extended abstract

  17. Niwa, K., Zhang, G., Kleijn, W.B., Harada, N., Sawada, H., Fujino, A.: Asynchronous decentralized optimization with implicit stochastic variance reduction. In: International Conference on Machine Learning, pp. 8195–8204. PMLR (2021)

    Google Scholar 

  18. Onoszko, N., Karlsson, G., Mogren, O., Zec, E.L.: Decentralized federated learning of deep neural networks on non-IID data. arXiv preprint arXiv:2107.08517 (2021)

  19. Ormándi, R., Hegedűs, I., Jelasity, M.: Gossip learning with linear models on fully distributed data. Concurr. Comput.: Pract. Exp. 25(4), 556–571 (2013)

    Article  Google Scholar 

  20. Partridge, C.: Gigabit Networking. Addison-Wesley Professional (1994)

    Google Scholar 

  21. Ramanan, P., Nakayama, K.: BAFFLE: blockchain based aggregator free federated learning. In: 2020 IEEE International Conference on Blockchain (Blockchain), pp. 72–81. IEEE (2020)

    Google Scholar 

  22. Shin, H., et al.: Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35(5), 1285–1298 (2016)

    Article  Google Scholar 

  23. Tölgyesi, N., Jelasity, M.: Adaptive peer sampling with newscast. In: Sips, H., Epema, D., Lin, H.-X. (eds.) Euro-Par 2009. LNCS, vol. 5704, pp. 523–534. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03869-3_50

    Chapter  Google Scholar 

  24. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

Download references

Author information

Authors and Affiliations

  1. University of Szeged, Szeged, Hungary

    Gábor Danner, István Hegedűs & Márk Jelasity

  2. ELKH SZTE Research Group on Artificial Intelligence, Szeged, Hungary

    Márk Jelasity

Authors
  1. Gábor Danner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. István Hegedűs
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Márk Jelasity
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to István Hegedűs .

Editor information

Editors and Affiliations

  1. Wrocław University of Science and Technology, Wrocław, Poland

    Ngoc Thanh Nguyen

  2. Eötvös Loránd University, Budapest, Hungary

    János Botzheim

  3. Eötvös Loránd University, Budapest, Hungary

    László Gulyás

  4. Universidad Complutense de Madrid, Madrid, Spain

    Manuel Nunez

  5. Vrije Universiteit Amsterdam, Amsterdam, The Netherlands

    Jan Treur

  6. University of Münster, Münster, Germany

    Gottfried Vossen

  7. Wrocław University of Science and Technology, Wrocław, Poland

    Adrianna Kozierkiewicz

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Danner, G., Hegedűs, I., Jelasity, M. (2023). Improving Gossip Learning via Limited Model Merging. In: Nguyen, N.T., et al. Advances in Computational Collective Intelligence. ICCCI 2023. Communications in Computer and Information Science, vol 1864. Springer, Cham. https://doi.org/10.1007/978-3-031-41774-0_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-41774-0_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-41773-3

  • Online ISBN: 978-3-031-41774-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation