Skip to main content
SpringerLink
Log in

Introduction to Federated Learning Systems

  • Chapter
  • First Online:
Federated Learning

Abstract

In this chapter, we introduce federated learning from a systems perspective. We go into the details of the different federated learning scenarios that have different system design considerations. We first introduce two most common but quite different federated learning scenarios, namely cross-device federated learning and cross-silo federated learning. Cross-device federated learning typically involves a significant number of parties (e.g., thousands to millions), who are usually less reliable and equipped with mobile or IoT devices that have various computing and communication capabilities. In cross-silo federated learning, the parties are usually a small number of organizations with ample computing power and reliable communications. We first describe the two very different problems that each of them address. We then describe the architectural differences between the two and their corresponding training steps. We also discuss the unique systems challenges that arise due to these properties and give a brief description of current works that have talked about these problems in detail.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abadi M, Chu A, Goodfellow I, McMahan HB, Mironov I, Talwar K, Zhang L (2016) Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. ACM, pp 308–318

    Google Scholar 

  2. Balakrishnan R, Akdeniz M, Dhakal S, Himayat N (2020) Resource management and fairness for federated learning over wireless edge networks. In: 2020 IEEE 21st international workshop on signal processing advances in wireless communications (SPAWC). IEEE, pp 1–5

    Google Scholar 

  3. Bonawitz K, Ivanov V, Kreuter B, Marcedone A, McMahan HB, Patel S, Ramage D, Segal A, Seth K (2017) Practical secure aggregation for privacy-preserving machine learning. In: Proceedings of the 2017 ACM SIGSAC conference on computer and communications security. ACM, pp 1175–1191

    Google Scholar 

  4. Bonawitz K, Eichner H, Grieskamp W, Huba D, Ingerman A, Ivanov V, Kiddon C, Konecný J, Mazzocchi S, McMahan B, Overveldt TV, Petrou D, Ramage D, Roselander J (2019) Towards federated learning at scale: System design. In: Talwalkar A, Smith V, Zaharia M (eds) Proceedings of machine learning and systems 2019, MLSys 2019, Stanford, CA, USA, March 31–April 2, 2019, mlsys.org. https://proceedings.mlsys.org/book/271.pdf

  5. Caldas S, Konečny J, McMahan HB, Talwalkar A (2018) Expanding the reach of federated learning by reducing client resource requirements. Preprint. arXiv:181207210

    Google Scholar 

  6. Chai Z, Ali A, Zawad S, Truex S, Anwar A, Baracaldo N, Zhou Y, Ludwig H, Yan F, Cheng Y (2020) Tifl: A tier-based federated learning system. In: Proceedings of the 29th international symposium on high-performance parallel and distributed computing, pp 125–136

    Google Scholar 

  7. Chan Z, Li J, Yang X, Chen X, Hu W, Zhao D, Yan R (2019) Modeling personalization in continuous space for response generation via augmented wasserstein autoencoders. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp 1931–1940

    Google Scholar 

  8. Chen T, Jin X, Sun Y, Yin W (2020) Vafl: a method of vertical asynchronous federated learning. e-prints. arXiv–2007

    Google Scholar 

  9. Cheng K, Fan T, Jin Y, Liu Y, Chen T, Papadopoulos D, Yang Q (2019) Secureboost: A lossless federated learning framework. Preprint. arXiv:190108755

    Google Scholar 

  10. Du W, Atallah MJ (2001) Secure multi-party computation problems and their applications: a review and open problems. In: Proceedings of the 2001 workshop on new security paradigms, pp 13–22

    Google Scholar 

  11. Du W, Zhan Z (2003) Using randomized response techniques for privacy-preserving data mining. In: Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining, pp 505–510

    Google Scholar 

  12. Dwork C, Hardt M, Pitassi T, Reingold O, Zemel R (2012) Fairness through awareness. In: Proceedings of the 3rd innovations in theoretical computer science conference, pp 214–226

    Google Scholar 

  13. Feng S, Yu H (2020) Multi-participant multi-class vertical federated learning. Preprint. arXiv:200111154

    Google Scholar 

  14. Gao D, Liu Y, Huang A, Ju C, Yu H, Yang Q (2019) Privacy-preserving heterogeneous federated transfer learning. In: 2019 IEEE international conference on big data (Big Data). IEEE, pp 2552–2559

    Google Scholar 

  15. Gentry C et al (2009) A fully homomorphic encryption scheme, vol 20. Stanford University, Stanford

    MATH  Google Scholar 

  16. Hao M, Li H, Xu G, Liu S, Yang H (2019) Towards efficient and privacy-preserving federated deep learning. In: ICC 2019-2019 IEEE international conference on communications (ICC). IEEE, pp 1–6

    Google Scholar 

  17. Hardy S, Henecka W, Ivey-Law H, Nock R, Patrini G, Smith G, Thorne B (2017) Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption. Preprint. arXiv:171110677

    Google Scholar 

  18. Hosseinalipour S, Brinton CG, Aggarwal V, Dai H, Chiang M (2020) From federated to fog learning: Distributed machine learning over heterogeneous wireless networks. IEEE Commun Mag 58(12):41–47. https://doi.org/10.1109/MCOM.001.2000410

    Article  Google Scholar 

  19. Kairouz P, McMahan HB, Avent B, Bellet A, Bennis M, Bhagoji AN, Bonawitz K, Charles Z, Cormode G, Cummings R et al (2019) Advances and open problems in federated learning. Preprint. arXiv:191204977

    Google Scholar 

  20. Konecnỳ J, McMahan HB, Felix XY, Richtárik P, Suresh AT, Bacon D (2016) Federated learning: Strategies for improving communication efficiency. CoRR

    Google Scholar 

  21. Lalitha A, Shekhar S, Javidi T, Koushanfar F (2018) Fully decentralized federated learning. In: Third workshop on bayesian deep learning (NeurIPS)

    Google Scholar 

  22. Li C, Shen H, Huang T (2016) Learning to diagnose stragglers in distributed computing. In: 2016 9th workshop on many-task computing on clouds, grids, and supercomputers (MTAGS). IEEE, pp 1–6

    Google Scholar 

  23. Li X, Huang K, Yang W, Wang S, Zhang Z (2019) On the convergence of fedavg on non-iid data. In: International conference on learning representations

    Google Scholar 

  24. Li T, Sahu AK, Talwalkar A, Smith V (2020) Federated learning: Challenges, methods, and future directions. IEEE Signal Process Mag 37(3):50–60

    Article  Google Scholar 

  25. Liang G, Chawathe SS (2004) Privacy-preserving inter-database operations. In: International conference on intelligence and security informatics. Springer, pp 66–82

    Google Scholar 

  26. Liu, Y., Kang, Y., Zhang, X., Li, L., Cheng, Y., Chen, T., …& Yang, Q. A Communication efficient vertical federated learning framework. 2019. arXiv preprint arXiv:1912.11187

    Google Scholar 

  27. Lo SK, Lu Q, Zhu L, Paik Hy, Xu X, Wang C (2021) Architectural patterns for the design of federated learning systems. Preprint. arXiv:210102373

    Google Scholar 

  28. Mao Y, You C, Zhang J, Huang K, Letaief KB (2017) A survey on mobile edge computing: The communication perspective. IEEE Commun Surv Tutorials 19(4):2322–2358

    Article  Google Scholar 

  29. McMahan HB, Moore E, Ramage D, Hampson S et al (2016) Communication-efficient learning of deep networks from decentralized data. Preprint. arXiv:160205629

    Google Scholar 

  30. McMahan HB, et al (2021) Advances and open problems in federated learning. Found Trends® Mach Learn 14(1):1

    Google Scholar 

  31. Nishio T, Yonetani R (2019) Client selection for federated learning with heterogeneous resources in mobile edge. In: ICC 2019-2019 IEEE international conference on communications (ICC). IEEE, pp 1–7

    Google Scholar 

  32. O’herrin JK, Fost N, Kudsk KA (2004) Health insurance portability accountability act (hipaa) regulations: effect on medical record research. Ann Surg 239(6):772

    Article  Google Scholar 

  33. Pathak MA, Rane S, Raj B (2010) Multiparty differential privacy via aggregation of locally trained classifiers. In: NIPS, Citeseer, pp 1876–1884

    Google Scholar 

  34. Scannapieco M, Figotin I, Bertino E, Elmagarmid AK (2007) Privacy preserving schema and data matching. In: Proceedings of the 2007 ACM SIGMOD international conference on Management of data, pp 653–664

    Google Scholar 

  35. Shi W, Dustdar S (2016) The promise of edge computing. Computer 49(5):78–81

    Article  Google Scholar 

  36. Shi W, Cao J, Zhang Q, Li Y, Xu L (2016) Edge computing: Vision and challenges. IEEE Internet Things J 3(5):637–646

    Article  Google Scholar 

  37. Sprague MR, Jalalirad A, Scavuzzo M, Capota C, Neun M, Do L, Kopp M (2018) Asynchronous federated learning for geospatial applications. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp 21–28

    Google Scholar 

  38. Tandon R, Lei Q, Dimakis AG, Karampatziakis N (2017) Gradient coding: Avoiding stragglers in distributed learning. In: International conference on machine learning, PMLR, pp 3368–3376

    Google Scholar 

  39. Tankard C (2016) What the GDPR means for businesses. Netw Secur 2016(6):5–8

    Article  Google Scholar 

  40. Wei K, Li J, Ding M, Ma C, Yang HH, Farokhi F, Jin S, Quek TQ, Poor HV (2020) Federated learning with differential privacy: Algorithms and performance analysis. IEEE Trans Inf Forensics Secur 15:3454–3469

    Article  Google Scholar 

  41. Wu W, He L, Lin W, Mao R, Maple C, Jarvis SA (2020) Safa: a semi-asynchronous protocol for fast federated learning with low overhead. IEEE Trans Comput 70:655

    Article  MathSciNet  Google Scholar 

  42. Xie C, Koyejo S, Gupta I (2019) Asynchronous federated optimization. Preprint. arXiv:190303934

    Google Scholar 

  43. Xu Z, Yang Z, Xiong J, Yang J, Chen X (2019) Elfish: Resource-aware federated learning on heterogeneous edge devices. Preprint. arXiv:191201684

    Google Scholar 

  44. Xu R, Baracaldo N, Zhou Y, Anwar A, Joshi J, Ludwig H (2021) Fedv: Privacy-preserving federated learning over vertically partitioned data. e-prints, pp arXiv–2103

    Google Scholar 

  45. Yang K, Fan T, Chen T, Shi Y, Yang Q (2019) A quasi-newton method based vertical federated learning framework for logistic regression. Preprint. arXiv:191200513

    Google Scholar 

  46. Yang Q, Liu Y, Chen T, Tong Y (2019) Federated machine learning: Concept and applications. ACM Trans Intell Syst Technol (TIST) 10(2):12

    Google Scholar 

  47. Yang R, Ouyang X, Chen Y, Townend P, Xu J (2018) Intelligent resource scheduling at scale: a machine learning perspective. In: 2018 IEEE symposium on service-oriented system engineering (SOSE). IEEE, pp 132–141

    Google Scholar 

  48. Yang S, Ren B, Zhou X, Liu L (2019) Parallel distributed logistic regression for vertical federated learning without third-party coordinator. Preprint. arXiv:191109824

    Google Scholar 

  49. Zhang C, Li S, Xia J, Wang W, Yan F, Liu Y (2020) Batchcrypt: Efficient homomorphic encryption for cross-silo federated learning. In: 2020 USENIX annual technical conference (USENIX ATC 20), pp 493–506

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Nevada, Reno, Reno, NV, USA

    Syed Zawad & Feng Yan

  2. IBM Research – Almaden, San Jose, CA, USA

    Ali Anwar

Authors
  1. Syed Zawad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ali Anwar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Syed Zawad .

Editor information

Editors and Affiliations

  1. IBM Research – Almaden, San Jose, CA, USA

    Heiko Ludwig

  2. IBM Research -- Almaden, San Jose, CA, USA

    Nathalie Baracaldo

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Zawad, S., Yan, F., Anwar, A. (2022). Introduction to Federated Learning Systems. In: Ludwig, H., Baracaldo, N. (eds) Federated Learning. Springer, Cham. https://doi.org/10.1007/978-3-030-96896-0_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-96896-0_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-96895-3

  • Online ISBN: 978-3-030-96896-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation