Abstract
In distributed machine learning, multiple machines or workers collaborate to train a model. However, prior research in cross-silo distributed learning with differential privacy has the drawback of requiring all workers to participate in each training iteration, hindering flexibility and efficiency. To overcome these limitations, we introduce a new algorithm that allows partial worker attendance in the training process, reducing communication costs by over 50% while preserving accuracy on benchmark data. The privacy of the workers is also improved because less data are exchanged between workers.
The work of LTP is partially supported by JST CREST Grant JPMJCR21M1, and JST AIP Accelerated Program Grant JPMJCR22U5, Japan. Parts of this work were done while TTP was at NICT.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
References
Agarwal, N., Suresh, A.T., Yu, F.X., Kumar, S., McMahan, B.: cpSGD: communication-efficient and differentially-private distributed SGD. In: Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pp. 7575–7586 (2018)
Balle, B., Barthe, G., Gaboardi, M.: Privacy amplification by subsampling: tight analyses via couplings and divergences. In: Bengio, S., Wallach, H.M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, Montréal, Canada, 3–8 December 2018, pp. 6280–6290 (2018)
Balle, B., Wang, Y.: Improving the Gaussian mechanism for differential privacy: analytical calibration and optimal denoising. In: Proceedings of the 35th International Conference on Machine Learning, ICML 2018, pp. 403–412 (2018)
Dwork, C., McSherry, F., Nissim, K., Smith, A.: Calibrating noise to sensitivity in private data analysis. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 265–284. Springer, Heidelberg (2006). https://doi.org/10.1007/11681878_14
Dwork, C., Roth, A.: The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. 9(3–4), 211–407 (2014). https://doi.org/10.1561/0400000042
Dwork, C., Rothblum, G.N., Vadhan, S.: Boosting and differential privacy. In: Proceedings of the 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, FOCS 2010, pp. 51–60. IEEE Computer Society (2010). https://doi.org/10.1109/FOCS.2010.12
Girgis, A.M., Data, D., Diggavi, S., Kairouz, P., Suresh, A.T.: Shuffled model of federated learning: privacy, accuracy and communication trade-offs. IEEE J. Sel. Areas Inf. Theory 2(1), 464–478 (2021). https://doi.org/10.1109/JSAIT.2021.3056102
Li, N., Qardaji, W., Su, D.: On sampling, anonymization, and differential privacy or, k-anonymization meets differential privacy. In: Proceedings of the 7th ACM Symposium on Information, Computer and Communications Security, ASIACCS 2012, pp. 32–33. ACM, New York (2012). https://doi.org/10.1145/2414456.2414474
Liu, R., Cao, Y., Chen, H., Guo, R., Yoshikawa, M.: FLAME: differentially private federated learning in the shuffle model. In: Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, 2–9 February 2021, pp. 8688–8696. AAAI Press (2021)
Phong, L.T., Phuong, T.T.: Distributed SignSGD with improved accuracy and network-fault tolerance. IEEE Access 8, 191839–191849 (2020). https://doi.org/10.1109/ACCESS.2020.3032637
Phuong, T.T., Phong, L.T.: Distributed SGD with flexible gradient compression. IEEE Access 8, 64707–64717 (2020). https://doi.org/10.1109/ACCESS.2020.2984633
Phuong, T.T., Phong, L.T.: Communication-efficient distributed SGD with error-feedback, revisited. Int. J. Comput. Intell. Syst. 14(1), 1373–1387 (2021). https://doi.org/10.2991/ijcis.d.210412.001
Phuong, T.T., Phong, L.T.: Decentralized stochastic optimization with random attendance. IEEE Sig. Process. Lett. 29, 1322–1326 (2022). https://doi.org/10.1109/LSP.2022.3179331
Phuong, T.T., Phong, L.T.: Distributed differentially-private learning with communication efficiency. J. Syst. Archit. 128, 102555 (2022). https://doi.org/10.1016/j.sysarc.2022.102555
Sattler, F., Wiedemann, S., Müller, K., Samek, W.: Robust and communication-efficient federated learning from non-i.i.d. data. IEEE Trans. Neural Netw. Learn. Syst. 31(9), 3400–3413 (2020). https://doi.org/10.1109/TNNLS.2019.2944481
Tang, H., Yu, C., Lian, X., Zhang, T., Liu, J.: DoubleSqueeze: parallel stochastic gradient descent with double-pass error-compensated compression. In: Proceedings of the 36th International Conference on Machine Learning, ICML 2019, pp. 6155–6165 (2019)
Wang, L., Jia, R., Song, D.: D2P-Fed: differentially private federated learning with efficient communication (2021). https://arxiv.org/abs/2006.13039
Wang, Y.X., Balle, B., Kasiviswanathan, S.P.: Subsampled Renyi differential privacy and analytical moments accountant. In: Chaudhuri, K., Sugiyama, M. (eds.) Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 89, pp. 1226–1235. PMLR, 16–18 April 2019
Zheng, S., Huang, Z., Kwok, J.T.: Communication-efficient distributed blockwise momentum SGD with error-feedback. In: Wallach, H.M., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E.B., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Vancouver, BC, Canada, 8–14 December 2019, pp. 11446–11456 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Phong, L.T., Phuong, T.T. (2023). Differentially-Private Distributed Machine Learning with Partial Worker Attendance: A Flexible and Efficient Approach. In: Nguyen, N.T., Le-Minh, H., Huynh, CP., Nguyen, QV. (eds) The 12th Conference on Information Technology and Its Applications. CITA 2023. Lecture Notes in Networks and Systems, vol 734. Springer, Cham. https://doi.org/10.1007/978-3-031-36886-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-36886-8_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-36885-1
Online ISBN: 978-3-031-36886-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)