Abstract
Differential privacy has been a common framework that provides an effective method of establishing privacy-guaranteed machine learning. Extensive research work has focused on differential privacy stochastic gradient descent (SGD-DP) and its variants under distributed machine learning to improve training efficiency and protect privacy. However, SGD-DP relies on the premise of convex optimization. In large-scale distributed machine learning, the objective function may be more a non-convex objective function, which not only makes the gradient calculation difficult and easy to fall into local optimization. It’s difficult to achieve truly global optimization. To address this issue, we propose a novel differential privacy optimization algorithm based on quantum particle swarm theory that suitable for both convex optimization and non-convex optimization. We further comprehensively apply adaptive contraction–expansion and chaotic search to overcome the premature problem, and provide theoretical analysis in terms of convergence and privacy protection. Also, we verify through experiments that the actual application performance of the algorithm is consistent with the theoretical analysis.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Chen J, Pan X, Monga R, Bengio S, Jozefowicz R (2017) Revisiting distributed synchronous SGD. arXiv Learning
Yao Q, Kwok JT, Wang T, Liu T (2019) Large-scale low-rank matrix learning with nonconvex regularizers. IEEE Trans Pattern Anal Mach Intell 41:2628–2643
Meng Q, Chen W, Wang Y, Ma Z, Liu T (2017) Convergence analysis of distributed stochastic gradient descent with shuffling. arXiv Machine Learning
Crandall PE, Quinn MJ (1993) Block data decomposition for data-parallel programming on a heterogeneous workstation network. In: High performance distributed computing, pp 42–49
Ofer D, Ran G-B, Ohad S, Lin X (2010) Optimal distributed online prediction using mini-batches. J Mach Learn Res 13(1):165–202
Bonomi F, Milito RA, Zhu J, Addepalli S (2012) Fog computing and its role in the internet of things. In: IEEE international conference on cloud computing technology and science, pp 13–16
Chaudhuri K, Sarwate AD, Sinha K (2013) A near-optimal algorithm for differentially-private principal components. J Mach Learn Res 14(1):2905–2943
Bassily R, Smith A, Thakurta A (2014) Private empirical risk minimization: efficient algorithms and tight error bounds. In: 2014 IEEE 55th annual symposium on foundations of computer science, pp 464–473
Jain P, Kulkarni V, Thakurta A, Williams O (2015) To drop or not to drop: robustness, consistency and differential privacy properties of dropout. arXiv Learning
Ming Y, Zhao Y, Wu C, Li K, Yin J (2017) Distributed and asynchronous stochastic gradient descent with variance reduction. Neurocomputing 281:S0925231217318039
Hegedüs I, Berta A, Jelasity M (2016) Robust decentralized differentially private stochastic gradient descent. J Wirel Mob Netw Ubiquitous Comput Depend Appl 7(2):20–40
Song S, Chaudhuri K, Sarwate AD (2013) Stochastic gradient descent with differentially private updates. In: 2013 IEEE global conference on signal and information processing, pp 245–248
Abadi M, Chu A, Goodfellow I, Brendan McMahan H, Mironov I, Talwar K, Zhang L (2016) Deep learning with differential privacy. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, CCS 16, New York, NY, USA. Association for Computing Machinery, p 308318
Hegedus I, Jelasity M (2016) Distributed differentially private stochastic gradient descent: an empirical study. In: 2016 24th Euromicro international conference on parallel, distributed, and network-based processing (PDP), pp 566–573
Bassily R, Smith A, Thakurta A (2014) Private empirical risk minimization: efficient algorithms and tight error bounds. In: Foundations of computer science, pp 464–473
Stone MH (1948) The generalized weierstrass approximation theorem. Math Mag 21(5):237
Clerc M, Kennedy J (2002) The particle swarm–explosion, stability, and convergence in a multidimensional complex space. IEEE Trans Evol Comput 6(1):58–73
Michael M, Michael S, Gisbert S (2006) Optimized particle swarm optimization (OPSO) and its application to artificial neural network training. BMC Bioinform 7(1):125–125
Zhang J, Xue F, Cai X, Zhihua CY, Chang WZ, Li W (2019) Privacy protection based on manyoptimization algorithm. Concurr Comput Pract Exp 31(20):e5342
Kalyani G, ChandraSekharaRao MVP, Janakiramaiah B (2018) Particle swarm intelligence and impact factor-based privacy preserving association rule mining for balancing data utility and knowledge privacy. Arab J Sci Eng 43(8):4161–4178
Clerc M (1999) The swarm and the queen: towards a deterministic and adaptive particle swarm optimization. In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), vol 3, pp 1951–1957
Sun J, Wu X, Palade V, Fang W, Lai CH, Xu W (2012) Convergence analysis and improvements of quantum-behaved particle swarm optimization. Inf ENCES 193:81–103
Yin C, Xi J, Sun R, Wang J (2018) Location privacy protection based on differential privacy strategy for big data in industrial internet of things. IEEE Trans Ind Inf 14(8):3628–3636
Nissim K, Raskhodnikova S, Smith A (2007) Smooth sensitivity and sampling in private data analysis. In: Proceedings of the thirty-ninth annual ACM symposium on theory of computing, STOC 07, New York, NY, USA. Association for Computing Machinery, p 7584
Berthold S, Abe S (1960) Statistical metric spaces. Pac J Math 10(10):313–334
Geyer RC, Klein T, Nabi M (2017) Differentially private federated learning: a client level perspective. arXiv Cryptography and Security
Sun J, Fang W, Wu X, Palade V (2012) evolutionary computation, quantum-behaved particle swarm optimization: analysis of individual particle behavior and parameter selection. Evol Comput 20:349–393
Tang K, Jun WU, Zhao J (2013) Adaptive particle swarm optimization algorithm based on diversity feedback. J Comput Appl 33(12):3372–3374
Wang YB (2010) An colony entropy-based adaptive genetic algorithm. Microcomput Inf
Sun X, Zhou DW, Zhang XW (2010) A chaos particle swarm optimization algorithm. Comput Eng Sci 32(12):272–277
Paranya A, Phayung M (2012) A multi-objective memetic algorithm based on chaos optimization. Appl Mech Mater 130–134:725–729
Tavazoei MS, Haeri M (2007) An optimization algorithm based on chaotic behavior and fractal nature. J Comput Appl Math 206(2):1070–1081
Lin J, Yang D, Li M, Xu J, Xue G (2016) Bidguard: a framework for privacy-preserving crowdsensing incentive mechanisms. In: 2016 IEEE conference on communications and network security (CNS), pp 145–153
Du M, Wang K, Xia Z, Zhang Y (2018) Differential privacy preserving of training model in wireless big data with edge computing. IEEE Trans Big Data 6:1
Acknowledgements
This work is sponsored by the National Key RD Program of China (No. 2018YFB1003201), the National Natural Science Foundation of P. R. China (Nos. 61672296, 61872196, 61872194, and 61902196), Scientific and Technological Support Project of Jiangsu Province (Nos. BE2017166, and BE2019740), Major Natural Science Research Projects in Colleges and Universities of Jiangsu Province (No. 18KJA520008), Six Talent Peaks Project of Jiangsu Province (RJFW-111), Postgraduate Research and Practice Innovation Program of Jiangsu Province (Nos. SJKY19_0759, SJKY19_0761).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Xie, Y., Li, P., Zhang, J. et al. Differential privacy distributed learning under chaotic quantum particle swarm optimization. Computing 103, 449–472 (2021). https://doi.org/10.1007/s00607-020-00853-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00607-020-00853-2