Abstract
The parameter server system (Parameter Server, PS) is the most widely used distributed machine learning system. Its core module is the communication module, which mainly includes three parts: communication content, communication frequency, and communication pace. The communication scheme of the existing PS system will have problems such as waste of computing power resources and a large proportion of communication overhead. This paper hopes to design and implement a distributed set that saves computing power and has a smaller communication overhead based on the communication module of the original PS system. Machine learning system, this article mainly has the following contributions: First, through the analysis and experiment of Bulk Synchronous Parallel (BSP) communication pace, based on the superiority of BSP algorithm level, it overcomes the waste of computing power of BSP at the system level, and designs and implements a new set of Communication pace, the new communication pace under the same training task can save more computing power and achieve higher model accuracy. Secondly, this article combines the quantization compression technology with the parameter server system, and uses the quantization technology to compress and decompress the communication content in the parameter server system, so that the volume of the communication content during the operation of the parameter server system is significantly reduced, thereby reducing the communication overhead and improving Speedup ratio of the system. Finally, based on the analysis and experiment of the change trend of the model accuracy rate, this paper designs and implements a dynamic communication frequency adjustment scheme based on the change trend of the model accuracy rate, which reduces the overall communication frequency during the operation of the parameter server system and reduces the parameter server. T Experiments show that the system designed in this paper is superior to the existing mainstream solutions in terms of communication pace, communication content, and communication frequency. When training the same model, this system compared with the existing parameter server system in model convergence time, model There is a noticeable improvement in the highest accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Liu, Y.: Disturbed ML: Algorithm, theory and Practice
Low, Y., Gonzalez, J., Kyrola, A., Bickson, D., Guestrin, C., Hellerstein, J.M.: Distributed graphlab: a framework for machine learning in the cloud. arXiv preprint arXiv:1204.6078 (2012)
Ahmed, A., Shervashidze, N., Narayanamurthy, S., Josifovski, V., Smola, A.J.: Distributed large-scale natural graph factorization. In: Proceedings of the 22nd international conference on World Wide Web, pp. 37–48 (2013)
Dean, J., et al.: Large scale distributed deep networks. In Advances in neural information processing systems, pp. 1223–1231 (2012)
Chen, W.-Y., Song, Y., Bai, H., Lin, C.-J., Chang, E.Y.: Parallel spectral clustering in distributed systems. IEEE Trans. Patt. Anal. Mach. Intell. 33(3), 568–586 (2010)
Agarwal, A., Wainwright, M.J., Duchi, J.C.: Distributed dual averaging in networks. In: Advances in Neural Information Processing Systems, pp. 550–558 (2010)
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149 (2015)
Gong, Y., Liu, L., Yang, M., Bourdev, L.: Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115 (2014)
Berkeley University of California. Ray. [EB/OL]. https://ray.io/
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ge, G., Zhu, F., Huang, Y. (2021). Asynchronous Compatible D-ML System for Edge Devices. In: Sun, X., Zhang, X., Xia, Z., Bertino, E. (eds) Advances in Artificial Intelligence and Security. ICAIS 2021. Communications in Computer and Information Science, vol 1422. Springer, Cham. https://doi.org/10.1007/978-3-030-78615-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-030-78615-1_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-78614-4
Online ISBN: 978-3-030-78615-1
eBook Packages: Computer ScienceComputer Science (R0)