Abstract
Deep convolutional neural networks (DCNNs) have been successfully used in many computer visions task. However, with the increasing complexity of the network and continuous growth of data scale, training a DCNN model suffers from the following three problems: excessive network parameters, insufficient capability of the parameter optimization, and inefficient parallelism. To overcome these obstacles, this paper develops an optimization algorithm for deep convolutional neural network (FP-DCNN) in the MapReduce framework. First, a pruning method based on Taylor’s loss (FMPTL) is designed to trim redundant parameters, which not only compresses the structure of DCNN, but also reduces the computational cost of training. Next, a glowworm swarm optimization algorithm based on information sharing strategy (IFAS) is presented, which improves the ability of parameter optimization by adjusting the initialization of weights. Finally, a dynamic load balancing strategy based on parallel computing entropy (DLBPCE) is proposed to achieve an even distribution of data and thus improve the parallel performance of the cluster. Our experiments show that compared with other parallelized algorithms, this algorithm not only reduces the computational cost of network training, but also obtains a higher processing speed.
Similar content being viewed by others
References
Hariri RH, Fredericks EM, Bowers KM (2019) Uncertainty in big data analytics: survey, opportunities, and challenges. Journal of Big Data 6:1–16
Sewak M, Sahay SK, Rathore H (2020) An Overview of Deep Learning Architecture of Deep Neural Networks and Autoencoders. J Comput Theor Nanosci 17:182–188
Dong, S., Wang, P., & Abbas, K. (2021). A survey on deep learning and its applications. Computer Science Review, 40, 100379
Maheshwaram S (2019) A Review on Deep Convolutional Neural Network and its Applications. International Journal of Advanced Research in Computer and Communication Engineering 8:174–179
Fredj, H.B., Bouguezzi, S., & Souani, C. (2020). Face recognition in unconstrained environment with CNN. The Visual Computer, 1–10
Tabernik D, Skočaj D (2020) Deep Learning for Large-Scale Traffic-Sign Detection and Recognition. IEEE Trans Intell Transp Syst 21:1427–1440
Cheng G, Zhou P, Han J (2016) Learning Rotation-Invariant Convolutional Neural Networks for Object Detection in VHR Optical Remote Sensing Images. IEEE Trans Geosci Remote Sens 54:7405–7415
Jha S, Dey A, Kumar R, Solanki VK (2019) A Novel Approach on Visual Question Answering by Parameter Prediction using Faster Region Based Convolutional Neural Network. Int J Interact Multim Artif Intell 5:30–37
Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both Weights and Connections for Efficient Neural Network. https://arxiv.org/abs/1506.02626
Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. (2017). Pruning Filters for Efficient ConvNets. https://arxiv.org/abs/1608.08710
He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018). Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks. IJCAI
Zhuo, H., Qian, X., Fu, Y., Yang, H., & Xue, X. (2018). SCSP: Spectral Clustering Filter Pruning with Soft Self-adaption Manners. https://arxiv.org/abs/1806.05320
Ye, J., Lu, X., Lin, Z.L., & Wang, J.Z. (2018). Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers. https://arxiv.org/abs/1802.00124
Chang, J., Lu, Y., Xue, P., Xu, Y., & Wei, Z. (2021). ACP: Automatic Channel Pruning via Clustering and Swarm Intelligence Optimization for CNN. https://arxiv.org/abs/2101.06407
Hidri, A. (2018). Optimization for training CNN deep models based on swarm intelligence. 2018 International Conference on Advanced Systems and Electric Technologies (IC_ASET), 284–289
Anaraki AK, Ayati M, Kazemi F (2019) Magnetic resonance imaging-based brain tumor grades classification and grading via convolutional neural networks and genetic algorithms. Biocybernetics and Biomedical Engineering 39:63–74
Banharnsakun A (2019) Towards improving the convolutional neural networks for deep learning using the distributed artificial bee colony method. Int J Mach Learn Cybern 10:1301–1311
Zeng K, Ding S, Jia W (2018) Single image super-resolution using a polymorphic parallel CNN. Appl Intell 49:292–300
Lee S, Kang Q, Madireddy S, Balaprakash P, Agrawal A, Choudhary A, Archibald R, Liao W (2019) Improving Scalability of Parallel CNN Training by Adjusting Mini-Batch Size at Run-Time. IEEE International Conference on Big Data 2019:830–839
Basit N, Zhang Y, Wu H, Liu H, Bin J, He Y, Hendawi AM (2016) MapReduce-based deep learning with handwritten digit recognition case study. IEEE International Conference on Big Data 2016:1690–1699
Maleki, N., Rahmani, A., & Conti, M. (2019). MapReduce: an infrastructure review and research insights. The Journal of Supercomputing, 1–69
El-Ajou A, Arqub OA, Al-Smadi M (2015) A general form of the generalized Taylor’s formula with some applications. Appl Math Comput 256:851–859
Rashmita Gupta, R.K.Bayal. (2020). A REVIEW ON GLOWWORM SWARM OPTIMIZATION TECHNIQUES[J]. Journal of Critical Reviews. Vol.7(No.11):3686–3694
Dong X, Chen C, Geng Q, Zhang W, Zhang X (2021) Fast Algorithm Based on Parallel Computing for Sample Entropy Calculation. IEEE Access 9:20223–20234
Yan C, Zhou L, Wan Y (2019) A Multi-Task Learning Model for Better Representation of Clothing Images. IEEE Access 7:34499–34507
Chen, C., Reiz, S., Yu, C.D., Bungartz, H., & Biros, G. (2019). Fast Evaluation and Approximation of the Gauss-Newton Hessian Matrix for the Multilayer Perceptron. https://arxiv.org/abs/1910.12184
Giuste, F., & Vizcarra, J.C. (2020). CIFAR-10 Image Classification Using Feature Ensembles. https://arxiv.org/abs/2002.03846
Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. https://arxiv.org/abs/1708.07747
Litjens G, Kooi T, Bejnordi BE, Setio A, Ciompi F, Ghafoorian M, Laak JV, Ginneken B, Sánchez C (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88
Cohen, G., Afshar, S., Tapson, J., & Schaik, A.V. (2017). EMNIST: an extension of MNIST to handwritten letters. https://arxiv.org/abs/1702.05373
Funding
This study was supported by the National Natural Science Foundation of China (41562019), the National Key Research and Development Program of China (2018YFC1504705) and the Natural Science Foundation of Jiangxi Province (2018BAB202004).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Le, Y., Nanehkaran, Y.A., Mwakapesa, D.S. et al. FP-DCNN: a parallel optimization algorithm for deep convolutional neural network. J Supercomput 78, 3791–3813 (2022). https://doi.org/10.1007/s11227-021-04012-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-021-04012-y