Abstract
Spark is an efficient big data processing platform based on memory computing. However, the default task scheduling algorithm in Spark does not take into account the difference in capability and resource usage of nodes under the Spark cluster. Therefore, an uneven load on the nodes might be resulted with the high-capability node in idle state and the low-capability node in high-load state which will affect the work efficiency. To this end, we propose an adaptive task execution node allocation algorithm based on the ant colony-simulated annealing algorithm. The proposed algorithm optimizes the Spark cluster task execution node allocation method based on the resource usage of the node, which is used to achieve the purpose of load balancing. Experiments show that in comparison with the task scheduling algorithm of the Spark cluster, the task scheduling algorithm proposed in this paper has a significant improvement in cluster load balancing and task completion time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Liu, Z., Zhang, Q., Ahmed, R., Boutaba, R., Liu, Y., Gong, Z.: Dynamic resource allocation for MapReduce with partitioning skew. IEEE Trans. Comput. 65(11), 3304–3317 (2016)
Xie, J., et al.: Improving MapReduce performance through data placement in heterogeneous Hadoop clusters. In: 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), pp. 1–9. IEEE, Atlanta (2010)
Mao, H., Hu, S., Zhang, Z., Xiao, L., Ruan, L.: A load-driven task scheduler with adaptive DSC for MapReduce. In: 2011 IEEE/ACM International Conference on Green Computing and Communications, pp. 28–33. IEEE/ACM, Sichuan (2011)
Shi, K.L.: Research on load balancing based on multi-tenant task scheduling in storm. Xinjiang University (2018)
Liu, M.Q.: Research on job scheduling method in Storm. Nanjing University of Posts and Telecommunications (2017)
Zhang, X.W., Li, Z.H., Liu, G.S., Xu, J.J., Xie, T.K., Nees, J.P.: A spark scheduling strategy for heterogeneous cluster. CMC Comput. Mater. Continua 55(3), 405–417 (2018)
Liu, W., Li, Z., Zhou, Y.: An efficient filter strategy for theta-join query in distributed environment. In: 46th International Conference on Parallel Processing Workshops (ICPPW), pp. 77–84. IEEE, Bristol (2017)
Wang, S.Z., Zhang, Y.P., Zhang, L., Cao, N., Pang, C.Y.: An improved memory cache management study based on spark. CMC: Comput., Mater. Continua 56(3), 415–431 (2018)
Verma, A., Mansuri, A.H., Jain, N.: Big data management processing with Hadoop MapReduce and spark technology: a comparison. In: 2016 Symposium on Colossal Data Analysis and Networking (CDAN), pp. 1–4. IEEE, Indore (2016)
Huang, C.J.: A research of load balancing algorithms for data skew in spark. University of Electronic Science and Technology of China (2018)
Ma, S.: Design and implementation of cross-terminal shopping mall system based on web components. Beijing University of Posts and Telecommunications (2018)
Yang, Z.W., Zheng, Q., Wang, S., et al.: Adaptive task scheduling strategy for heterogeneous spark cluster. Comput. Eng. 42(1), 31–35 (2016)
Liu, W.J., Wang, X.Y., Qu, H.C., Meng, Y.: Research on server cluster resource scheduling based on improved ant colony algorithm. Microelectron. Comput. 33(03), 98–101 (2016)
Song, X.Q., Gao, L., Wang, J.P.: Job scheduling based on ant colony optimization in cloud computing. In: 2011 International Conference on Computer Science and Service System (CSSS), pp. 3309–3312. IEEE, Nanjing (2011)
Tawfeek, M.A., El-Sisi, A., Keshk, A.E., Torkey, F.A.: Cloud task scheduling based on ant colony optimization. In: 2013 8th International Conference on Computer Engineering & Systems (ICCES), pp. 64–69. IEEE, Cairo (2013)
Cao, Y., Liu, Y.J., Yu, Y.: Task scheduling and optimization of cloud computing based on genetic algorithm and ant colony algorithm. J. Jilin Univ. (Science Edition) 54(05), 1077–1081 (2016)
Qin, J., Dong, Q.Q., Hao, T.S.: Improvement of algorithm for cloud task scheduling based on ant colony optimization and simulated annealing. Comput. Technol. Dev. 27(03), 117–121 (2017)
Zhang, H.R., Chen, P.H., Xiong, J.B.: Task scheduling algorithm based on simulated annealing ant colony algorithm in cloud computing environment. J. Guangdong Univ. Technol. 31(03), 77–82 (2014)
Sun, W., Zhang, N., Wang, H., Yin, W., Qiu, T.: PACO: a period ACO based scheduling algorithm in cloud computing. In: 2013 International Conference on Cloud Computing and Big Data, pp. 482–486. IEEE, Fuzhou (2013)
Gupta, A., Garg, R.: Load balancing based task scheduling with ACO in cloud computing. In: 2017 International Conference on Computer and Applications (ICCA), pp. 174–179. IEEE, Doha (2017)
Jia, R.X.: Research on hybrid task scheduling algorithm simulation of ant colony algorithm and simulated annealing algorithm in virtual environment. In: 2015 10th International Conference on Computer Science & Education (ICCSE), pp. 562–565. IEEE, Cambridge (2015)
Acknowledgements
This paper is partially supported by the Education technology Foundation of the Ministry of Education (No. 2017A01020), and the Major Project of the Hebei Province Education Department (No. 2017GJJG083)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, S., Zhang, Z., Geng, S. (2019). Research on Load Balancing Algorithm Optimization Based on Spark Platform. In: Sun, X., Pan, Z., Bertino, E. (eds) Artificial Intelligence and Security. ICAIS 2019. Lecture Notes in Computer Science(), vol 11633. Springer, Cham. https://doi.org/10.1007/978-3-030-24265-7_39
Download citation
DOI: https://doi.org/10.1007/978-3-030-24265-7_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-24264-0
Online ISBN: 978-3-030-24265-7
eBook Packages: Computer ScienceComputer Science (R0)