Abstract
Hadoop YARN is one of the most commonly used frameworks for implementing MapReduce distributed computing model. The current resource allocation modes in YARN are triggered by events, which are executed when every slave sent heartbeat message to the master. In another word, the resource allocation is based on the order of every slave node, rather than the global information. A global resource allocation can achieve a better outcome than the allocation method based on every single node. In reality, resource allocation is a complicated issue and many influencing factors need to be considered. Based on the YARNs existing cluster architecture and allocation mode, this paper designs the mechanism of resource allocation and carries out work schedules to optimize the running time of cluster mainly focuses on network bandwidth and node execution rate. We make an improvement on the basis of the existing algorithm, and propose an algorithm used strategy based on the greedy choice to make resource allocation. We designed an experimental simulation of the operation of the clusters. Compared to the existing resource allocation model, the result shows our algorithm has improved the performance and shortens the execution time for the whole cluster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Castillo, C., Carrera, D., Becerra, Y., Whalley, I., Steinder, M., Torres, J.: Resource-aware adaptive scheduling for mapreduce clusters. In: International MIDDLEWARE Conference, pp. 180–199 (2011)
Chen, T.Y., Wei, H.W., Wei, M.F., Chen, Y.J., Hsu, T.S., Shih, W.K.: LaSA: a locality-aware scheduling algorithm for hadoop-mapreduce resource assignment. In: International Conference on Collaboration Technologies and Systems, pp. 342–346 (2013)
Chun, B.G., Ihm, S., Maniatis, P., Naik, M., Patti, A.: Clonecloud: elastic execution between mobile device and cloud. In: Conference on Computer Systems, pp. 301–314 (2011)
Vavilapalli, V.K., Murthy, A.C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., Saha, B., Curino, C., O’Malley, O., Radia, S., Reed, B., Baldeschwieler, B.: Apache hadoop yarn: yet another resource negotiator. In: Symposium on Cloud Computing, p. 5 (2013)
Li, H., Wei, X., Qingwu, F., Luo, Y.: Mapreduce delay scheduling with deadline constraint. Concurr. Comput. Pract. Exp. 26(3), 766–778 (2014)
Shi, W., Cao, J., Zhang, Q., Li, Y., Lanyu, X.: Edge computing: vision and challenges. IEEE Internet Things J. 3(5), 637–646 (2016)
Tan, J., Meng, X., Zhang, L.: Coupling task progress for mapreduce resource-aware scheduling. In: 2013 Proceedings of IEEE INFOCOM, pp. 1618–1626 (2013)
Zhang, Q., Zhani, M.F., Yang, Y., Boutaba, R., Wong, B.: Prism: fine-grained resource-aware scheduling for mapreduce. IEEE Trans. Cloud Comput. 3(2), 182–194 (2015)
Zhao, H., Yang, S., Fan, H., Chen, Z., Xu, J.: An efficiency-aware scheduling for data-intensive computations on mapreduce clusters. IEICE Trans. Inf. Syst. E96.D(12), 2654–2662 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Xu, D., Li, Y., Wang, S., Li, X., Qian, Z. (2017). On Global Resource Allocation in Clusters for Data Analytics. In: Wang, G., Atiquzzaman, M., Yan, Z., Choo, KK. (eds) Security, Privacy, and Anonymity in Computation, Communication, and Storage. SpaCCS 2017. Lecture Notes in Computer Science(), vol 10658. Springer, Cham. https://doi.org/10.1007/978-3-319-72395-2_25
Download citation
DOI: https://doi.org/10.1007/978-3-319-72395-2_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-72394-5
Online ISBN: 978-3-319-72395-2
eBook Packages: Computer ScienceComputer Science (R0)