Skip to main content
SpringerLink
Log in

Hybrid Gradient Descent Golden Eagle Optimization (HGDGEO) Algorithm-Based Efficient Heterogeneous Resource Scheduling for Big Data Processing on Clouds

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Resource scheduling is indispensable for enhancing the system performance during big data processing on clouds. It is highly useful for attaining significant utilization of computing resources completely concentrating towards the facilitation of resource scalability and on-demand services. The resources essential for running different applications is determined to be maximum heterogeneous in cloud computing. This heterogeneous resource demand introduces a resource gap in which some of the resource potentialities are drained on par with the other resource potentialities still available in the same server resulting in imbalanced resource utilization. This imbalanced resource allocation condition is more apparent when the computing resources are more heterogeneous. At this juncture, intelligent resource scheduling strategy becomes essential to distribute resources for big data processing by adopting a potential decision-making process that focusses on the objective of achieving necessitated tasks over time. In this paper, Hybrid Gradient Descent Golden Eagle Optimization (HGDGEO) algorithm-based efficient heterogeneous resource scheduling process is proposed for handling the challenges that are highly possible during big data processing in the Hadoop heterogenous cloud environment. This HGDGEO algorithm is proposed as an adaptive resource scheduling strategy that handles the dynamic characteristics of the resources and users’ fluctuating demand during big data stream processing by mimicking the golden eagles’ intelligence which alternates the speed of tuning at different spiral trajectory stages of hunting. It handles big data processing by adopting two adaptive parameters which completely concentrates on optimal resource allocation to suitable VMs in the shortest time possible depending on their requirements. The simulation results of HGDGEO algorithm confirmed its predominance in terms of makespan, load balance and throughput on par with the competitive resource scheduling algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data Availability

Enquiries about data availability should be directed to the authors.

References

  1. Tuli, S., Sandhu, R., & Buyya, R. (2020). Shared data-aware dynamic resource provisioning and task scheduling for data intensive applications on hybrid clouds using Aneka. Future Generation Computer Systems, 106, 595–606.

    Article  Google Scholar 

  2. Souravlas, S., & Anastasiadou, S. (2020). Pipelined dynamic scheduling of big data streams. Applied Sciences, 10(14), 4796.

    Article  Google Scholar 

  3. Nivitha Pabitha, K., Parameshwaran (2022) C-DRM: Coalesced P-TOPSIS entropy technique addressing uncertainty in cloud service selection. Information Technology and Control, 51(3), 592–605. https://doi.org/10.5755/j01.itc.51.3.30881

  4. Kang, Y., Pan, L., & Liu, S. (2022). An online algorithm for scheduling big data analysis jobs in cloud environments. Knowledge-Based Systems, 245(4), 108628.

    Article  Google Scholar 

  5. Jagatheswari Praveen, S., Ramalingam, J, Chandra Priya (2022). Improved grey relational analysis-based TOPSIS method for cooperation enforcing scheme to guarantee quality of service in MANETs. International Journal of Information Technology, 14(2), 887–897. https://doi.org/10.1007/s41870-022-00865-5

  6. Kaladevi, P., Janakiraman, S., Ramalingam, P., & Muthusankar, D. (2023). An improved ensemble classificationbased secure two stage bagging pruning technique for guaranteeing privacy preservation of DNA sequences in electronic health records. Journal of Intelligent & Fuzzy Systems, 44(1), 149–166.

  7. Mashayekhy, L., Nejad, M. M., Grosu, D., Lu, D., & Shi, W. (2014). Energy-aware scheduling of MapReduce jobs. IEEE International Congress on Big Data, 3(4), 12–24.

    Google Scholar 

  8. Rajalakshmi, Shenbaga Moorthy P., Pabitha (2019). Optimal provisioning and scheduling of analytics as a service in cloud computing. Transactions on Emerging Telecommunications Technologies, 30(9). https://doi.org/10.1002/ett.3609

  9. Sun, D., Zhang, G., Yang, S., Zheng, W., Khan, S. U., & Li, K. (2015). Re-stream: Real-time and energy-efficient resource scheduling in big data stream computing environments. Information Sciences, 319(4), 92–112.

    Article  MathSciNet  Google Scholar 

  10. Kaur, N., & Sood, S. K. (2017). Dynamic resource allocation for big data streams based on data characteristics (5Vs). International Journal of Network Management, 27(4), e1978.

    Article  Google Scholar 

  11. Upadhyay, U., & Sikka, G. (2020). MRS-DP: Improving performance and resource utilization of big data applications with deadlines and priorities. Big Data, 8(4), 323–331.

    Article  Google Scholar 

  12. Yin, L., Zhou, J., & Sun, J. (2022). A stochastic algorithm for scheduling bag-of-tasks applications on hybrid clouds under task duration variations. Journal of Systems and Software, 184(4), 111123.

    Article  Google Scholar 

  13. Li, H., Fang, H., Dai, H., Zhou, T., Shi, W., Wang, J., & Xu, C. (2021). A cost-efficient scheduling algorithm for streaming processing applications on cloud. Cluster Computing, 25(2), 781–803.

    Article  Google Scholar 

  14. Shabestari, F., Rahmani, A. M., Navimipour, N. J., & Jabbehdari, S. (2019). A taxonomy of software-based and hardware-based approaches for energy efficiency management in the Hadoop. Journal of Network and Computer Applications, 126(4), 162–177.

    Article  Google Scholar 

  15. Gokuldhev, M., & Singaravel, G. (2020). Local pollination-based moth search algorithm for task-scheduling heterogeneous cloud environment. The Computer Journal, 65(2), 382–395.

    Article  Google Scholar 

  16. Khallouli, W., & Huang, J. (2021). Cluster resource scheduling in cloud computing: Literature review and research challenges. The Journal of Supercomputing, 78(5), 6898–6943.

    Article  Google Scholar 

  17. Abualigah, L., Yousri, D., Abd Elaziz, M., Ewees, A. A., Al-qaness, M. A., & Gandomi, A. H. (2021). Aquila optimizer: A novel meta-heuristic optimization algorithm. Computers & Industrial Engineering, 157(4), 107250.

    Article  Google Scholar 

  18. Mohammadi-Balani, A., Dehghan Nayeri, M., Azar, A., & Taghizadeh-Yazdi, M. (2021). Golden eagle optimizer: A nature-inspired metaheuristic algorithm. Computers & Industrial Engineering, 152(2), 107050.

    Article  Google Scholar 

  19. Wang, S., Jia, H., Abualigah, L., Liu, Q., & Zheng, R. (2021). An improved hybrid Aquila optimizer and Harris hawks algorithm for solving industrial engineering optimization problems. Processes, 9(9), 1551.

    Article  Google Scholar 

  20. Mashayekhy, L., Nejad, M. M., Grosu, D., Zhang, Q., & Shi, W. (2015). Energy-aware scheduling of MapReduce jobs for big data applications. IEEE Transactions on Parallel and Distributed Systems, 26(10), 2720–2733.

    Article  Google Scholar 

  21. Lu, Q., Li, S., Zhang, W., & Zhang, L. (2016). A genetic algorithm-based job scheduling model for big data analytics. EURASIP Journal on Wireless Communications and Networking, 2016(1), 89–98.

    Article  Google Scholar 

  22. Lim, N., & Majumdar, S. (2017). Resource management for MapReduce jobs performing big data analytics. Big Data Management and Processing, 3(4), 105–134.

    Article  Google Scholar 

  23. Hashem, I. A., Anuar, N. B., Marjani, M., Gani, A., Sangaiah, A. K., & Sakariyah, A. K. (2017). Multi-objective scheduling of MapReduce jobs in big data processing. Multimedia Tools and Applications, 77(8), 9979–9994.

    Article  Google Scholar 

  24. Shao, Y., Li, C., Gu, J., Zhang, J., & Luo, Y. (2018). Efficient jobs scheduling approach for big data applications. Computers & Industrial Engineering, 117(2018), 249–261.

    Article  Google Scholar 

  25. Hu, Y., Wang, H., & Ma, W. (2020). Intelligent cloud workflow management and scheduling method for big data applications. Journal of Cloud Computing, 9(1), 2251–2272.

    Google Scholar 

  26. Seethalakshmi, V., Govindasamy, V., & Akila, V. (2020). Hybrid gradient descent spider monkey optimization (HGDSMO) algorithm for efficient resource scheduling for big data processing in heterogenous environment. Journal of Big Data, 7(1), 34–48.

    Article  Google Scholar 

  27. Islam, M. T., Srirama, S. N., Karunasekera, S., & Buyya, R. (2020). Cost-efficient dynamic scheduling of big data applications in Apache spark on cloud. Journal of Systems and Software, 162, 110515.

    Article  Google Scholar 

  28. Abualigah, L., Diabat, A., & Elaziz, M. A. (2021). Intelligent workflow scheduling for big data applications in IoT cloud computing environments. Cluster Computing, 24(4), 2957–2976.

    Article  Google Scholar 

  29. Zhao, Y., Calheiros, R. N., Gange, G., Bailey, J., & Sinnott, R. O. (2021). SLA-based profit optimization resource scheduling for big data analytics-as-a-Service platforms in cloud computing environments. IEEE Transactions on Cloud Computing, 9(3), 1236–1253.

    Article  Google Scholar 

  30. Viswanathan, Ramasamy Jagatheswari, Srirangan Praveen, Ramalingam (2021) Fuzzy and Position Particle Swarm Optimized Routing in VANET. International journal of electrical and computer engineering systems, 12(4), 199–206. https://doi.org/10.32985/ijeces.12.4.3

Download references

Funding

The authors have not disclosed any funding.

Author information

Authors and Affiliations

  1. Department of Information Technology, Velammal Institute of Technology, Tamil Nadu, Chennai, 601204, India

    N. Jagadish Kumar

  2. Department of Computer Science and Engineering, P.S.R. Engineering College, Tamil Nadu, Sivakasi, 626140, India

    C. Balasubramanian

Authors
  1. N. Jagadish Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. C. Balasubramanian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to N. Jagadish Kumar.

Ethics declarations

Competing Interests

The authors have not disclosed any competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jagadish Kumar, N., Balasubramanian, C. Hybrid Gradient Descent Golden Eagle Optimization (HGDGEO) Algorithm-Based Efficient Heterogeneous Resource Scheduling for Big Data Processing on Clouds. Wireless Pers Commun 129, 1175–1195 (2023). https://doi.org/10.1007/s11277-023-10182-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-023-10182-0

Keywords

Search

Navigation