Abstract
Big data analytics in cloud environments introduces challenges such as real-time load balancing besides security, privacy, and energy efficiency. This paper proposes a novel load balancing algorithm in cloud environments that performs resource allocation and task scheduling efficiently. The proposed load balancer reduces the execution response time in big data applications performed on clouds. Scheduling, in general, is an NP-hard problem. Our proposed algorithm provides solutions to reduce the search area that leads to reduced complexity of the load balancing. We recommend two mathematical optimization models to perform dynamic resource allocation to virtual machines and task scheduling. The provided solution is based on the hill-climbing algorithm to minimize response time. We evaluate the performance of proposed algorithms in terms of response time, turnaround time, throughput metrics, and request distribution with some of the existing algorithms that show significant improvements.
Similar content being viewed by others
Change history
10 September 2021
A Correction to this paper has been published: https://doi.org/10.1007/s11227-021-04053-3
References
Klous S, Wielaard N (2016) We are big data: the future of the information society
Skourletopoulos G, Mavromoustakis C, Mastorakis G, Batalla JM, Dobre C, Panagiotakis S, Pallis E (2017) Big data and cloud computing: a survey of the state-of-the-art and research challenges
Zhang R (2020) The impacts of cloud computing architecture on cloud service performance. J Comput Inf Syst 60:166–174
Chang WL, Laszewski G (2019) NIST big data interoperability framework: volume 8, reference architecture interfaces
Khan S, Shakil KA, Alam M (2017) Big data computing using cloud-based technologies, challenges and future perspectives. ArXiv, abs/1712.05233
Wang L, Jones R (2020) Big data analytics in cyber security: network traffic and attacks. J Comput Inf Syst, pp 1–8
Khatibi E, Mirtaheri SL (2019) A dynamic data dissemination mechanism for Cassandra NoSQL data store. J Supercomput 75(11):7479–7496. https://doi.org/10.1007/s11227-019-02959-7
Neves PC, Schmerl B, Cámara J, Bernardino J (2016) Big data in cloud computing. Features Issues, IoTBD
Xiong H, Wang Y, Li W, Chen C (2019) Flexible, efficient, and secure access delegation in cloud computing. ACM Trans Manag. Inf Syst 10(2):1–2
Yadav V, Yadav MP, Yadav DK (2012) Reliable task allocation in heterogeneous distributed system with random node failure: load sharing approach. Int Conf Comput Sci 2012:187–192
Patel N, Chauhan S (2014) A survey on load balancing and scheduling in cloud computing
Singh A, Juneja D, Malhotra M (2015) Autonomous agent based load balancing algorithm in cloud computing. Procedia Comput Sci 45:832–841
Mata-Toledo RA, Madison J, Gupta P (2010) Green data center: How green can we perform?. J Technol Res
Chen Y, Argentinis JD, Weber G (2016) IBM Watson: how cognitive computing can be applied to big data challenges in life sciences research. Clin Ther 38(4):688–701
Shah N, Farik M (2015) Static load balancing algorithms in cloud computing: challenges and solutions. Int J Sci Technol Res 4:365–367
Fox G, Qiu J, Jha S, Ekanayake S, Kamburugamuve S (2015) Big data. Simulations and HPC convergence, WBDB
Lohr Steve (2012) The age of big data. New York Times
Kansal N, Chana I (2012) Cloud load balancing techniques a step towards green computing. Int J Comput Sci Issues
Baliga J, Ayre R, Hinton K, Tucker R (2011) Green cloud computing: balancing energy in processing, storage, and transport. Proc IEEE 99:149–167
Hwang K, Dongarra J, Fox G (2011) Distributed and cloud computing: from parallel processing to the internet of things. Morgan Kaufmann
Ghomi EJ, Rahmani A, Qader N (2017) Load-balancing algorithms in cloud computing: a survey. J Netw Comput Appl 88:50–71
Rastogi G, Sushil R (2015) Analytical literature survey on existing load balancing schemes in cloud computing. Int Conf Green Comput Internet Things ICGCIoT 2015:1506–1510
Wang S, Yan K, Liao W, Wang S (2010) Towards a load balancing in a three-level cloud computing network. In: 2010 3rd International Conference on Computer Science and Information Technology, 1:108–113
Mirtaheri SL, Grandinetti L (2017) Dynamic load balancing in distributed exascale computing systems. Cluster Comput 20:3677–3689
Kumar M, Sharma S (2017) Dynamic load balancing algorithm for balancing the workload among virtual machine in cloud computing. Procedia Comput Sci 115:322–329
Karthick A, Ramaraj E, Subramanian R (2014) An efficient multi queue job scheduling for cloud computing. World Cong Comput Commun Technol 2014:164–166
Kaur S, Kaur G (2015) A review of load balancing strategies for distributed systems. Int J Comput Appl 121:45–47
Mell P, Grance T (2011) The NIST definition of cloud computing
Sharma M, Bhatia J (2013) A review on different approaches for load balancing in computational grid. J Global Res Comput Sci 4:82–85
Liu G, Li J, Xu J (2013) Liu G, Li J, Xu J (2013) An improved min–min algorithm in cloud computing. In: Proceedings of the 2012 International Conference of Modern Computer Science and Applications, pp 47–52
El-Zoghdy SF, Ghoniemy S (2014) A survey of load balancing in high-performance distributed computing systems. Int J Adv Comput Res
Elzeki OM, Reshad M, Elsoud MA (2012) Improved max–min algorithm in cloud computing. Int J Comput Appl 50:22–27
Sharma N, Tyagi S, Atri S (2017) A comparative analysis of min–min and max–min algorithms based on the makespan parameter. Int J Adv Res Comput Sci 8:1038–1041
Kumar R, Prashar T (2015) Performance analysis of load balancing algorithms in cloud computing. Int J Comput Appl 120(7):19–27
Domanal SG, Reddy GR (2014) Optimal load balancing in cloud computing by efficient utilization of virtual machines. Sixth Int Conf Commun Syst Netw COMSNETS 2014:1–4
Li J, Ma T, Tang M, Shen W, Jin Y (2017) Improved FIFO scheduling algorithm based on fuzzy clustering in cloud computing. Information 8:25
Hamdani M, Aklouf Y, Bouarara HA (2019) Improved fuzzy load-balancing algorithm for cloud computing system. In: Proceedings of the 9th International Conference on Information Systems and Technologies
Daraghmi E, Yuan S (2015) A small world based overlay network for improving dynamic load-balancing. J Syst Softw 107:187–203
Saleh H, Nashaat H, Saber W, Harb H (2019) IPSO task scheduling algorithm for large scale data in cloud computing environment. IEEE Access 7:5412–5420
Sanaj MS, Prathap P (2020) Nature inspired chaotic squirrel search algorithm (CSSA) for multi objective task scheduling in an IAAS cloud computing atmosphere. Eng Sci Technol Int J 23:891–902
Samadi Y, Zbakh M (2017) Threshold-based load balancing algorithm for big data on a cloud environment. BDCA’17
Manikandan N, Pravin A (2019) LGSA: hybrid task scheduling in multi objective functionality in cloud computing environment 3D. Research 10:1–16
Mousavi, S., Mosavi, A., Várkonyi-Káczy, A. (2017). A load balancing algorithm for resource allocation in cloud computing
Vashishth V, Chhabra A, Sood A (2017) A predictive approach to task scheduling for big data in cloud environments using classification algorithms. In: 2017 7th International Conference on Cloud Computing, Data Science and Engineering—Confluence, pp 188–192
Lagwal M, Bhardwaj N (2017) Load balancing in cloud computing using genetic algorithm. Int Conf Intell Comput Control Syst ICICCS 2017:560–565
Ebadifard F, Babamir SM, Barani S (2020) A dynamic task scheduling algorithm improved by load balancing in cloud computing. In: 2020 6th International Conference on Web Research (ICWR)
Tadi AA, Khayyambashi M, Farsani HK (2020) OASM: An overload-aware workload scheduling method for cloud computing based on biogeographical optimization. Int J Netw Manag, p 30
Kaur G, Bala A (2021) OPSA: an optimized prediction based scheduling approach for scientific applications in cloud environment. Clust Comput, pp 1–20
Russell S, Norvig P (2011) Artificial intelligence: a modern approach, 3rd edn. Artif Intell 175:122–125
Wolpert D, Macready W (1995) No free lunch theorems for search
Wolpert D, Macready W (1997) No free lunch theorems for optimization. IEEE Trans Evol Comput 1:67–82
Lattimore T, Hutter M (2011) No Free lunch versus Occam’s razor in supervised learning. Algorithmic probability and friends. Bayes Predict Artif Intell, pp. 223–235
Yang X (2012) Swarm-based metaheuristic algorithms and no-free-lunch theorems
Calheiros R, Ranjan R, Beloglazov A, Rose C, Buyya R (2011) CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Pract Exp Softw 41
Author information
Authors and Affiliations
Corresponding author
Additional information
The original online version of this article was revised: In this article ref. 7 was incorrect.
Rights and permissions
About this article
Cite this article
Aghdashi, A., Mirtaheri, S.L. Novel dynamic load balancing algorithm for cloud-based big data analytics. J Supercomput 78, 4131–4156 (2022). https://doi.org/10.1007/s11227-021-04024-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-021-04024-8