Abstract
Frost is an agro-meteorological event which causes both damage in crops and important economic losses, therefore frost prediction applications (FPA) are very important to help farmers to mitigate possible damages. FPA involves the execution of many CPU-intensive jobs. This work focuses on efficiently running FPAs in paid federated Clouds, where custom virtual machines (VM) are launched in appropriate resources belonging to different providers. The goal of this work is to minimize both the makespan and monetary cost. We follow a federated Cloud model where scheduling is performed at three levels. First, at the broker level, a datacenter is selected taking into account certain criteria established by the user, such as lower costs or lower latencies. Second, at the infrastructure level, a specialized scheduler is responsible for mapping VMs to datacenter hosts. Finally, at the VM level, jobs are assigned for execution into the preallocated VMs. Our proposal mainly contributes to implementing bio-inspired strategies at two levels. Specifically, two broker-level schedulers based on Ant Colony Optimization (ACO) and Particle Swarm Optimization (PSO), which aim to select the datacenters taking into account the network latencies, monetary cost and the availability of computational resources in datacenters, are implemented. Then, VMs are allocated in the physical machines of that datacenter by another intra-datacenter scheduler also based on ACO and PSO. Performed experiments show that our bio-inspired scheduler succeed in reducing both the makespan and the monetary cost with average gains of around 50% compared to genetic algorithms.
Similar content being viewed by others
Notes
Analogous to the broker level in our paper.
Analogous to the infrastructure level in our paper.
Historical statistics can be found here (in Spanish): http://acovi.com.ar/observatorio/wp-content/uploads/2014/09/Ambiental5.xlsx.
References
Snyder, R.L., de Melo-Abreu, J.P.: Frost Protection: Fundamentals, Practice and Economics, Volume 1 of Environment and Natural Resources Series. Food and Agriculture Organization of the United Nations (FAO), Rome (2005)
Bishop, C.: Pattern Recognition and Machine Learning, Volume 20 of Information Science and Statistics. Springer, Berlin (2006)
Oliveira, L., Rodrigues, J.: Wireless sensor networks: a survey on environmental monitoring. J. Commun. 6(2), 143–151 (2011)
Rehman, A., Abbasi, A.Z., Islam, N., Shaikh, Z.A.: A review of wireless sensors and networks’ applications in agriculture. Comput. Stand. Interfaces 36(2), 263–270 (2014)
Buyya, R., Yeo, C., Venugopal, S., Broberg, J., Brandic, I.: Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility. Future Gener. Comput. Syst. 25(6), 599–616 (2009)
Mauch, V., Kunze, M., Hillenbrand, M.: High performance cloud computing. Future Gener. Comput. Syst. 29(6), 1408–1416 (2013) (Including Special sections: High Performance Computing in the Cloud & Resource Discovery Mechanisms for P2P Systems)
Zhai, Y., Liu, M., Zhai, J., Ma, X., Chen, W.: Cloud versus in-house cluster: evaluating Amazon cluster compute instances for running mpi applications. In: State of the Practice Reports, vol. 11, pp. 1–11. ACM (2011)
Coutinho, R.C., Drummond, L.M., Frota, Y., de Oliveira, D.: Optimizing virtual machine allocation for parallel scientific workflows in federated clouds. Future Gener. Comput. Syst. 46, 51–68 (2014)
Petri, I., Beach, T., Mengsong, Z., Montes, J.D., Rana, O., Parashar, M.: Exploring models and mechanisms for exchanging resources in a federated cloud. In: IEEE International Conference on Cloud Engineering (IC2E), pp. 215–224. IEEE (2014)
Celesti, A., Fazio, M., Villari, M., Puliafito, A.: Virtual machine provisioning through satellite communications in federated cloud environments. Future Gener. Comput. Syst. 28(1), 85–93 (2012)
Pacini, E., Mateos, C., García Garino, C., Careglio, C., Mirasso, A.: A bio-inspired scheduler for minimizing makespan and flowtime of computational mechanics applications on federated clouds. J. Intell. Fuzzy Syst. 31(3), 1731–1743 (2016)
Manasrah, A.M., Smadi, T., ALmomani, A.: A variable service broker routing policy for data center selection in cloud analyst. J. King Saud Univ. Comput. Inf. Sci. 29(3), 365–377 (2017)
Woeginger, G.: Exact algorithms for NP-Hard problems: a survey. In: Junger, M., Reinelt, G., Rinaldi, G. (eds.) Combinatorial Optimization—Eureka, You Shrink!, volume 2570 of Lecture Notes in Computer Science, pp. 185–207. Springer (2003)
Kennedy, J.: Swarm Intelligence. In: Zomaya, Albert Y. (ed.) Handbook of Nature-Inspired and Innovative Computing, pp. 187–219. Springer, New York (2006)
Pacini, E., Mateos, C., García Garino, C.: Distributed job scheduling based on Swarm Intelligence: a survey. Comput. Electr. Eng. 40(1), 252–269 (2014). 40th-year commemorative issue
Pacini, E., Mateos, C., García Garino, C.: Balancing throughput and response time in online scientific clouds via ant colony optimization. Adv. Eng. Softw. 84, 31–47 (2015)
Pacini, E., Mateos, C., García Garino, C.: SI-based scheduling of parameter sweep experiments on federated clouds. In: Hernandez, G., et. al. (eds.) First HPCLATAM—CLCAR Joint Conference Latin American High Performance Computing Conference (CARLA), volume 845 of High Performance Computing. Communications in Computer and Information Science, pp. 28–42. Springer (2014)
Pacini, E., Mateos, C., García Garino, C.: A three-level scheduler to execute scientific experiments on federated clouds. IEEE Latin Am. Trans. 13(10), 3359–3369 (2015)
Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., Zaharia, M.: A view of cloud computing. Commun. ACM 53(4), 50–58 (2010)
Iosup, A., Ostermann, S., Yigitbasi, N., Prodan, R., Fahringer, T., Epema, D.: Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Trans. Parallel Distrib. Syst. 22(6), 931–945 (2011)
Calheiros, R., Ranjan, R., Beloglazov, A., De Rose, C., Buyya, R.: Cloudsim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms. Softw. Pract. Exp. 41(1), 23–50 (2011)
Pacini, E., Mateos, C., García Garino, C.: Multi-objective Swarm Intelligence schedulers for online scientific clouds. Special Issue on Cloud Computing. Computing, pp. 1–28 (2014)
Agostinho, L., Feliciano, G., Olivi, L., Cardozo, E., Guimaraes, E.: A Bio-inspired approach to provisioning of virtual resources in federated Clouds. In: Ninth International Conference on Dependable, Autonomic and Secure Computing (DASC), DASC 11, pp. 598–604, Washington, DC, USA, 12–14 December 2011. IEEE Computer Socienty (2011)
Chandra Mohan, B., Baskaran, R.: A survey: ant colony optimization based recent research and implementation on several engineering domain. Expert Syst. Appl. 39(4), 4618–4627 (2012)
Mahdiyeh, E., Hussain, S., Mohammad, K., Azah, M.: A survey of the state of the art in Particle Swarm Optimization. Res. J. Appl. Sci. Eng. Technol. 4(9), 1181–1197 (2012)
Pedemonte, M., Nesmachnow, S., Cancela, H.: A survey on parallel Ant Colony Optimization. Appl. Soft Comput. 11(8), 5181–5197 (2011)
Poli, R.: Analysis of the publications on the applications of Particle Swarm Optimisation. J. Artif. Evol. Appl. 2008(4), 1–10 (2008)
Tavares Neto, R.F., Godinho Filho, M.: Literature review regarding Ant Colony Optimization applied to scheduling problems: guidelines for implementation and directions for future research. Eng. Appl. Artif. Intell. 26(1), 150–161 (2013)
Vlachos, A.: Ant colony system algorithm solving a thermal generator maintenance scheduling problem. J. Intell. Fuzzy Syst. 24(4), 713–723 (2013)
Masdari, M., Salehi, F., Jalali, M., Bidaki, M.: A survey of PSO-based scheduling algorithms in cloud computing. J. Netw. Syst. Manag. 25(1), 122–158 (2017)
Singha, U., Jain, S.: An analysis of swarm intelligence based load balancing algorithms in a cloud computing environment. Int. J. Hybrid Inf. Technol 8(1), 249–256 (2015)
Zhan, Z.H., Liu, X.F., Gong, Y.J., Zhang, J., Chung, H.S.H., Li, Y.: Cloud computing resource scheduling and a survey of its evolutionary approaches. ACM Comput. Surv. 47(4), 63:1–63:33 (2015)
Marosi, A.C., Kecskemeti, G., Kertesz, A., Kacsuk, P.: Fcm: anarchitecture for integrating iaas cloud systems. In: Cloud computing 2011: the second international conference on cloud computing, GRIDs, and virtualization, pp. 7–12. IARIA (2011)
Villegas, D., Bobroff, N., Rodero, I., Delgado, J., Liu, Y., Devarakonda, A., Fong, L., Sadjadi, S.M., Parashar, M.: Cloud federation in a layered service model. J. Comput. Syst. Sci. 78(5), 1330–1344 (2012). JCSS Special Issue: Cloud Computing 2011
Tordsson, J., Montero, R.S., Moreno Vozmediano, Rl, Llorente, I.M.: Cloud brokering mechanisms for optimized placement of virtual machines across multiple providers. Future Gener. Comput. Syst. 28(2), 358–367 (2012)
Kessaci, Y., Melab, N., Talbi, E.-G.: A pareto-based metaheuristic for scheduling HPC applications on a geographically distributed cloud federation. Cluster Comput. 16(3), 451–468 (2013)
Lucas-Simarro, J.L., Moreno-Vozmediano, R., Montero, R.S., Llorente, I.M.: Scheduling strategies for optimal service deployment across multiple clouds. Future Gener. Comput. Syst. 29(6), 1431–1441 (2013) (Including Special sections: High Performance Computing in the Cloud & Resource Discovery Mechanisms for P2P Systems)
Song, Y., Peng, J., Liu, K., Jiang, F., Liu, W., Huang, Z.: A hybrid particle swarm ant colony based resource reservation for geo-distributed cloud service. In: 2016 IEEE Global Communications Conference (GLOBECOM), pp 1–6. IEEE (2016)
Kumrai, T., Ota, K., Dong, M., Kishigami, J., Sung, D.K.: Multiobjective optimization in cloud brokering systems for connected internet of things. IEEE Internet Things J. 4(2), 404–413 (2016)
Feller, E., Rilling, L., Morin, C.: Energy-Aware Ant Colony based workload placement in clouds. In: 12th International Conference on Grid Computing, number 8 in Grid ’11, pp. 26–33. IEEE Computer Society (2011)
Gao, Y., Guan, H., Qi, Z., Hou, Y., Liu, L.: A multi-objective ant colony system algorithm for virtual machine placement in cloud computing. J. Comput. Syst. Sci. 79(8), 1230–1242 (2013)
Jeyarani, R., Nagaveni, N., Vasanth Ram, R.: Design and implementation of adaptive power-aware virtual machine provisioner (APA-VMP) using swarm intelligence. Future Gener. Comput. Syst. 28(5), 811–821 (2012)
Dhinesh Babu, L.D., Venkata Krishna, P.: Honey bee behavior inspired load balancing of tasks in cloud computing environments. Appl. Soft Comput. 13(5), 2292–2303 (2013)
de Oliveira, G.S., Ribeiro, E., Ferreira, D.A., Araújo, A.P.,Holanda, M., Walter, M.E.: ACOsched: a scheduling algorithm in afederated Cloud infrastructure for bioinformatics applications. In: International Conference on Bioinformatics and Biomedicine, pp. 8–14. IEEE (2013)
Zhang, G., Zuo, X.: Deadline constrained task scheduling based on standard-pso in a hybrid cloud. In: Tan, Y., Shi, Y., Mo, H. (eds.) Advances in Swarm Intelligence: 4th International Conference, ICSI 2013, pp. 200–209, Harbin, China. Springer, Berlin (2013)
Gabaldon, E., Vila, S., Guirado, F., Lerida, J.L., Planes, J.: Energy efficient scheduling on heterogeneous federated clusters using a fuzzy multi-objective meta-heuristic. In: IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1–6 (2017)
Sedeño Noda, A., Raith, A.: A dijkstra-like method computing all extreme supported non-dominated solutions of the biobjective shortest path problem. Comput. Oper. Res. 57, 83–94 (2015)
Breque, F., Nemer, M.: Frosting modeling on a cold flat plate: comparison of the different assumptions and impacts on frost growth predictions. Int. J. Refrig. 69, 340–360 (2016)
Brun Laguna, K., Diedrichs, A.L., Chaar, J.E., Dujovne, D., Taffernaberry, J.C., Mercado, G., Watteyne, T.: A demo of the peach iot-based frost event prediction system for precision agriculture. In: 2016 13th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON), pp. 1–3. IEEE (2016)
Iacono, L., Vázquez-Poletti, J.L., García Garino, C., Llorente, I.M.: A Model to Calculate Amazon EC2 Instance Performance in Frost Prediction Applications. In: Hernández, G., et al. (eds.) High Performance Computing: First HPCLATAM—CLCAR Latin American Joint Conference, CARLA 2014, pp. 68–82. Springer, Berlin (2014)
Iacono, L., Vázquez-Poletti, J.L., García Garino, C., Llorente, I.M.: A performance models for frost prediction on public cloud infrastructures. Comput. Inf. 37(4), 815–837 (2018)
Monge, D.A., Pacini, E., Mateos, C., García Garino, C.: Meta-heuristic based autoscaling of cloud-based parameter sweep experiments with unreliable virtual machines instances. Comput. Electr. Eng. 69, 364–377 (2018)
Jung, J.K., Jung, S.M., Kim, T.K., Chung, T.M.: A study on the cloud simulation with a network topology generator. World Acad. Sci. Eng. Technol. 6(11), 303–306 (2012)
Madivi, R., Kamath, S.S.: An hybrid bio-inspired task scheduling algorithm in cloud environment. In Fifth International Conference on Computing, Communications and Networking Technologies (ICCCNT), pp. 1–7. IEEE (2014)
Karaboga, D., Gorkemli, B., Ozturk, C., Karaboga, N.: A comprehensive survey: artificial bee colony (ABC) algorithm and applications. Artif. Intell. Rev. (2012)
Ghafarian, T., Javadi, B.: Cloud-aware data intensive workflow scheduling on volunteer computing systems. Future Gener. Comput. Syst. 51, 87–97 (2015)
Zhao, Y., Li, Y., Raicu, L., Lu, S., Tian, W., Liu, H.: Enabling scalable scientific workflow management in the cloud. Future Gener. Comput. Syst. 46, 3–16 (2015)
Philip Chen, C.L., Zhang, C.Y.: Data-intensive applications, challenges, techniques and technologies: a survey on big data. Inf. Sci. 275, 314–347 (2014)
Neshat, M., Sepidnam, G., Sargolzaei, M., Toosi, A.N.: Artificial fish swarm algorithm: a survey of the state-of-the-art, hybridization, combinatorial and indicative applications. Artif. Intell. Rev. 42(4), 965–997 (2014)
Zhou, A., Wang, S., Yang, C., Sun, L., Sun, Q., Yang, F.: Ftcloudsim: support for cloud service reliability enhancement simulation. Int. J. Web Grid Serv. 11(4), 347–361 (2015)
Bonabeau, E., Dorigo, M., Theraulaz, G.: Swarm Intelligence: From Natural to Artificial Systems. Oxford University Press, Oxford (1999)
Dorigo, M., Birattari, M.: Swarm intelligence. Scholarpedia 2(9), 1462 (2007)
Dorigo, M., Stützle, T.: The ant colony optimization metaheuristic: algorithms, applications, and advances. In: Handbook of Metaheuristics, volume 57 of International Series in Operations Research and Management Science, chapter 9, pp. 250–285. Springer (2003)
Kennedy, J., Eberhart, R.: Swarm Intelligence. Morgan Kaufmann Publishers Inc., San Francisco (2001)
Acknowledgements
We acknowledge the financial support provided by ANPCyT through grants PICT-2012-2731, PICT-2014-1430 and PICT-2015-1435, and UNCuyo project project 06/B308. We want to thank the anonymous reviewers for their valuable comments and suggestions that helped to improve the quality of this paper.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendix A: Swarm Intelligence Techniques
SI [61] bases on studying collective behaviors that emerge from interactions between individuals and the environment which they live in to solve optimization problems. Examples of systems in which SI inspires are ants colonies, fish schools, birds flocks, and herds of land animals, where the whole group of individuals perform a desired task (i.e., feeding), which might not be made individually. For example, an ant is relatively unintelligent, but when it is part of a colony, some behaviors emerge from the interactions between ants, such as searching for food. Moreover, an individual fish makes dynamic decisions to swim in one direction or another but only up to a certain point. If there are several fishes, when the first fish swims near a food source, the other fishes will listen to the first fish instead of following other instincts.
According to M. Dorigo and M. Birattari [62], in an SI system (a) there are many individuals, (b) the individuals are relatively homogeneous, i.e., they are either all identical or they belong to a few typologies, (c) the interactions among the individuals are based on simple behavioral rules that exploit only local information that the individuals exchange directly or via the environment, and (d) the overall behavior of the system results from the interactions of individuals with each other and with their environment, i.e., the group behavior self-organizes.
The following subsections briefly describe the most popular SI techniques—i.e., ACO and PSO—that are widely used in job scheduling problems such as the one addressed in this paper. This appendix has been partially extracted from a paper of our own [15].
1.1 Ant Colony Optimization
The ACO algorithm [63] arises from the way real ants behave in nature. An interesting aspect of this behavior is how ants manage to locate short paths to reach a food source from their nest. The ACO algorithm can solve computational problems since the algorithm has the ability to reduce paths and precisely to find the shortest paths. In nature, ants move randomly from one place to another to search for food. On the return to its nest each ant leaves an hormone—called pheromone—that lures other working ants to the same course. When more and more ants choose the same path, the pheromone trail is intensified and even more ants will further choose it. Over time the shortest paths will be intensified by the pheromone faster. That is because the ants will both reach the food source and travel back to their nest at a faster rate. Furthermore, if over time ants do not follow a certain path, its pheromone trail evaporates. From an algorithmic point of view, the pheromone evaporation process is useful for preventing the convergence to a local optimum solution.
Figure 6 shows two possible nest-food source paths. Figure 6a shows that ants will move randomly at the beginning and choose one of the two paths. The ants that follow the faster path will naturally get to the objective before other ants, and in doing so the former group of ants will leave a pheromone trail. Moreover, the ants that perform the round-trip faster, strengthen more quickly the quantity of pheromone in the shorter path (see Fig. 6b). The ants that reach the food source through the slower path will find attractive to return to the nest using the faster path. Eventually, most ants will choose the left path as shown in Fig. 6c.
ACO employs artificial pheromone trails that play the role of information that is dynamically updated by ants to reflect their accumulated experience in contributing to solve an entire problem. In practice, to optimize job scheduling problems, the ACO algorithm is mapped to graphical representations, usually graphs. A graph for example may include jobs and executing physical resources or machines (nodes) and scheduling decisions (arcs). Each job can be carried out by an ant to search for machines with available computing resources.
1.2 Particle Swarm Optimization
PSO [64] mimics the behavior of natural processes from animals such as birds and insects such as bees. In PSO the general term “particle” is used to represent birds, bees or any other individuals who exhibit social and group behavior. Suppose a group of bees flies over the countryside looking for flowers. Their goal is to find as many flowers as possible. At the beginning, bees do not have knowledge of the field and fly to random locations with random velocities looking for flowers. Each bee has the capability of remember the places where it saw more flowers, and moreover, somehow knows the places where other bees have found a high density of flowers. These two pieces of information—nostalgia and socialknowledge—are used by the bees to continually modify their trajectory, i.e., each bee alters its path between the two directions to fly somewhere between the two points and find a greater density of flowers.
A bee may fly over a place with more flowers than it had been found by any bee in the swarm. The whole swarm would then be drawn toward that location by considering the bees own individual discovery. In this way the bees explore the field by overflying locations with the greatest flower concentration and then being pulled back toward them. Constantly, they are checking the territory they fly over against previously found locations of highest concentration hoping to find the absolute highest concentration of flowers. Eventually, the process leads all bees to the one single place F in the field with the highest concentration of flowers. After that, bees will be unable to find any points of higher flower concentration, so they will be always drawn back to F. This social behavior is shown in Fig. 7.
When used to model job scheduling, a PSO algorithm instantiation consists of particles, each maintaining one potential solution to the entire scheduling problem. The design of the representation of a particle is different for each type of problem. An example of how to represent particles when modeling the job scheduling problem is placing the position of a particle in an n-dimensional search space. Each dimension i is the job to schedule, and the corresponding value represents the machine in which a job can be allocated. Furthermore, the global best position will indicate the best possible schedule.
Appendix B: Data Information from Weather Stations and Sensor Nodes
Table 6 illustrates the data structure used for running the FPA. This data were extracted from a raw dataset provided by the National Oceanic and Atmospheric AdministrationFootnote 6 database. The dataset contains 10 years of atmospheric data (from 2003 to 2013) that correspond to the the weather station of the Plumerillo Airport, Mendoza. Atmospheric data are expressed in METeorological Aerodrome Report (METAR) format. METAR format is commonly used by pilots in pre-flight weather briefing and by meteorologists for weather forecasting.
In the Table 6, the first five columns show the acquisition time data including year, month, day, hour and minute, respectively. Column six indicates cloud coverage (in METAR notation) and columns seven and eight indicate the temperature and humidity levels, respectively.
Table 7 details METAR cloud coverage notation which is reported by the number of “oktas” (eighths) of the sky that is occupied by clouds. First column indicates the METAR abbreviation and the second column represents the abbreviation description. For example, in the first row of Table 6, OVC indicates that on 5/3/2003 at 06:00 am, the sky was overcast, i.e., with full cloud coverage. Regarding okta, this is a unit of measurement to describe the amount of cloud coverage at any given location, e.g., five oktas are equal to five octaves of the sky covered with clouds.
Rights and permissions
About this article
Cite this article
Pacini, E., Iacono, L., Mateos, C. et al. A Bio-inspired Datacenter Selection Scheduler for Federated Clouds and Its Application to Frost Prediction. J Netw Syst Manage 27, 688–729 (2019). https://doi.org/10.1007/s10922-018-9481-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10922-018-9481-0