A Survey on the Current Challenges of Energy-Efficient Cloud Resources Management


Given the perpetual surging of cloud services’ requests, energy consumption of cloud data centers with their related CO2 emissions still represents major issues. Efficient use of cloud’s resources becomes then the driven force ensuring both, the satisfaction of service-level agreements and the sobriety of cloud’s energy consumption. This paper purports to survey for the first time, a comprehensive literature of some actual challenges facing various cloud’s resources utilization scheduling approaches dealing with energy conservation. Indeed, cloud resources involve not only computing servers, but also a wide set of intra- and inter-cloud network’s resources to be considered. These resources are mainly provisioned through two well-known technologies: virtualization and/or containerization. As existing studies in the area of cloud resources provisioning are generally categorized into different groups, this research depicts a complete taxonomy of energy-efficient cloud resources scheduling. In this same perspective, we survey some recent efforts made in energy-efficient virtual and containerized resources, by emphasizing first the most used reactive then proactive techniques for managing the whole cloud resources energy efficiency scheduling.


Given the perpetual evolution of heterogeneous on-demand requests and the huge need of an elastic cloud, the so-called green or sustainable cloud computing has been getting a lot of interest in recent years. According to a recent Cisco report [1], annual global cloud IP traffic will reach 14.1 ZB (1.2 ZB per month) by the end of 2020 and total hyperscale data centers (DCs) number in this year will extend to 485. The more we send email, watch online videos, conduct business online and use social media like Facebook, the more demands on data centers will grow [2]. Cloud provisioning is the allocation of a cloud provider’s infrastructures and services to their customers. The growing challenge is how to efficiently provision resources to meet these huge demands with better quality of service (QoS), minimum energy consumption and minimum greenhouse gases (GHG) emissions. Reasons behind this challenge are multiple. First, servers are the most energy intensive in all IT DCs elements [3]. Even worse, they still consume more than half their maximum energy consumption even in idle state [4].

Consolidating workload on an optimized number of resources and switching off idle ones are one of the most common solutions. Shutdown techniques can save at least 84% of energy that would be otherwise lost to power idle nodes [4]. Such solutions can be easily exploited thanks to virtualization and/or containerization technologies by deploying multiple instances of resources hosted over a physical server. However, as shown below in the state-of-the-art techniques, these various instances require energy-efficient scheduling that can be either: static or dynamic, vertical or horizontal and reactive or proactive. However, cloud computing resources not only settle for DCs infrastructures, but also refer to a wide range of network resources, since cloud is mainly a distributed and clustering system that extends over a wide geographical area. Moreover, these network links are also highly underutilized operating just between 5 and 25% [5]. All these lead toward two serious issues, which are excessively high-energy consumption and carbon emissions.

Again, the production variability of most renewable sources pushes DCs to only partially rely on them and to depend on the regular electrical grid as a backup or main supplier [6]. Nevertheless, even when supplying the entire DC energy source with renewable energy as the case of actual Google DCs, it is crucial to support this with a regular energy-efficient scheduling of the entire inter- and intra-DCs cloud infrastructure and platform utilization. In this paper, we survey the most relevant energy-aware resources provisioning algorithms with a focus on current challenges facing this field.

In this paper, we survey the most relevant energy-aware resources provisioning algorithms with a focus on current challenges facing this field. The remainder of the paper is organized as illustrated in Fig. 1.

Fig. 1

Organization of the remainder of this paper

Section 2 surveys some energy-efficient virtual computing and network resources management researches.

Section 3 represents some recent effort made in energy-efficient container resources management for both configurations: containers on bare metal and containers on the top of VMs.

As for the last section, it regroups an exhaustive taxonomy of the widely used predictive techniques of future workload and tasks or future resource demands.

Energy-Efficient Virtual Resources Management Techniques

Recent emerging researches focus on reducing energy consumption while provisioning resources dynamically and without sacrificing the quality of data center services. Dynamic resources provisioning is not a trivial problem to manage, since it requires several processes as repeated migrations and VMs reallocation. In general, dynamic techniques are the most solicited because of the enormous variability of on-demand workloads. Dynamic scheduling can be decomposed into hardware level or software level. Scheduling at virtualization or software level can be either reactively: which means planning takes place after the arrival of customer’s requests, or proactively: when we talk about an anticipated scheduling that predicts future demands and takes place in real time with customer’s demands. Again, we usually hear about horizontal planning: when it comes to creating multiple VMs and allocating resources to them, as for vertical scheduling: it refers to the reallocation of resources to a single VM as workloads change. The graph (Fig. 2) illustrates all existing solutions and manners for managing resource’s energy consumption in cloud DCs.

Fig. 2

Energy management techniques in cloud DCs

In this part, we focus on dynamic resources management approaches aiming primarily to minimize energy consumption or carbon emissions as well. The first three “Bin-Packing Technique, Evolutionary Genetic Algorithm (GA) and Natural Inspired Algorithms (NIA)” sections represent the most common techniques used to allocate and provision VMs resources efficiently in order to manage underutilized DCs and servers. But to also integrate the underutilization of the entire network infrastructures, the fourth part (2.4) represents the state of the art of VNE techniques dealing with the energy-efficient allocation of virtual nodes resources (switches and routers) and virtual links resources at network level.

Bin-Packing Technique

Bin-packing (BP) technique is widely used to fulfill a main objective, which is consolidating VMs on the minimum number of servers when allocating resources to VMs. As shown in Fig. 3, this technique considers servers or hosts as bins, VMs as items to be packed, while physical resources (such as CPU, memory, bandwidth) are viewed as dimensions. Recent BP algorithms use or improve generally some of the classical BP heuristics such as first fit decreasing (FFD) or best fit decreasing (BFD) combined with various optimization methods. In this section, we will leverage the most relevant research in this field, starting with multicloud until servers’ management.

Fig. 3

Homogeneous and heterogeneous multidimensional bin packing

For multicloud provider or distributed DCs, the author in [7] proposes first an energy- and carbon-efficient (ECE) architecture containing a cloud information service where distributed DCs register their information (such as available resources, current utilization, physical server’s proportional power and carbon footprint rate). Based on such information, cloud ECE broker performs VMs request placement in cloud computing using a derivation of best fit algorithm that can be seen as a bin-packing problem with different bin sizes. Thus, this algorithm allocates VM to three decision levels: the DC, cluster and host having, respectively, the minimum PUE value, cluster’s carbon footprint rate and dynamic power consumption. This algorithm reduces carbon footprint until 45% and power consumption until 20% comparing with other heuristics. However, results show that the SLA violation (number of rejected VMs) under low system load for ECE is slightly higher than other algorithms. Another proposed service composition [8] consists on a multidimensional bin-packing problem. The proposed broker acts as an intermediary bridge between user and subscribed datacenters, finds first the most energy-efficient cloud service provider, and examines within this cloud the most energy-efficient single atomic service or predefined composition that much the user request. Otherwise, the broker will create a composition plan of different energy-efficient services from several cloud service providers. Cloud-SEnergy algorithm maintains a low number of examined services and composite clouds. Nevertheless, other energy efficiency relevant factors, such as the power consumption of switches and links in DCs, have to be incorporated.

However, to consolidate VMs inside a DC and powering off unused PMs, it was found [9] that different sorting PMs strategies of FFD could lead to different levels of optimized consolidation and migration. It turned out that the proposed server load strategy that consists of sorting PMs/VMS in descending order of their occupied magnitude (of CPU and memory), is the overall best performing one. It outperforms other strategies when it comes to migration efficiency and server consolidation. To prevent performance degradation, migration process should occur sparingly, and paper [10] suggests limiting the upper bound of resources by keeping CPU utilization below a certain threshold value (between 50 and 80%). This author proposes then an offline algorithm to address bin-packing problem for a dynamic consolidation that tries finding the exactly free CPU available server into which a VM can fit or the closely one using best fit. This way, the algorithm prevents the wastage of free CPU available. Once there is no server capable to accommodate current task, the algorithm hands over the control to migration algorithm, which has as a main objective to allocate resources efficiently without turning on new servers, hence minimizing energy consumption while maintaining a certain level of performance.

Another approach to optimally solve the bin-packing problem in a polynomial time was proposed by Fares Alharbi in [11]. He proposes OptDiv an offline algorithm that extends the FFD heuristic with informed decisions related to power consumption, and the optimal bin chosen is the one that has the minimum cost function. This algorithm returns an optimal solution if and only if the input is weakly divisible which means (i.e., in a non-increasing order of items size, the last item size is a divisor of the size of all items that were packed before) and the cost functions are monotonously increasing and concave. Thus, OptDiv leads to higher power consumption saving; however, it is still not fast compared to FFD and other heuristics, and it is not clear whether it can carry out heterogeneous bin. However, resorting to online algorithms is the best way for solving bin-packing problem at optimality by taking immediate decisions for heterogeneous resources. Mazumdar [12] finds an alternative approach by formulating an offline problem that can be perceived as an online one: It is about a snapshot-based approach used in (near) real time to avoid ineffective migration cost, since VMs operating times are not known in advance. In addition, VMs load’s profiles are also not known; thus, VMs are provisioned based on the worst-case scenario, i.e., the case in which VM consumes all the required resources for all the time. Here, the modeled server consolidation problem (SCP) consists of allocating two sets of VMs: VMs of the current snapshot and VM0 of the previous one to a set of servers such that the total assignment cost is minimized. Results are very closer to optimality since the gaps (relative error from the lower bound) to optimality are far less than 1%, but this is at the expense of time. In addition, VMs were provisioned based on the worst-case scenario because VMs load’s profiles are also not known.

Obtaining good results in a reasonable period of time and considering heterogonous resources are therefore more significant. In [13], the author proposes a new energy-aware DVMC dynamic VM consolidation process that can be performed by heterogeneous hosts and different kinds of VMs. This process consists of four phases: a monitoring phase to collect information related to resources, energy consumption, on–off status and workload. The second one is a workload analysis and estimation phase for detecting overloaded and light-loaded hosts using several policies as: interquartile range (IQR), median absolute deviation (MAD), local regression (LR), and robust local regression (LRR). The third one is decisions on VM migration selection and migrated VMs placement phase. Finally, actuation phase is to carry out VM migration and the turning on or off of the respective physical host(s). The author in [14] improves previous result by incorporating even more useful information into the profile-based dynamic VM placement algorithm that enables updating VMs profiles dynamically at each time interval by removing the expired VMs and adding the incoming one. Then, it invokes FFD during each interval to obtain all active PMs. Next, during each slot of interval, this algorithm uses a more improved heuristic based on ant colony system (ACS) to generate VMs placement plan for the current interval. At the end, the total energy consumption is derived after ACS solution. Summarizing the results from the entire intervals gives a final solution to the profile-based dynamic VM placement problem. Results show 10% improvement over FFD in terms of energy efficiency and almost 5% improvement over previous ACS. However, the high complexity of this algorithm clearly affects the execution time. Furthermore, the number of requests for VMs is unknown in most cases. Hence, the number of slots required for a VM should be predicted. As shown above combining BP with other inspired nature algorithm gives better result, for example [15], combining bin-packing problem using best fit strategy with an improved PSO for solving efficiently VMs consolidation problem. This later follows a levy distribution LFPSO to update velocity. Results show that this combination PSOLBP performs better than simple PSO and LFPSO algorithms. However, nature-inspired techniques will be discussed in more detail in “Natural Inspired Algorithms (NIA)” section.

Unlike previous research, this author [16] considers the case where several hosts use the same energy, also takes into account the efficiency of a particular workload, while allocating VMs. This allocation is performed using a combined best fit decreasing algorithm (Cbfd), based on a power efficiency formula combining the energy efficiency of a host and the well-known linear power model depending on server utilization. Afterward, CMUR algorithm selects VMs of various types and sizes for migration process, combining three selection policies (minimum migration time (MMT), maximum correlation (MC), and random selection (RS)). Then, it avoids repeatable migration using a control migration policy verifying one simple condition: “if the saving power while turning off a host is higher than the amount of energy to migrate the VM,” the migration occurs. Until now, seldom are those researches that studied multidimensional and heterogeneous resources problem. Zhu [17] proposes a three-dimensional virtual resource-scheduling method for energy saving in cloud computing (TVRSM), in which the process of virtual resource scheduling is divided into three stages. The first one is a multidimensional vector bin packing that uses a heuristic based on the WDP (weight dot product) to allocate PMs to VMs. The second power-aware virtual resource scheduling algorithm consists of migrating VMs from overloaded host to host providing the least increase of energy using a new multidimensional host overloading detection strategy (MHOD) based on the popular autoregression model. The third process is VROA algorithm that migrates VMs from underutilized host to switch them into sleep mode. With the proposed TVRSM processes, energy consumption and SLA violation can be effectively reduced in the cloud DC thanks to the usefulness of prediction models. However, VMs runtimes as well as the cost of switching on/off hosts were largely neglected. In “Predictive Techniques” section, other predictive techniques will be discussed.

As concluded from these studies, energy-efficient bin-packing problem was widely used in cloud computing, adopting various power models. In addition, several allocation and migrations policies were proved above. However, the challenging features of cloud computing make such BP method facing some impressive challenges. First, seldom are those researches that studied multidimensional and heterogeneous resources problem with different DC topologies including network dimension. Second, as this method sorts one solution iteratively, it would be better to use or combine this method with population-based methods that manipulate several solutions (chromosomes) at once. Beyond the most common population-based metaheuristics, genetic algorithms will be discussed in the next subsection.

Evolutionary Genetic Algorithm (GA)

Genetic algorithms (GAs) are one of the most used computational intelligence branches that can potentially work for a large-scale and multiobjective optimization problems. GAs are iterative stochastic optimization methods inspired by evolution theory. In general, GAs represent the first population by encoding chromosomes; in our case, chromosome can represent a set of VMs mapped to PMs. It selects then a population part according to fitness value/values or objective function in order to reproduce descendants with the same genes in crossover phase. These descendants undergo a mutation stage that randomly modifies their genotypes and then are further evaluated to generate the best solutions among the entire population. The graph illustrates the main operators used in each of encoding, selection, crossover and mutation phases (Fig. 4).

Fig. 4

The main operators used in the GA phases

Starting with a static resource allocation (single core servers) approach for cloud DC considering fat tree topology, a multiobjective GA adopting non-dominated sorting genetic algorithm (NSGA-II) was addressed [18] to find a trade-off between tasks completion time and system power consumption. This NSGA-II consists of a ranking procedure that classifies solutions according to their non-dominance level or (Pareto-optimal solutions). However, this approach does not necessarily sort feasible solution. To this end, constrained domination was also introduced so that better rank is given to solution with smaller constraint violation. In addition, when a rank of non-dominated solutions exceeds population size, crowding distance method has been carried out to only choose the largest cuboid surrounding this solutions rank. The same static allocation and experimental setup were performed in [19] using MOGA and simulated annealing heuristics. However, NSGA-II gives better results. These two approaches still have to adapt to dynamic task allocation in order to reallocate or migrate tasks. As recent works focus on multiobjective (MO) to cover multiple aspects of virtual machine placement (VMP), the same concept of adapted Pareto filter appeared again in [20]. This concept was accompanied with an alternative variable neighborhoods search (VNS) that ignores VMs hindering the search for better scenario, so that no migration will be needed. The aim is to quickly get the best non-dominated scenarios in at least one objective without being worse in any others. Nevertheless, task migration remains a main feature in pervasive cloud computing, and a number of possibly conflicting objectives should be considered in order to find an optimal migration configuration. This author [21] chooses simulation tool (CloudSim) to prepare experimental data and get each task’s running time on all VMs. Then, this time and other information such as capacity, resource, and energy consumption are used on the GAs framework JMetal to give optimized solutions for task migration paths in the form of chromosome vector. Results are compared using CloudSim tool to evaluate migration strategies so that the cloud system can finish all user tasks within minimum amount of time.

Concerning dynamic allocation approach, a combination of DVFS and GA techniques was introduced in [22] for energy-efficient VMs allocation at both single-server and cluster-level heterogeneous environment. The proposed GA first randomly allocates VMs on PMs and applies fitness evaluation on each chromosome according to how VMs are fitted on less number of low configured servers. Then, a roulette wheel selection operator selects chromosome having the biggest fitness in order to participate to the crossover phase. After completion of the crossover and mutation phases, random workloads are applied on VMs. Results show that the mean shutdown time of PMs has been changed from 920 s in case of DVFS random allocation to 640 s by using GA. The proposed hybrid GA and DVFS resources allocation consumed 22.4% less power over DVFS random and 43.1% over genetic without DVFS, respectively, and 0% SLA violations since no migration was considered. The same author adds a bin-packing FFD approach in [23] to apply workload to VMs after allocating VMs to PMs. The average utilization of PMs was improved because of using both GA and bin packing, thereby allocating VMs on the lower configured PMs. However, a problem of classical FFD algorithm is that different orders of service requests may result in different outputs. This is way; another similar combined FFD with classical GA approach [24] uses the first chromosome encoding as a representation of the service request’s order. Result shows better performance in solving the server consolidation problem. However, the hybrid GA costs more time than FFD algorithm. So to reduce the algorithm’s computation time, a novel improved genetic algorithm (IGA) combined with modified first decreasing (MFD) technique [25] is proposed to determine the near-optimal solution for multiobjective heterogeneous resources allocation, using Euclidian distance as fitness value for achieving both energy reduction and minimization of resources wastage at the green cloud DC. In this study, MFD technique was used to better generate a part of chromosomes population, while the remaining part was generated randomly to ensure diversity. Then, the same GA’s process occurs, except that this later uses elitism policy to avoid losing best chromosomes. Results demonstrate that the proposed IGA allocates VMs without disturbing already allocated VMs at the DC and performs better than FF, MFD, and other GGA algorithm in terms of energy consumption, time execution, and scalability. Again, to manage energy consumption meanwhile focusing on investigating a decrease of SLA violations (SLAVs), another research [26] presents a GA for heterogeneous resource allocation (GARA) that keeps CPU utilization under 70% and satisfies host, RAM, and BW capacities. Afterward the same GA process is repeated except that in this study, mutation phase was performed before crossover phase. However, this study still has to improve compared to DVFS.

One of the main points noticed in GA is that the number of iterations of a GA is always set based on experience. However, the accuracy of results depends on chosen parameters. An energy-efficient virtual resource dynamic integration (VRDI) framework following the three steps was proposed in [27]. The first one is a PM selection according to resource utilization and the corresponding variable thresholds predefined by local regression. Second is a VM selection for either a full migration or a partial one using a minimum migration (MM) policy. As for VM placement, an improved GA denoted IGAVP was proposed. However, in this study the iteration number is low. So, to obtain an optimal solution as in the case of high iteration number, IGAVP terminates if the optimal fitness of the offspring individual is no longer increasing. The proposed VRDI has a significant advantage in terms of reducing the energy consumption and selects less number of VMs for migration. Thus, it can be useful in the construction of a green DC. Another load balance GA [28] can also get only fewer VM migration amounts using a new GA mapping tree. This mapping can make the load change of host meet predetermined load constraint conditions with minimum number of VMs migration. After setting first the amount of VMs into the smallest node load to produce the leaf node (i.e., the most loaded node having no child nodes) of the initial spanning tree, it evaluates the individual VM fitness according to load ratio. Then, it selects randomly two-crossover host using RWS with part of VM to combine them. Thereafter, it deletes VMs with two or more links to form a new tree and allocates deleted VMs on under loaded hosts. Finally, a small mutation probability of individuals is set in order to avoid local optimum. Comparing to traditional methods, result shows that LB-GA not only can obtain a load balance scheme, but also get least VM migration amount. A further research believes that consolidation not always lead to energy reduction. Therefore, the proposed load balancing approach to resource provisioning [29] aims at distributing load evenly across available PMs instead of consolidating them on one single PM. The initial population of GA are generated either randomly or using another version of FFD where task is not mapped to the same PMs, but in an arbitrary order without consideration of the objective function. The selected PMs are those that have less difference of the current load and load variance. Then to form the chromosome child, the algorithm removes first the set of requests, which have been mapped multiple times, and reallocates them to PMs in chromosome child having the maximum difference between current load and variance. This empirical measure heuristic of distance is used for crossover, so that the selected mapping beyond the two parents is the one whose difference value is lower. Result shows that the proposed crossover and mutation operators perform better as compared to generic crossover and mutation operators and hence produces a better pool of childs in succeeding generations. Table 1 provides a synthesis listing the various approaches mentioned in this section.

Table 1 Comparison between the existing energy-aware GAs approaches

Nevertheless, as concluded several times from previous researches, GA takes more convergence time, especially when the solution space is large and when the fitness values of parent chromosomes are low. In addition, even the random migration of VMs in chromosome during crossover operation can lead over poor-quality solution in GA. To deal with this problem, paper [30] proposes a combination of GA with particle swarm optimization (PSO heuristic, since PSO has fast convergence. Therefore, to improve fitness value of parent chromosome, PSO should be applied before applying the crossover and mutation operations of GA, and this is the main process of the proposed HGAPSO algorithm. Next infeasible solution was reallocated using FFD. Results indicate that the proposed hybrid combination not only saves the energy consumption by 9.8% less but also improves the average resource utilization by 13% more in homogeneous and heterogeneous environments, when compared to bench mark solution achieved by exact algorithm.

To sum up, GAs often avoid premature convergence due to the diversification of solutions established in the crossover and mutation phases. However, it should not be neglected that these two phases are carried out on the best selected chromosomes. Hence, chromosomes quality can be degraded and low-quality solutions can be produced. Sometimes, combining GAs with NIAs can converge toward more optimal solutions, especially when finding an intersection between the convergence speeds of these two kinds of metaheuristics. More details on natural inspired heuristics will be demonstrated in the next “Natural Inspired Algorithms (NIA)” section.

Natural Inspired Algorithms (NIA)

This section aims to illustrate some of the main nature-inspired metaheuristics that are belonging to the field of swarm intelligence, namely (ACO) ant colony optimization, (ABC) artificial bees colony and (PSO) particle swarm optimization. Figure 5 demonstrates some of the typical parameters, and operators used in these 3 main used natural inspired algorithms. This section highlights especially these metaheuristics through energy-efficient cloud resource management researches.

Fig. 5

Typical parameters and operators of the three most known NIAs

Starting with ant colony optimization, this metaheuristic imitates the indirect communication between ants by the use of pheromone. At the beginning, ants choose a random path to follow, but after some time, the entire colony converges toward the shortest path having the largest pheromone concentration, since on longest path, pheromone tends to evaporate quickly. So as we already know that GA has a fast global searching ability in the beginning, whereas ACO is not effective at the primary stage, and the presented paper [31] finds an intersection between the convergence speeds of these two algorithms. This means, when the evolutionary rate of successive iterations decreases, GA is replaced by ACO to guarantee the solving precision. So, once several optimal solutions are produced by GA, pheromone is initialized to place ants on these optimal resources. Then, they start searching for another point by transition probability until completing tasks allocation. Result demonstrates that the proposed algorithm’s completion time is smaller than the traditional ACO in a large scale of cloud computation. Additionally, since ACO is highly compatible with distributed cloud, this author [32] applied ACO metaheuristic through an improved three-architecture load balancing. The proposed architecture divides tasks received by load balancer into elementary subtasks using MapReduce in order to minimize the size of users’ requests. This is due to Taguchi plan results suggesting that request per user and data size are the most influential factor in optimizing response time. Indeed, the proposed selection policy is ACO, where the initial pheromone contains processing capacity, storage capacity, and the enable memory and bandwidth ability indicators of a certain VM. After this step each ant selects a VM for an appropriate user’ tasks according to probability of weighted pheromone, cost, and load balancer indicator. Then, the best solution is saved and pheromone updated till the end of maximum iterations. However, this study does not focus on energy efficiency as much as response time. So, to minimize total energy consumption of all active PMs in DC, a more improved ACO algorithm was formulated as a solution to profile-based VM placement problem as mentioned in [14] from “Bin-Packing Technique” section.

A further most organized and smart insects are bees. ABC is another metaheuristic inspired by the smart behavior of honey bees. There exist two predefined types of bees: first, the employed bees that exploit food source and share the information with certain probability by waggle dance or chemical pheromone, and these employed bees tend to become scout bees after reaching exhausted source. As for the unemployed bees, they are onlooker bees that just observe the dance of employed bees and calculate fitness value of sources. A power-aware artificial bee colony VMs allocation technique was proposed in [33] for private cloud. As the request of cloud users generated, the VMs list is submitted to the DCs broker, and accordingly VMs are created with parameters such as number of processing elements and total of Mips. The VMs allocation is then done using ABC, where population is initialized as a first VMs allocation solution. Then new positions/solutions are produced on their neighborhood by employed bees using greedy selection. These new solutions are evaluated by onlooker bees using fitness cost function. Then if an abandoned solution exists, it will be replaced by scout bee solution. At the end, after evaluating power consumption of CPU utilization, results concluded that the proposed ABC algorithm produces better results than the existing GA for power-aware VMs allocation. But given the weight of energy consumption recorded by DCs, another paper [34] extends the previous study [32] by suggesting an energy efficiency solution for cloud computing adopting a fast and smart bee colony scheduling algorithm that respects both response time and economic energy system. So first, in order to identify the key factor that really affects energy consumption, a Taguchi plan was experienced to carry out a deep evaluation of the system. Almost all scenarios highlight that switch core is the indicator having the impact on energy consumed by switch followed by access to host factor that most influences the energy consumed by servers. According to these results, ABC was proposed to identify an optimal host for a given VM in a very short time. So after initializing population and first best solution, an acceleration coefficient is randomly chosen to find new position. Once new solution was found, fitness evaluation and selection likelihood are calculated to keep better bees and update best solution and scout bees. Moreover, several observations and conclusion are still not exploited. Similarly, a new approach for a three-tier architecture cloud broker containing three scheduling levels was presented in [35], to obtain the resource required for tasks without wasting energy. The proposed algorithm inherits the basic idea from ABC algorithm to enhance the execution of cloud tasks and the performance of a non-preemptive scheduling. Same, new initial population (solutions for task allocation) was generated randomly. In this study, the fitness value of each task represents the ratio of the length of task on the number of processor and processing speed of a certain VM. Scout bees that offer the highest fitness are then selected, and all related VMs, which they have visited, are memorized. Afterward, onlooker bees are updated based on their fitness value that in addition to the previous fitness adds a bandwidth ability term. In addition, this algorithm is an effective cost and context (database)-aware scheme.

Again, beyond the smartest heuristic search technique: the particle swarm optimization (PSO). This technique is also a population-based search, which simulates the movement of a swarm of birds that aim to find food. Similar to previous two techniques, each particle is firstly positioned randomly in the search space. Particles are characterized by their positions and velocities, so that each particle moves according to its current speed, its best solution and the best solution obtained from its neighbors. This work [36] proposes an accurate power model for a static and dynamic server consumption that was validated with the real measurement taken in the enterprise servers. Since this nonlinear model incorporates several parameters such as temperature, CPU dynamic voltage, memory constant voltage, frequency, and other further constants, a multiobjective particle swarm optimization (PSO) was selected to find a set of solution with less error. Notably, the average and maximum errors of the proposed model are compared to the real power consumption. After conducting tests gathering real data from an enterprise server adapted to cloud, the proposed model was validated by executing three different tests representing real workload of a cloud computing data center. Thus, this model allows targeting many optimization problems that work on setting an energy-efficient working point. Even so, most researches apply the common linear model of server power consumption that depends on CPU usage. One more power-aware PSO design [37] took as an input this linear power model, dimension, and the current utilization of PMs in order to get out an efficient VM provisioning in cloud environment.

However, to tune vagueness of the inertia weight of the PSO, a multiobjective smart fuzzy particle swarm optimization [38] was supported to minimize power consumption, idle resources (CPU, memory) and VM transfer time during the load balancing issue. So, after creating an initial population array of every particle with random positions and velocities in the search space, vector position value was converted from continuous value to discrete one to better determine the optimal pattern for mapping. Then, idle resources constraint was verified in order to calculate a normalized fitness value as well as the values of energy consumption objective function. After comparing each particle’s fitness evaluation, personal best fitness function value and global neighbor fitness function value are updated. And the proper choice of inertia weight is also updated by through a FLS that uses some IF–THEN rules to perform operations on data by a series of inferences, to reach an appropriate conclusion. Results indicate that the fuzzy logic system leads to better results when compared with typical PSO. Nevertheless, in order to minimize primitive mathematical operators and reduce dependency on fine parameter tuning, another modified binary particle swarm optimization [39] was proposed and applied for the VM placement problem to minimize energy and resource wastage. Since the initial population in the binary search space is generated randomly, it is not necessary that the particle has to satisfy primarily predefined constraints. Therefore, a generator algorithm was used to assign randomly particular VM to a PM in a binary way. The same PSO process occurs, until completing the preset iteration number. Finally, Pareto-optimal solutions are computed using predominance concept. The obtained results established that the overall performance of the proposed method is improved due to dynamic neighborhood concept.

From the above state of the art on the three most known natural inspired heuristics, we can conclude that each heuristic has its own aspect of intelligence. Results obviously differ when combining techniques rather than using typical heuristic. Again solutions differ from one approach to another depending on parameters’ setting (Fig. 5) and on problems specifications. Hence, it is quite complex to judge on the relevance of each of these three heuristics. Nevertheless, paper [40] investigates these three approaches to compare them using well-known existed algorithms for dynamic task scheduling problem in cloud computing. The experimental results prove that ABC algorithm is the superior; then, PSO and ACO can be putted in second level and third level, respectively, in terms of makespan, degree of imbalance, and standard deviation. So the high adoption of these three swarm heuristics is principally due to their intelligent exploration of search space. But, they are not the only ones. Recently, new nature-inspired metaheuristics such as “firefly algorithm; cuckoo search; bat algorithm; artificial social spiders and monkey search” have also appeared in, respectively [41,42,43], to deal with scheduling problem in computing environment. However, despite being very efficient they are merely explored and never been used for the purpose of energy-efficient scheduling in clouds.

All researches mentioned above demonstrate how to deal with underutilized servers. But most of them forget that network infrastructures are also often underutilized. Next subsection will be dedicated specifically to this subject, in order to extend the allocation of resources to all virtual network infrastructures.

Virtual Network Embedding (VNE)

Decisions made around the choice between local and remote cloud have been so complicated, to the point of thinking about an optimization model resolving this choice issue [44]. Nonetheless, remote cloud becomes nowadays indispensable. Even more, the future of cloud industry requires the consideration of multicloud DCs interconnected by large optical networks.

As reported from an energy profiling system simulating power consumption at network multiprotocol label switching (MPLS) level [45], the packet-processing rate is the main power driver instead of bit-processing rate. This makes senses given the CPU overhead coming from packet headers and also further supports the proposal to schedule energy efficiency of resources uses in large backbone network.

Hence, if we observe the growing number of dispersed and interconnected DCs, network resources virtualization becomes the promising technology that surpasses internet ossification. It enables flexibly extending existing network without involving much configuration to support heterogeneous and dynamic applications. Toward this end, many physical nodes and links should be taken into consideration while allocating virtual nodes and virtual networks/links (VNs) as illustrated in Fig. 6: These allocations refer to (VNE) virtual network embedding. Usually, VNE could be divided into two subproblems: virtual nodes mapping into either substrate network devices or DCs servers, and virtual links mapping into substrate paths.

Fig. 6

Virtual network embedding architecture

Certainly, network virtualization has largely served internet service provider (ISP). However, previous work focused on maximizing just the ISP’s revenues. The first paper focusing on minimizing the energy cost of the whole system; meanwhile, maximizing the number of VNs that can be accommodated by an ISP is demonstrated in [46]. In this work, VNE was performed as a one-to-one mapping that includes first the virtual router mapping heuristic on a substrate router satisfying quantity constraint and router location and generating low electricity price. Secondly, a host mapping satisfying nodes requirements such as CPU speed, memory size, and storage capacity was selected using a best fit strategy. As for virtual links, a weighted shortest path calculated with Floyd’s algorithm with preference of active nodes was used to find substrate links with available bandwidth and loop-free paths between two routers and also between host and router. When the whole network components are efficiently selected, this study obtains the global optimal VNE solution using a variant of discrete PSO metaheuristic that initializes population by an energy-aware local selection (EALS) strategy and determines the quality of these solutions by calculating the additional energy cost of them. Finally, results conclude that the proposed algorithms can save up to 50% energy cost for ISPs than existing state-of-the-art algorithms. However, to optimize energy consumption over the VN entire life cycle and in a realistic DC topology, this work [47] designs a novel VNE model developed through a topology and migration-aware energy-efficient heuristics in order to minimize energy usage caused by operations and future migration in DCs. Here, the requested computational resource of virtual node and bandwidth was modeled for day and night times since users’ workloads often change predictably with time. And physical nodes can be either at sleep or awake mode according to the real fat tree DC topology. In addition, the proposed heuristic selects first optimal DCs with enough available resources on awake physical nodes to avoid intermigrations that cost very high on energy. Next this heuristic roughly estimates the energy consumption, resulting in embedding a virtual node in one DC and then to another one after certain time. The two DC sets with the smallest estimated energy consumption are then picked, and virtual nodes with largest consumption difference are embedded first. Once embedded, each virtual link will be embedded to a physical link with the shortest path under resource constraints. Comparing to other two heuristics, the proposed heuristic achieves the smallest total energy consumption.

As shown above, the first study [46] used a worst-fit scheme to select substrate nodes with higher degree and bandwidth and the second [47] selects DCs with largely available resources. However, these selection types often lead to resource wastage and fragmentation. So to deal with this issue, this author [48] presents a combination of network embedding and server consolidation considering the total data center virtualization DCV. This approach considers fat tree topology for its minimum oversubscription ratio and keeps the DC network maintains a minimum spanning tree (MST) for minimum connectivity between servers. Three groups of servers, namely near traffic that flows between two interconnected servers, middle traffic that should flow between aggregation switches and far traffic flowing in core switches, were defined. The proposed heuristics first uses virtual machine mapping by finding first all possible mapping groups that have the number of available physical servers. It chooses next candidate groups with the least physical servers which need to turn on then a group that has as many servers that are near to each other as possible. Accordingly, virtual links mapping selects the path that has least available bandwidth and consumes the least additional energy. Server consolidation becomes necessary when VDCs are left underutilized. Toward this end, three VMs migration strategies were presented in this work (partial migration, on arrival migration (MoA), and full migration). Result shows that the proposed MoA can be the most suitable strategy in terms of energy efficiency as well as performance. Nevertheless, for a granular power consumption consideration, this work [49] takes a very generic, detailed, and accurate approach toward energy-efficient VNE (EEVNE). This approach allows the model to decide the optimum way to minimize the total backbone network and server power consumption considering router, OXC switch, transponders, multiplexer and demultiplexer, and EDFAs for long-distance transmission. Here, each substrate node is considered to host a DC, IP, and optical equipment under several constraints. The proposed heuristic tries to consolidate the embedding in DCs by filling the ones with least residual capacity before switching on others. Then, it distributes the bandwidth demands between candidate substrate nodes using the minimum hop algorithm and the minimum distance between nodes to reduce number of EDFAs, thus minimizing the network power consumption. Results show that the EEVNE model achieves a maximum power saving. In addition, O-OFDM-based cloud networks were also developed and shown that they have improved power and spectral efficiency compared to conventional WDM-based networks. Again, considering the WDM network due to its ultrahigh capacity transmission, the author in [50] takes into account energy consumption and load balance through multiobjective VNE algorithms. This author defines primarily the static and dynamic power consumption of a power-up server, the energy consumption of substrate fiber links based on link’s length, and the power consumed by repeaters. Then, this author considers also the additional power caused by the micro-engine used for forwarding bulk of packets when a virtual link passes through a substrate node. Unlink classical node ranking method, the proposed node mapping algorithm incorporates node energy consumption and node resource utilization to propose a comprehensive node ranking method called CNRM. It is used to measure the substrate node importance so that the lower the node resource utilization, the less the additional energy consumption, the larger the value of CNRM. As for virtual mapping stage, this work utilizes an improved differentiated pricing strategy to assign different weights to substrate links according to their resource utilization ratios comparing to predefined threshold. Finally, this algorithm adopts the shortest path to select substrate path with minimum link energy consumption during the link mapping process with the aim of balancing load. Extensive simulation results demonstrated that the proposed method (E-LB-VNE) can reduce the energy consumption and improve the load balance of substrate network while slightly deteriorating the long-term average revenue, acceptance ratio, and revenue to cost ratio. Again, to minimize total power consumption (router port, EDFAs, OXCs, transponders and servers in DCs) in software-defined optical data center networks (OCDN), study in [51] proposes a global topology resource (GTR) reflecting the selection of each node and link in the substrate network. Based on this GTR method, two energy-efficient VNE algorithms were composed to improve the performance in the scalability of requests and topology. The first one is a two-stage (node mapping, link mapping), and the second one is a coordinated VNE algorithm based on ACO and GTR node ranking. Simulation result demonstrated that GTR is an efficient node ranking method that can obtain 5% increase of acceptance ratio compared with benchmarks.

Additionally, as MapReduce becomes a cloud computing paradigm that is widely deployed in many DCs, the author in [52] presents EnergyMap: a novel energy-efficient embedding method that requires splitting the computation-based virtual nodes (mappers/reducers) except (shufflers), in order to control the introduced incast queuing delay. In addition, EnergyMap enables mapping heterogeneous MapReduce-based VNs onto heterogeneous data center network. There is also a possibility to collocate virtual nodes (mappers/shufflers/reducers) of a VN in a single relevant substrate node. Using Waxman algorithm to generate random virtual network topologies and considering heterogeneous DCs, the power consumption of EnergyMap was significantly reduced due to the fact that it allows node splitting or node collocation. However, further efforts should improve the algorithm’ scalability to large network sizes. Dissimilar to above studies, another VNs splitting approach not only focuses on maximizing revenue and energy cost reduction, but also moves its full attention to CO2 emissions factor resulting from electricity generation. This work [53] comes up with a potential trade-off between centralized and distributed approaches. It is about a hybrid approach that reduces the communication overhead while facilitating more optimal embedding. The proposed hybrid two-tier hierarchical VNE contains three levels: on the top, a service provider that receives VNs request and specifies the virtual nodes and links constraint. Then, the second level is a virtual infrastructure provider containing central nodes. These nodes split VNs into a set of small subnetworks and then forward each subnetwork service type to management nodes. These last nodes find the closest processing capacities to the capacity requested and having the smallest CO2 emission factor. In this study, virtual links were mapped to the minimum number of physical links and shortest paths. As for the last level, it contains physical infrastructure provider offering substrate resources categorized into a set of sectors predefined by their energy source type and CO2 emission factor. The proposed hybrid algorithm not only reduces the number of resources and time required for VN request mapping, but also provides the smallest carbon footprint of the VNE operation, as well as resources’ consolidation and energy savings.

To sum up this subsection, when it comes to optimizing resources utilization through global virtual resources scheduling, the global granular power consumer’ components must be taken into account. However scarce are studies digging in depth in the entire cloud DC virtualization (VDC), considering servers, switches, routers, and large backbone network components’ power consumptions.

In fact, it still turns out that there is another path of investigation into mobile network environment, which are trying nowadays to improve VNE’s energy efficiency. Indeed, it is worthwhile to have a 5G wireless network seamlessly integrated with the optical DC infrastructure to meet the requirement to interconnect users with DCs. Instead of consuming a large amount of energy to build inter-DC paths, this author [54] presents a novel design framework of location recommendation-aware virtual network embedding to minimize energy consumption. This design consists of embedding virtual networks owned by combined users group with high interdependency in terms of location and recommendation: point of interest into the same DC and this through an optical wireless hybrid networks supporting 5G models. Through an energy-efficient VNE heuristic, VNs of user groups with similar POI trajectories were put together into a set and embedded into the same DC that has a high node degree and available computing resources. Compared to traditional VNE without the awareness of location recommendations, extensive simulation results show that the improvement ratios of profitability and energy efficiency are 20% and 38%, respectively.

To conclude this entire section, significant efforts have been made on developing resource allocation policies for cloud virtual resources. However, these policies can or cannot adapt to containers. Therefore, existing VMs resource allocation policies may have to be refined and further optimized for containers.

Energy-Efficient Container Resources Management

Container as a service is a new layer that becomes recently majorly deployed in cloud environment for its large benefit on energy efficiency. In fact, The Cisco Containerized Data Center showed a significant improvement, since it delivers a power usage efficiency (PUE) from 1.05 to 1.25 [55]. Similarly to VMs, containers are also characterized by their requests of several resources. However, containers startup time usually takes only a couple of seconds, since application in a container is partially dissociated from operating system. Thus, these are more lightweight and easy to move (migrate). But unlike VM migrations, [56] in practice containers migration can be more complicated since it requires large amount of OS state that must be copied along with the memory pages. Researches that address the efficient allocation of resources to containers are scarce and still lead to confusion. Some believes [57,58,59] that the use of container on bare metal is more optimal in terms of energy consumption (Fig. 7b). Others think [60,61,62] that the combination of the two sand-boxing technologies (VM containers) is more efficient (Fig. 7a). The question becomes then how to configure containers, and whether migrating a VM, a container, or a function inside a container to better achieve energy efficiency.

Fig. 7

a Containers on the top of VMs, b containers on bare metal

Actually, the most known containers manager is Docker. This study [63] indicates that VM containers are very close to native Docker containers deployment or, in certain cases, slightly better in terms of latency. In addition, Google Container EngineFootnote 1 has recently introduced the combination of the two sand-boxing technologies (i.e., container and virtualization), which uses the open-source technology Kubernetes.Footnote 2

Starting with VM container combination, a framework consisting of “Host Status” and “Consolidation” modules was addressed in [60]. Host status module starts with a host under-/overload detector component that compares utilization with two static thresholds. Then a container selector is based either on correlation analysis between container load and host workload (Mcor policy) or on the container having the biggest utilization (Mu policy). Concerning the second consolidation module, it selects hosts destination for these containers using two separates algorithms based on least full host selection policy and correlation threshold host selection policy (CorHS). Results of several experiments show that “CorHS” is the most energy-efficient host selection algorithm with 7.45% less consumption and less than 5% SLA violation comparing to other algorithms (random, FF, least CPU utilization). However, these results show the same number of container migrations, since migration decisions were based on host load rather than VM load. So as shown, more focus was just on host level. Again in this study [61], a suitable VM for containers in terms of type and operating system was selected first using first fit (FF) algorithm regarding both CPU and memory. However, PM’s selection was more largely performed using PSO. During this step, particles (allocation solutions) move using three PSO search operators (subtraction, addition, multiplication) following the local and global best positions and based on fitness value which is the total energy consumption in a DC. Evaluation results show that the proposed algorithm generates solutions with lowest energy consumption. However, VM level was not examined sufficiently; in addition, this later paper assumed that VMs real sizes follow resources requirement. Nevertheless, unlike traditional VMs fixed size configuration, the combination VM container requires a VM consolidation level that should examine VMs variable sizes configuration. Yet, this configuration could be inefficient due to user’s overestimation of required resources on virtual machine level. To better investigate this problem, this author [62] focuses on finding efficient virtual machine sizes and suitable type for hosting containers with minimum wastage of resources on VM level. The proposed system architecture finds first the number of task’s cluster with similarities using X-means: an extended version of K-means. This extended version uses either a whole features set (WFS) as input such as task length, submission rate, scheduling class, priority, and resource usage or just the most effective ones such as the average CPU and memory utilization (RFS). Once task’s cluster defined, a VM sizing strategy based on average amount of CPU required per hour and VM memory size was performed to define a suitable VM size. Results indicate that RFS results in 81% less energy consumption in comparison with WFS due to the least number of instantiated VMs, rejection rate, and resource fragmentation. In addition, comparing the proposed usage-based approach with the baseline scenario (resource requirement), the energy consumption was improved by almost 50%. However, the proposed usage-based approach cannot adapt to new customers, where there is no historical information about usage.

Other studies view containerization as a good alternative to traditional hypervisor-based virtualization, especially using Docker container technology. A workload and energy-efficient container (WEEC) brokering system categorized into two events was formulated in [57]. The first one is a request arrival event, and then a periodical resource monitoring event estimates the average value of global resources utilization (CPU, RAM, Disk) of a container, and this, for all containers in whole servers, is using the exponential weighted moving average (EWMA). Afterward, the optimal cluster of current workload is determined using PAM (k-medoids) algorithm. Then, the optimized request allocation plan for arrival requests is found by solving the objective cost function relative to minimum ppA of linear programming solver. After experimenting environment on three heterogeneous servers (1, 2, 3) having, respectively, increasing performances, and three bench application types (CPU intensive, memory intensive, disk intensive) workloads, results show that server 3 is the optimal one beyond others, since it has good performance level and requires less energy consumption than server 2. Therefore, servers have to be chosen adaptively based on request type. However, as the energy consumption and performance of a container are primarily impacted by the number of cores assigned to it and their frequencies, this author [58] focuses on the energy efficiency of cores allocation problem for containers in a DC and decides the number of cores, core frequencies, and the core deployment per container. This study considers two scenarios: single server and multiple servers; both scenarios take into account DVFS per server and DVFS per container. As the problem was divided into two subproblem scenarios, dynamic programming was first proposed to obtain the optimal cores allocation, which takes into account even the energy consumption of the previous cores allocation, by recurrence. Thus, the number of cores and their frequencies are derived from the triplet FEC table. However, for more scalable, fast, and efficient result, greedy heuristic and other consecutive allocation heuristics were proposed. Evaluation on different sets of container and servers under the three proposed algorithms shows that the greedy heuristic has the smallest execution time. Therefore, as for nearly optimal energy consumption, it only takes 0.123 s to generate optimal energy using consecutive allocation algorithm. However, these experiments were set just on homogenous server. Managing data center heterogeneous resources becomes among the main things. Another paper [59] investigates the possibility of using a single-resource manager to manage DC heterogeneous resources, energy, and performance efficiently while considering several sand-boxing strategies. Therefore, the proposed H2 framework consists of a H2 scheduler that predicts workload and defines whether it should fall within bare-metal HPC, VM, or container based on historical data. Then a H2 orchestrator predicts whether a workload should be migrated based on energy consumption, previous runtime (execution time), and performance requirements in order to minimize energy consumption and performance loss. After extending CloudSim simulator to simulate VMs and containers, results show that the proposed H2 is approximately 14.61–37.97% more energy efficient and 7.23–20.0% more performance efficient than only virtualization and containerization technology, respectively. This is because the allocation/scheduler of H2 is aware of the workload type. However, this research suggests that even though containers running inside VMs provide the highest level of resource utilization, their performance is limited to the performance of VM, thus consuming more energy. So, the approach “containers run on bare metal” provides optimal energy and performance efficiencies. Table 2 provides a comparative synthesis of all approaches mentioned in this section, for both containers configurations.

Table 2 Existing techniques for scheduling container resources

As outlined above, researches on containers resource management are scarce and still cause considerable confusion. The configuration of containers on bar metal can be more energy efficient in most cases and tend to replace virtual machines. However, for many security reasons or when it comes to testing something outside the container engine (exp. firewall configuration), it is crucial to combine both VMs with containers to gain strengths provided from both sand-boxing technologies. Therefore, further significant researches must be investigated in this direction.

Just as indicated in previous “Energy-Efficient Virtual Resources Management Techniques and Energy-Efficient Container Resources Management” sections, several new techniques were involved or invented to optimize quantities of resources deployed with respect to energy consumption. Some of them consolidate hosts utilization to switch off under-utilized resources, while others handle load balancing by setting high and low threshold to readjust load assignments through migration processes. Moreover, prediction of future resources demands is a crucial phase for efficient resource assignments. Hence, observing traffic trends in order to predict future resources demands or future request is necessary to determine the reassignment plans in a timely manner. This last category represents one of the growing challenges for cloud computing, since it provides a significant improvement in cloud system energy saving and performance.

Predictive Techniques

Since a better allocation of resources requires prior knowledge of future workloads and VMs execution times, this last section highlights some of the most popular predictive parametric (i.e., with a particular function form) and nonparametric (i.e., having no specific mapping function) techniques. The main process of predicting techniques is described globally in Fig. 8

Fig. 8

The main process of predicting techniques


Regression Techniques

Forecasting regression methods are usually used for predicting and inferring the causal relationship between dependent and independent variables. Linear regression is the simple regression model; it gives a clear and greater understanding of factors influencing the estimated future values.

To increase resource utilization and reduce resources cost while meeting the changing demand, an autoscaling algorithm based on load prediction was proposed in [64]. Here, linear regression predicts load next time according to load in recent period of time. However, to predict low resource utilization rate during extension, an improved Knuth–Morris–Pratt string matching was used to predict the next moment load based on load sequence trend instead of absolute value. It takes the first historical data and current data as an input and then completes fast automatic vertical and horizontal extension before changes in resource requirement occur. Comparing the proposed algorithm with both ARMA and ESM models, results show that the proposed scaling method provides the minimum error and total resources usage with the least cost. Still, understanding how the resource usage affects the power consumption is required. A new energy-aware model was introduced in [65] to attribute fairly PMs idle and active energy consumption to homogeneous and heterogeneous VMs concurrently running on the same PM. The VMs workload (CPU, memory, disk, and network) was firstly predicted using seasonal ARIMA model based on stationary historical time-series workload data. Then, the PM CPU utilization was also predicted since it is correlated with VMs workload. Next, applying linear regression model, the PM power consumption was estimated and accordingly the predicted VM power consumption. After conducting experiment for three types of VMs, the proposed framework shows good prediction accuracy under various error metrics.

However, since linear regression in some cases may predict values outside the range, logistic regression can be used to always produce values within the valid range. To this end, a host overload detection algorithm is proposed in [66]. It is based on MAD to determine the upper utilization threshold and logistic regression model to predict host overloading. Effectively the proposed HOPBLR algorithm outperforms overall performance metrics comparing to other benchmark algorithms, again, in order to reduce SLA violation and at the same time energy consumption, while adapting the changes of different types of workloads running on the cloud provider DC. This proposed algorithm [67] uses adapted linear predicted value of host utilization multiplied by a safety parameter as an input to the logistic regression model, to predict the probability of host being overloaded. The proposed model performs better in power consumption reduction, SLA violation as well as the number of VM migrations comparing with the RS, MU, MMT, and MC.

Furthermore, since heterogeneity and fluctuations are the main characteristics of cloud workload, this author [68] forecasts the number of task arrivals of request, during each target period in several pattern periods to achieve higher prediction accuracy. First, the Poisson distribution model was decided based on the histogram analysis of data. Then, their parameters were obtained in every observed period using the maximum likelihood estimation (MLE) that finds the maximum likelihood between original model and another one identically distributed to choose the best parameter position. After enough dataset is accumulated for prediction, the nonparametric statistical local linear regression (LLR) was used to predict parameters of future requests. It uses first the least square weight function to obtain local observed parameter data that should be estimated around the given data. Result shows that the proposed predictor algorithm performs better than the fractal modeling and autoregressive predictor. Again, because of dynamic changes of application workloads, an adaptive placement strategy based on robust local weight regression was presented in [69]. It adjusts the value of static upper utilization threshold based on a statistical analysis of historical data collected during VMs lifetime. This method adds few degrees of smoothing to increase the number of observation points so as to better identify the adjusted value, but this is at the expense of a small increase of energy consumption.

Another prediction framework called CloudInsight for irregular workloads was created in [70]. This framework combines in total eight workload predictors in one prediction pool, which are linear regression also applied as a local regression because only limited amount of past workload was used, time-series predictors: WMA, BDES, AR, ARMA, ARIMA, then linear and Gaussian classifier support vector machines to predict workload with clear trend factors or with nonlinear characteristics (Gaussian). It was demonstrated in this later study that using single models such as ARIMA, SVM, FFT (Fast Fourier Transform), or RSLR (Robust Stepwise Linear Regression) perform differently under various kinds of workloads (cluster workload, web workload and HPC workload). However, the proposed multiple predictors’ framework is more accurate and constant since it keeps adjusting weights for each predictor and creates new ensemble model periodically to fit the changes in the workload. Thus, regression model can be combined with all other kinds of prediction techniques to give more accurate result depending on problem instances. (More ensemble learning will be discussed in the last subsections.) Thus, we can conclude that even if inferring future relationships between factors can help to easily predict future values, predicting future fluctuations in a very short term or in a real time requires sometimes more appropriate models referring to ordered data series. Next section describes the most common time-series techniques used to predict future real-time cloud resource utilization.

Time-Series Techniques

Underlining systematic variations in the specific time-series data is sometimes primordial to describe several characteristics such as possible trends, seasonal fluctuations, and relation between sequential observations in order to forecast future values.

As shown in “Energy-Efficient Container Resources Management” section, Docker containers can handle resource more efficiently than VMs, since using Docker means: the possibility to package a web application and all its components into the same Docker image, and then deploy it anywhere. This proposed system [71] adjusts the number of Docker application containers by predicting the number of user requests in the next time period. Web request number was defined as a time-series historical data, so that the next time period of data can be predicted using the autoregressive moving average (ARMA) model. This model usually adopts previous top key times period that highly correlates with the predicted value, by applying a certain weight to each time period. Next, by dividing the predicted request number by the number of requests that a single container can support, the estimated number of containers on demand can be obtained. Furthermore, [72] explored the performance of the adaptive ARMA model in order to predict in advance CPU consumptions during next hour using sliding window. Here, parameter orders were identified using the most frequently used error criterions: Akaike’s information criterion (AIC) and the Bayesian information criteria (BIC). The one providing the smallest value was taken. Then, applying recursive prediction error method, parameters were estimated recursively according to previous one. Moreover, nonlinear cost function (error) was minimized using the well-known Gauss–Newton recursive algorithm. Results demonstrate this model constitutes a reliable prediction when parameters are well determined (not biased and almost Gaussian distributed).

ARIMA is another improved model established from the autoregressive (AR) model and the (ARMA) model, with the only difference that it adds a differentiation term to transform a non-stationary series into a stationary one. As an improvement to the previous research [73], placing anti-correlated VMs separately together on a single host requires fewer resources and saves 10% of capacity [74]. Here, a CPU and bandwidth resource prediction model infer first the probability of host overload and select after migrating VMs using the popular local regression minimum migration time (LR-MMT) algorithm. The prediction models are then used to generate both CPU forecasts for the migrating VMs during each time step of the scheduled migration time and the estimated network load applying two forecasting model (nonlinear ANN and linear ARIMA). Results conclude that the nonlinear models (ANNs) perform best overall and generates the smallest error values across the four metrics. However, when a time series deploys potential seasonality, SARIMA remains to be an advantageous model. The multiplied SARIMA model takes six parameters in total, since for each ARIMA parameters, it grants a seasonality parameter. To forecast 1 week of hourly server workload time series [75], SARIMA model was introduced by first obtaining seasonal period using spectrum analysis and then determining differencing parameters using unit root test or augmented Dickey–Fuller. Other parameters were determined next using autocorrelation function (ACF) and partial autocorrelation function (PACF). Results deduced that SARIMA model generates good prediction of the week for both two server workload traces. Except that, there is no need to use it for daily forecasting because of difference between days of the week. Toward this end, many machine learning prediction modules will be highlighted in the next section.

Supervised and Unsupervised Machine Learning Techniques

Supervised techniques are the simplest one since they require a prior knowledge of outputs. These techniques are generally trained and evaluated by a dataset composed of inputs with their corresponding output values. Below are state-of-the-art studies using some of supervised learning techniques:

Decision tree is one of the supervised classification techniques, and it was used in [76] for energy-aware allocation of virtual machines. In this case, classification models have been learnt by separating the training set in 5 and 10 bins representing different ranges of CPU and RAM usages. Such bins corresponded to several interval labels used in J48 (C4.5) algorithm instances of Weka toolkit to have for each VM a decision tree classifier for CPU and another one for RAM. After evaluating the average error rate for both cases (5 bins and 10 bins), prediction errors were quite low for CPU usage (8.1% and 11.7%), while show higher values for RAM (11.7% and 25.7%). Another classification was implemented using SVM-based classification [77] to classify user request into two classes (delay sensitive, delay tolerant). This method primarily finds a hyperplane that maximizes margin between classifier hyperplane and the other data points, meanwhile preventing data point to fall in this margin. For nonlinearly separable data, a nonlinear Gaussian radial basis function (RBF) kernel was used to easily construct a decision boundary considering two additionally modeling parameters: the trade-off between classification error and margin distance, and then the differencing objective parameters to achieve best cross-validation accuracy. Once the classifier is trained, it is called to affect a certain class to user workload. However, this proposed model does not address workload prediction. Another paper [78] combines the wavelet transform with the support vector machine (WSVM) to come with the advantage of analyzing the cycle and frequency of the input signal by replacing the kernel function of SVM by wavelet function. Experimental results indicate that the proposed WSVM cloud computing load forecasting outperforms the standard SVM and also ARIMA model.

In order to achieve a green cloud without needless VMs migrations and SLA violations, predicting future host resource utilization can be achieved using a k-nearest neighbor regression-based model [79]. This model takes the local average of the k-samples nearest training data to the new sample. Each sample of dataset of historical CPU usage consists of one input variable representing the used CPU capacity at the current time and one output variable indicating the CPU utilization at the next time instance. To evaluate the prediction model accuracy, two error metrics (RMSE and MAPE) were calculated and represented (0.24 and 0.26, respectively) which means that the proposed model provides an accurate prediction of utilization.

To incorporate dynamic scaling up and scaling down of resources without violating the service-level agreements (SLA), this author [80] shows another predictive supervised machine learning approaches which is the Bayesian network. The proposed BN model was obtained using the NPC algorithm; it contains five variables nodes, namely (workload demand, CPU usage, memory usage, availability, and response time). Results successfully demonstrated that BN is capable of representing highly dynamic systems, by efficiently modeling unknown and complex relationships metrics monitored in the DC.

Nonetheless, the artificial neural network remains to be the most used supervised learning approach. An end-to-end neural network-based adaptive selection of VM consolidation algorithms was suggested [81]. It adaptively chooses appropriate dynamic VMs consolidation algorithm according to cloud provider’s goal priority and environment parameters. Normalized environment parameters and priority were used in input layer as a raw data. After evaluating priority, the full dataset is then generated in the hidden layer to choose the best adaptive selection. For instance, the output layer comes out with appropriate method in the form of a combination of some dynamic VM consolidation approaches collected from some benchmark algorithms. To create training and testing dataset, DeepLearning4j library was used for running neural network in adaptive selector, and then Xavier weight initialization and ReLu activation function were applied with one hidden layer of 10 nodes. Another model [82] establishes multilayer perceptron (MLP) neural network that is known by his back-propagation training procedures by calculating error and adjusting parameters of prior layers. This model predicts workloads of a certain node, at a certain interval of time intervals for training based on the previous selected workload time series. Then it classifies load into high, moderate, or low categories based on the predicted resource usage (e.g., CPU utilization) for a specific time period with some predefined rules. Experimental results show that predicted values are very close to the actual data and MLP model exhibits lower RMSE than ARIMA and long short-term memory recurrent neural network. Nevertheless, this model should be explored on real-time cloud testbed to better conclude his accuracy. So to gain one and two-step ahead predictions for horizons of 5, 10, and 15 min, a multiplicative component model was used in [83]. It divides time series by a trend component determined by least square method and then by a cyclical component computed using the phase-average trend method. All these to better focus on the residual component as a training data which contain fluctuations that is hard to predict for a horizons of 5, 10, and 15 min for example. The network was trained using Levenberg–Marquardt algorithm, and better result was achieved using many input neurons networks and just few hidden neurons. However, to forecast the client’s requests (data center traffic) for a very short-term duration of 1 s and deal with resource allocation and power management issues, a forecasting accuracy as high as 99.81% has been attained in [84]. The proposed model adopts a Cartesian Genetic Programming Evolved Artificial Neural Network (CGPANN) that presents four parameters (node functions, connections, encoding weights, and node’s input) in a single genotype in order to train the network and obtain the desired genotype behavior. The proposed CGPANN model gives better results in terms of RMSE and MAPE comparing to Levenberg–Marquardt algorithm and many other state-of-the-art models. As shown that neural network is the most used supervised techniques, it can be presented under many other variants. They are widely used due to their weights and biases enabling more complex neural connections to be more precise. However, complexity becomes much higher when it comes to unsupervised techniques that have any predefined knowledge about output values.

Unsupervised learning is the second more complex ML techniques that produce previously unknown patterns in dataset without pre-existing labels. Beyond the most common techniques using in this learning type, we can cite clustering approach, deep neural network, and some blind signal separation techniques such as ACP.

Before moving to clustering approach, a new approach for correlated VMs workload prediction was presented in [85], based on deep learning designed with a deep belief network (DBN) composed of multiple-layered restricted Boltzmann machines (RBMs) and a regression layer. Each hidden layer of a RBM was used as the visible layer of the following RBM using the contrastive divergence to obtain connection weights and minimize reconstruction error. The top layer of the proposed model consists of a sigmoid regression layer to fine-tune the whole deep learning model in a supervised way. Afterward, the network has been trained using the fast unsupervised greedy layerwise pretraining method to come up with the vanishing gradient problem. With little prior knowledge, the proposed DBN could learn features efficiently for the VM workload prediction in an unsupervised fashion.

As for clustering techniques, the K-means method is still one of the most widely used clustering algorithms. It contains three parameters, namely the number K of data subset, initial clustering center selection, similarity measurement. It is considered as a non-deterministic polynomial time hard problem (NP-hard) that consists first on selecting K-centroids showing improvement in accuracy and then assigns each point to its closest centroid. A model of workload characteristic based on K-means clustering analysis using Google workload trace dataset was proposed in [86]. This research adopts firstly the method of increasing K value with Euclidean distance criterion. Then, it selects the initial centroid by extracting K clusters from hierarchical clustering ensemble for each sample and using the cluster centers as the initial centroid. In this study, VM samples were divided into two clusters regular and irregular categories, and then each cluster was divided into other clusters according to workloads level. It was demonstrated that the model gives the more optimal resources for the most urgent resource requirements needed by using the clustering analysis of workload characteristics. Another paper [87] combines k-means clustering techniques and extreme learning machines (ELMs) to find the optimal weights for the predictor in a single step. Three categories of VM clusters were created according to workload characteristics using K-means algorithm during an observation window. The other part of the proposed data clustering is the trace decomposer for mapping each request to one cluster. Then once clusters obtained, one ELM neural network was applied to each category or cluster to predict the number of VM requests during the future prediction window. Nevertheless, workload in cloud DC is not only induced by load characteristics but also by user behaviors. Toward this end, the following paper [88] incorporates user behaviors’ to improve prediction results considering three phases: data clustering on different data sizes cluster labeled with ranges of workload percentages, data filtering by discovering workload patterns relationships, and calculating Pearson’s correlation between user clusters. The last phase predicts the incoming workload of VMs for a future time interval using a multiway (tensor) technique.

K-means clustering algorithm is a greedy algorithm that could not obtain global optimum in the polynomial time but only local optimum. It has been shown that the K-medoids technique converges toward the most obvious classification comparing to K-means. Predicting workloads of new tasks using knowledge about workloads of a pool of tasks is a new concept that was developed using a clustering K-medoids learning-based approach [89]. This algorithm clusters workload patterns of tasks in the dataset using the dynamic time warp (DTW) distance. Then, a multilayer perceptron neural network was used to learn characteristics of workloads of each cluster. Results confirmed that learning based on clustered data could effectively improve the accuracy in workload time-series prediction. All these clustering methods in unsupervised learning generally present one big challenge, which is the incapacities to evaluate the usefulness of what has been learned, since there is no previous knowledge about correct outputs.

Before moving to the last subsection, it is crucial to present some predictive green resource management techniques using another area of ML techniques that are neither supervised nor unsupervised. It is about reinforcement learning technique; it is the third ML paradigm that concerns software agents and their abilities to take actions in such a way to rapidly improve cumulative reward. A reinforcement learning-based dynamic consolidation method was presented in [90], in which the agent learns how to perform an action by interactions with dynamic observed state through Q-learning. The aim is to specify host state and selects the power mode of hosts that is provided the minimum Q-value (energy consumption and SLA). Then, it calculates the action penalty as the reinforcement signal after changing all power modes. The proposed RL-DC does not require any prior information about workload and dynamically achieve online energy and performance management. Another paper [91] combines Q-learning with novel two-phase (learning aggregation) distributed algorithm. Here, each PM computes Q-values and converges to their unified value through a gossip-based protocol. Using both techniques, each PM cooperates with its follow PMs to improve its status to a new state incrementally, by migrating VMs. Studies using RL in this field usually refer to states as calibrated PMs load state, rewards, but unlike previous study, this one defines actions as moving out or migrating any specific VM and also adds a transition probabilities (P). Again, on the basic of reinforcement learning mechanism and fuzzy logic for green solutions [92], each host state in this study has four dimensions: the current energy consumption of DC, candidate host, utilization of candidate host in MIPS, and then CPU demand of VM in MIPS. Possible actions were either to allocate or not allocate. Fuzzy logic rules were used for the mapping of energy parameters into reward value. Then, reward function was defined by the Markov decision process. There exist many other RL instances; however, studies in this field are scarce. To sum up, RL can be located in the middle of supervised and unsupervised learning, since unlike unsupervised learning RL can provide some partial information to the model but not a correct answer or action as supervised learning outputs. Here the challenge is how to find interesting patterns and how to decide rapidly choices based on what it was discovered. There is yet another hybrid field of ML techniques that have mainly been used recently because of their high precision adaptability. It is about ensemble learning methods that are most suited to various prediction instances.

Ensemble Learning Methods

Ensemble learning is another extensively used category in the prediction field. It is the process that combines many classifiers to achieve better prediction accuracy with high confidence estimation. Ensemble learning aims to overcome: data size issues, features heterogeneity in dataset, or even the complexity of single classifier’s decision boundaries. Ensemble methods can be generated by one of the most common following techniques: bagging, boosting, voting, and stacking. However, in this last subsection, we will discuss some of the widely used and founded ensemble prediction techniques in a green cloud computing environment.

Since each cloud provider (CP) has its own implementation idiosyncrasies, this author [93] provides a new very generic elasticity framework called VADARA that transparently connects and abstracts several CPs API behaviors. Then, to forecast future workload demand using these various elasticity strategies, it introduces a padding of several individual forecasting methods combined in a single forecasting technique. The proposed combination uses a weighted k-nearest neighbors algorithm, which outputs a weighted average of the k methods with the lowest error value for recent forecasts. Another framework [94] was designed to predict application execution times with a given resource (VM number) in a Hadoop virtual environment using three-phase prediction process. The first is an application’s behavior analysis phase, the second is a creation of training dataset phase, and the last phase trains various ML algorithms using an ensemble learning bagging combined with M5P (regression tree) and LR (linear regression) to find the best classifiers. Experimental results concluded that linear regression model shows less accuracy than M5P model, because the training dataset attributes used in this study have nonlinear effects depending on the system behavior, resource consumption, and workload. Again, the bagging approach has been used in [95] through an improved version, to reduce the prediction error for learning the selected algorithms set with default parameters: (SVM, random forest, K-nearest neighbor, and neural network). In this improved version, instead of just randomly taking samples from dataset and using each one to build a separate model, it applies re-weighting on the training set to prevent under-performing the ensemble members. Finally, to select the best fit n models from improved and unstable models, the hill climbing approach was taken into consideration.

To predict workload time series in cloud, this author [96] uses another performed ensemble learning approaches called stacking combining first the unsupervised autoencoder used to learn a “representation” of the input, and a recurrent neural network (RNN) layer trained in order to provide final predictions. In order to avoid the exploding and vanishing gradient issues in learning dependencies, three models were selected as candidates for this prediction model. The first one is identity recurrent neural network (IRNN), the second is long short-term memory (LSTM), and the last is gated recurrent unit (GRU). Since input data were already normalized, a tenfold cross-validation was used to partition data into equal size training subsamples. Experimental results show that GRU is more computational efficient and faster to train because of his less complicated structure. Another study [97] proposes a linear stacking resources prediction (LSRP) based on stacking framework that trains in the first level some time-series-based models (last predicting, AR, ARMA, and ARIMA). Then, it constructs a new training set using linear model by combining the outputs of the first stage and real workloads and updates weights using constrained convex optimization methods. In different workload types, the proposed model showed more robustness and accuracy. One more ensemble-based workload prediction mechanism is proposed [98] based on stack generalization (stacking) of K-nearest neighbor and decision tree. However, instead of using an optimization approach for constructing and selecting the best set of methods, an over produce and choose (OPC) approach was adopted to produce first a large set of candidate classifiers and then select the subset of classifiers that can be combined to achieve optimal accuracy based on the degree of error diversity rather than a specific combination rules. However, to come up with a prediction method that can be trained in real time and that can capture the latest trends and workload characteristics, an online ensemble workload prediction mechanism based again on time series and machine learning techniques was proposed [99]. Here each method was fitted to a history window for the last tactual data points. Next, an exponential smoothing model was used to fit errors and calculate method’s contribution.

Since determining the best time-series model for predicting cloud resource usage not only depends on time but also on cloud resource usage trend, this study [100] presents an ensemble cloud resource usage prediction framework including multiple prediction models that each has a learning automata (LA) that aim at tuning weights. The ensemble prediction is provided by combining the multiplication of prediction values of individual models to their weights. Finally, an intelligent regressive ensemble approach (REAP) [101] has been proposed to achieve high performance. It integrates first feature selection and then a resource usage prediction through the train of eight different ML techniques. Afterward, an ensemble learning algorithm was developed to obtain the best set or combination of models giving the highest accuracy, while task dependencies were modeled using polynomial regression. We can deduct from this section that combining decisions from multiple models improve the overall performance regardless of the way of combination. Here the challenge is how to find the appropriate combination strategy to improve the stability and accuracy of the model.

Analysis and Conclusion

In this paper, we survey cloud resources provisioning algorithms and techniques with as the first objective the energy saving, meanwhile maintaining performance level. Different approaches and policies aforementioned are adopted to address either one objective or a combination of different kinds of specifications. The majority of these studies consider just computing resources, while just a small part of them deal with various combinations of cloud computing and networking resources. Again, from technical and metaheuristic point of view, for solving NP-hard reactive or proactive cloud resources provisioning problem, it is difficult to deduce what each technique performs best in. None of them can solve all problems, since each of them works best in an environment of specific parameters and specifications. Yet, it was concluded several times that a smart combination of techniques could alleviate over-fitting and local convergence issues. Moreover, having regard to cloud challenges that are still actually under investigations, we have noticed several open challenges, which can be briefly summarized as follows:

  • First as reported previously, a complete optimization of resources usage through an energy-efficient virtual resources allocation management is not only restricted to computing resources, but also is expected to extend beyond local and backbone network resources.

  • Moreover, since energy efficiency of such resources depends mainly on traffic invading them, it has been found that there is no well-established taxonomy defining the complex nature of dynamic workloads crossing the cloud. For example, some studies classify the traffic types based on their intensity in need of computation or their intensity in need of data transfer… Others divide workloads’ natures by either their temporal or spatial events. Another palpable workloads’ natures’ classification consists simply on dividing traffic per services type. Confronted with such a multitude of cloud traffic taxonomies, the question arising is how to reflect on energy-efficient policies for virtual resources selection and allocation, satisfying any of those types?

  • On the other hand, the joint deployment of IaaS and PaaS is still evolving over time, further pushing the adoption of containerization technology, which has proven such efficient use of resources. So how can we find a good configuration of this new technology in terms of energy efficiency of resources? In addition, what are the most appropriate policies for containers resources’ selection and allocation?

  • Once again, one of the open challenges that still affecting cloud’s energy efficiency and elasticity is the inter- and intra-migration of virtual resources, where only very little research has succeeded in removing them entirely.

  • This last challenge opens on the need to develop in-depth proactive models as stated in the last section. The question remains how to perfectly predict in real-time different types of future customer requests, taking into account requests interferences? Yet much still can be done in this broad area.

  • As each green proposed model or policies should not be performed at the expense of the saved energy consumption, the last encountered challenge neglected by the majority of researches, is how to guarantee a green whole implementation?


  1. 1.

    Google Cloud: https://cloud.google.com/kubernetes-engine/docs/.

  2. 2.




Artificial bees colony


Ant colony optimization


Principal component analysis


Ant colony system


Artificial neural network


Application program interface


Autoregressive model


Autoregressive integrated moving average


Autoregressive moving average


Best fit


Best fit decreasing


Bayesian network


Bin packing




Combined best fit decreasing


Constant fixed selection


Cloud providers


Data centers


Dynamic management algorithm


Dynamic voltage frequency scaling


Dynamic virtual machine consolidation


Energy and carbon efficient


Frequency core energy


First fit decreasing


Fuzzy logic system


Genetic algorithms


Greenhouse gas


Global topology resource


High-performance computing


Interquartile range


Internet service provider


K-nearest neighbor


Load balancing


Levy flight PSO


Local regression


Link state adaptation


Regression tree


Median absolute deviation


Mean absolute error


Mean absolute percentage error


Maximum correlation


Modified first decreasing


Million instructions per second


Machine learning


Multilayer perceptron


Minimum migration


Minimum migration time


On arrival migration


Mean squared error


Natural inspired algorithms


Necessary path condition


Non-deterministic polynomial time hardness


Optical data center network


Orthogonal frequency division multiplexing


Partitioning around medoids


Physical machines


Point of interest


Power consumption per application


Particle swarm optimization


PSO levy bin packing


Power usage effectiveness


Quality of service


Radial basis function


Restricted Boltzmann machine


Reinforcement learning


Robust local regression


Root-mean-square error


Random selection/random choice


Roulette wheel selection


Seasonal autoregressive integrated moving average


Service-level agreement


Support vector machine


Virtual data centers


Virtual machines


Virtual network embedding


Virtual networks


Wave length division multiplexing


Workload and energy-efficient container


Wavelet support vector machine


  1. 1.

    Cisco Public. Cisco global cloud index: forecast and methodology. 2016. 2015–2020 from https://www.iotjournaal.nl/wp-content/uploads/2017/02/white-paper-c11-738085.pdf. Accessed 20 May 2019.

  2. 2.

    Whitney J, Delforge D. Data center efficiency assessment–scaling up energy efficiency across the data center industry: evaluating key drivers and barriers, Rep. IP NRDC and Anthesis, pp. 14–08, 2014.

  3. 3.

    Kliazovich D, Bouvry P, Khan SUJ. GreenCloud: a packet-level simulator of energy-aware cloud computing data centers. J Supercomput. 2012;62:1263–83. https://doi.org/10.1007/s11227-010-0504-1.

    Article  Google Scholar 

  4. 4.

    Raïs I, Orgerie AC, Quinson M, Lefèvre L. Quantifying the impact of shutdown techniques for energy-efficient data centers. Concurr Comput Pract Exp. 2018. https://doi.org/10.1002/cpe.4471.

    Article  Google Scholar 

  5. 5.

    Carrega A, Singh S, Bruschi R, Bolla R. Traffic merging for energy-efficient datacenter networks. In: Proceedings of the international symposium of performance evaluation of computing and telecommunication systems. IEEE; 2012. pp. 1–5.

  6. 6.

    Li Y, Orgerie AC, Menaud JM. Balancing the use of batteries and opportunistic scheduling policies for maximizing renewable energy consumption in a Cloud data center. In: 25th Euromicro international conference on parallel, distributed, and network-based processing. St Petersburg, Russia; 2017.

  7. 7.

    Khosravi A, Garg SK, Buyya R. Energy and carbon-efficient placement of virtual machines in distributed cloud data centers. In: Wolf F, Mohr B, Mey D, editors. Euro-Par 2013 parallel processing. Euro-Par 2013. Lecture notes in computer science. Berlin: Springer; 2013. vol. 8097, pp. 115–134. https://doi.org/10.1016/b978-0-12-801476-9.00006-9.

  8. 8.

    Baker T, Aldawsari B, Asim M, Tawfik H, Maamar Z, Buyya R. Cloud-SEnergy: a bin-packing based multi-cloud service broker for energy efficient composition and execution of data-intensive applications. Sustain Comput Inform Syst. 2018. https://doi.org/10.1016/j.suscom.2018.05.011.

    Article  Google Scholar 

  9. 9.

    Lei S, Furlong J, Wang R. Empirical evaluation of vector bin packing algorithms for energy efficient data centers. IEEE Symp Comput Commun. 2013. https://doi.org/10.1109/iscc.2013.6754915.

    Article  Google Scholar 

  10. 10.

    Ngenzi A, Rangasamy RS, Suchithra R. Improving server consolidation using physical consolidation concept. Int J Adv Appl Sci. 2016.

  11. 11.

    Halácsy G, Zoltán ÁM. Optimal energy-efficient placement of virtual machines with divisible sizes. Inf Process Lett. 2018;138:51–6. https://doi.org/10.1016/j.ipl.2018.06.003.

    MathSciNet  Article  MATH  Google Scholar 

  12. 12.

    Somnath M, Marco P. Power efficient server consolidation for Cloud data center. Future Gener Comput Syst. 2017;70:4–16. https://doi.org/10.1016/j.future.2016.12.022.

    Article  Google Scholar 

  13. 13.

    Wang H, Tianfield H. Energy-aware dynamic virtual machine consolidation for cloud datacenters. IEEE Access. 2018;6:15259–73. https://doi.org/10.1109/access.2018.2813541.

    Article  Google Scholar 

  14. 14.

    Fares A, Yu-Chu T, Maolin T, Wei-Zhe Z, Chen P, Minrui F. An ant colony system for energy-efficient dynamic virtual machine placement in data centers. Expert Syst Appl. 2019;120:228–38. https://doi.org/10.1016/j.eswa.2018.11.029.

    Article  Google Scholar 

  15. 15.

    Fatima A, Javaid N, Sultana T, Aalsalem M, Shabbir S, Durre-Adan. An efficient virtual machine placement via bin packing in cloud data centers. In: Barolli L, Takizawa M, Xhafa F, Enokido T, editors. Advanced information and applications AINA 2019. Advances in intelligent systems and computing, vol. 926. Cham: Springer; 2019. https://doi.org/10.1007/978-3-030-15032-7_82.

    Google Scholar 

  16. 16.

    Ayaz AK, Muhammad Z, Rahim K. Energy-aware dynamic resource management in elastic cloud datacenters. Simul Model Pract Theory. 2019;92:82–99. https://doi.org/10.1016/j.simpat.2018.12.001.

    Article  Google Scholar 

  17. 17.

    Zhu W, Zhuang Y, Zhang L. A three-dimensional virtual resource scheduling method for energy saving in cloud computing. Future Gener Comput Syst. 2017;69:66–74. https://doi.org/10.1016/j.future.2016.10.034.

    Article  Google Scholar 

  18. 18.

    Portaluri G, Giordano S, Kliazovich D, Dorronsoro B. A power efficient genetic algorithm for resource allocation in cloud computing data centers. In: 2014 IEEE 3rd international conference on cloud networking (CloudNet). Luxembourg; 2014. pp. 58–63. https://doi.org/10.1109/cloudnet.2014.6968969.

  19. 19.

    Portaluri G, Giordano S. Power efficient resource allocation in cloud computing data centers using multi-objective genetic algorithms and simulated annealing. In: IEEE 4th international conference on cloud networking (CloudNet). Niagara Falls; 2015. pp. 319–321. https://doi.org/10.1109/cloudnet.2015.7335329.

  20. 20.

    Geronimo GA, Brundo Uriarte R, Westphall C. Order@Cloud: an agnostic meta-heuristic for VM provisioning, adaptation, and organization. J Netw Manag. 2019. https://doi.org/10.1002/nem.2085.

    Article  Google Scholar 

  21. 21.

    Zhang W, Tan S, Lu G, Liu X. Towards a genetic algorithm based approach for task migrations. In: 2014 international conference on identification, information and knowledge in the internet of things. Beijing; 2014. pp. 182–187. https://doi.org/10.1109/iiki.2014.45.

  22. 22.

    Sharma NK, Reddy GRM. Novel energy efficient virtual machine allocation at data center using Genetic algorithm. In: 3rd International conference on signal processing, communication and networking (ICSCN). Chennai; 2015. pp. 1–6. https://doi.org/10.1109/icscn.2015.7219897.

  23. 23.

    Sharma NK, Reddy GRM. A novel energy efficient resource allocation using hybrid approach of genetic DVFS with bin packing. In: Fifth international conference on communication systems and network technologies. Gwalior; 2015. pp. 111–115. https://doi.org/10.1109/csnt.2015.156.

  24. 24.

    Zha J, Wang C, Chen Q, Lu X. Lai J Server consolidation based on hybrid genetic algorithm. In: Ninth international conference on frontier of computer science and technology. Dalian; 2015. pp. 370–375. https://doi.org/10.1109/fcst.2015.43.

  25. 25.

    Sharma NK, Guddeti RMR. Multi-objective resources allocation using improved genetic algorithm at cloud data center. In: IEEE international conference on cloud computing in emerging markets (CCEM). Bangalore; 2016. pp. 73–77. https://doi.org/10.1109/ccem.2016.021.

  26. 26.

    Bakalla M, Al-Jami H, Kurdi H, Alsalamah S. A QoS-aware and energy-efficient genetic resource allocation algorithm for cloud data centers. In: 9th International congress on ultra modern telecommunications and control systems and workshops (ICUMT). Munich; 2017. pp. 244–249. https://doi.org/10.1109/icumt.2017.8255166.

  27. 27.

    Wen Y, Li L, Jin S, Lin C, Liu Z. Energy-efficient virtual resource dynamic integration method in cloud computing. IEEE Access. 2017;5:12214–23. https://doi.org/10.1109/ACCESS.2017.2721548.

    Article  Google Scholar 

  28. 28.

    Xiaoqing Z. Efficient and balanced virtualized resource allocation based on genetic algorithm in cloud. In: The 10th international symposium on computational intelligence and design. Hangzhou; 2017. pp. 374–377. https://doi.org/10.1109/iscid.2017.187.

  29. 29.

    Kaustabha R, Sunanda B, Nandini M. A load balancing approach to resource provisioning in cloud infrastructure with a grouping genetic algorithm. In: International conference on current trends towards converging technologies (ICCTCT). Coimbatore: IEEE; 2018. pp. 1–6. https://doi.org/10.1109/icctct.2018.8550885.

  30. 30.

    Sharma NK, Reddy GRM. Multi-objective energy efficient virtual machines allocation at the cloud data center. IEEE Trans Serv Comput. 2019;12(1):158–71. https://doi.org/10.1109/TSC.2016.2596289.

    Article  Google Scholar 

  31. 31.

    Hu W, Li K, Xu J, Bao Q. Cloud-Computing-based resource allocation research on the perspective of improved ant colony algorithm. In: International conference on computer science and mechanical automation (CSMA). Hangzhou, China: IEEE; 2015. pp. 76–80. https://doi.org/10.1109/csma.2015.22.

  32. 32.

    Ragmani A, Omri AE, Abghour N, Moussaid K, Rida M. A performed load balancing algorithm for public Cloud computing using ant colony optimization. In: 2nd International conference on cloud computing technologies and applications (CloudTech). Morocco: Marrakech; 2016. pp. 221–228. https://doi.org/10.1109/cloudtech.2016.7847703.

  33. 33.

    Agrawal K, Tripathi P. Power aware artificial bee colony virtual machine allocation for private cloud systems. In: International conference on computational intelligence and communication networks (CICN). Jabalpur, India: IEEE; 2015. pp. 947–950. https://doi.org/10.1109/cicn.2015.186.

  34. 34.

    Ragmani A, Omri A.E, Abghour N, Moussaid K and Rida M. An intelligent scheduling algorithm for energy efficiency in cloud environment based on artificial bee colony, In: 3rd International conference of cloud computing technologies and applications (CloudTech). Rabat, Morocco, IEEE; 2017. pp. 1–8. https://doi.org/10.1109/cloudtech.2017.8284708.

  35. 35.

    Benali A, El Asri B and Kriouile H. A pareto-based Artificial Bee Colony and product line for optimizing scheduling of VM on cloud computing. In: International conference on cloud technologies and applications (CloudTech). Marrakech, Morocco; 2015. pp. 1–7. https://doi.org/10.1109/cloudtech.2015.7336980.

  36. 36.

    Arroba P, Risco-Martín JL, Zapater M, Moya JM, Ayala JL, Olcoz K. Server power modeling for run-time energy optimization of cloud computing facilities. Energy Procedia. 2014;62:401–10. https://doi.org/10.1016/j.egypro.2014.12.402.

    Article  Google Scholar 

  37. 37.

    Aruna P, Vasantha S. A particle swarm optimization algorithm for power-aware virtual machine allocation. In: 6th International conference on computing, communication and networking technologies (ICCCNT). Denton, TX, USA; 2015. pp. 1–6. https://doi.org/10.1109/icccnt.2015.7395196.

  38. 38.

    Ramezani F, Naderpour M, Lu J. A multi-objective optimization model for virtual machine mapping in cloud data centres. In: IEEE international conference on fuzzy systems (FUZZ-IEEE), Vancouver, BC, Canada; 2016. pp. 1259–1265. https://doi.org/10.1109/fuzz-ieee.2016.7737833.

  39. 39.

    Tripathi A, Pathak I, Vidyarthi DP. Energy efficient VM placement for effective resource utilization using modified binary PSO. Comput J. 2017;61(6):832–46. https://doi.org/10.1093/comjnl/bxx096.

    Article  Google Scholar 

  40. 40.

    Elhady GF, Tawfeek MA. A comparative study into swarm intelligence algorithms for dynamic tasks scheduling in cloud computing. In: IEEE seventh international conference on intelligent computing and information systems (ICICIS), Cairo, Egypt: 2015. pp. 362–369. https://doi.org/10.1109/intelcis.2015.7397246.

  41. 41.

    Kaur N, Chhabra A. Analytical review of three latest nature inspired algorithms for scheduling in clouds. In: International conference on electrical, electronics, and optimization techniques (ICEEOT). Chennai, India; 2016. pp. 3296–3300. https://doi.org/10.1109/iceeot.2016.7755315.

  42. 42.

    Abrol P, Gupta S, Kaur K. Analysis of Resource Management and placement policies using a new nature inspired meta heuristic SSCWA avoiding premature convergence in Cloud. In: International conference on computational techniques in information and communication technologies (ICCTICT), New Delhi, India; 2016. pp. 653–658. https://doi.org/10.1109/icctict.2016.7514659.

  43. 43.

    Gupta P, Tewari P. Monkey search algorithm for task scheduling in cloud IaaS, In: Fourth international conference on image information processing (ICIIP). Shimla, India; 2017. pp. 1–6. https://doi.org/10.1109/iciip.2017.8313789.

  44. 44.

    Gelenbe E, Lent R, Douratsos M. Choosing a local or remote cloud. In: The second symposium on network cloud computing and applications. London; 2012. pp. 25–30. https://doi.org/10.1109/ncca.2012.16.

  45. 45.

    Lent R. Simulating the power consumption of computer networks. In: The 15th IEEE international workshop on computer aided modeling, analysis and design of communication links and networks (CAMAD). Miami, FL; 2010. pp. 96–100. https://doi.org/10.1109/camad.2010.5686955.

  46. 46.

    Su S, Zhang Z, Liu AX, Cheng X, Wang Y, Zhao X. Energy-aware virtual network embedding. IEEE/ACM Trans Netw. 2014;22(5):1607–20. https://doi.org/10.1109/TNET.2013.2286156.

    Article  Google Scholar 

  47. 47.

    Guan X, Choi BY, Song S. Energy efficient virtual network embedding for green data centers using data center topology and future migration. Comput Commun. 2015;69:50–9. https://doi.org/10.1016/j.comcom.2015.05.003.

    Article  Google Scholar 

  48. 48.

    Manh Nam T, Huu Thanh N, Trung Hieu H, Tien Manh N, Van Huynh N, Duong Tuan H. Joint network embedding and server consolidation for energy–efficient dynamic data center virtualization. Comput Netw. 2017;125:76–89. https://doi.org/10.1016/j.comnet.2017.06.007.

    Article  Google Scholar 

  49. 49.

    Nonde L, El-Gorashi TEH, Elmirghani JMH. Energy efficient virtual network embedding for cloud networks. J Lightwave Technol. 2015;33(9):1828–49. https://doi.org/10.1109/JLT.2014.2380777.

    Article  Google Scholar 

  50. 50.

    Zhang P. Incorporating energy and load balance into virtual network embedding process. Comput Commun. 2018;129:80–8. https://doi.org/10.1016/j.comcom.2018.07.027.

    Article  Google Scholar 

  51. 51.

    Zong Y, et al. Location-aware energy efficient virtual network embedding in software-defined optical data center networks. IEEE/OSA J Opt Commun Netw. 2018;10(7):58–70. https://doi.org/10.1364/JOCN.10.000B58.

    Article  Google Scholar 

  52. 52.

    Ghazisaeedi E, Huang C. EnergyMap: energy-efficient embedding of MapReduce-based virtual networks and controlling incast queuing delay, In: 8th IEEE international conference on communication software and networks (ICCSN), Beijing, China; 2016. pp. 698–702. https://doi.org/10.1109/iccsn.2016.7586614.

  53. 53.

    Triki N, Kara N, El Barachi M, Hadjres S. A green energy-aware hybrid virtual network embedding approach. Comput Netw. 2015;91:712–37. https://doi.org/10.1016/j.comnet.2015.08.016.

    Article  Google Scholar 

  54. 54.

    Gong X, Ning Z, Guo L, Wei X, Song Q. Location-recommendation-aware virtual network embedding in energy-efficient optical-wireless hybrid networks supporting 5G models. IEEE Access. 2016;4:3065–75. https://doi.org/10.1109/ACCESS.2016.2580615.

    Article  Google Scholar 

  55. 55.

    Cisco Solution Overview. Containerized data centers: compelling economics and efficiency. https://www.cisco.com/c/dam/en_us/solutions/industries/docs/gov/Containerized_Data_Centers_Solution_Overview.pdf.

  56. 56.

    Zakarya M. Energy and performance aware resource management in heterogeneous cloud datacenters. A thesis submitted for the degree of Doctor of Philosophy in Computer Science from the University of Surrey; Guildford, Surrey GU2 7XH, United Kingdom; 2017.

  57. 57.

    Dong-Ki K, Gyu-Beom C, Seong-Hwan K, Il-Sun H, Chan-Hyun Y. Workload-aware resource management for energy efficient heterogeneous Docker containers. In: IEEE region 10 conference (TENCON). Singapore; 2016. pp. 2428–2431. https://doi.org/10.1109/tencon.2016.7848467.

  58. 58.

    Ching-Chi L, Jian-Jia C, Pangfeng L, Jan-Jan W. Energy-efficient core allocation and deployment for container-based virtualization. In: IEEE 24th international conference on parallel and distributed systems (ICPADS). Singapore; 2018. pp. 93–101. https://doi.org/10.1109/padsw.2018.8644537.

  59. 59.

    Ayaz A, Zakarya M, Khan R. H2—a hybrid heterogeneity aware resource orchestrator for cloud platforms. IEEE Syst J. 2019. https://doi.org/10.1109/jsyst.2019.2899913.

    Article  Google Scholar 

  60. 60.

    Piraghaj SF, Dastjerdi AV, Calheiros RN, Buyya R. A framework and algorithm for energy efficient container consolidation in cloud data centers. In: IEEE international conference on data science and data intensive systems. Sydney; 2015. pp. 368–375. https://doi.org/10.1109/dsdis.2015.67.

  61. 61.

    Tao S, Hui Ma, Gang CH. Energy-aware container consolidation based on PSO in cloud data centers. In: IEEE congress on evolutionary computation (CEC). Rio de Janeiro; 2018. pp. 1–8. https://doi.org/10.1109/cec.2018.8477708.

  62. 62.

    Piraghaj SF, Dastjerdi AV, Calheiros RN, Buyya R. Efficient virtual machine sizing for hosting containers as a service. In: IEEE world congress on services. New York, NY; 2015. pp. 31–38. https://doi.org/10.1109/SERVICES.2015.14.

  63. 63.

    Qasim A. Scaling web 2.0 applications using docker containers on vsphere 6.0. [Online]; 2015. http://blogs.vmware.com/performance/2015/04/scaling-web-2-0-applications-using-docker-containers-vsphere-6-0.html.

  64. 64.

    Li T, Wang J, Li W, Xu T, Qi Q. Load prediction-based automatic scaling cloud computing. In: The international conference on networking and network applications (NaNA). Hakodate, Japan; 2016. pp. 330–335. https://doi.org/10.1109/nana.2016.49.

  65. 65.

    Aldossary M, Djemame K, Alzamil I, Kostopoulos A, Dimakis A, Agiatzidou E. Energy-aware cost prediction and pricing of virtual machines in cloud computing environments. Future Gener Comput Syst. 2019;93:442–59. https://doi.org/10.1016/j.future.2018.10.027.

    Article  Google Scholar 

  66. 66.

    Jararweh Y, Bani Issa M, Daraghmeh M, Al-Ayyoub M, Alsmirat MA. Energy efficient dynamic resource management in cloud computing based on logistic regression model and median absolute deviation. Sustain Comput Inform Syst. 2018;19:262–74. https://doi.org/10.1016/j.suscom.2018.07.005.

    Article  Google Scholar 

  67. 67.

    Daraghmeh M, Bani Melhem S, Agarwal A, Goel N, Zaman M. Linear and logistic regression based monitoring for resource management in cloud networks. In: The IEEE 6th international conference on future internet of things and cloud (FiCloud). Barcelona, Spain; 2018. pp. 259–266. https://doi.org/10.1109/ficloud.2018.00045.

  68. 68.

    Yoon MS, Kamal AE, Zhu Z. Requests prediction in cloud with a cyclic window learning algorithm. In: The IEEE globecom workshops (GC Wkshps). Washington, DC USA; 2016. pp. 1–6. https://doi.org/10.1109/glocomw.2016.7849022.

  69. 69.

    Xiaoqing Z. Energy-aware virtual machine management optimization in clouds. In: 2017 3rd IEEE international conference on computer and communications (ICCC). Chengdu, China; 2017; pp. 2434–2438. https://doi.org/10.1109/compcomm.2017.8322972.

  70. 70.

    Kim IK, Wang W, Qi Y, Humphrey M. CloudInsight: utilizing a council of experts to predict future cloud application workloads. In: IEEE 11th international conference on cloud computing (CLOUD). San Francisco, CA USA; 2018. pp. 41–48. https://doi.org/10.1109/cloud.2018.00013.

  71. 71.

    Li Y, Xia Y. Auto-scaling web applications in hybrid cloud based on Docker. In: 5th International conference on computer science and network technology (ICCSNT). Changchun, China; 2016. pp. 75–79. https://doi.org/10.1109/iccsnt.2016.8070122.

  72. 72.

    Renault E, Boumerdassi S, Bouzefrane S, Gbaguid FAR, Milocco R, Ezin EC. Adaptive ARMA based prediction of CPU consumption of servers into datacenters. Cham, Springer International Publishing; 2019. pp. 277–288. https://doi.org/10.1007/978-3-030-03101-5_23.

  73. 73.

    Shaw R, Howley E, Barrett E. A predictive anti-correlated virtual machine placement algorithm for green cloud computing. In: IEEE/ACM 11th international conference on utility and cloud computing (UCC). Zurich, IEEE; 2018. pp. 267–276. https://doi.org/10.1109/ucc.2018.00035.

  74. 74.

    Shaw R, Howley E, Barrett E. An energy efficient anti-correlated virtual machine placement algorithm using resource usage predictions. Simul Model Pract Theory. 2019;93:322–42. https://doi.org/10.1016/j.simpat.2018.09.019.

    Article  Google Scholar 

  75. 75.

    Tran VG, Debusschere V, Bacha S. Hourly server workload forecasting up to 168 hours ahead using Seasonal ARIMA model. In: 2012 IEEE international conference on industrial technology. Athens; 2012. pp. 1127–1131. https://doi.org/10.1109/icit.2012.6210091.

  76. 76.

    Altomare A, Cesario E. Predictive models for energy-efficient clouds: an analysis on real-life and synthetic data. In: IEEE international conference on computer and information technology; ubiquitous computing and communications; dependable, autonomic and secure computing; pervasive intelligence and computing. Liverpool, UK; 2015. pp. 1538–1543. https://doi.org/10.1109/cit/iucc/dasc/picom.2015.231.

  77. 77.

    Gagangeet SA, Neeraj K. MEnSuS: an efficient scheme for energy management with sustainability of cloud data centers in edge–cloud environment. Future Gener Comput Syst. 2018;86:1279–300. https://doi.org/10.1016/j.future.2017.09.066.

    Article  Google Scholar 

  78. 78.

    Zhong W, Zhuang Y, Sun J, and Gu J. The cloud computing load forecasting algorithm based on wavelet support vector machine. In: Proceedings of the Australasian computer science week multiconference (ACSW ‘17). New York, ACM; 2017. p. 5. https://doi.org/10.1145/3014812.3014852.

  79. 79.

    Farahnakian F, Pahikkala T, Liljeberg P, Plosila J, Tenhunen H. Utilization prediction aware VM consolidation approach for green cloud computing, In: IEEE 8th international conference on cloud computing. New York, USA; 2015. pp. 381–388. https://doi.org/10.1109/cloud.2015.58.

  80. 80.

    Bashar A. Autonomic scaling of cloud computing resources using BN-based prediction models. In: IEEE 2nd international conference on cloud networking (CloudNet). San Francisco, CA; 2013. pp. 200–204. https://doi.org/10.1109/cloudnet.2013.6710578.

  81. 81.

    Nathanael Witanto J, Lim H, Atiquzzaman M. Adaptive selection of dynamic VM consolidation algorithm using neural network for cloud resource management. Future Gener Comput Syst. 2018;87:35–42. https://doi.org/10.1016/j.future.2018.04.075.

    Article  Google Scholar 

  82. 82.

    Ahammad T, Acharjee UK, Hasan MM. Energy-effective service-oriented cloud resource allocation model based on workload prediction. In: 21st International conference of computer and information technology (ICCIT), Dhaka, Bangladesh; 2018. pp. 1–6. https://doi.org/10.1109/iccitechn.2018.8631953.

  83. 83.

    Lenhardt J, Schiffmann W, Jannevers S. Prediction of future loads using neural networks for energy-efficient computing. In: Fourth international symposium on computing and networking (CANDAR), Hiroshima, Japan; 2016. pp. 579–585. https://doi.org/10.1109/candar.2016.0105.

  84. 84.

    Ali J, Zafari F, Khan GM, Mahmud SA. Future clients’ requests estimation for dynamic resource allocation in cloud data center using CGPANN. In: 12th International conference on machine learning and applications, Miami, FL, USA; 2013. pp. 331–334. https://doi.org/10.1109/icmla.2013.189.

  85. 85.

    Qiu F, Zhang B, Guo J. A deep learning approach for VM workload prediction in the cloud. In: 17th IEEE/ACIS international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD). Shanghai, China; 2016. pp. 319–324. https://doi.org/10.1109/snpd.2016.7515919.

  86. 86.

    Xia Q, Lan Y, Zhao L, Xiao L. Energy-saving analysis of Cloud workload based on K-means clustering. In: IEEE computers, communications and IT applications conference, Beijing, China; 2014. pp. 305–309. https://doi.org/10.1109/comcomap.2014.7017215.

  87. 87.

    Ismaeel S, Miri A. Using ELM techniques to predict data centre VM requests. In: 2015 IEEE 2nd international conference on cyber security and cloud computing. New York, NY, USA; 2015. pp. 80–86. https://doi.org/10.1109/cscloud.2015.82.

  88. 88.

    Raed K, Salam I, Miri A. Energy-efficient resource allocation for cloud data centers using a multi-way data analysis technique, human-computer interaction. Theory Des Dev Pract. 2016;9731:577–85. https://doi.org/10.1007/978-3-319-39510-4_53.

    Article  Google Scholar 

  89. 89.

    Yu Y, Jindal V, Yen I, Bastani F Integrating clustering and learning for improved workload prediction in the cloud. In: IEEE 9th international conference on cloud computing (CLOUD), San Francisco, CA, USA; 2016. pp. 876–879. https://doi.org/10.1109/cloud.2016.0127.

  90. 90.

    Farahnakian F, Liljeberg P, Plosila J. Energy-efficient virtual machines consolidation in cloud data centers using reinforcement learning. In: 22nd Euromicro international conference on parallel, distributed, and network-based processing. Torino, Italy; 2014. pp. 500–507. https://doi.org/10.1109/pdp.2014.109.

  91. 91.

    Khelghatdoust M, Gramoli V, Sun D. GLAP: distributed dynamic workload consolidation through gossip-based learning. In: IEEE international conference on cluster computing (CLUSTER). Taipei, Taiwan; 2016. pp. 80–89. https://doi.org/10.1109/cluster.2016.24.

  92. 92.

    Thein T, Myat My M, Parvin S, Gawanmeh A. Reinforcement learning based methodology for energy-efficient resource allocation in cloud data centers. J King Saud Univ Comput Inf Sci. 2018. https://doi.org/10.1016/j.jksuci.2018.11.005.

    Article  Google Scholar 

  93. 93.

    Loff J, Garcia J. Vadara: predictive elasticity for cloud applications. In: IEEE 6th international conference on cloud computing technology and science. Singapore; 2014. pp. 541–546. https://doi.org/10.1109/cloudcom.2014.161.

  94. 94.

    Al-Rawahi M, Edirisinghe EA, Jeyarajan T. Machine learning-based framework for resource management and modelling for video analytic in cloud-based hadoop environment. In: International IEEE conferences on ubiquitous intelligence & computing, advanced and trusted computing, scalable computing and communications, cloud and big data computing, internet of people, and smart world congress. Toulouse, France; 2016. pp. 801–807. https://doi.org/10.1109/uic-atc-scalcom-cbdcom-iop-smartworld.2016.0128.

  95. 95.

    Hirwa JS, Rugwiro U, Stammers M, Gu C. Cloud-based clusters: multivariate optimization techniques for resource performance prediction. In: 15th International symposium on parallel and distributed computing (ISPDC). Fuzhou, China; 2016. pp. 165–171. https://doi.org/10.1109/ispdc.2016.29.

  96. 96.

    Nguyen HM, Woo S, Im J, Jun J, Kim D. A workload prediction approach using models stacking based on recurrent neural network and autoencoder. In: IEEE 18th international conference on high performance computing and communications; IEEE 14th international conference on smart city; IEEE 2nd international conference on data science and systems (HPCC/SmartCity/DSS). Sydney, Australia; 2016. pp. 929–936. https://doi.org/10.1109/hpcc-smartcity-dss.2016.0133.

  97. 97.

    Liao S, Zhang H, Shu G, Li J. Adaptive resource prediction in the cloud using linear stacking model. In: 2017 Fifth international conference on advanced cloud and big data (CBD). Shanghai, China; 2017. pp. 33–38. https://doi.org/10.1109/cbd.2017.14.

  98. 98.

    Borkowski M, Schulte S, Hochreiner C. Predicting cloud resource utilization. In: 2018 15th international conference on smart cities: improving quality of life using ICT & IoT (HONET-ICT). Islamabad; 2016. pp. 38–42. https://doi.org/10.1145/2996890.2996907.

  99. 99.

    Shariffdeen RS, Munasinghe DTSP, Bhathiya HS, Bandara UKJ, Bandara HMND. Adaptive workload prediction for proactive auto scaling in PaaS systems. In: 2nd International conference on cloud computing technologies and applications (CloudTech). Marrakech, Morocco; 2016. pp. 22–29. https://doi.org/10.1109/cloudtech.2016.7847713.

  100. 100.

    Rahmanian AA, Ghobaei-Arani M, Tofighy S. A learning automata-based ensemble resource usage prediction algorithm for cloud computing environment. Future Gener Comput Syst. 2018;79:54–71. https://doi.org/10.1016/j.future.2017.09.049.

    Article  Google Scholar 

  101. 101.

    Kaur G, Bala A, Chana I. An intelligent regressive ensemble approach for predicting resource usage in cloud computing. J Parallel Distrib Comput. 2019;123:1–12. https://doi.org/10.1016/j.jpdc.2018.08.008.

    Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Ikhlasse Hamzaoui.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hamzaoui, I., Duthil, B., Courboulay, V. et al. A Survey on the Current Challenges of Energy-Efficient Cloud Resources Management. SN COMPUT. SCI. 1, 73 (2020). https://doi.org/10.1007/s42979-020-0078-9

Download citation


  • Energy-aware computing
  • Virtual resources management
  • Container resources management
  • Predictive approaches