Energy-Efficient and Load-Aware VM Placement in Cloud Data Centers

VM consolidation has been proposed as an effective solution to improve resource utilization and energy efficiency through VM migration. Improper VM placement during consolidation may cause frequent VM migrations and constant on–off switching of PMs, which can significantly hurt QoS and increase energy consumption. Most existing algorithms on efficient VM placement suffer the problem of easily falling into a sub-optimum prematurely since they are heuristic. Also, they do not achieve a good balance between multiple different goals, such as resource utilization, QoS, and energy efficiency. To address this problem, we propose an effective and efficient VM placement approach called MOEA/D-based VM placement, with the goal of optimizing energy efficiency and resource utilization. We develop an improved MOEA/D algorithm to search for a Pareto-compromise solution for VM placement. Our experiment results demonstrate that the proposed multi-objective optimization (MOO) model and VM placement solution have immense potential as it offers significant cost savings and a significant improvement in energy efficiency and resource utilization under dynamic workload scenarios.


Introduction
As cloud computing technology becomes increasingly mature [1], more applications are migrating toward cloud data centers. To satisfy the increasing demand for storage and computing resources, cloud data centers are growing larger, bringing many challenges in terms of resource management, resource scheduling, and energy-consumption management. As a popular approach for virtual resource scheduling and management in cloud data centers, virtual machine (VM) consolidation with VM placement and migration achieves virtual resources scheduling through live VM migration. In fact, VM consolidation is a stage-separating operation [1,2]. Generally, VM migration incurs VM placement. Both of them are the approach to technical enablement of Virtual Machines (VMs) consolidation. VM migration facilitates performing load balancing, but it is impacted by those of the stochastic workload, the dynamic and uncertain resource demands of VMs.
Abstract VM consolidation has been proposed as an effective solution to improve resource utilization and energy efficiency through VM migration. Improper VM placement during consolidation may cause frequent VM migrations and constant on-off switching of PMs, which can significantly hurt QoS and increase energy consumption. Most existing algorithms on efficient VM placement suffer the problem of easily falling into a sub-optimum prematurely since they are heuristic. Also, they do not achieve a good balance between multiple different goals, such as resource utilization, QoS, and energy efficiency. To address this problem, we propose an effective and efficient VM placement approach called MOEA/Dbased VM placement, with the goal of optimizing energy efficiency and resource utilization. We develop an improved MOEA/D algorithm to search for a Pareto-compromise solution for VM placement. Our experiment results demonstrate that the Of which results in that, VM placement is impacted by various factors, such as CPU utilization, memory utilization, bandwidth, disk size, and so on. Thus VM placement is a multi-resource constraint issue [3,4]. Additionally, the highly efficient VMs consolidation in cloud data centers attempts to perform an ideal balance among degrading energy consumption, optimizing resource utilization, and guaranteeing QoS, which needs to be addressed from a entirety and globalization aspect and difficult. From this perspective, VMs placement is a multi-objective optimization problem with multiple resource constraints. During the past ten years, there are significant research efforts to optimize energy consumption, quality of service (QoS), and resource utilization through VM migration, VM placement or VM consolidation respectively or their combinations [2,[4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21].
Cloud service providers worldwide are challenged by huge amounts of energy consumed by cloud data centers [22]. The average CPU utilization of all running physical machines (PMs) in cloud data centers is only 15-20% [23]. Most running PMs are seriously wasted, with idle PMs responsible for 70% of peak energy consumption [4]. Hence energy consumption can be effectively reduced by migrating all VMs out of under-loaded PMs and set these PMs to hibernate. Where to migrate those VMs is the key problem for improving resource utilization, guaranteeing QoS, and avoiding the resource overloading risk. Moreover, virtual machines can have uncertain and random demands for resources. When many virtual machines request resources simultaneously, a data center's demand for resources surges sharply. Turning off or hibernating too many PMs during this time can easily lead to overloading of PMs and frequent VM migration, which negatively affects QoS. Thus, it is critical for VM placement to minimize the probability of resource overloading while guaranteeing QoS.
A number of studies have been conducted on VM placement [2,4,6,7,11].The best fit decreasing (BFD) algorithm and its extensions [4] are best known, sorting the migrated VMs in descending order of CPU resource utilization, finding a destination host for them, and minimizing the increase in the energy consumption. Utilization prediction-aware BFD (UP-BFD) [2] predicts the CPU utilization through applying the k-nearest neighbor algorithm, arranges virtual machines in overloaded PMs or PMs with overloading risk according to the descending order of their resource requests, and migrates VMs to the physical machine with the highest load. If the destination host experiences or risks overload after VM placement, the algorithm will select the host with the most load in the remaining candidates. However, these algorithms are heuristic and are more likely to fall into a local optimum prematurely.
In this paper, we propose a multi-objective heuristic evolutionary algorithm for the VM placement method, MOEA/D-VMP, which can guarantee QoS and reduce energy consumption. Our method seeks the optimal balance among guaranteeing QoS, energy consumption reduction, and resource utilization improvement through the dynamic optimization model for VM placement, with multi-resource constraints with a multi-objective optimization (MOO) algorithm based on decomposition. In summarization, our main contributions are listed below, (1) We model VM placement as a multi-objective optimization problem with multi-resource constraints, with the goals to reduce energy consumption at data centers, lower the probability of resource overloading on PMs, and improve resource utilization. (2) We propose a method to solve the problem based on the improved MOEA/D algorithm. A set of Pareto optimal solutions is obtained as candidate solutions for the deployment of migrated VMs. (3) A compromise solution selection strategy is proposed to compare candidate schemes for VM placement based on decision-makers' preferences, and a compromise solution selected from the above set of Pareto optimization solutions is the optimal solution of VM placement. (4) We demonstrate the effectiveness and efficiency of MOEA/D-VMP using the CloudSim simulation platform. The experimental results show that the method achieves the optimal balance among reducing energy consumption, increasing resource utilization, and improving QoS.

Related Works
The growing scale of cloud data centers directly leads to the difficulty of virtual resource management and scheduling and the inefficiency of virtual resource management. In order to improve efficiency for virtual resource management and scheduling, variant VM placement problems have been studied from traditional single-resource constraints to multiple-resource constraints and from single-objective optimization to MOO. Several studies [15,16,[24][25][26] have proposed utilizing intelligent algorithms to resolve the issue of VM placement problems. Their results have shown that intelligent algorithms are able to obtain desirable solutions for optimal VM placement. Combining the Genetic Algorithm (GA) and First Fit (FF) algorithm, Hallawi et al. [9] propose a Combinatorial Order First-Fit Genetic Algorithm (COFFGA) and find the optimal VM placement. In order to improve resource utilization and reduce energy consumption, it considers an optimization model that minimizes the number of running hosts and the remaining amount of resources. COFFGA encodes chromosomes in the way that the gene position represents the host and the information on the gene position represents a VM. FF algorithm is used to allocate the host for each virtual machine. Accordingly, the optimum mapping relationship between VMs and hosts is obtained by updating the population iteratively and selecting the optimum chromosome sequence according to the optimization model. Since GA can suffer insufficient training and easily fall into the local optimal, COFFGA cannot effectively find the global optimum mapping relationship between VMs and hosts.
Li et al. [15] establish an optimization model to reduce energy consumption and virtual machine migrations (VMMs), and decrease the overloading probability of resources of hosts. The VM consolidation is abstracted as a MOO problem with multiple resource constraints, and the ABC-like (Artificial Bee Colony) algorithm is used to search for the optimal mapping relationships between hosts and VMs. Considering both energy heterogeneity and overload probability of hosts after VM placement, the EOP-VMP algorithm is proposed, which effectively reduces energy consumption and VMMs.
To reduce energy consumption and avoid the resource overloading risk on hosts in cloud data centers, Li et al. [16] address the optimal placement problem of VMs by improving the classical differential evolution (DE) algorithm. An improved discrete DE algorithm is proposed to minimize energy consumption and overloading probability for VM placement.
In order to accelerate the convergence speed of the algorithm, the heuristic information matrix is employed to record the experience information in the process of iterative optimization to guide the evolution of the algorithm, which significantly enhances the convergence speed of the algorithm. The proposed method effectively reduces energy consumption and improves the QoS of data centers.
Ding et al. [7] propose an effective and secure VM placement strategy (DFA-VMP) based on the discrete firefly algorithm (DFA) in order to ensure the security of the data center. DFA-VMP considers a MOO model for system security, energy consumption and resource utilization and normalizes it as a single objective optimization model using linear weighting and penalty function. The DFA algorithm is applied to solve the single objective model iteratively to get the global optimization solution. The prominent contribution of the DFA-VMP method is to deploy malicious tenants and target tenants in different Hosts to avoid bypass attacks, aiming to improve the performance and security of the data center.
Fares et al. [4] propose an Ant Colony System (ACS) algorithm to address the problem of VMs placement. This method defines the local heuristic information of the artificial ant colony, updates the pheromone accumulated in the iterative circles, and generates random state transition rules. It selects the destination hosts for the migration VMs using these rules, thereby perform VM placement. In [4], the linear weighting of CPU resources and energy consumption is used to define local heuristic information. The virtual machine can be deployed on the host with relatively high resource utilization to avoid turning on new hosts and realize the reduction of energy consumption. Farahnakianet al. [27] defines the local heuristic information according to the comprehensive workload of the host. The higher the resource utilization of the host is, the greater its local heuristic information is. This method targets placing the virtual machine on the destination hosts with high resource utilization first to reduce energy consumption. Ferdaus et al. [28] define the local heuristic information by using the resource imbalance vector (RIV) and the comprehensive resource utilization of the hosts [29] to reduce energy consumption and resource waste and improve load balancing between different computing resources.
To solve the MOO model for VM placement, many methods [7,9,15,16] combine multiple objectives into a single objective by linear weighting. They give different weight values according to the importance of each objective to optimize the combined single objective, namely, transform the issue of MOO into a single objective optimization problem. Unfortunately, the optimization of an objective often causes the degradation of the other objectives due to the MOO problems [30]. Therefore, it is often desirable to find a set of Pareto optimal solutions for MOO problems. The Pareto optimization solutions mean that there are no better solutions on one or more objective functions without degrading the optimal solutions of other objective functions. Obviously, this scheme is more suitable for solving the optimization problems of VM placement.
Additionally, several new strategy and scheme of VM placement [18-21, 31, 32] have emerged, of which are under the consideration of the actual bandwidth and traffic. Samaneh Sadegh et al. [18] derived a novel two-phase strategy for VM placement by modeling the issue of VM placement as the minimum weight K-vertex-connected induced subgraph. The presented algorithm effectively performs the defficiency of host resource utilization reduction and workload imbalance incurred by compact VM placement; Huanlai Xing et al. [19] present an energy-and trafficaware ant colony optimization ETA-ACO algorithm. Further, the ETA-ACO algorithm integrates three separated schemes on resolving the VM placement optimization problem, of which results in the final VM placement scheme; Wei Zhang et al. [20] developed a model of initial VM fault-tolerant placement for star topological cloud data centers. The model is built on the basis of multiple factors including the servicelevel agreement violation rate, resource remaining rate, power consumption rate, failure rate, and fault tolerance cost. Additionally, Marwa Thabet et al. [21] developed an online secure optimization VM placement strategy and algorithm. To minimize the total number of running physical machines, the CLR VM placement algorithm takes a way that guarantees-apriori-a specified level of security. Lin Wang et al. [31,32] proposed a new general framework GreenDCN integrates purposeful VM assignment, two-phase routing in data center networks, reduction of the active switches on achieving energy conservation. GreenDCN primarily addressed on the traffic engineering to reduce power consumption with optimization VM placement, simplization data center network infrastructure and balance traffic flows.

Estimation of Resource Utilization
Let V j be the set of all VMs deployed on the host h j (1 ≤ j ≤ n) in a cloud data center. The current amount of resources consumed by a host is the sum of the resources used by all VMs deployed on the host. The utilization of a specific type of resource on host h j can be expressed by (1), where r ∈ RS , RS = {cpu, mem, bw} is the set of all kinds of resources in the host, U r j is the utilization of r -type resources in host h j , C r j is the configured capacity of r-type resources in host h j , and D r j is the amount of r-type resources used in host h j , i.e., the sum of r-type resources used by all VMs deployed in host h j .

Estimation of the Resource Overloading Probability on Hosts
A host that experiences overloaded resources will negatively impact the service quality of a cloud data center. While given plenty of virtual machines, the workload will change dynamically with users' computing requests. A statistical analysis of the load records in the PlanetLab [33] and Google Cluster trace [34] datasets found that these workloads had a Gaussian distribution [35]. Therefore, we assume various resource demand models of VMs follow Gaussian distribution. Since the number of resources used by the host D r j is aggregated by VMs, D r j has a Gauss- where r j and r j are respectively the expected value and standard deviation of the number of resources used by host h j . The host is overloaded when D r j is greater than C r j , so the probability of r-type overloaded resources in host h j can be estimated based on (2), where Pr D r j > C r j means that the probability that D r j , the total number of r-type resources requested by all VMs that are deployed on host h j , does not exceed C r j , the configured capacity of h j , and Φ is a normal distribution function.

Estimation of Energy Consumption
There is an approximately linear or sublinear relationship between the power of the host and the CPU utilization [1]. Hence, the power of host h j at a certain time can be calculated based on (3), where P full j is the power that host h j h j is at the status of full load, the running host consumes energy even when it carries no load, and P idel j is the power of the non-load host. P j U cpu j is the power of host h j when the CPU resource utilization is U cpu j . The energy consumption of the host h j from time period t 0 to t 1 can be calculated according to (4), where EC h j is the energy consumption of the host h j , and U cpu j (t) is the CPU resource utilization of the host h j at time t.

Problem Formulation
Expenditure related to energy consumption has become the largest source of operation cost for cloud data centers [36]. Only the optimal balance between reducing energy consumption, increasing resource utilization, and improving QoS can maximize both cloud service providers and users' interests.
Proper VM placement is the key to reducing energy consumption and improving the resource utilization of cloud data centers. Inappropriate VM placement will increase the resource overloading risk in the destination host, thereby impacting QoS. Hence a reasonable VM placement scheme must: 1) improve the resource utilization; 2) prevent increased energy consumption due to wasted resources, and 3) avoid the negative impact on service quality of an overloaded host. According to the above analysis, VM placement is defined as the multi-objective optimization problem shown in (5), where X is the deployment mapping matrix between VMs and hosts, and m is the total amount of VMs in the cloud data center. The description of the notations we use in the paper can be found in Table 1.
Equation (5) is essentially the joint optimization of various resources and their probabilities of getting overloaded. The function f 1 (X) combines energy consumption and the probability of CPU overload; f 2 (X) is a joint function of the memory resource utilization and the probability of memory overload, and f 3 (X) combines the bandwidth resource utilization and the probability of bandwidth overload. f 1 (X) , f 2 (X) , and f 3 (X) essentially constrain CPU, memory, and bandwidth resources through corresponding overload probabilities. In (6), (7), and (8), P r over h j ∈ [0, 1] is the probability of the overload of r-type resources in the host h j . When the amount of r-type resources requested by the host h j is very small, i.e., P r over h j is close to 0, then it is difficult for a virtual machine to choose the optimal destination host among multiple under-loaded hosts. Therefore, the range of the probability is adjusted by an exponential function such that as P r over h j increases, the value of the exponential function will increase rapidly. The parameter i can be used to control the influence of the resource overloading probability on the value of the objective function. The larger i is, the more sensitive the objective function is to the probability of resource overload. Equation (9) is the constraint of the MOO problem. Equation (9) means that one virtual machine can only be deployed on one host. If the virtual machine vm i is deployed on the host h j , then x i,j = 1 ; otherwise, x i,j = 0 and x ∈ X . According to Eq. (9), the total amount of r-type resources that are requested by all the VMs deployed on host h j cannot exceed C r j , the total capacity of r-type resources in this host, where d r i is the capacity of r-type resources requested by virtual machine vm i .

Our VM Placement Approach
MOEA/D [37], a classical multi-objective optimization (MOO) algorithm based on decomposition, provides a desirable solution to complicated large-scale optimization problems. Its core is to transform the complicated MOO problem into a series of simple sub-problems and obtain the approximate solution set through joint optimization [38]. MOEA/D uses a set of weight vectors to decompose the MOO problem into a set of single-objective optimization sub-problems and solves them simultaneously [39]. The solutions correspond to different points on the Pareto dominating front, i.e., the optimal solution set for the MOO problem. With searching for the optimal solution in a flexible and extensive space, MOEA/D is well suited The set of migrated VMs The energy consumption of h j after hosting vm i on it ΔEC(h j ) Energy consumption growth of PM to solving MOO problems with a large solution space, such as VM placement. Furthermore, it can quickly converge to the Pareto dominating front, so it is suitable for problems with real-time requirements. Here, we use an improved MOEA/D algorithm, namely, discrete differential evolution aided neighborhood control in MOEA/D (DDE-MOEA/D), to solve the VM placement problem given in Eq. (5). The process includes individual coding, initialization, and restructuring and deployment strategies.

Discrete Differential Evolution Aided
Neighborhood Control in MOEA/D

Individual Coding
For the problem of VM placement, each individual denotes a mapping relationship between VMs and hosts, and the set of all possible individuals (i.e., mapping relationships) is called a population. The chromosome codes of each individual are not exactly the same, and together they constitute the solution space of the optimization problem for VM placement. The constraints shown in (9) must be satisfied when individual chromosomes are encoded, i.e., one virtual machine can only be employed on one host. In contrast, one host can hold multiple virtual machines. Figure 1 is an example of an individual coding scheme. The length of a chromosome is equivalent to the size of the set of VMs to be deployed, namely, g(e.g.,0 ≤ g ≤ M ) items of VMs to be migrated. The i-th position loci(i) of the chromosome stands for the i-th virtual machine vm i vm i in the set of VMs that will be migrated. The jth host h j that holds the virtual machine vm i is stored in the i-th position of a gene in the chromosome. At this time, only one host can be assigned on a gene locus, but the host and host ID can emerge on multiple gene loci. Exactly, the emerged hosts are the current candidate destination hosts. Therefore, this coding method meets the constraints of Eq. (9).

Initialization
In the beginning, the algorithm initializes the population, neighborhood, and reference point. During the population initialization, the algorithm determines the set of destination hosts of VMs to be migrated. During initialization, a host is randomly assigned to each gene in the individual, checking whether the constraints of (9) are satisfied. If the conditions hold, the ID of the host is added to the gene locus. Otherwise, the random allocation process continues. Finally, the algorithm generates an initial population S is one of the mapping matrix between VMs and hosts, the index of the matrix represents the ID number of the virtual machine to be assigned. The element of the matrix refers to the ID number of the destination host.
For neighborhood initialization, a systematic method [40] is generally applied to yield a set of uniformly distributed weight vectors λ = 1 , … , s , where s for each individual X t s . Once the weight vectors are generated, the Euclidean distance can be calculated to determine the neighborhood set B(s) of each weight vector s .B(s) contains the index numbers of the T items of weight vectors closest to s .
Once the initial population is formed, the problem is decomposed into S single-objective optimization sub-problems by a Chebyshev function, as shown in (10), where g te (⋅) is the Chebyshev function, K is the amount of objective functions f k , and the reference (10) ming te X t s | s , z * = max 1≤k≤K s,k |f k X t s − z * k | Fig. 1 Individual coding demo point z * k = minf k X t s , s = 1, 2, … , S . During initialization, z * k = minf k X 0 s . The s-th individual X t s in the population represents the solution to the s-th subproblem after the t-th iteration.

Restructing and Deployment Strategies
After initialization, the individuals in the population are iteratively updated through restructuring and deployment to yield the next generation for individuals. Since the weight vectors correspond to the individuals in the population, each individual's neighborhood is generated once the neighborhood of the weight vector is generated, i.e., the index numbers contained in B(s) are the neighborhood for the individuals. We use the differential evolution operator to realize the restructuring as follows. For example, as an individual X t s , we randomly select two indices γ and β from the set B(s) and find the corresponding individuals X t γ and X t β in the population set X . The restructuring and generation of the next generation of individuals is computed as in (11), where F ∈ [0, 2] is a scaling factor, and X t+1 s is the new individual generated by X t s after t + 1 iterations. The classical differential evolution operator is only suitable for continuous optimization problems, but VM placement is a discrete problem, so we need to redefine the addition and subtraction operations of (11). We determine the difference matrix X Δ from the subtraction of any two individuals, such as X t γ and X t β , based on (12), where x Δ,i , x ,i , and x ,i are respectively the i-th elements in matrices X Δ ,X t γ , and X t β ; the i-th element represents the ID number of the destination host to hold the migrated virtual machine vm i . Instead of subtracting vectors, (12) obtains the difference matrix by checking whether the destination hosts have the same ID number, i.e., if the value of x Δ,i is − 1, then virtual machine vm i must be re-assigned to a destination host, and the next generation of individuals is generated by (13) once the matrix X Δ is obtained, where x t+1 s,i denotes the ID number of the host to hold the virtual machine vm i with the ID number i in the individual X t+1 s , where randf ∈ [0, 1] is a random number.
New individuals are generated after implementing the restructuring strategy, and they execute the local deployment strategy, i.e., parent X t s generates child X t+1 s , which is compared to each solution X u (u ∈ B(s)) in the neighborhood of X t s , and the deployment strategy is implemented based on (14),

MOEA/D-based VM Placement Algorithm
Usually, the set of non-dominant solutions can be obtained by MOEA/D, expressed as EP , from which a compromise solution can be derived based on the decision-makers' real-time preferences, and we use it as the final VM placement scheme. We note it is important to establish a compromise solution selection strategy. The hosts have heterogeneous energy consumption, i.e., the energy consumed by hosts with different resource utilization for a given workload is not necessarily the same. Therefore, if the same virtual machine is assigned to different destination hosts, these hosts may have different energy consumption. Thus we use (15) to calculate the energy consumption increment by host h j after virtual machine vm i is deployed on it, where EC h j +vm i and ΔEC h j are the energy consumption of host h j after deployment of virtual machine vm i and the increment of energy consumption, respectively.
To further guarantee QoS, a running host with a low overloading probability after deploying the migrated virtual machines is a priority selected as the destination host. Hence, based on a full consideration of the energy consumption growth and resource overloading probability of running hosts, a selection strategy for the final compromise solution is proposed. The ΔEC h j = EC h j +vm i − EC h j energy consumption and overloading probability of the destination host after the placement of all VMs are evaluated by (16). At the end of each iteration, select the solution with the smallest F opt value as the optimal solution, i.e., the Pareto-compromise solution.
According to the above analyses, we present a MOEA/D-based VM placement (MOEA/D-VMP) algorithm, which is described as a pseudo-code-representing algorithm. Exactly, the presented MOEA/D-VMP algorithm integrates the combination consideration with the actual demand of VM placement on the definition of neighborhood, the definition of reference point, the alternative strategy and the reorganization strategy. After doneing the algorithm, the non-dominated solution set of the population is also obtained, i.e. EP. As for the specific issue of VM placement, it continues to select a compromise solution from the EP depending on the real-time preference of the decision maker. Further, under the comprehensive consideration of the energy consumption growth and hosts' overloading probability, the final compromise solution is determined in terms of the Eq. (16). Namely, after the iteration is over, the final solution with the smallest F_opt is recommended as the pareto-compromise solution.
Here, the pareto-compromise solution is identical to the scheme of VM placement. The time complexity of the proposed MOEA/D-VMP algorithm is mainly decided by the 13th to 15th lines in the body, i.e., updating the neighborhood set of individuals. Here, the algorithm needs to calculate g te g te for T items of individuals in the neighborhood, and m comparisons are required for each calculation.

Simulation Setting
The Cloudsim [41] simulation platform is employed to perform the experiments. We simulate a cloud data center, which consists of 800 heterogeneous hosts. According to different resource configurations, we choose two types of hosts, as shown in Table 2. Besides, as shown in Table 3, four different types of VMs are considered. In order to confirm the effectiveness of the proposed algorithms, we use six indices [1,15]  In order to properly simulate the real resource request and response in data centers, two data sets (Bitbrains trace [42], Alibaba cluster data [1]) are employed for the simulation experiments. Bitbrains include CPU usage, memory, and bandwidth workload of selected 10-day traces, as shown in Table 4. Alibaba cluster data, as a large-scale data set, also includes CPU and memory usage. We randomly select 3-day data traces, and 3600 items of data-trace are selected randomly from each day. Further, a total of 10,800 items of data-trace are separated into nine sub-data traces. The selected sub-data trace is shown in Table 5. Clearly, the average consumption of memory resources by hosts is about three times of that the CPU resources. This situation is under the consideration of the actual background of Alibaba Cloud Core. In Alibaba Cloud, a large number of small and medium-scale merchants locate in it. The merchants' business calculations are not complicated, but there are many user requests for their business transactions, which directly results in large memory consumption.

Experiment Results
We compare the proposed MOEA/D-VMP algorithm with Combinatorial Order First-Fit Genetic Algorithm (COFFGA) [9], FFD [2], Energy-aware and Overloaded Probability-estimation VM Placement (EOP-VMP) algorithm [15] in this Sect. 3 overloading detection algorithms, i.e., Interquartile Range (IQR) [2], Median Absolute Deviation (MAD) [4], Static Threshold (ST) [4] and 3 VM selection algorithms, i.e., Random Selection (RS) [2], Maximum Correlation (MC) [2], Minimum Migration Time (MMT) [2] are combined with each other, yielding a total of 9 combination methods for VM overloading detection and selection. The 4 virtual machine placement algorithms in our comparison are combined with each of these 9 methods, yielding a total of 36 complete VM consolidation methods. For the total 36 VM consolidation methods, the pre-copy mechanism [43] is employed to treat the live VM migration. We analyze each method and evaluate its effectiveness and efficiency.

Effectiveness
The six evaluation indices [1,15], are measured to show the validity of our algorithm on Alibaba Cluster Data V2018 and Bitbrains data. Figure 2 shows the results of the energy consumption of 36 VM consolidation methods on the two datasets. Figures 2(a) and (b) respectively show the energy consumption on the Bitbrains Trace and Alibaba Cluster Data V2018 datasets. It can be seen from Fig. 2(a) that, under different overload detection algorithms and VM selection algorithms, the energy consumption of EOP-VMP is always the minimum. In contrast, when ST and MAD are the overload detection algorithms, the energy consumption is basically maintained at 75-90 kWh. EOP-VMP takes energy consumption as an optimization objective and  Regarding VM placement as a packing problem, FFD reduced energy consumption by minimizing the number of running hosts, which is basically its only optimization objective. Regarding VM placement as a multi-dimensional packing problem, COFFGA combined the genetic and first fit (FF) algorithms to solve it to minimize the number of running hosts and surplus resources. However, when VM placement is solved as a packing problem, the algorithm is likely to fall into local optimization [4,11], which would increase the amount of energy consumption and running hosts. Figure 2(a) shows that when IQR is used as the overload detection algorithm, the hosts consume plenty of energy after implementing COFFGA. As an overload detection and VM selection algorithm, IQR-RS incurs about 27% more energy consumption than EOP-VMP, which implies that COFFGA cannot find the global optimal VM placement scheme. In contrast, MOEA/D-VMP performs better in saving energy because it establishes an optimization model to minimize energy consumption and achieve optimal global solutions through iteration. The energy consumption of MOEA/D-VMP is slightly higher than that of FFD or COFFGA, as shown in Fig. 2(b). This is because MOEA/D-VMP considers the resource overloading risk in the modeling process, so the number of under-loaded hosts turned off by MOEA/D-VMP is less than that of the other algorithms. As a result, a great number of running hosts run in the data center, directly increasing energy consumption. When IQR is applied for host overload detection, COFFGA consumes much more energy than the other algorithms, demonstrating that COFFGA is likely to fall into local optimization. When EOP-VMP is used to deploy the VMs, it consumes much less energy than the other algorithms, indicating that EOP-VMP effectively turns off the under-loaded running hosts and improves the average resource utilization of hosts. Since the VM migration process suspends services, degrading the total amount of VM migrations can effectively improve QoS. The average number of VMMs obtained by the compared algorithms on two datasets is shown in Fig. 3. Figures 3 (a) and (b) respectively show the average number of VMMs on Bitbrains datasets and Alibaba cluster data V2018 datasets. For Fig. 3(a), when using different overload detection algorithms and VM selection algorithms, the VMMs of MOEA/D-VMP remain stable, less than those of FFD and COFFGA algorithms. The reason for this result is that the MOEA/D-VMP algorithm takes constraints on the overloading probability of hosts, effectively reducing the overloading risk of the host. Consequently, the hosts have enough reserved resources to deal with the rapid increase of the requested resource from VMs, thereby reducing the insignificant VM migration. The VMMs of COFFGA and FFD algorithms are higher than those of the other comparison algorithms. This is because both algorithms aim to minimize the number of running hosts, which is very likely to trigger too many VM migrations. As a result, too many VMs are deployed on a single destination host. In this process, if the resource overloading risk of the destination host is further Fig. 2 Comparison of EC increased, there will be more insignificant VM migrations. When selecting different overload detection algorithms and VM selection algorithms, the VMMs of the EOP-VMP algorithm are less than those of the other compared algorithms. This is because the algorithm takes minimizing VMMs as the optimization objective and the algorithm can effectively identify the change mode of workload. This mechanism effectively reduces the possibility of host overloading, which diminishes the VMMs. As we can see from Fig. 3(b), when choosing different overload detection algorithms and VM selection algorithms, the VMMs of the MOEA/D-VMP algorithm remain stable and less than those of the other compared algorithms. Combining it with Fig. 2(b), it could be found that the energy consumption of the MOEA/D-VMP algorithm is higher than that of the other compared algorithms, indicating that the number of running hosts is more than that of the other comparison algorithms, so that the overloading probability of hosts is lower than that of the other algorithms. Hence, the VMMs are far lower than that of the other comparison algorithms. For the EOP-VMP algorithm, the performance of VMMs is poor. According to the analysis of Fig. 2(b), the energy consumption of the algorithm is far lower than that of the other comparison algorithms because, on Alibaba Cluster Data V2018, too many running hosts are closed. It results in high resource utilization of the remainder hosts, which easily incurs overloading hosts, consequently resulting in insignificant frequent VM migration. Figure 4(a) shows the SLATAH values obtained by different algorithms on the Bitbrains Trace dataset, and Fig. 4(b) shows the SLATAH values obtained by different algorithms on the Alibaba Cluster Data V2018 dataset. When different overload detection algorithms and VM selection algorithms are used, the SLATAH values obtained by MOEA/D-VMP and EOP-VMP are much smaller than those calculated by the other algorithms, as shown in Fig. 4(a). It can be seen from Fig. 3(a) that these two algorithms have a relatively small number of VM migrations, implying a small resource overloading probability on the host and a negative impact on QoS. COFFGA has the largest SLATAH value. As shown in the box diagrams, i.e., Fig. 4(a), when the IQR-MMT combination is utilized for overload detection and VM selection, after the VMs are deployed by COFFGA, the SLATAH value changes remarkably. This is because COFFGA is likely to fall into local optimization, causing the number of running hosts to increase. As a result, it has fewer VM migrations, and the SLA-TAH value is comparatively low. Conversely, upon finding the global optimal VM placement scheme, COFFGA turns off redundant under-loaded hosts, resulting in the migration of plenty of VMs and an increase in the SLATAH value. Figure 4(b) shows the SLATAH values obtained by different algorithms on the Alibaba Cluster Data V2018. MOEA/D-VMP has the optimal SLATAH when different overload detection algorithms and VM selection algorithms are used. As illustrated in Fig. 3(b), MOEA/D-VMP has much less frequent VM migration than the other algorithms. Additionally, as illustrated in Fig. 4(b),   Fig. 4 Comparison of SLATAH when MOEA/D-VMP runs with the data on different days in the same dataset, the SLATAH value has the smallest variation amplitude, indicating that MOEA/ D-VMP is more stable than the other algorithms. Though EOP-VMP has a larger number of VM migrations than the other algorithms, the SLATAH value is smaller than those of FFD and COFFGA, which shows that this algorithm can find the destination host for the migrated VMs in the overloaded host in a timely manner, reducing the impact of the violation of the service level agreement on QoS. Figure 5 shows the PDM indices obtained by the comparison algorithms. The PDM on the Bitbrains dataset with different algorithms is shown in Fig. 5  (a), and the PDM on Alibaba cluster data V2018 dataset with different algorithms is shown in Fig. 5(b). For Fig. 5 (a), in combination with Fig. 3 (a), it can be found that when using different overload detection strategies and VM selection strategies, the times of VM migrations of MOEA/D -VMP are less than those of FFD and COFFGA algorithms, so the PDM of MOEA/D-VMP algorithm can remain stable and lower than most of them. The PDM of the COFFGA algorithm is the highest because the times of VM migrations of the COFFGA algorithm are much higher than those of the other comparison algorithms, which impacts the PDM indice. Figure 5 (b) shows the PDM from different algorithms on the Alibaba cluster data V2018 dataset. Since the VMMs of MOEA/D-VMP are very small, the PDM index is far lower than that of the other compared algorithms; after running the FFD algorithm, the PDM value is lower than the EOP-VMP and COFFGA algorithm, which is because the VMMs of the FFD algorithm is less.
The SLAV indices of different algorithms are shown in Fig. 6, the SLAV on Bitbrains data set is shown in Fig. 6 (a), and the SLAV on Alibaba cluster data V2018 is shown in Fig. 6 (b). For Fig. 6(a), when selecting different overload detection strategies and VM selection strategies, the SLAV of MOEA/D-VMP is the best one on both of the two datasets. This is because the MOEA/ D-VMP algorithm establishes the optimization model of resource overloading probability and resource utilization and finds the Pareto optimum solution of the optimization model through the inside mechanism of the MOEA / D algorithm itself, iterative updating, leading to the more stable workload on hosts. As a result, the overloading risk is not likely to emerge and the number of VM migration decreases. Therefore, for MOEA/D algorithm combined with different overload detection algorithms and VM selection algorithms, the SLAV indices on both data sets are stable and lower than those of the FFD, COFFGA algorithm. For the Bitbrains dataset, the SLAV value of the EOP-VMP algorithm is smaller than that of the other comparison algorithms because the EOP-VMP algorithm restricts the utilization of each resource, thus guaranteeing QoS effectively. For the Alibaba cluster data V2018 dataset, after the EOP-VMP algorithm completes VM placement, it turns off too many running hosts, resulting in frequent VMs migration, which negatively impacts QoS.
The ESV index is impacted by both QoS and energy consumption.  Fig. 7(b), based on Figs. 2(b) and 6(b), we know that during the selection of different overload detection and VM selection algorithms, MOEA/ D-VMP has the smallest ESV value, implying that MOEA/D-VMP has the optimal overall performance.
Based on the above analysis, EOP-VMP performs well on the Bitbrains dataset based on the six indices but poorly on most indices on the Alibaba dataset. On the Bitbrains dataset, the six indices of MOEA/D-VMP rank second to EOP-VMP and are better than those of the other algorithms on Alibaba Cluster Data V2018. This result demonstrates the validity of MOEA/D-VMP, which can effectively reduce energy consumption and guarantee QoS.

Efficiency
The efficiency of the proposed algorithm is validated from the average resource utilization of running hosts, including CPU, memory and bandwidth. Usually, if the CPU and memory utilization of hosts achieve a relatively high degree simultaneously, the loaded tasks are relatively concentrated, and the number of running hosts is small. Consequently, the corresponding energy consumption of the cloud data center gets lower. Moreover, if the cost bandwidth resource is low, the number of live VM migration is relatively rare, a perfect service quality. Therefore, the resource utilization of hosts is capable of evaluating the workload stability and QoS of hosts in the data center comprehensively.
According to the effectiveness analysis of MOEA/ D-VMP and all the other algorithms in Sect. 5.2.1, when IQR-MMT is used for overload detection and VM selection, there are significant differences in the values of indices obtained by the implementation of different combination VM consolidation methods. Thus, IQR-MMT is employed as the host overload detection and VM selection algorithm in this experiment, and the four VM placement algorithms are combined with IQR-MMT to compare VM consolidation experimentally. The experiment was conducted on a day's workload randomly selected from the two datasets, and the VMs are assumed to be consolidated every five minutes. The total times of 288 VM consolidations are carried out in one day.
There generally exists a close relationship between the memory resource utilization and CPU resource utilization of hosts. The abscissas in Figs. 8 and 9 represent the cycle of VM consolidation. Figure 8 shows the change in the average CPU utilization for different VM consolidation methods, and Fig. 9 shows changes in the average memory resource utilization of running hosts. In order to better illustrate these algorithms' utilization of CPU and memory resources, Figs. 8 and 9 are combined for comparative analysis. Figures 8(a) and 9(a) show the changes in the average utilization of CPU resources and memory resources of the four VM consolidation methods on the Bitbrains dataset. As can be seen from the two figures, the VM consolidation method combined with MOEA/D-VMP has the highest resource utilization, with the CPU utilization curve experiencing small fluctuations. This is because the optimization model established by MOEA/D-VMP considered two important factors, i.e., the resource overloading probability and resource utilization. Through the evolutionary mechanism of MOEA/D, this model stabilizes the load in the running hosts, thereby reducing the resource overloading risk. For the VM consolidation method combined with COFFGA, the average CPU utilization of running hosts is about 60%. The utilization of memory resources ranged from 30 to 40%, less than that of the other algorithms, which indicates that there are too many running hosts, and the resource utilization remains to be further improved. Figures 8(b) and 9(b) show the changes in the average utilization of CPU and memory resources of the four algorithms on the Alibaba Cluster Data V2018 dataset. For Fig. 8(b), all the algorithms have a CPU utilization of less than 30%, indicating that the data center requests a small number of CPU resources and the application of different VM consolidation methods has little impact on their utilization. As shown in Fig. 9(b), when MOEA/D-VMP is used, the memory resource utilization reaches about 80%. Based on the energy consumption data in Fig. 2(b), MOEA/D-VMP has more running hosts than the other algorithms, contributing to reducing the resource overloading risk on the hosts and guaranteeing QoS. Combining the VM consolidation method and EOP-VMP, the memory resource utilization is maintained between 40 and 80% because there are insufficient reserved resources for the host because too many running hosts are turned off after the VM consolidation. According to the joint analysis based on Figs. 3(b) and 6(b), the VM consolidation method combined with EOP-VMP increased the resource overloading risk on the hosts, causing frequent but insignificant VM migration and negatively impacting the QoS. For the VM Fig. 6 Comparison of SLAV consolidation method combined with FFD, memory resource utilization is kept at about 90%. This algorithm reduces energy consumption and improves resource utilization by minimizing the number of running hosts without considering the QoS of the cloud data center. Combining the VM consolidation method with COFFGA, the resultant optimal VM placement scheme leads to the optimal number of running hosts with a relatively high average resource utilization. Nonetheless, as the algorithm converges to the optimal local value, more running hosts can be turned on, decreasing resource utilization, so the average utilization of memory resources fluctuates greatly. Figure 10 shows changes in the average bandwidth resource utilization of VM consolidation methods combined with different VM placement algorithms. As shown in the figure, all the algorithms have an average bandwidth resource utilization of less than 6%, which indicates that the data center requested relatively few bandwidth resources. This scenario proves that the live VM migrations rarely emerge, which results in low bandwidth resource consumption. The experimental results in Fig. 3 of Sect. 5.2.1 have also proved it from another aspect. The VM consolidation method, combined with COFFGA, has the lowest bandwidth resource Fig. 7 Comparison of ESV utilization. According to the joint analysis based on Figs. 8(a) and 9(a), COFFGA is likely to appear as a local optimum. Once the algorithm achieves the local optimal, more hosts can be turned on, thereby no live VM migration and decreasing resource utilization. Thus, COFFGA has the lowest average bandwidth resource utilization. For the VM consolidation methods related to EOP-VMP and MOEA/D-VMP, the average bandwidth resource utilization is higher but quite close to those of COFFGA and FFD. This is because that EOP-VMP effectively identifies the changing pattern of the load, reduces the resource overloading risk on the host, and ensures the balance of the load. However, MOEA/D-VMP establishes an optimization model for resource utilization and the resource overloading probability, thus realizing a balanced load of the hosts.
This section presents the VM consolidation method combined with MOEA/D-VMP compared with currently available VM placement methods. According to the experimental results, MOEA/D-VMP can significantly reduce the number of VM migrations, thereby balancing loads of different resources by stabilizing the average utilization of CPU and memory resources of the running hosts. In this way, the running hosts have sufficient remaining resources to meet the random requests for increased resources by the VMs, to effectively reduce the resource overloading risk on the hosts and guarantee the QoS. This fully demonstrates the efficiency of MOEA/D-VMP.

Conclusion and Future Works
The optimal VM placement with multi-objectives is an essential but challenging problem for cloud data centers, usually regarded as NP-hard issues. This paper focused on VM placement with an improved multi-objective evolutionary algorithm based on decomposition (MOEA/D) mechanism. We addressed both energy consumption and resource utilization of the hosts in data centers and proposed a discrete differential evolution aided neighborhood control in the MOEA/D algorithm to resolve the problem in the field of multi-objective optimization. Depending on the Pareto optimal solution obtained by the proposed algorithms, a selection criterion for the Pareto-compromise solutions in combination with the energy consumption and QoS was proposed. The experimental results demonstrated the apparent improvement of MOEA/D-VMP in various performance indices, including PDM, SLATAH, SLAV, EC, ESV, QoS and the number of VM migrations. In addition, the proposed scheme also achieved optimum balancing among reducing energy consumption, improving resource utilization, and guaranteeing QoS.
However, in our future works, there are several limitations that need to be further studied. The current model uses exponential amplification to enhance the impact of resource overloading probability. This is not exactly the same as the actual scenario in cloud data centers. A practical scheme needs to be developed. Additionally, MOEA/D is a flexible optimization framework that can be combined with various singleobjective optimization algorithms to achieve Paretocompromise solutions. This point also requires further in-depth research on the issue of VM placement.