1 Introduction

Cloud computing is a large-scale distributed computing paradigm in which a pool of computing resources, e.g., computation, storage and networking, is available to cloud users via the Internet [1, 2]. With the development of virtualization technology, cloud providers enable their users to submit job requests with specific resource demands and software stack (e.g., operation systems and applications) and then package them all together into virtual machines (VMs) [3, 4]. By submitting job requests to cloud providers, users no longer need to purchase and maintain sophisticated hardware for the resource usage in their peak load, thus reducing their total cost of ownership [5]. Cloud computing has now become the most emphasized information and communications technology (ICT) paradigm and is directly or indirectly used by almost every online users [6].

In cloud environments, users can submit their job requests anytime and anywhere. Once receiving a job request, the cloud provider should create a VM on a suitable physical machine and allocate required resources to guarantee quality of service (QoS), according to the user’s job demands [7]. How to allocate VMs to suitable physical machines according to cloud users’ QoS requirements is the VM scheduling problem studied in this paper. Since cloud computing is a market-oriented utility, optimal VM scheduling in cloud computing should allow the cloud provider and cloud users to focus on their own businesses to optimize their own incentives, respectively [8]. Job requests submitted by different cloud users may demand different amounts of resources. For instance, job requests for high-performance computing require more CPU cores, while big data processing applications require more memory [9, 10]. Furthermore, the cloud provider provisions heterogeneous virtual machine types with different resource configurations. If the cloud provider underestimates the provision of resources to cloud users, it would result in service-level agreement (SLA) violations and thus cause penalties. Otherwise, overestimating the provision would lead to resource under utilization and losing revenue as well.

VM scheduling in cloud computing is a complex problem, in which different concerns should be taken into account and then be properly addressed. Basically, these concerns can be classified into two categories, i.e., concerns of cloud users and those of cloud providers. From the perspective of cloud users, there are two major concerns in VM scheduling, i.e., successful execution rate of the VM requests (SERoV) (e.g., [11,12,13,14]), and the combined cost (which is the total execution cost of all users’ job requests) incurred (e.g., [3, 7, 11, 15]). These two concerns are important ones since cloud users generally hope to successfully complete their submitted job requests (i.e., to meet their respective deadlines and budgets), and at the same time, at the lowest possible cost. If, on the contrary, users’ job requests frequently miss their deadlines, or the cost incurred is high, then cloud users experience low degree of user satisfaction and tend to lose interest in the cloud system and may finally leave it. Therefore, increasing SERoV is an incentive for cloud users, and so is reducing combined cost, which shall be considered in VM scheduling.

On the other hand, from the perspective of the cloud provider, the most interesting thing is to make the best use of the provided computing resources to make profits. However, if the workload on a computing resource (CR) is too high, this makes it likely that the VMs residing on the given CR cannot receive the required resources, which may lead to SLA violation and degrade the degree of user satisfaction. Frequent failures to fulfill cloud users’ expectations inevitably damage the reputation of the cloud provider, which in the long run will lead to profit loss. In this respect, a major concern of the cloud provider is to make sure all the provided CRs have equal opportunities to offer their resources and make profits according to their resource capacities. In this paper, we call this concern as profit fairness among all the provided CRs. Therefore, increasing profit fairness, or equivalently reducing the fairness deviation of profits (FDoP, details are shown in Sect. 4.1.2) among all the provided CRs, is desirable from the perspective of the cloud provider [16, 17], which is an incentive for the cloud provider that shall also be considered in VM scheduling.

From the above discussions, it can be observed that, in cloud environments, there are different incentives for the two parties, i.e., cloud users and the cloud provider. These incentives for both parties should be considered and properly addressed so that a more comprehensive VM scheduling algorithm can be developed, which could help to provide both parties with sufficient incentives to stay and play in the cloud, leading to a sustainable system [12, 17]. Much attention has been paid on the field of VM scheduling in cloud computing [3, 7, 9,10,11, 14]. (Details are given in Sect. 2.) However, most of them are either from the incentive for cloud users or from the ones for the cloud provider to address this problem, and very few studies take into consideration the incentives for both parties.

This paper, in contrast, makes an endeavor to investigate the VM scheduling problem in cloud computing by addressing the major incentives for both parties, i.e., maximizing the SERoV and minimizing the combined cost (incentives for cloud users), and at the same time minimizing the FDoP (incentive for cloud provider). The problem of incentive-aware VM scheduling is thus formulated as a multi-objective optimization problem. Then, we develop a heuristic scheduling algorithm, called Cost-Greedy Dynamic Price Scheduling (CGDPS) algorithm, which can effectively allocate VMs to suitable physical resources. Simulation results show that the proposed CGDPS algorithm is effective and can achieve higher SERoV, lower cost, smaller FDoP and most important, higher degree of user satisfaction, compared with some popular algorithms (i.e., PSO [11], FCFS [15], MinCTT [18]) in most cases.

The rest of this paper is organized as follows: A detailed review of related work on VM scheduling in cloud computing is given in Sect. 2. The system model used in the paper is introduced in Sect. 3. Section 4 puts forward the incentives for cloud users and the cloud provider and proposes a heuristic scheduling algorithm called CGDPS algorithm. Simulations, results and analyses are given in Sect. 5. We conclude this paper in Sect. 6.

2 Related work

Many efforts have been put into the research on VM scheduling in cloud computing. For example, Yang and Chen expected [13] to enhance the successful ratio of backup tasks for scheduling MapReduce tasks and proposed an adaptive task allocation scheduler (ATAS) to determine the response time of backup tasks in heterogeneous cloud system. Their research focused on optimizing the execution time for MapReduce tasks. However, in cloud computing, there is another major concern of cloud users other than execution time, i.e., the combined cost. In order to reduce cloud users’ execution cost, some cost-based approaches [3, 7, 14] have been proposed. For example, by considering different plans for renting resources from the cloud provider, Chaisiri et al. [3] formulated the VM scheduling problem as a stochastic programming model and proposed an OCRP algorithm with the aim of minimizing the total cost of cloud users. Zhang et al. [7] proposed a cost-efficient algorithm (ROSA) to find the feasible schedule that can minimize cloud users’ total execution cost. Li et al. [14] also investigated the cost minimization-oriented VM scheduling problem in hybrid clouds and proposed an ODPA algorithm. On the basis of these research works, this paper further takes users’ satisfactions into consideration.

To further improve the degree of users’ satisfaction, some studies try to optimize both the execution time and cost for cloud users’ job requests [11, 18,19,20,21,22,23,24]. For example, Garg et al. [18] presented a heuristic algorithm, min–min cost time trade-off (MinCTT), to manage the trade-off between the execution time and cost spent to execute users’ jobs. Somasundaram and Govindarajan [11] proposed a particle swarm optimization (PSO)-based scheduling mechanism to minimize the total execution time and the total execution cost of users’ submitted job requests. In [19], Singh and Chana developed a cloud workload management framework, which is actually a cloud workload analyzer (CWZ). Then, based on the developed CWZ, they [20] proposed a QoS metric-based scheduling algorithm to optimize the execution cost and execution time by analyzing and categorizing the workloads submitted by cloud users. Recently, Ran et al. [21] proposed a dynamic VM provisioning strategy for determining the number of the purchased VMs dynamically in order to minimize the total cost while keeping QoS. Wang et al. [22] also focused on the performance metrics from the view of users and proposed a resource provisioning algorithm, called max–min-cloud algorithm, to minimize the mean of the stochastic response time of users’ request. Moreover, researches on workflow scheduling (e.g., [23] and [24]) mostly focused on optimizing the total execution time and the combined cost of users’ workflows, as well. It can be noted that, although these researches cloud improve the degree of users’ satisfaction to a certain extent, they all only focused on addressing the incentives for cloud users.

Some other existing studies tried to address the VM scheduling problem by considering the incentives for the cloud provider. For example, Zheng et al. [25] studied the VM provisioning jointed with physical servers’ maintenance scheduling problem and proposed a heuristic scheduling algorithm to maximize the revenue of the cloud provider. Albagli-Kim et al. [26] proposed a dwindling job scheduling algorithm to maximize the profit of the cloud provider for processing cloud users’ job requests. Wei et al. [9, 10] made an endeavor to best utilize cloud provider’s physical resources by trying to avoid resource starvation where dominant resources are starved while non-dominant resources are wasted. They proposed a heterogeneous resource allocation algorithm, SAMR, from the cloud provider’s point of view. Yu et al. [27] propose a probabilistic guarantee scheme with the objective of minimizing the total migration overhead caused by inefficient migrations catering to dynamic and bursty resource demands of VMs. Besides, some researches (e.g., [5] and [28, 29]), which tried to optimize energy consumption of cloud data centers, are also from the cloud provider’s respective to address the VM scheduling problem.

From the above review of the related works, we know that most of existing studies on VM scheduling in cloud computing only address the incentive for one party, i.e., either the cloud users or the cloud provider. However, cloud users and the cloud provider as the two main participated entities of a cloud system may have the different incentives. These incentives for both parties should be considered and properly addressed for providing the both parties with sufficient incentives to stay and play in the cloud and leading to a sustainable cloud system. Nevertheless, very few studies consider the incentives for both parties. For example, Tian et al. [30] investigated the placement of web applications, which is a similar problem as VM scheduling in cloud computing. They proposed a heuristic algorithm with the objectives of satisfying the resource demands of users’ web applications as much as possible and at the same time, keeping load balance among the resource provider’s physical machines. Recently, Liu and Shen [31] proposed a dependency-aware and resource-efficient scheduling (DRS) algorithm with the objectives of decreasing the response time and simultaneously improving the resource utilization. Gregory and Majumdar [32] investigated the resource management technique for scheduling batches of MapReduce tasks and proposed a resource management techniques by considering concerns of both cloud users (QoS requirement, i.e., deadline) and the cloud provider (i.e., energy consumption). These researches are from the perspectives of both the users and the resource provider; nevertheless, the combined cost, which is one of the most attractive incentives for cloud users [17], is not considered in deriving the scheduling algorithms. Although much attention has been paid on the field of VM scheduling, there are few studies considering incentives for both cloud users and the cloud provider, in which the combined cost, one of the most attractive incentives for cloud users, is not addressed. To the best of our knowledge, there is no existing research on addressing VM scheduling problem to optimize the SERoV and the combined cost for cloud users and simultaneously optimize FDoP for the cloud provider before. Therefore, this paper investigates the incentive-aware VM scheduling in cloud computing by taking into account the incentives for both cloud users and the cloud provider.

3 System model

This paper investigates the VM scheduling problem in cloud computing, where multiple customers may submit job requests at random instants with various workloads that should be fulfilled before the specified deadlines [7]. The system model generally consists of three main active entities, namely the cloud provider who owns the infrastructure resources, cloud users who are the resource consumers, and cloud scheduler.

Cloud users (such as individuals, small and medium enterprises) have demands to execute their jobs with certain QoS requirements. Generally, some important QoS factors that users most concern about are budget and deadline [7, 15, 17]. Each job may consist of several subtasks, and each subtask requires one CPU core to execute it. Subtasks in the same job should be executed concurrently on the same computing node [5, 12]. Denote by J = {J1, J2,…, Jm} the n(n ≥ 1) job requests submitted by cloud users during the scheduling interval [0, T). Each job request Ji(1 ≤ i≤n) can be characterized by a six-tuple Ji = (Ki, Li, memi, ti, bi, di) [7, 11, 15, 18, 33], in which Ki(Ki ≥ 1) is the number of subtasks that job request Ji contains; Wi = {wik|1 ≤ k≤Ki} is the workload set of job request Ji’s Ki subtasks, in which wik is the workload of the k:th subtask in job request Ji, in terms of millions of instructions (MI); memi represents the memory size required by job request Ji; ti(0 ≤ ti < T) is the arrival time that job request Ji arrives at the cloud provider’s data center; bi is job request Ji’s budget constraint, which means that the cost of executing job request Ji must not exceed bi; di represents its deadline by which the cloud user desires job request Ji to be completed. Before the jobs are executed, the desirable resources should be allocated.

The cloud provider owns infrastructure resources in its data center, which could be composed of hundreds or thousands of computing nodes. These computing nodes are heterogeneous since different computing nodes may have different CPU cores, different CPU processing speeds, different memory sizes and different prices. Without loss of generality, suppose that the cloud data center consists of m(m ≥ 1) heterogeneous computing nodes, denoted by CN = {CN1, CN2,…, CNm}. Each computing node CNj(1  ≤  j ≤ m) can be characterized by a four-tuple CNj = (Corej, memj, sj, pj) [12, 34, 35], in which Corej and memj represent the number of CPU cores and memory size provided by computing node CNj, respectively; sj is the processing speed of computing node CNj’s CPU cores, in terms of million instructions per second (MIPS); pj is the price of a CPU core, which equals to the cost of using a single CPU core of computing node CNj per second. To provision resources for users’ job requests, the cloud provider should first encapsulate all the job requests integrated with their own specific resource demands into different VM requests [3, 36]. Thus, each job request Ji can be packaged as a VM request Vi according to Ji’s characterization Ji = (Ki, Li, memi, ti, bi, di). The packaged VM requests are then submitted to the cloud scheduler for scheduling. For convenience of reading, we use VM request instead of job request in the rest of the paper.

The cloud scheduler manages the collections of VM requests, gathering all computing nodes’ resource information (such as available CPU cores, available memory size, the price, etc.), mapping each VM request to a suitable computing node and provisioning the required resources for the VM requests based on their QoS requirements. Once a VM request is successfully mapped onto a computing node and starts to execute, it will non-preemptively occupy the provisioned resources until its workloads are completed. After successfully completing the workloads of a VM request, the cloud scheduler is also responsible for releasing the provisioned resources. Then, the cloud provider charges cloud users for the provisioned resources.

Generally, the problem of VM scheduling in cloud computing can be described as follows: Suppose that a cloud data center consists of m heterogeneous computing nodes CN = {CN1, CN2,…, CNm}, and there are n VM requests V = {V1, V2, …, Vn} submitted by cloud users, the problem VM scheduling is to allocate each VM request to a suitable computing node, as to optimize the incentives for both the cloud users and the cloud provider.

4 Problem formation and proposed algorithm

This paper focuses on the VM scheduling problem, i.e., how to determine the allocations of VM requests to computing nodes, taking into account the QoS guarantees, as well as the incentives for both the cloud users and the cloud provider. Cloud users may have various QoS requirements, and different computing nodes may have different processing speeds and different prices. As the two main active entities of a cloud system, cloud users and the cloud provider may also have different concerned incentives. For example, cloud users always desire their VM requests can be successfully allocated with low cost and high QoS guarantees. However, for the cloud provider, the most important concern is to make full use of its computing node resources to obtain high profit. All the above-mentioned challenges make VM scheduling problem more complex to be resolved.

4.1 Problem formulation

This paper formulates the VM scheduling problem in cloud computing as a multi-objective optimization model. Before presenting the details of the optimization model, we first introduce the incentives for both cloud users and the cloud provider.

4.1.1 Incentives for cloud users

From the perspective of cloud users, many objectives could be defined, but what attracts them most is that their VM requests can be successfully allocated and executed at low cost with high QoS guarantees. If the cost of VM allocation is too high or the job executions frequently miss their deadlines, users will lose interests in the cloud system. Consequently, this paper chooses the successful execution ratio of VM requests (SERoV) and the combined cost as the cloud users’ incentives. The SERoV, denoted by θ, is the value of the number of successfully allocated VM requests divided by the total number of all VM requests submitted by cloud users, which can be given by

$$ \theta = \frac{{\sum\nolimits_{i = 1}^{n} {\varphi_{i} } }}{n}, $$
(1)

where φi indicates whether VM request Vi is completed before its deadline. If VM request Vi is completed before its deadline, then \( \varphi_{i} = 1 \); otherwise, \( \varphi_{i} = 0 \).

The combined cost, denoted by C, is the sum of the costs for all the successfully allocated VM requests, which can be given by

$$ C = \sum\limits_{i = 1}^{n} {\sum\limits_{j = 1}^{m} {\left[ {x_{ij} \cdot K_{i} \cdot \mathop {\hbox{max} }\limits_{{1 \le k \le K_{i} }} (w_{ik} /s_{j} ) \cdot p_{j} } \right]} } , $$
(2)

where xij indicates whether VM request Vi is allocated to computing node CNj. If VM request Vi is allocated to computing node CNj, then \( x_{ij} = 1 \); otherwise, \( x_{ij} = 0 \). The term \( w_{ik} /s_{j} \) denotes the execution time of the k:th subtask of VM request \( V_{i} \) on computing node CNj. The term \( { \hbox{max} }_{{1 \le k \le K_{i} }} (w_{ik} /s_{j} ) \) is the execution time of VM request Vi on computing node CNj, which is equal to the maximum execution time among VM request Vi’s Ki subtasks. It means that if VM request Vi is successfully allocated to computing node CNj, it will occupy all the provisioned resources until all its workloads are completed.

4.1.2 Incentive for the cloud provider

From the perspective of cloud provider, due to the limitation of the cost of investment, cloud provider’s data center consists of many heterogeneous computing nodes. What’s more, the cloud provider must offer customizable services, which makes itself face a problem, i.e., how to make full use of all its computing node resources to obtain profit. If a computing node can hardly get the opportunity to execute any VM request, it cannot be profitable for the cloud provider. Otherwise, if there are too many VM requests executing on a computing node, it may violate the QoS guarantees promised to the users, which cloud also reduce the cloud provider’s profit. To make sure the fairness of obtaining profit among all the provided computing nodes can effectively avoid the above-mentioned situation [17, 37, 38]. Fairness Deviation of Profits (FDoP) among the computing nodes can ensure that each computing node should have equal opportunity to offer its resources and gain a fair profit according to its resource capacity. It means that a computing node can obtain the share of profit proportional to the capacity that it invests to the cloud system. The FDoP is attractive to both computing nodes with low resource capacity and those with high capacity. Therefore, this paper chooses the FDoP as the incentive for the cloud provider.

The obtained profit of computing node CNj, denoted by profitj, is the sum of the costs charged from the cloud users for successfully completing their VM requests, so profitj can be given by

$$ {\text{profit}}_{j} = \sum\limits_{i = 1}^{n} {\left[ {x_{ij} \cdot K_{i} \cdot \mathop {\hbox{max} }\limits_{{1 \le k \le K_{i} }} ({{w_{ik} } \mathord{\left/ {\vphantom {{w_{ik} } {s_{j} }}} \right. \kern-0pt} {s_{j} }}) \cdot p_{j} )} \right]} . $$
(3)

The profit rate of computing node CNj, denoted by μj, is defined as the ratio of the obtained profit of CNj divided by its total CPU capacity, so the calculation of μj can be given by

$$ \mu_{j} = \frac{{{\text{profit}}_{j} }}{{{\text{Core}}_{j} \cdot s_{j} }}. $$
(4)

The FDoP among the computing nodes can be measured by the standard deviation of the profit rates of all the computing nodes. Denote by σ the FDoP among the computing nodes, then we have

$$ \begin{aligned} \sigma & = {\text{std}}\_{\text{dev}}(\mu_{1} ,\mu_{2} , \ldots ,\mu_{m} ) \\ & = \sqrt {\frac{1}{m}\sum\limits_{j = 1}^{m} {(\overline{\mu } } - \mu_{j} )^{2} ,} \\ \end{aligned} $$
(5)

where \( \overline{\mu } \) is the average value of the profit rates of all the computing nodes.

4.1.3 Multi-objective optimization model

In this paper, we make an endeavor to investigate the problem of VM scheduling by addressing the major incentives for both parties in cloud computing, i.e., maximizing the SERoV and minimizing the combined cost (incentives for cloud users), and at the same time minimizing the FDoP (incentive for the cloud provider). The problem of incentive-aware VM scheduling in cloud computing can thus be formulated as the following multi-objective optimization model.

Objectives:

$$ {\text{Max}}\quad \theta = \frac{{\sum\nolimits_{i = 1}^{n} {\varphi_{i} } }}{n}\quad ({\text{I}}) $$
$$ {\text{Min}}\quad C = \sum\limits_{i = 1}^{n} {\sum\limits_{j = 1}^{m} {\left[ {x_{ij} \cdot K_{i} \cdot \mathop {\hbox{max} }\limits_{{1 \le k \le K_{i} }} ({{w_{ik} } \mathord{\left/ {\vphantom {{w_{ik} } {s_{j} }}} \right. \kern-0pt} {s_{j} }}) \cdot p_{j} } \right]} } ;\quad ({\text{II}}) $$
$$ {\text{Min}}\quad \sigma = \sqrt {\frac{1}{m}\sum\limits_{j = 1}^{m} {(\overline{\mu } } - \mu_{j} )^{2} } .\quad ({\text{III}}) $$

Subject to:

  1. (i)
    $$ x_{ij} \in \{ 0,1\} ,\quad \forall i \in \{ 1,2, \ldots ,n\} ,\;j \in \{ 1,2, \ldots ,m\} ; $$
  2. (ii)
    $$ \sum\limits_{j = 1}^{m} {x_{ij} \le 1} ,\quad \forall i \in \{ 1,2, \ldots ,n\} ; $$
  3. (iii)
    $$ \sum\limits_{i = 1}^{n} {x_{ij} \cdot K_{i}^{{}} \le {\text{Core}}_{j} } ,\quad \forall j \in \{ 1,2, \ldots ,m\} ; $$
  4. (iv)
    $$ \sum\limits_{i = 1}^{n} {x_{ij} \cdot {\text{mem}}_{i} \le {\text{Mem}}_{j} } ,\quad \forall j \in \{ 1,2, \ldots ,m\} ; $$
  5. (v)

    \( \forall i \in \{ 1,2, \ldots ,n\} ,\quad {\text{if}}\;x_{ij} = 1, \) then computing node CNj must satisfy the following two restrictions:

$$ wt_{i} + \mathop {\hbox{max} }\limits_{{1 \le k \le K_{i} }} (w_{ik} /s_{j} ) \le d_{i} ; $$
(6)
$$ K_{i} \cdot \mathop {\hbox{max} }\limits_{{1 \le k \le K_{i} }} (w_{ik} /s_{j} ) \cdot p_{j} \le b_{i} . $$
(7)

In the proposed multi-objective optimization model, optimization objective (I) maximizes the successful execution rate of VM requests, which tries to make sure that all the VM requests can be successfully executed. Optimization objective (II) minimizes the combined cost, which is another incentive for cloud users. It tries to reduce the total execution cost of all cloud users for their submitted job requests. Optimization objective (III) minimizes the fairness deviation of profits among cloud provider’s all computing nodes. This objective tries to ensure that all the computing nodes’ profit rates are the same, which means that all the computing nodes can get profits for the cloud provider according to their resource capacity. The proposed multi-objective optimization model can offer sufficient incentives for both the cloud users and the cloud provider, encourage the two parties to stay and play in the cloud system and keep the cloud system sustainable.

The first constraint defines the feasible range of decision variable xij. Constraint (ii) ensures that a VM request should be assigned to no more than one computing node. Constraint (iii) is CPU capacity restriction, which specifies that the total provisioned CPU cores should not exceed its available number. Constraint (iv) is memory capacity restriction, which is similar to constraint (iii). The last constraint indicates that if VM request Vi is allocated to computing node CNj, then computing node CNj must satisfy VM request Vi’s deadline and budget restrictions. In Eq. (6), the term wti is the waiting time of VM request Vi; the term \( {\hbox{max} }_{{1 \le k \le K_{i} }} (w_{ik} /s_{j} ) \) is the execution time of VM request Vi on computing node CNj, which is equal to the maximum execution time among VM request Vi’s Ki subtasks. Therefore, Eq. (6) constrains that the sum of VM request Vi’s waiting time and execution time must be less than its deadline. Equation (7) constrains that the execution cost of VM request Vi on computing node CNj must be less than its budget.

This paper formulates the incentive-aware VM scheduling in cloud computing as a multi-objective optimization problem, i.e., maximizing the SERoV [objective (I)], minimizing the combined cost [objective (II)] and minimizing the FDoP [objective (III)]. It can be noted that, when substituting Eq. (3) and Eq. (4) into Eq. (5), the last optimization objective, (III), is not linear concerning the decision variable xij. Therefore, the proposed optimization model is a nonlinear optimization problem. Moreover, from the first two constraints we can see that the problem is a kind of combinatorial optimization problem, which has been proved to be a NP-hard problem [30]. The intractability of the problem increases exponentially with the number of variables if being handled with deterministic algorithms, such as exhaustive search. It becomes more challenging with the increase in the proliferation and complexity of cloud data centers [8]. Using an heuristic-based algorithm to tackle VM scheduling problem in cloud computing has received increasing attentions in recent years, as such an algorithm offers an NP-hard problem global solution acceptable in a time frame proportional to the number of variables [17, 39, 40]. Therefore, this paper develops a heuristic-based algorithm, called cost-greedy dynamic price scheduling (CGDPS) algorithm, to tackle the incentive-aware VM scheduling problem.

4.2 Proposed algorithm

Before giving the detailed description of the developed CGDPS algorithm, two definitions should be explicitly understood.

Definition 1

Candidate Node Set (CNS): For an arbitrary VM request \( V_{i} \in {\mathbf{V}} \), if the resource capacity of a computing node \( {\text{CN}}_{j} \in {\mathbf{CN}} \) can satisfy VM request \( V_{i} \)’s resource demands and, at the same time, the computing node \( {\text{CN}}_{j} \) can successfully complete VM request \( V_{i} \)’s workloads before its deadline with the cost less than its budget, then the computing node \( {\text{CN}}_{j} \) is a candidate node for VM request \( V_{i} \). All the candidate nodes of VM request \( V_{i} \) compose the candidate node set of VM request \( V_{i} \), which can be denoted by \( {\text{CNS}}_{i} \). If there is more than one candidate node for a VM request, then the developed algorithm will allocate the VM request to the candidate node with the lowest cost.

Definition 2

Scheduling Matrix (SM): Let \( {\mathbf{X}} = (x_{ij} )_{n \times m} \) be the scheduling matrix, in which \( x_{ij} \) is the decision variable of the proposed multi-objective optimization model (shown in Sect. 4.1.3). If VM request \( V_{i} \) is allocated to computing node \( {\text{CN}}_{j} \), then \( x_{ij} = 1 \); otherwise, \( x_{ij} = 0 \). Actually, the scheduling matrix is the scheduling result obtained by a certain algorithm.

The pseudo-code of the developed CGDPS algorithm is shown in Algorithm 1. The inputs of CGDPS algorithm are the set of VM requests, V = {V1, V2, …, Vn}, and the set of computing nodes, CN = {CN1, CN2,…, CNm}. The output is the obtained scheduling matrix X = (xij)n×m. The detailed scheduling processes of CGDPS algorithm are as follows.

figure a

Initially, all VM requests are marked with unscheduled state, and the values of all the elements of scheduling matrix X are set as 0 (line 1, Algorithm 1). Then, CGDPS algorithm iteratively executes the following processes (lines 2–27, Algorithm 1):

Firstly, for each unscheduled VM request Vi, CGDPS algorithm orderly checks m computing nodes, {CN1, CN2, …, CNm} whose subscripts are randomly generated, to find its candidate node set CNSi (lines 3–7, Algorithm 1). During this process, if a certain candidate node set CNSi is an empty set, which means that there is not any candidate node for VM request Vi and thus Vi cannot be successfully scheduled, then CGDPS algorithm changes VM request Vi’s state to failed scheduling (lines 8–10, Algorithm 1); otherwise, if a certain candidate node set CNSi contains only one candidate node CNj, then CGDPS algorithm allocates VM request Vi to the computing node CNj and changes the relevant parameters, namely VM request Vi’s state, the value of the decision variable xij and computing node CNj’s available resource capacity, such as CPU cores and memory size (lines 11–16, Algorithm 1).

Secondly, for VM requests whose candidate node sets contain more than one candidate node, CGDPS algorithm orderly selects the VM request Vi whose candidate node set CNSi contains the minimal number of candidate nodes [as shown in Eq. (8)] and allocates Vi to the candidate node CNj (in CNSi) which can minimize Vi’s execution cost (lines 19–25, Algorithm 1). Then, CGDPS algorithm invokes the AdjustComputingNodePrice( ) function, as shown in Algorithm 2, to adjust the prices of corresponding computing nodes related to VM request Vi (line 26, Algorithm 1).

Continue the two steps mentioned above until there is no VM request in unscheduled state.

$$ \left| {{\text{CNS}}_{i} } \right| = \mathop { \hbox{min} }\limits_{1 \le k \le n} \left\{ {|{\text{CNS}}_{k} |} \right\},\;{\text{s}} . {\text{t}} .\;\left| {{\text{CNS}}_{k} } \right| > 1. $$
(8)

The developed CGDPS algorithm contains a price adjusting function AdjustComputingNodePrice( ) whose aim is to adjust the prices of some related computing nodes to ensure the fairness of obtaining profits for all computing nodes. The pseudo-code of AdjustComputingNodePrice( ) function is shown in Algorithm 2. The inputs are the set of computing nodes CN = {CN1, CN2,…, CNm}, a certain successfully scheduled VM request Vi, the coefficient of increasing price α and the coefficient of decreasing price β. This function is invoked once a certain VM request Vi is successfully scheduled.

The detailed executing processes of the price adjusting function are as follows:

Firstly, the function needs to find all the qualified computing nodes, which can satisfy both the resource demands and deadline restriction of VM request Vi (lines 1–5, Algorithm 2). These qualified computing nodes compose VM require Vi’s qualified computing node set, which can be denoted by QCSi.

figure b

Secondly, the function changes the prices of the computing nodes in QCSi. For each computing node CNj (CNj∈QCSi), if CNj is the computing node that executes VM request Vi, then the price of CNj increases by a increasing coefficient α, which is a decimal slightly greater than 1, to avoid CNj always being selected in the following steps (lines 7 and 8, Algorithm 2); otherwise, the price of CNj decreases by a decreasing coefficient β, which is a decimal slightly less than 1, to avoid CNj never being selected in the following steps (lines 9–11, Algorithm 2).

5 Performance evaluation

In this section, we discuss the performance evaluation of CGDPS algorithm by a series of experiments. All the experiments share the following configurations.

5.1 Simulation configurations

The simulated cloud data center totally consists of 100 computing nodes, and each node is characterized by four parameters, i.e., number of CPU cores, memory size, processing speed and price. The number of CPU cores of each computing node is an integer uniformly distributed in the range of {2, 4, 8, 16}. Each computing node also has different memory sizes whose value is generated by a uniform distribution in the range of {4 GB, 8 GB, 16 GB, 32 GB}. The processing speed of each computing node is uniformly distributed within the range [100, 200] with the average speed of 150 MIPS. The initial prices of all computing nodes are generated by a uniform distribution within the range [0.35, 1]. Generally, the prices of computing nodes have roughly linear relationship with their processing speed, which makes sure that a faster node needs more execution cost than a slower one for executing a same task.

The parameters of each VM request includes arrival time, required memory size, number of subtasks, workloads of each subtask, deadline and budget. In the experiments, we assume all the VM requests needed to be scheduled are submitted within a scheduling interval (T = 100 s) and their arrival times are generated by a uniform distribution within the range (0, 100]. The required memory sizes are randomly distributed within the range of {1 GB,2 GB,3 GB,4 GB}. Each VM request consists of 1–7 subtasks whose value is uniformly generated with the average number of four subtasks. The workload of each subtask is considered as a random integer uniformly distributed within the range of {100,000, 120,000, 140,000, 160,000,…, 500,000} MI. The deadline restriction allocated to each VM request is set as the sum of average estimated runtime with a 10% variation. The budget allocated to each VM request is set as the value of the maximum workload of all its subtasks divided by the average processing speed of all computing nodes, multiplying by the number of subtask and the average price with a 10% variation. Most of the configurations mentioned above are similar to those adopted in [11, 15, 17, 18]. In order to avoid the influence of causal factors, we run each experiment 1000 times, and the results presented in this paper are the mean value of the results obtained by the 1000 experiments.

It is to be noted that the resource attributes of CNs can be obtained according to their configurations and the attribute information (such as CPU core and memory requirements, task length, etc.) of VM requests can also be achieved by some prediction techniques, such as [41, 42]. The two twofold inputs of the proposed algorithm can be both obtained by existing techniques and thus the simulation configurations are feasible.

5.2 Performance metrics

In order to verify the efficiency of the developed CGDPS algorithm, we compare it with the other three related algorithms, i.e., PSO [11], FCFS [15] and MinCTT [18], with the following five performance metrics. The first performance metric is the successful execution ratio of VM requests (SERoV), which can be calculated by Eq. (1). The second metric is the average execution cost (AEC), which is the total combined cost [calculated by Eq. (2)] divided by the number of successfully executed VM requests. The third metric is the average makespan (AMS), which is the average value of all the successfully executed VM requests’ makespans [11]. The makespan of a successfully executed VM request is defined as the time span between its submitted time and completed time. The fourth metric is the fairness deviation of profit (FDoP) among the computing nodes, which can be calculated by Eq. (5). The last performance metric is the overall user satisfaction (OUS), which is the sum of the satisfactions of all cloud users according to the obtained scheduling result. Denote by usi the satisfaction of the cloud user, who submits the VM require Vi, according to the scheduling result, so we have

$$ {\text{US}} = \sum\limits_{i = 1}^{n} {{\text{us}}_{i} } . $$
(9)

In Eq. (9), usi is determined by three factors, i.e., whether VM request Vi is successfully executed or not, the makespan and the execution cost of VM request Vi. Denote by makespani and costi the makespan and the execution cost of VM request Vi, respectively, so we have

$$ {\text{us}}_{i} = \varphi_{i} \times \left[ {(d_{i} - {\text{makespan}}_{i} )/d_{i} + (b_{i} - {\text{cost}}_{i} )/b_{i} } \right]. $$
(10)

5.3 Simulation results

5.3.1 Experiment 1

To evaluate the performance of the proposed CGDPS algorithm, we randomly generate six groups of VM requests in this experiment by the methods mentioned above. In the six VM request groups, the number of computing nodes is the same, i.e., 100 computing nodes, and the numbers of VM requests are 75, 100, 125, 150, 175 and 200, respectively. The six groups simulate the cloud environments with general system workloads, which can be used to evaluate the performance of the proposed algorithm under general system workloads.

5.3.1.1 Successful execution ratio of VM requests (SERoV)

Successful execution of a VM request means that the VM request is completed before its deadline with the cost less than its budget. The higher SERoV a cloud data center provides, the more interest users have in performing their businesses through the data center. The results of SERoV obtained by the four compared algorithms are listed in Table 1.

Table 1 Results of SERoV obtained by different algorithms with varied system workloads
Full size table

From the results listed in Table 1, it can be observed that the proposed CGDPS algorithm can obtain the highest successful execution rate. Specifically, when the numbers of VM requests are 100, 125, 150, 175 and 200, the SERoV obtained by CGDPS algorithm is 10.7, 15.36, 17.6, 17.32 and 18.75%, respectively, higher than that of FCFS algorithm, 1.22, 3.55, 7.75, 7.16 and 6.01%, respectively, higher than that of MinCTT algorithm and 1.43, 4.79, 8.87, 8.09 and 6.99%, respectively, higher than that of PSO algorithm.

The reason is as follows: For the VM requests with tight QoS constraints, few computing nodes can satisfy their requirements. Assigning high priority to the VM requests with tight QoS constraints can increase the possibility of these VM requests to be scheduled. The proposed CGDPS algorithm exactly adopts the idea mentioned above and preferentially schedules the VM requests with tight QoS constraints, as presented in lines 11–16 and 19–22 of Algorithm 1. Therefore, CGDPS algorithm can obtain the highest successful execution ration of VM requests among the compared algorithms.

Also, from Table 1, it can be seen that the SERoV obtained by the four algorithms all decreases with the increasing of the number of VM requests. This is because with the increasing of VM requests, the system workload increases, but the resource capacity of the data center is fixed (i.e., 100 computing nodes), and thus, it leads to the decrease in the SERoV.

Furthermore, Fig. 1 shows the results of the number of successfully executed VM requests (SEVR) obtained by different algorithms. As shown in Fig. 1, the proposed CGDPS algorithm can successfully execute more VM requests than the other compared algorithms. Specifically, when the numbers of VM requests are 125, 150, 175 and 200, the SEVR obtained by CGDPS algorithm are 12.97, 15.6, 14.65 and 14.98%, respectively, higher than that of FCFS algorithm, 5.17, 8.38, 7.98 and 7.18%, respectively, higher than that of MinCTT algorithm and 6.39, 9.36, 8.94 and 7.73%, respectively, higher than that of PSO algorithm. The trend of the metric of SEVR is the same with that of the metric of SERoV. The reason lies in that the proposed CGDPS algorithm preferentially schedules the VM requests with tight QoS constraints.

Fig. 1
figure 1

Number of SEVR obtained by different algorithms with varied system workloads

Full size image
5.3.1.2 Average execution cost (AEC)

Average execution cost of all VM requests can be calculated by the total combined cost [Eq. (2)] divided by the number of successfully executed VM requests. The smaller the average execution cost is, the lower the cost (users need to pay for their VM requests) is. The results of AEC obtained by different algorithms are listed in Table 2. It can be seen that when the numbers of VM requests are 75, 100, 125, 150 and 175, the average execution cost obtained by CGDPS algorithm is 18.42, 20.84, 20.06, 14.7 and 10.54%, respectively, smaller than that obtained by FCFS algorithm, 12.36, 10.74, 8.58, 4.35 and 6.54%, respectively, smaller than that obtained by MinCTT algorithm, and 8.8, 8.36, 5.6, 0.68 and 1.04%, respectively, smaller than that obtained by PSO algorithm.

Table 2 Results of AEC obtained by different algorithms with varied system workloads
Full size table

The proposed CGDPS algorithm can obtain the lowest average execution cost among the compared algorithms in almost all the varied system workload situations. This is because the proposed CGDPS algorithm tries to schedule a VM request to the candidate computing node with the lowest cost (shown in line 21 of Algorithm 1), which can lead to a lower execution cost.

5.3.1.3 Average makespan (AMS)

This performance metric indicates the average time that an algorithm needs to complete a VM request. The smaller the value of average makespan is, the shorter the time that the corresponding algorithm needs to complete users’ VM requests is. Table 3 shows the results of average makespan of all successfully executed VM requests obtained by different algorithms. It can be seen that MinCTT algorithm can obtain the shortest average makespan and the proposed CGDPS algorithm does not perform well on this metric. The reason for this is that makespan is one of the optimization objectives of MinCTT algorithm and the algorithm always schedules a VM request to the computing node which can minimize the weighting sum of execution cost and makespan. However, when CGDPS algorithm schedules a VM request, makespan is just as one of the restrictions [shown in Eq. (6)], which is used to determine whether a computing node is the candidate node for the VM request (shown in Definition 1). Thus, the proposed CGDPS algorithm cannot perform well on the performance metric of average makespan.

Table 3 Results of AMS obtained by different algorithms with varied system workloads
Full size table

Moreover, although FCFS algorithm can obtain relatively low average execution cost and short average makespan, the successful execution rate of VM requests obtained by the algorithm is much lower than other algorithms. The VM requests rejected by FCFS algorithm are those with tight budget or deadline constriction. In other words, these rejected VM requests are either the ones who contain too many subtasks or the ones whose workloads of subtasks are too long. Most of its successfully executed VM requests are the ones with a few subtasks or with short subtask workload. Thus, FCFS algorithm performs relatively well on the two metrics.

5.3.1.4 Fairness deviation of profits (FDoP)

The fairness deviation of profits means that all computing nodes provided by the cloud provider have equal opportunities to offer their resources and can obtain fair profits according to their resource capacities. FDoP, shown in Eq. (5), indicates the dispersion of all computing nodes’ profit rates. The smaller the value of FDoP is, the fairer all the computing nodes’ profits are. Table 4 shows the results of FDoP obtained by different algorithms with varied system workloads. As listed in Table 4, when the numbers of VM requests are 75, 100, 125, 150, 175 and 200, the FDoP obtained by CGDPS algorithm is 45.7, 41.7, 38.2, 30.5, 30.3 and 29.9%, respectively, smaller than that of FCFS algorithm, 32.1%, 28.9.4, 25.6, 18.3, 18.7 and 20.5%, respectively, smaller than that of MinCTT algorithm, and 27.5, 24.7, 23.1, 16.4, 18.7 and 20.8%, respectively, smaller than that of PSO algorithm.

Table 4 Results of FDoP obtained by different algorithms with varied system workloads
Full size table

It can be seen that the proposed CGDPS algorithm can obtain the lowest fairness deviation of profits. The reason is that once CGDPS algorithm successfully schedules a VM request, the price adjusting function (shown in Algorithm 2) will be invoked to execute to dynamically adjust the price of all computing nodes in its candidate node set. If a candidate computing node obtains the VM request, then CGDPS algorithm slightly increases its price to prevent it from always being selected in following scheduling; otherwise, CGDPS algorithm slightly decreases its price to avoid the computing node never being selected in following scheduling. Thus, the proposed algorithm can ensure FDoP better than the other compared algorithms.

5.3.1.5 Overall user satisfaction (OUS)

The OUS represents the degree of users’ satisfaction based on the obtained scheduling result (i.e., the scheduling matrix). Higher OUS means meeting user satisfaction better. The comparison results of OUS obtained by different algorithms are presented in Fig. 2. It can be seen that the proposed CGDPS algorithm can obtain the highest degree of user satisfaction. Specifically, when the numbers of VM requests are 75, 100, 125, 150, 175 and 200, the OUS of CGDPS algorithm are 38.4, 37.1, 45.6, 48.5, 34.6 and 34.2%, respectively, higher than that of FCFS algorithm, 1.2, 10, 19.3, 24.2, 18.8 and 15.1%, respectively, higher than that of MinCTT algorithm, and 18.6, 22.4, 29.7, 31.7, 22.5 and 17.8%, respectively, higher than that of PSO algorithm. It is evident that the proposed CGDPS algorithm can meet users’ satisfaction best.

Fig. 2
figure 2

Overall user satisfaction obtained by different algorithms with varied system workloads

Full size image

5.3.2 Experiment 2

To evaluate the performance of the proposed algorithm in a large-scale cloud environment, this experiment randomly generates a cloud data center with 600 computing nodes and 1000 VM requests submitted by cloud users, using the methods discussed in Sect. 5.1.

Table 5 shows the results of the five performance metrics obtained by the compared algorithms under a large-scale cloud data center. It can be seen that, in most cases, the comparison results are similar to the results presented in Sect. 5.3.1. In particular, the overall user satisfaction (OUS) obtained by CGDPS algorithm is comparatively 44.3% higher than FCFS algorithm, 18.6% higher than MinCTT algorithm and 24.1% higher than PSO algorithm. It means that CGDPS algorithm can guarantee cloud users’ QoS requirements better. Therefore, we can conclude that the proposed CGDPS algorithm is effective to be adopted in large-scale cloud data centers.

Table 5 Results of all metrics obtained by different algorithms under a large-scale data center
Full size table

5.3.3 Experiment 3

In this section, we verify the practicability of the proposed CGDPS algorithm under real-life workload traces [43, 44], which are also widely adopted in related research. We use DAS2 fs1 trace as the simulated workload because it records the similar resource attributes required by a VM request. There are total 40,315 records in the workload, from which we choose 196 ones as VM requests in the experiment to limit the simulation time. The 196 records are selected by a principle that the number of subtasks contained in a record must be in the range of 1–7. Each record has detailed descriptions of the submitted time, runtime, required CPU cores and memory size, and some other descriptions. However, some QoS requirements, such as deadline and budget constraints, still need to be randomly generated using the methods discussed in Sect. 5.1 since the real-life workload trace does not contain any information about them. Also, we also normalize the submitted times of the selected records within the range of [0, 100] so that we can schedule all of them in a scheduling interval.

Table 6 shows the results of the five performance metrics obtained by different algorithms under the real-life workload trace of DAS2 fs1. As listed in Table 6, the proposed CGDPS algorithm can achieve the highest successful execution rate of VM requests, relatively lower average execution cost, the lowest fairness deviation of profit, and, most important, the highest degree of user satisfaction.

Table 6 Results of all metrics obtained by different algorithms under real-life workload trace
Full size table

This section conducts experiment under the real workload trace. From the results and analysis presented above, we can find that the proposed algorithm can also perform pretty well under the real-life workload trace. Therefore, it can be concluded that the proposed CGDPS algorithm can be practically applied to real-life cloud environments.

6 Conclusions

Cloud users and cloud provider are two important entities in a cloud system, and they make autonomous scheduling decisions with different incentives of interest, which makes the problem of VM scheduling in cloud computing more complex than ever before.

This paper formulates VM scheduling in cloud computing as a multi-objective optimization problem and develops a heuristic-based CGDPS algorithm. In each iterative step, CGDPS algorithm sets high priority to VM requests with few candidate nodes and schedules a VM request to the candidate computing node with the lowest cost. After successfully scheduling a VM request, CGDPS algorithm adopts the price adjusting function to dynamically adjust the price of all qualified computing nodes to avoid the fairness deviation becoming worse. The simulation results clearly illustrate that the developed method can achieve the highest degree of users’ satisfaction.

Although this paper considers the scenario of multi-core physical machines, there is a constraint that each of PM’s CPU cores cannot be allocated to more than one VM. However, in modern cloud environments, multi-core physical machines are more promising and each of PM’s CPU cores can be shared by cores of multiple multi-core VMs. So as a future work, we will conduct research on the more real-life scenario of multi-core PMs, which may lead to a more complex scheduling issue.