1 Introduction

Cloud computing has emerged as a paradigm to deliver, on demand computing resources to cloud consumers, similar to other utilities (e.g., water, electricity and gas) [1]. The size of the cloud computing market is growing rapidly in recent years. In this huge cloud market, it is difficult for cloud users to deal with different types of virtual machines (configurations and virtualization software’s running under them), interface managers, pricing schemes, levels of service quality, variation in availability of resources and other value-added services. Some cloud providers also enforce restrictions on the number of virtual machines (VMs) that a user can utilize. The complexity of selecting and provisioning right cloud services is intermediated by cloud broker services, which manages the use, performance and delivery of cloud services and often negotiates the relationships between cloud providers and cloud consumers [2]. Cloud broker eases the task of consumers to select suitable resources based on their computational needs.

A cloud broker has to provide the best deployment plan, where VMs are placed in an independent cloud or in multiple clouds based on the consumers requirements. To carry out this, the broker must take into account the attributes such as the configuration of resources, service performance, total cost, security. Also, the consumer can specify constraints regarding geographical locations, load balancing criteria, service configurations and legal regulations for data placement. The cloud broker synthesizes an optimized deployment plan for the placement of VMs among multiple clouds, which adheres to the user criteria and placement constraints. Deploying VMs over multiple clouds offers several benefits such as scalability of services, improved reliability cost reduction and avoid vendor lock-in. The cloud broker implicitly considers the possibility of multi-cloud deployment. Based on the requirements, the cloud broker can provide a single-cloud or multi-cloud deployment plan. Usually, a multi-cloud deployment is preferred when consumers application has loosely coupled services with less communication overhead. In contrast, a single-cloud deployment plan is preferred when a tightly coupled set of application components with a high communication overhead is involved.

For optimal resource provision in heterogeneous cloud environments, a three-phase approach cloud broker architecture is proposed. In the first phase, the cloud broker has to get the service request description, and relative weights of the required service measurement index (SMI) attribute from the consumers. The service request description consists of the type of application, virtual machine configuration, number of VMs required, location, minimum required SMI score and other Quality-of-Service (QoS) parameters. The possible set of cloud resources and services satisfying service request is identified in the second phase. In the third phase, the cloud providers are evaluated based on SMI, developed by Cloud Service Measurement Index Consortium (CSMIC) [3], and a cost optimized deployment plan is developed.

In this paper, the resource provisioning over multiple clouds is addressed by mixed integer programming model formulation using AIMMS [4] modeling language, which provides access to a wide range of solvers including CPLEX and GUROBI. Benders decomposition [5] is discussed as a possible way to solve the optimization problem efficiently for a large number of cloud providers. The cloud broker limits the number of providers for provisioning resources based on application requirements or consumers specification. The consumer can specify minimum score for each SMI category and attributes along with their weights to enable the selection cloud provider for the optimal resource provisioning phase. The objective is to minimize the total cost of the deployment plan, which satisfies consumers requirements. To illustrate the effectiveness of the optimization mechanism, numerical analysis of the model is performed using a large synthetic data set involving one thousand cloud providers.

In this work, cloud brokering architecture for provisioning resources on multiple-cloud environments is considered, and the major contributions are as follows:

  • The problem of minimizing the cost of resource provisioning in the multi-cloud environment is formulated as a mixed integer programming(MIP) problem, and its specification is modeled with AIMMS modeling language.

  • Those cloud providers satisfying the consumers minimum requirement of SMI category and attribute score as per the metrics defined by CSMIC are evaluated for cost optimal deployment plan.

  • Benders decomposition algorithm has been applied to the MIP formulation to solve the model efficiently and to prove the scalability of the model.

  • Numerical evaluation and sensitivity analysis are performed to prove the effectiveness and scalability of the proposed model.

The rest of this paper is organized as follows. Related work is discussed in Sect. 2. The proposed cloud brokering architecture is described in Sect. 3. In Sect. 4, a mixed integer programming model formulation is presented. Benders decomposition algorithm is presented in Sect. 5. In Sect. 6, numerical evaluation and sensitivity analysis of the proposed model are presented. Finally, the conclusions and future research directions are discussed in Sect. 7.

2 Related work

In recent years, researchers and cloud brokers have focused on developing models and methods for federated cloud service and product selection based on either minimizing the total deployment cost or maximizing the QoS. Cloud service provisioning and cost optimization based on static demand, price and availability have been discussed in [6–11], where [12, 13] deal with the uncertainty in the demand, price and availability of cloud services.

Tordsson et al. [6] proposed a cloud brokering mechanism that performs two operations: (i) the optimal placement of the virtual resources of a virtual infrastructure across a set of cloud providers and (ii) management and monitoring of these virtual resources by providing a unified management user interface. By considering the demand and price of resources to be static, a 0–1 integer programming model is developed to minimize the cost and maximize the performance. Experimental results confirm that multi-cloud deployment provides better performance and lower costs compared to the usage of a single-cloud deployment. Simarro et al. [7] provided a cloud brokering architecture that can work with different scheduling strategies for optimal deployment of virtual services across multiple clouds based on different optimization criteria and several user constraints. Binary integer programming formulation is used in [7].

Papagianni et al. [8] and Breitgand et al. [9] used integer linear programming with approximation algorithms to optimize the cost and improve the QoS. In [8], the authors focused on the benefits for cloud consumers in contrast to [9], where the focus is on maximizing profit and QoS of the cloud providers by utilizing the resources of partnering cloud providers to meet the peak demand. Malawski et al. [10] developed a mixed integer nonlinear programming model to optimize the total cost of resource provisioning under the time constraint. The authors also considered the maximum number of resources provided by a single-cloud provider. Wright et al. [11] introduced a two-phase constraint-based approach in a multi-cloud environment for discovering the most appropriate set of infrastructure resources for a given application. In the first phase, suitable resources are identified for the application, and in second phase, heuristic approach is used to select the best services based on cost and performance.

Simarro et al. proposed a methodology in [12] which considers the dynamic price and demand of cloud service in contrast to the static one in [7]. The authors also considered cloud migration overhead issues for service provisioning among multiple clouds. Chaisiri et al. [13] proposed an optimization of resource provisioning cost in the federated cloud with future demand and price uncertainty. There is a trade-off between reservation and on-demand pricing plan where the focus is on minimizing on-demand and over subscription cost [13].

Javadi et al. [14] consider the problem of QoS-based resource provisioning in a hybrid-cloud computing system where the private cloud is failure-prone and to overcome that they had developed a hybrid-cloud architecture. They proposed a brokering strategy in the hybrid-cloud system where an organization that operates its private cloud aims to improve the QoS by utilizing the public cloud resources.

In case of evaluation of infrastructure providers, proximity aware service selection methods for effective service selection for IaaS and geographically distributed clouds were proposed in [15, 16]. The CSMIC [3, 17] are developing a standard measurement framework for the cloud services, called the SMI. SMI compares cloud services with relevant and meaningful cloud characteristics. The cloud characteristics are categorized into seven categories such as accountability, agility, assurance, financial, performance, security and privacy, and usability. Each category has three or more attributes, and in total, SMI addresses 51 attributes. A framework for ranking IaaS cloud services using the AHP method considering the elements quantitative criteria has been developed by [18].

Wu et al. [19] proposed a service selection method based on qualitative evaluation criteria with quality-of-service aspects (such as response time and availability) as well as social perspectives of services. Rehman et al. [20] applied MCDM for service selection, which is effective for services offered with similar specifications but only differ in performance. User feedback-based MCDM approach for effective cloud service monitoring and selection has been proposed in [21]. Yan et al. [22] developed an MCDM approach for recommendation and selection of cloud services for hybrid-cloud computing environment. Cloud service evaluation and service selection method combining interval-valued fuzzy sets with VIKOR method for cloud services have been proposed in [23]. A fuzzy AHP-based service selection method has been proposed in [24].

In this work, the optimal resource provisioning over multiple clouds is addressed by evaluating the cloud providers using SMI attributes and final deployment plan using MIP formulation. The proposed approach considers both technical- and business-level criteria, while existing works consider only QoS attributes and cost for resource provisioning. Also, the model considers the location of service provided, jurisdictional regulations and the lower and upper bound for the number of VMs provided by a cloud provider. The proposed model considers families of VMs rather than individual VMs as such. It helps to overcome the minor variations among the VM configurations offered by various cloud providers in reality.

3 System model

3.1 Cloud broker architecture

Cloud brokering architecture outlined in Fig. 1 consists of three main actors, namely consumer, cloud provider and cloud broker. The consumer has the demand for computing infrastructure to execute jobs, which they can obtain from the cloud provider. Consumers request the cloud broker for virtualized infrastructure, with service request description. The service request description consists of a required set of VMs, optimization criteria, location of data center or availability zones (where VMs are placed) and required level of SMI attributes that may include performance, security level, accountability, usability.

The broker then filters the cloud providers that meet the criteria based on the consumers or applications requirements as described in service request description. The filtered cloud providers are ranked based on CSMIC SMI attributes on a 10-point scale, using weighted sum model [25]. After ranking the providers, the broker selects those providers who satisfy the minimum SMI score in either category, attribute, measure or total score based on consumers specification. Then, the broker implements an algorithm to make an optimal deployment plan to reduce the total infrastructure cost.

Fig. 1
figure 1

Proposed cloud broker architecture

Full size image

In this work, there are multiple VM classes that are used to categorize the different types of VMs. Let \(i\in \mathbb {N}\) denote the set of VM classes. It is assumed that one VM class represents a family of VM configuration within a range. In real-world scenario, VMs are offered in both predefined and customized configurations to suit customer needs. Providers like Amazon [26] offer only predefined instances where ElasticHosts [27] offers only customized VMs. Customer may request the broker with different VM configurations to run their jobs. With this requirement, the cloud broker can select the best computing resources from the available cloud providers to cater the actual demand.

Let \(p\in \mathbb {N}\) denote the set of cloud providers. Each cloud provider has a pool of resources with predefined or customizable VM class. Let r denote the set of resource types provided by the cloud providers. Resource types can be computing power, storage, memory, network bandwidth, etc. Each VM class has its specification of required resource type. Let \(b_{ir}\) be the amount of resource type r required by the VM in class i. Let \(l\in \mathbb {N}\) denote the set of geographical locations where the cloud provider offers services. It is assumed that every cloud provider prepares facilities such as virtualization management software, network facility and load balancer to support the consumer using the VMs. Schematic diagram of the proposed model is provided in Fig. 2. The key notations used in this paper are listed in Table 1.

Table 1 List of key notations
Full size table
Fig. 2
figure 2

Work flow of the cloud brokering approach

Full size image

3.2 Virtual machine cost

The cloud providers offer VMs either in predefined configurations or as customized configurations. Amazon EC2 [26] and GoGrid [28] offer VMs in predefined configurations. Amazon EC2 offerings are grouped into eight families: standard, micro, high-memory, high-CPU, cluster compute, cluster GPU and high I/O [26]. Each family has its configuration to cater the needs of different types of application requirements. When choosing VMs, the broker considers the characteristics of the application with regard to resource utilization and selects a suitable one. For example, cluster computer and cluster GPU family VMs are selected for high-performance computing (HPC) applications, while micro VMs are well suited for lower throughput applications and Web sites that consume significant compute cycles periodically. Cloud providers like ElasticHosts [27] offer customized VMs, where VMs are configured based on application needs.

In general, the cloud services are offered in nonlinear pricing plans to serve consumer heterogeneity. Cloud services are offered in one of the following ways.

Pay per use Pay per use component consists only of a per unit rate for every utilized unit (i.e., pay per hour), known as linear tariff, normally offered by many cloud providers. The pay per use tariff is also called as usage price, marginal price or per unit charge.

Flat rate The flat rate component with fixed fee is independent of the consumer’s consumption that is charged on a regular seasonal basis (either monthly, quarterly, half yearly or yearly).

Two-part pricing The consumers have to pay an upfront cost for the period (either monthly, quarterly, half yearly or yearly) and will be charged with pay per usage unit rate for every utilized unit.

The VMs are charged based on the resource configuration (number of CPU cores, memory size, storage capacity and network bandwidth), licensing cost of the software running on them and location of the data center. Normally, VMs are charged per usage hour. The pricing charged by the cloud providers are in US dollars ($) per resource unit per usage hour. Let \(C_{rpl}\) denote the unit price of resource type r provided by cloud provider p in location l. The cost of VM class \(C_{ipl}\) is the cost for provisioning every resource type defined as follows:

$$\begin{aligned} C_{ipl} =\sum _{r} b_{ir}\ C_{rpl} \end{aligned}$$
(1)

where \(b_{ir}\) is amount of resource type r required by VMs i.

3.3 Location and legal constrains

Data-center location of cloud services determines the performance of the service offered to the end user. The cloud providers build multiple data centers that are distributed geographically to meet the availability and reliability of the service. It plays an important role in the performance of the applications hosted. Response time is the key issue that can be reduced when the end user of the hosted applications has less geographical distance from the data center. Applications like telephony, video conferencing, online gaming and finance are delay sensitive, and they are benefited from the local data-center that is closer to the end user.

Another key issue is being the legal constraints and compliance requirements that govern the data. Local states have regulations and juristic limitations on where data can be stored and how data can be accessed. In the USA, regulations such as HIPAA [29], FERPA [30], PATRIOT Act [31] and GLBA [32] control how data can be stored, as well as who may access that data. In the European Union (EU), the Data Protection Directive (EUDPD) [33] governs sensitive private data and flatly forbids the transfer of data to other jurisdictions not explicitly approved [34]. Some state laws like Israeli law [35] permit data reside in other jurisdictions when adequate and sufficient levels of protection are met. The local jurisdictions have restrictions on permitting trans-border data crossing when the other jurisdiction has equivalent or better levels of protection. Many cloud providers started offering services in different geographical locations based on local state-specific regulations. Amazon offers AWS GovCloud [36] for US government agencies and contractors to move more sensitive workloads into the cloud by addressing their specific regulatory and compliance requirements.

The consumers should select the required regulations and compliance laws during service request description. Let \(j\in \mathbb {N}_1\) be a juristic regulation and compliance laws for the data placement. Let \(Y_{jpl}\) be a binary decision variable representing the cloud provider p available in location l is compliance with data placement regulation j is defined as follows:

$$\begin{aligned} Y_{jpl} = \left\{ \begin{array}{l l} 1 &{} \quad \text {if provider { p} has compliance with }\\ &{} \quad \text {regulation { j} in location l }\\ 0 &{} \quad \text {otherwise} \end{array} \right. \end{aligned}$$
(2)

3.4 Service measurement index (SMI)

The SMI is a framework of critical characteristics, associated attributes, and measures that decision-makers may apply to enable comparison of cloud services available from multiple providers. SMI is devised to be a standard method to measure any cloud service based on critical business and technical requirements of the consumers. The SMI starts with a hierarchical framework. The top level divides into seven categories, where each category is further refined by three or more attributes as defined in Table 2. Then, within each attribute one or more measures are being defined to enable the use of cloud service provider data to inform selection decisions. Some of the attributes and measures will be service specific, while others such as the security, financial will apply to all types of cloud services [3, 37]. The seven categories are defined below:

Table 2 SMI categories and attributes
Full size table

Accountability attributes used to measure the properties related to a service provider organization. These properties may be independent of the service being provided.

Agility attributes indicating the impact of a service upon the consumers ability to change direction, strategy or tactics quickly with minimal disruption.

Assurance attributes that indicate how likely it is that the service will be available as specified.

Financial the amount spent on the service by the consumer.

Performance attributes that indicate the performance characteristics of the provided services.

Security and privacy attributes that indicate the effectiveness of a service provider in controlling access to services, service data and physical facilities from which services are provided.

Usability the ease with which a service can be used by the consumers.

3.5 SMI score

SMI scores for the categories and attributes are rated from 0 to 10 where zero is the least score. The rating formula for attribute and measure is defined by CSMIC [38] and will change periodically based on the evolution of standards. SMI framework consists of both qualitative and quantitative measures. The score of the cloud provider is calculated using a weighted sum model [25], where each category, attribute and measure has their own weight based on consumer’s preferences. This provides flexibility to the consumer who provides their weights based on importance. Comparison of points scored by providers of a particular attribute to the consumers required score for a particular category or attribute is done to select the providers who meet the minimum criteria. The rating formula for suitability attribute of performance category and learnability attribute of usability category is defined in Tables 3 and 4, respectively.

Table 3 Rating formula for suitability attribute
Full size table
Table 4 Rating formula for learnability attribute
Full size table
Table 5 Scale for pairwise comparisons
Full size table

3.5.1 Relative weight calculation

SMI attribute scores are calculated using the weighted sum model of relevant measures. The weights are assigned to SMI attributes either based on predefined configuration settings (such as high performance, high security, cost-effective) or by using relative weights based on consumer preferences. For customization, they can either use pairwise comparison method proposed by Saaty [39, 40] or can provide direct arbitrary weight.

Pairwise comparison The pairwise comparisons are made depending on the scale shown in Table 5. In the pairwise comparison matrix, the score of \(s_{uv}\) represents the relative importance of the component on row (u) over the component on column (v); i.e., \(s_{uv} = w_u/w_v\). The reciprocal value of the expression \((1/s_{uv})\) is used when the component v is more important than the component u. The comparison matrix S is defined as

$$\begin{aligned} S= & {} \begin{bmatrix} w_1/w_1&\quad w_1/w_2&\quad \cdots&\quad w_1/w_n \\ w_2/w_1&\quad w_2/w_2&\quad \cdots&\quad w_2/w_n \\ \vdots&\quad \vdots&\quad \ddots&\quad \vdots \\ w_n/w_1&\quad w_n/w_2&\quad \cdots&\quad w_n/w_n \end{bmatrix}\nonumber \\= & {} \begin{bmatrix} 1&\quad s_{12}&\quad \cdots&\quad s_{1n} \\ s_{21}&\quad 1&\quad \cdots&\quad s_{2n} \\ \vdots&\quad \vdots&\quad \ddots&\quad \vdots \\ s_{n1}&\quad s_{n2}&\quad \cdots&\quad 1 \end{bmatrix} \end{aligned}$$
(3)

Then, a local priority vector (eigenvector) w is computed as an estimate of the relative importance accompanied by the elements being compared by solving the following equation:

$$\begin{aligned} Sw = \lambda _\mathrm{max}w, \end{aligned}$$
(4)

where \(\lambda _\mathrm{max}\) is the largest eigenvalue of matrix S.

Direct arbitrary consumer assigned weights The consumer can assign weights on their own scale rather than using the pairwise comparison. In this case, the weights are normalized. Let \(uw_c\) denote the user-assigned weight for category c, and then SMI category weights \(W_c\) is calculated as follows:

$$\begin{aligned} W_c = \dfrac{uw_c}{\sum _{c} uw_c}, \ \forall c \end{aligned}$$
(5)

Let \(uw_{ca}\) denote the user-assigned weight for attribute a and \(uw_{am}\) denote the user-assigned weight for measures m. Let \(W_a\) and \(W_{am}\), defined similarly as (1), which denote the normalized attribute and measure weight, respectively.

3.5.2 SMI score calculation

Let \(S_{pa}\) denote the SMI attribute score of an attribute a for a cloud provider p and \(S_{pam}\) denote the measure score of a measure m belong to attribute a. The attribute score \(S_{pa}\) for every attribute type is calculated as follows:

$$\begin{aligned} S_{pa} = \sum _{m} W_{am} \ S_{pam},\ \forall (p,a) \end{aligned}$$
(6)

subject to

$$\begin{aligned} \sum _{m} W_{am} = 1,\ \forall a \end{aligned}$$
(7)

Let \(S_{pac}\) denote the score of the category c for the cloud provider p. The category score \(S_{pac}\) is calculated using weighted sum approach of relevant SMI attributes score \(S_{pa}\) and measures m. The SMI category score \(S_{pc}\) for every category type is defined as follows:

$$\begin{aligned} S_{pc} = \sum _{a} W_{ca} \ S_{pa},\ \forall (p,c) \end{aligned}$$
(8)

subject to

$$\begin{aligned} \sum _{a} W_{ca} = 1,\ \forall c \end{aligned}$$
(9)

Let \(Ts_p\) denote the total SMI score for a cloud provider p. Total SMI score \(Ts_p\) is calculated by using weighted sum approach of all SMI category scores \(S_{pac}\). The SMI total score for every cloud provider p is defined as follows.

$$\begin{aligned} Ts_{p} = \sum _{c} W_{c} \ S_{pc},\ \forall p \end{aligned}$$
(10)

subject to

$$\begin{aligned} \sum _{c} W_{c} = 1 \end{aligned}$$
(11)

4 Mixed integer programming model

In this section, the mixed integer programming is presented as the core formulation.

Minimize:

$$\begin{aligned} \sum _{ipl} C^o_{ipl}\ x_{ipl} +\sum _{ipl} C^f_{ipl}\ x_{ipl} + \sum _{p} F_{p} y_{p} \end{aligned}$$
(12)

subject to:

$$\begin{aligned}&\sum _{p} x_{ipl} \ge D_{il},\ \forall (i,l) \end{aligned}$$
(13)
$$\begin{aligned}&x_{ipl} \le A_{ipl},\ \forall (i,p,l) \end{aligned}$$
(14)
$$\begin{aligned}&\sum _{p} y_p \le n,\ \forall p \end{aligned}$$
(15)
$$\begin{aligned}&Y_{jpl} \ge uy_{jl},\ \forall (j,p,l) \end{aligned}$$
(16)
$$\begin{aligned}&\underline{M}_{ipl}y_p \le x_{ipl} \ge \overline{M}_{ipl}y_p, \ \forall (i,p,l) \end{aligned}$$
(17)
$$\begin{aligned}&S_{pc} \ge u_c ,\ \forall (p,c)\end{aligned}$$
(18)
$$\begin{aligned}&S_{pa} \ge u_a ,\ \forall (p,a)\end{aligned}$$
(19)
$$\begin{aligned}&TS_{p} \ge u_{ts} ,\ \forall p\end{aligned}$$
(20)
$$\begin{aligned}&x_{ipl} \in \mathbb {N}_0 ,\ \forall (i,p,l)\end{aligned}$$
(21)
$$\begin{aligned}&y _p \in \left\{ 0,1\right\} , \ \forall p\end{aligned}$$
(22)
$$\begin{aligned}&Y_{jpl}\in \left\{ 0,1\right\} ,\ \forall (j,p,l) \end{aligned}$$
(23)

The general form of the optimization algorithm is formulated in Eqs. (12) to (23). The goal of the objective function Eq. (12) is to minimize the total deployment cost of VMs among multiple-cloud providers. The decision variable \(x_{ipl}\) denotes the number of VMs provisioned, and \(C^o_{ipl}\) denotes the pay per use cost of the VMs i offered by the cloud provider p in location l. The parameter \(C^f_{ipl}\) denotes the flat rate cost of the VMs, and \(F_p\) denotes the fixed cost of the provider p. The constraint in Eq. (13) maintains that the consumers demand for VMs i in location l is satisfied. In Eq. (14), the constraint states that the allocation of resource for VMs must not exceed the maximum resource capacity offered by the cloud provider p in location l. Constraint Eq. (15) indicates that the number of cloud providers is limited for deployment of VMs according to the consumer specification and application requirements. The constraint in (16) ensures that the providers compliance with legal and juristic regulations is met. The minimum and maximum number of VMs that can be provisioned by the cloud provider p in location l is limited by the constraint Eq. (17). Constraints in Eqs. (18) and (19) ensure that the cloud providers are having SMI category and attribute score greater than or equal to the consumer requirement. In Eq. (20), the constraint implies that the cloud provider having a total SMI score greater than or equal to the consumer specified score alone is considered for resource provisioning. Constraint Eq. (21) indicates that the variables accept values from a set of nonnegative integers.

A multitude of modeling languages and solvers could be used to solve the specified optimization problem. Our choice of modeling language is AIMMS [4]. It offers some advanced modeling concepts not found in other languages, as well as a full graphical user interface for both developers and end users. It can be used with world class solvers and personal solvers. CPLEX [41] solver is used for MIP formulation in this paper. AIMMS modeler can be incorporated with existing cloud brokers using a Web service-based interface. External database and data sets can be incorporated with AIMMS, and it is adaptable to any architecture.

5 Benders decomposition

In this section, Benders decomposition algorithm [5] is applied to solve the mixed integer programming formulation. Benders decomposition is an approach to solve complicated mathematical programming problems by splitting them into a master problem and multiple subproblems that can be solved in parallel. The master problem contains integer variables while continuous variables become a part of the subproblem. The classic approach of Benders decomposition algorithm is implemented, and it solves an alternative sequence of master and subproblems.

To apply Benders decomposition, it is necessary to divide the variables and constraints of the MIP formulation P(x, y) into two groups. The binary variable \(y_{p}\), together with the constraint Eqs. (15)–(17) and (22), represents the set Y. The continuous variable \(x_{ipl}\), together with the constraint Eqs. (13), (14) and (21), represents the linear part to be dualized. The flowchart of the benders decomposition algorithm is presented in Fig. 3.

Fig. 3
figure 3

Benders decomposition algorithm flowchart

Full size image

The initial master problem \(M(y, m=0)\) does not contain any Benders cuts (i.e., \(m = 0\)) and can be stated as follows.

Minimize:

$$\begin{aligned} \sum _{p} F_{p} \ y_{p} \end{aligned}$$
(24)

subject to : Eqs. (15)–(17), (22).

The problem to be dualized (i.e., linear formation), the equivalent of the inner optimization problem can be stated as follows.

Minimize:

$$\begin{aligned} \sum _{ipl} x_{ipl} \ C^{o}_{ipl} + \sum _{ipl} x_{ipl} \ C^{f}_{ipl} \end{aligned}$$
(25)

subject to: Eqs. (13), (14), (21).

By introducing the dual variables [5] \( \sigma _{il}\) and \( \pi _{ipl}\) corresponding to the two constraints Eqs. (13) and (14), the dual formation \(S(\sigma ,\pi |y)\) of the problem in Eq. (25) can be written as follows.

Maximize:

$$\begin{aligned} \sum _{il} \sigma _{il}\ D_{il} + \sum _{ipl} \pi _{ipl} \ A_{ipl} \end{aligned}$$
(26)

subject to:

$$\begin{aligned}&\sigma _{il} + \pi _{ipl} \ge C^{o}_{ipl} ,\ \forall i, \forall p, \forall l \end{aligned}$$
(27)
$$\begin{aligned}&\sigma _{il} + \pi _{ipl} \ge C^{f}_{ipl} ,\ \forall i, \forall p, \forall l \end{aligned}$$
(28)
$$\begin{aligned}&\sigma _{il} \le 0, \ \pi _{ipl} \ge 0. \end{aligned}$$
(29)

The Benders cuts [5] added to master problem at each iteration are derived from the objective function of the subproblem \(S(\sigma ,\pi |y)\), and a new constraint is derived as follows.

$$\begin{aligned} \sum _{il} \sigma _{il}\ D_{il} + \sum _{ipl} \pi _{ipl} \ A_{ipl} \le m \end{aligned}$$
(30)

The relaxed master problem M(y, m) can be obtained by adding the Benders cuts b to the initial master problem \(M(y,m=0)\) after introducing the set of Benders cuts B generated so far. The resulting master problem is developed by adding the variables stated as follows.

Minimize:

$$\begin{aligned} \sum _{p} F_{p} \ y_{p} + m \end{aligned}$$
(31)

subject to: Eqs. (15)–(17), (22)

$$\begin{aligned} \sum _{il} \sigma _{bil}\ D_{il} + \sum _{ipl} \pi _{bipl} \ A_{ipl} \le m, \forall b \end{aligned}$$
(32)
Table 6 VM demand (\(D_{il}\))
Full size table
Table 7 Criteria for service selection
Full size table
Table 8 Pairwise comparison
Full size table
Table 9 Attribute score of providers
Full size table
Table 10 The weighted attribute score of providers
Full size table
Table 11 Deployment plan (\(x_{ipl}\))
Full size table

6 Numerical evaluation

To validate the MIP formulation, numerical analysis is performed as follows. Due to the unavailability of benchmarking data sets, a synthetic data set is created using the uniform distribution function available in the AIMMS programming language. For the evaluation purpose, the set of cloud providers p is loaded with one thousand cloud providers (i.e., P-1 \(\ldots \) P-1000). The set locations l, set VM class i and set regulation j are loaded with nine locations (i.e., Loc-1 \(\ldots \) Loc-9), nine VM classes (i.e., VM-1 \(\ldots \) VM-9) and nine regulations (i.e., R-1 \(\ldots \) R-9), respectively. The parameter VM availability \(A_{ipl}\) is generated by using pseudo random number generator based on uniform distribution with lower bound set to 40 and an upper bound of 50. The data for parameter \(S_{pc}\) are generated using normal distribution with lower and upper bound set to 4 and 10, respectively. The cost of VM class for per hour usage \(C^o_{ipl}\) in \(\$ \) is derived using uniform distribution with lower and upper bound to be 1 and 4, respectively. The values of parameters \(C^r_{ipl}\), \(C^f_p\) and \(Y_{jpl}\) have been generated using uniform distribution with range of (50, 80), (40, 60) and (0, 1), respectively. The VM demand parameter \(D_{il}\) is defined as in Table 6. The parameter \(u_{ts}\) user required minimum SMI score is set to eighty. For the illustrative purpose, thirteen SMI criteria (in Table 7) and six cloud service providers are considered. The relative weight of the criteria is computed using pairwise comparison method as shown in Table 8. SMI score and the normalized weighted score of the cloud service provider are shown in Tables 9 and 10, respectively. Then, the objective function is solved using Benders decomposition method. The final optimal deployment plan \(x_{ipl}\) is listed in Table 11.

The evaluation is performed with combinations of different required SMI level and limiting resource provisioning in multiple clouds. The required SMI levels such as high performance, high security, balanced and financial stability are considered, while multi-cloud provider deployment limit is varied from 2 to 5. The final results are shown in Fig. 4.

6.1 Comparison with other resource provisioning algorithms

In this section, the proposed resource provisioning method is compared with existing resource provisioning algorithms (i.e., SMICloud [18], OCRP [13], Tordsson et al. [6], Breitgand et al. [9] and Wright et al. [11]). In SMICloud, the cloud providers are evaluated using analytical network processing (AHP) for quantitative attributes, and finally, the service provisioning is done based on value/cost ratio. ORCP considers the trade-off between pay per use and reservation pricing plan with uncertain demand. Tordsson method uses integer programming by considering the hardware configuration, minimum and a maximum number of VMs allocation and load balancing constraints. Policy-based resource provisioning is performed by Breitgand et al. [9]. Wright et al. [11] use constraint optimization engine with two-phase constraint-based discovery approach.

All the methods are coded in AIMMS modeling language with defined input parameters in the above section. The solution from each method yields the deployment plan with optimal provisioning costs. A simulation program is developed to evaluate the solution of each method. The simulation contains multiple iterations of three scenarios. In the first scenario, these methods are evaluated with various SMI requirement levels such as high performance, high security, balanced and financial stability. The optimal deployment plan of the first scenario is shown in Fig. 5. The proposed approach yields cost-effective deployment plan in the least computational time compared to other methods and considers all the attributes and measures to evaluate the providers. The second evaluation scenario is based on the limiting number of cloud providers for resource provisioning. The limit is initially set to two providers and increased up to five and evaluated for all the algorithms and is shown in Fig. 6. Proposed approach and Tordsson method provide the best optimal deployment plan compared to other methods. Finally, these methods are evaluated with different types of jurisdiction and legal regulation requirements which is shown in Fig. 7. For most of the evaluation scenarios, the proposed approach provides best optimal deployment plan in the least time compared to other methods.

Fig. 4
figure 4

Optimal deployment plan with various required SMI levels and multi-cloud provider deployment limit

Full size image

6.2 Sensitivity analysis

Sensitivity analysis investigates the changes in the objective function value of a model as the result of changes in the input data. The marginal values derived from simplex algorithm give the additional information on the variability of an optimal solution of the model to changes in the data. The marginal values are divided into shadow prices (associated with constraints and their right-hand side) and reduced costs (associated with the decision variables and their bounds) [42].

Fig. 5
figure 5

SMI evaluation level

Full size image
Fig. 6
figure 6

Cloud provider limit

Full size image
Fig. 7
figure 7

Legal regulation requirements

Full size image

Shadow price Defined as the rate of change of the objective function from a unit increase in the right-hand side. A positive shadow price of the constraint indicates that the objective function will increase with a unit increase in the right-hand side of the constraint, while negative shadow price indicates that the objective will decrease. The shadow price will be zero for a non-binding constraint since its right-hand side is not constraining the optimal solution [42].

Reduced costs Defined as the rate of change of the objective function for a unit increase in the bounds of a variable. A positive reduced cost of a non-basic variable increases the objective function with a unit increase in the binding domain. The objective function will decrease if a non-basic variable has a negative reduced cost. The reduced cost of a basic variable is zero since its bounds are non-binding and, therefore, do not constrain the optimal solution [42].

In the proposed MIP formulation, sensitivity analysis is done on continuous variables by fixing all the integer variables to an optimal solution. The decision variable \(x_{ipl}\) has positive reduced cost for non-basic variables and zero for the basic variables since its bounds are non-binding. The shadow price of the constraint in Eq. (13) is positive which indicates that the objective will increase with a unit increase in the right-hand side of the constraint (i.e., \(D_{il}\)). The shadow prices of the other constraints are zero since the right-hand side is not constraining the optimal solution.

7 Conclusion and future work

In this work, a novel cloud brokering architecture for optimal deployment of resources among multiple-cloud providers has been proposed. The consumer sends the service request description, which contains required configuration and quantity of virtual machines along with constraints such as location, performance, legal and jurisdictional regulations. Furthermore, consumer assigns weight for the SMI categories and attributes, either by pairwise comparison method or by direct arbitrary weighting method. They also define the required SMI total, category or attribute score based on their application requirement. The proposed broker filters the cloud services based on the constraints given in the service request description. Then, the SMI score for the filtered cloud services is calculated. Optimal deployment plan is obtained by formulating and solving the mixed integer programming. The efficiency of the model is improved by implementing the Benders decomposition algorithm by decomposing into multiple smaller problems and solving it in parallel solver sessions. The evaluation of the proposed model has been performed using numerical analysis and sensitivity analysis to show the robustness and scalability of the proposed model.

In future, dynamic cloud deployment plan for applications with dynamic workloads will be investigated. One such example is a Web server, where the load for the server can vary significantly over time. Another aspect of the study is to minimize the effect of virtual machine migrations, as it involves performance degradation and incurs migration cost.