1 Introduction

Currently cloud computing is considered as the step after clusters or grids for massive and intensive computation in parallel or distributed environments. More precisely, according to Buyya et al. (2009) a cloud can be viewed as a distributed collection of inter-connected virtual computers that can be provisioned by users in a dynamic way. Even if clouds and grids share some characteristics, clouds have some specificities as pointed by Foster et al. (2008): (i) they are ‘massively scalable”, (ii) they are “abstract entities” able to deliver different levels of services, (iii) they are driven by “economies of scale” and (iv) cloud services can be configured and delivered dynamically on demand.

Public cloud providers offer different types of instances of virtual machines (VMs) which can be provisioned to meet the computational demands of users. The prices vary with the type of hardware, the amount of memory required, the operating system, the storage capacity, the availability zone (the geographical area where the services physically stay), the bandwidth, etc.

Cloud providers offer different types of computing power services in a Platform-as-a-Service (PaaS) market and different modalities for hiring VMs. In the “on-demand” modality, instances can be hired or released when needed. In the “reserved” modality users pre-pay for long-term reservations to insure lower tariffs and a better availability. In most cases, there is a maximum number of instances that can be used at a given moment. There is also an opportunistic market in which providers offer instances when they have some excess of capacity (unused resources), such as the Spot instances of Amazon web services or Preemptible VM of Google cloud. And more recently, also “burstable” instances could be hired. A burstable instance provides a baseline level of performance, but that can burst to a higher level for occasional spikes in usage.

From the cloud user’s point of view, the challenge consists in efficiently managing cloud resources while integrating prices and performance criteria. Díaz et al. (2017) solve an allocation problem with reserved VMs and on-demand VMs that takes into account the predicted load of each time slot to minimize the cost while a required level of performance is guaranteed for the service. The load is expressed in a “request per second” metric and it determines firstly the optimal allocation of reserved VMs followed by the number of on-demand VMs if necessary, i.e. the later decision is made for only one time slot depending on its load. Genez et al. (2020) define an integer linear programming model to determine the VMs to be leased from multiple cloud providers to execute workflows while meeting the user’s deadline and minimizing the leasing costs. The loads are scientific workflows decomposed into smaller tasks to be processed in a defined order. In the proposed model only one VM can execute each processing demand.

The focus of this work is related to another very popular type of workload named a Bag-of-Task (BoT). It consists in a set of identical independent tasks that can be operated in parallel. The execution of a BoT usually consumes a large amount of computational time as well as data storage and transfer. Determining the allocation of VM instances to tasks is a scheduling problem which is NP-hard (Abdi et al. 2018; Abed-alguni and Alawad 2021). The goal of the paper is to solve a BoT scheduling problem in a multi-cloud environment.

The primary motivation of this work is to develop an integrated model to consider both cost and makespan criteria in the provisioning and scheduling procedure of BoT applications in multi-cloud environments with capacity constraints using on-demand instances. For this purpose, the contributions of the paper are the following:

  • We define new optimization models integrating three type of decisions: instance type selection, resource scaling and workload allocation.

  • We introduce two optimization criteria: the cost and the execution makespan. A lexicographical approach is used to deal with both criteria.

  • To the difference of previous works by Abdi et al. (2017) and Abdi et al. (2018), tasks of the same BoT are allowed to run on different instance types and the requirement for an instance to be hired from the beginning to the deadline is relaxed.

  • We propose and compare two approaches to introduce capacity constraints into the model.

The paper is organized as follows. Section 2 presents related work. In Sect. 3, we first introduce a new integer programming model to schedule BoTs in a multi-cloud environment in order to minimize the cost of executing the whole set of tasks while satisfying a specified deadline. Next, a new criteria related to the minimization of the makespan is introduced into the objective function. In Sect. 4, we extend the models defined in Sect. 3 to integrate instantaneous capacity constraints and present a comprehensive study of its properties. In Sect. 5, we put into highlight through computational experiments the performance of the proposed models and, in particular, the impact of including instantaneous capacity constraints. Section 6 is devoted to conclusions and future research proposals.

2 Related work

The scheduling of general tasks in (multi) cloud environments has been deeply studied (Alkhanak et al. 2015; Mann 2015; Keivani and Tapamo 2019; Kumar et al. 2019. In a survey and taxonomy of resource optimization for BoTs, (Thai et al. 2018) put into highlight three relevant characteristics to be taken into account: the type of instances to hire, the time during which each instance must work and the workload to be allocated to each hired instance. The special characteristics of BoT problems has made it possible to develop solutions specifically adapted to this field. We next review some recent papers involving a mathematical model. All these articles and the references they contain provide an update on this subject that is currently attracting so much interest.

Wang et al. (2016) present a heuristic algorithm for solving a binary non-linear program whose goal is to minimize the cost of renting VMs while respecting the respective deadlines. Abdi et al. (2017) and Abdi et al. (2018) provide a binary linear programming formulation for the scheduling of BoT applications with a fixed deadline. The model consists in allocating an instance type of one of the clouds to each BoT of the submitted applications. The restrictive assumption that only one instance type can be used allows them to compute the coefficients of the mathematical model. More precisely, the authors determine the number of required VMs to complete the BoT for each instance type. For this purpose, the so-called master VMs are run until the deadline is met. If necessary, the required time of running an extra VM is added to execute remaining tasks. Wang et al. (2019) solve the optimization problem maximizing the global utilization of the servers by using a scheduling method with different degrees of task parallelism. Each task is characterized by a deadline requirement and its parallel degree. More precisely, the number of cores executing the task at a time, is tuned between one and its maximum value according to the available cores of the server during its execution. The authors define a nonlinear program solved by heuristic methods.

Teylo et al. (2021) propose a framework for the execution of BoT applications with deadline constraints including, both spot and on-demand burstable VMs, aiming at minimizing the monetary cost and the execution time. The objective function is a weighted function of both terms. In order to ensure the deadline is respected, the framework considers the possibility of migrating tasks in the hibernated spot instances and those not yet executed to on-demand instances. They use an iterated local search to obtain an approximate solution at an acceptable computing time of the non linear model.

Karaja (2022) focus on solving the BoT scheduling problem in a dynamic multi-cloud environment. They rely on a dynamic batch approach: tasks to be scheduled are collected and scheduled in batches at predefined times. They propose a bi-level multi-follower model. The upper level represents the multi-cloud scheduler that aims to minimize the completion time of the BoT while taking into consideration the budget constraints. The lower level aims to minimize simultaneously the execution cost and the completion time of tasks belonging to the BoT. Metaheuristics are defined to efficiently solve the problem.

Chhabra et al. (2022) also adopt a solution based on a heuristic approach. The problem of scheduling concurrent BoTs is considered as a biobjective optimization problem to minimize the makespan and energy consumption. The biobjective problem is reformulated as a single objective one and a heuristic algorithm is defined on the basis of the fitness function. More precisely, they define the Whale Particle Swarm Optimization algorithm integrating the search mechanism of opposition-based learning and particle swarm optimization mechanisms together with the standard whale optimization algorithm.

Yin et al. (2022) focus on the uncertainty around the real time required to execute a BoT task and the performance of any given VM. The time required can vary depending on many aspects such as the concrete data to be processed. In addition, due to virtualization, the same VM can result in varying execution times for the same task. These authors consider such times as random variables and propose a BoT scheduling problem whose aim is to maximize the profit of the cloud provider while considering the deadline constraints imposed by the user. The problem takes the point of view of a a service cloud provider, and considers the integration of the private cloud resources within public clouds. The stochastic optimization problem is solved by a metaheuristic named Immune Algorithm-based BoT Scheduling (IABS).

All previous works based on mathematical optimization models, linear or non linear, rely on heuristic-based solution approaches. In this work the problem is modeled as a single level biobjective linear mathematical optimization model integrating time dependent decision variables which can be solved by one of the shelf software. Moreover, the biobjective optimization problem is solved by a lexicographic approach leading to the generation of two specific efficient solutions.

3 Mathematical models for scheduling BoT applications

A BoT application is composed of a set of BoTs. In a multi-cloud environment, assigning each application to a single cloud makes sense, as it is assumed that all the BoTs in an aplication share some common data and this single allocation avoids the time and monetary costs of transferring and storing the same data in multiple clouds. BoTs in an application are independent, that is, there are not precedence constraints among them, and all the tasks can be executed in parallel.

In this section we define three models generalizing the integer optimization problem IP-NDS defined by Abdi et al. (2017). They pre-process the problem for a given deadline by computing for each instance type the number of VMs needed to run each BoT. The decisions in model IP-NDS are thus reduced to deciding which type of instance is selected for each BoT. For completeness, this model is included in the Appendix and notations in Table 1. In this paper, we first relax the assumption that only one type of instance can be allocated to a given BoT imposed in Abdi et al. (2017) to formulate the allocation problem. Relaxing this assumption allows to explore all the feasible solutions involving one or more instance types of one cloud, while IP-NDS model only computes a subset of feasible solutions. The second extension consists in considering a makespan criteria into the objective function in place of considering a pre-established deadline D. Finally the third one integrates simultaneously a cost minimization and makespan criteria into the objective function.

As a general rule, in cloud environments, the cost depends on the type of CPU, the amount of memory required, the storage capacity used for data management, etc. Usually, CPU and memory are considered as a whole by the cloud providers in the set of the proposed VM instances. In the following, we use the classical CCU metric developed by CloudHarmony (as mentionned in Abdi et al. 2017) to evaluate the performance of instance types belonging to a variety of clouds rented for executing different tasks. A value of 1 CCU indicates a CPU capacity of 1.0\(-\)1.2 GHz Opteron or 2007 Xeon processor.

Table 1 shows the notations used in the mathematical formulation of the models.

Table 1 Notation used in the mathematical problem formulations
Full size table

Let K be the set of clouds. Each cloud \(k\in K\) provides a set \(J_k\) of VM instance types which are characterized by the fee \(P_{kj}\) of running per time unit, and the performance level, \(ccu_{kj}\), \(k\in K,\,j\in J_k\). We assume a cloud provider-level restriction scenario, i.e there exist some capacity constraints. More precisely, resources are allocated in accordance to the limits defined by providers. For each cloud \(k\in K\), \(U_k\) represents the maximal number of instances allowed by the cloud, while \(U_{kj}\) is the corresponding limit for each instance type \(j\in J_k\).

On the other hand, let I be a set of applications. Each application \(i\in I\) consists of a set of independent BoTs \(\mathcal {B}_i\). A BoT \(b\in \mathcal {B}_i\) includes \(\delta _{ib}\) independent tasks and one task requires \(ET_{ib}\) execution time in a VM with a performance of 1 CCU. Therefore, the execution time of each task of BoT \(b\in \mathcal {B}_i\) on an instance type \(j\in J_k,\,k\in K\) is computed as:

$$\begin{aligned} \tau _{ibkj}=\displaystyle \frac{ET_{ib}}{ccu_{kj}} \end{aligned}$$

Given a deadline D, we define the set \(T=\{1,2,\cdots ,D\}\). Even if Abdi et al. (2017) consider an hour as the time unit, the models are valid for any time unit, such as minutes or seconds. A higher resolution in the time unit increases the cardinality of T. Without loss of generality we next use the term time unit. The number of tasks of BoT \(b\in \mathcal {B}_i,\,i\in I\) executed by an instance type \(j\in J_k,\,k\in K\) during exactly \(d\in T\) time units is computed as follows:

$$\begin{aligned} w_{ibkj}^d=\left\lfloor \displaystyle \frac{d}{\tau _{ibkj}}\right\rfloor \end{aligned}$$

The decision variables are:

$$\begin{aligned} \begin{array}{l} \begin{array}{ll} y_{ik}=&{}\left\{ \begin{array}{ll} 1&{}\hbox {if application }i\hbox { is submitted to cloud }k,\\ 0&{}\hbox {otherwise}, \end{array} i\in I,\,k\in K \right. \end{array} \\ \begin{array}{ll} N_{ibkj}^d=&{}\hbox {Number of VMs of instance type }j\hbox { in cloud }k\hbox { working on BoT }b \\ &{}\hbox {during exactly }d\hbox { time units}, k\in K,\,j\in J_k,\,i\in I,\,b\in \mathcal {B}_i,\,d\in T \end{array} \end{array} \end{aligned}$$

The Multi-Cloud Bot Scheduling (MCBS) problem for a fixed deadline D is formulated as follows:

$$\begin{aligned} \min _{y,N}&\displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{d\in T} (P_{kj}\times d)N_{ibkj}^d \end{aligned}$$
(1a)
$$\begin{aligned} \hbox {s.t.:}&\displaystyle \sum _{k\in K} y_{ik}=1,\quad i\in I&\end{aligned}$$
(1b)
$$\begin{aligned}&N_{ibkj}^d\le U_{kj}y_{ik},\quad i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d\in T&\end{aligned}$$
(1c)
$$\begin{aligned}&\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d\in T} N_{ibkj}^d\le U_k,\quad k\in K&\end{aligned}$$
(1d)
$$\begin{aligned}&\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{d\in T} N_{ibkj}^d\le U_{kj},\quad k\in K,\,j\in J_k&\end{aligned}$$
(1e)
$$\begin{aligned}&\displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d\in T} w_{ibkj}^d N_{ibkj}^d\ge \delta _{ib},\quad i\in I,\,b\in \mathcal {B}_i&\end{aligned}$$
(1f)
$$\begin{aligned}&y_{ik}\in \{0,1\},\quad i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d\in T{} & {} \end{aligned}$$
(1g)
$$\begin{aligned}&N_{ibkj}^d\ge 0,\,\hbox { integer},\quad k\in K,\,j\in J_k,\,d\in T&\end{aligned}$$
(1h)

The objective function (1a) consists in minimizing the cost. Constraints (1b) guarantee that each application is submitted to only one cloud in the federation. If an application is not allocated to a cloud, none of the instances belonging to that cloud are chosen to run tasks from that application by constraints (1c). Constraints (1d) and (1e) are cloud and type capacity constraints. Constraints (1f) guarantee that the whole set of tasks are executed.

The optimal value of model MCBS is lower or equal to the one of model IP-NDS since the feasible set of model IP-NDS is included in the feasible region of model MCBS. Indeed the optimal solution of IP-NDS is always feasible for MCBS. Moreover as shown in Example 1 at the end of this section, the inequality can be strict, which means that model MCBS can provide a better scheduling than model IP-NDS.

Starting from MCBS we can define a model MCBS-M including a makespan criteria in place of a fixed deadline. The makespan is the length of time needed for executing all BoTs. In order to compute it, a binary variable, \(z^d\), is added for each \(d\in T\) to determine whether there is an instance of any type running at that time d. In that case, the variable is equal to 1:

$$\begin{aligned} \begin{array}{ll} z^d=&{}\left\{ \begin{array}{ll} 1&{}\hbox {if any }N_{ibkj}^d>0,\,k\in K,\,j\in J_k,\,i\in I,\,b\in \mathcal {B}_i,\\ 0&{}\hbox {otherwise} \end{array} \right. \\ \end{array} \end{aligned}$$

Then, the objective function of model MCBS-M consists of minimizing the maximum value of \(d\in T\) in which some instance type j is used, \(j\in J_k,\,k\in K\):

$$\begin{aligned} \min _{y,z,N} \max _{d\in T} dz^d \end{aligned}$$
(2)

In order to avoid the non-linearity of the objective function (2), we introduce a continuous decision variable \(\theta\) and we include a set of constraints in model MCBS-M to guarantee that \(\theta\) is greater than or equal to \(dz^d\), for all \(d\in T\). Therefore, the model MCBS-M is linear and it is formulated as follows:

$$\begin{aligned} {{\textbf {MCBS-M}}: }\min _{y,z,N,\theta }&\theta \end{aligned}$$
(3a)
$$\begin{aligned} \hbox {s.t.:}&\displaystyle \sum _{k\in K} y_{ik}=1,\quad i\in I&\end{aligned}$$
(3b)
$$\begin{aligned}&N_{ibkj}^d\le U_{kj}y_{ik},\quad i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d\in T&\end{aligned}$$
(3c)
$$\begin{aligned}&\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d\in T} N_{ibkj}^d\le U_k,\quad k\in K&\end{aligned}$$
(3d)
$$\begin{aligned}&\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{d\in T} N_{ibkj}^d\le U_{kj},\quad k\in K,\,j\in J_k&\end{aligned}$$
(3e)
$$\begin{aligned}&\displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d\in T} w_{ibkj}^d N_{ibkj}^d\ge \delta _{ib},\quad i\in I,\,b\in \mathcal {B}_i&\end{aligned}$$
(3f)
$$\begin{aligned}&N_{ibkj}^d\le U_{kj}z^d,\quad i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d\in T&\end{aligned}$$
(3g)
$$\begin{aligned}&\theta \ge d z^d,\quad d\in T&\end{aligned}$$
(3h)
$$\begin{aligned}&y_{ik}\in \{0,1\},\quad i\in I, k\in K{} & {} \end{aligned}$$
(3i)
$$\begin{aligned}&N_{ibkj}^d\ge 0,\,\hbox { integer}\quad i\in I, b\in \mathcal {B}_i, k\in K,\,j\in J_k, d\in T&\end{aligned}$$
(3j)
$$\begin{aligned}&\theta \ge 0&\end{aligned}$$
(3k)

Note that if there exists a variable \(N_{ibkj}^d>0\) then at least one VM is active during d time units. The corresponding constraint (3g) guarantees that \(z^d=1\) and by constraint (3h) \(\theta \ge d\). It results that the makespan will be more than d time units provided that there are VMs working d time units. The minimum makespan, denoted by \(\theta ^*\), is thus implicitly computed. Constraint (3k) is a sign constraint on the decision variable \(\theta\). Constraints (3b–3f) and (3i–3j) are the same as in MCBS problem.

Finally, both cost and deadline minimization can be addressed simultaneously from a biobjective perspective. In a biobjective problem, it is generally not possible to compute a single solution generating the best value for both objectives. In this context, reducing the makespan leads to an increase in cost. A solution is efficient for a biobjective problem if there is no other feasible solution which a lower value for both objective functions, one of which is strictly lower. The set of efficient solutions provides different trade-offs between cost and makespan. In particular, a lexicographic approach can be used to compute an efficient solution (Ehrgott 2005). The lexicographic approach is a nonscalarizing method for finding efficient solutions based on the ordering of the objectives according to some priorities. Model MCBS-1 prioritizes cost over makespan and the MCBS-2 model takes the reverse order. Thus, this approach allows either i) to minimize the makespan in which the tasks are executed at minimum cost with a given deadline D:

$$\begin{aligned} {{\textbf {MCBS-1}}:Lex}\min _{y,z,N,\theta }\left( \displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{d\in T} (P_{kj}\times d)N_{ibkj}^d,\,\theta \right) , \end{aligned}$$

or ii) to minimize the cost at the lowest feasible makespan:

$$\begin{aligned} {{\textbf {MCBS-2}}:Lex}\min _{y,z,N,\theta }\left( \theta ,\,\displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{d\in T} (P_{kj}\times d)N_{ibkj}^d\right) . \end{aligned}$$

In the next example, we compare the solutions provided by the models MCBS and MCBS-M to the solution resulting from the IP-NDS model.

Example 1

Table 2 displays the input data with one cloud (\(\vert K\vert =1\)), three instance types (\(\vert J_1\vert =3\)) and one application (\(\vert I\vert =1\)) with two BoTs (\(\vert \mathcal {B}_1\vert =2\)). Let \(\hbox {VM}_{j}\) denote a VM of instance type j.

Table 2 Instance types specifications and BoTs of the Example 1
Full size table

Auxiliary parameters are required to formulate IP-NDS model: \(\eta _{bj}\) is the maximum number of tasks of BoT \(b\in \mathcal {B}_1\) solved by one \(\hbox {VM}_{j}\), \(j\in J_1\) for a given deadline D; although a master VM is run until the deadline is met, \(T_{bj}\) is the exact running time ot the master VMs and \(\nu _{bj}\) is the number of required master VMs of instance type \(\hbox {VM}_{j}\) for executing tasks of BoT \(b\in \mathcal {B}_1\). If \(\tau _{bj}\eta _{bj}<\delta _{b}\) the master \(\hbox {VM}_{j}\)s cannot carry out the computation of all tasks of BoT \(b\in \mathcal {B}_1\). In this case, one extra \(\hbox {VM}_{j}\) is required during \(TR_{bj}\) in time units. These auxiliary paramaters are mathematically defined in the Appendix. For the example the corresponding values are defined in Table 3. The first column gives the name of the instance; columns 2 to 7 (resp. 8 to 13 ) present the corresponding data to \(\hbox {BoT}_1\) (resp. \(\hbox {BoT}_2\)).

The optimal solution of IP-NDS is displayed in bold in Table 3. The minimum cost is equal to 40.8$. Both BoTs are executed by \(\hbox {VM}_1\): 5 master and 1 extra (5 time units) for \(\hbox {BoT}_1\) and 1 master and 1 extra (3 time units) for \(\hbox {BoT}_2\).

Table 3 Auxiliary parameters in the IP-NDS model for a deadline \(D=10\) and the optimal solution with cost=40.8$
Full size table

Table 4 shows the value of the auxiliary parameters \(\{w_{bj}^d\}\) involved in problem MCBS. For each number of time units shown at the first row, the number of tasks executed by each VM is computed. As an example, a \(\hbox {VM}_1\) instance executes during 5 time units 41 tasks of \(\hbox {BoT}_2\).

Table 4 Auxiliary parameters of the MCBS model for Example 1
Full size table

In the solution proposed by MCBS model, there is an intensive use of \(\hbox {VM}_1\) instances for both, \(\hbox {BoT}_1\) and \(\hbox {BoT}_2\): 6 instances executing tasks for \(\hbox {BoT}_1\) during 9 time units, solving 600 tasks with a cost of 32.4$; 1 instance executing \(\hbox {BoT}_2\) tasks for 9 time units and another one for 3 time units, solving 75 and 25 tasks at a cost of 5.4$ and 1.8$, respectively. The total cost is 39.6$  which is lower than 40.8$. Moreover, the makespan is 9 time units, one time unit less than the deadline \(D=10\) assumed in IP-NDS model.

Let us define upper bounds on the number of VMs of each instance type: \(U_1=U_2=4\) and \(U_3=2\). With these constraints, IP-NDS problem is no feasible any more, although MCBS problem finds a solution for a deadline \(D=10\). The optimal cost is 42.2$. Tasks of \(\hbox {BoT}_1\) are executed by 4 \(\hbox {VM}_1\) instances during 10 time units (444 tasks), 2 \(\hbox {VM}_2\) during 10 time units (132 tasks) and 1 \(\hbox {VM}_3\) during 10 time units (22 tasks) and another one during 1 time unit (2 tasks); \(\hbox {BoT}_2\) is executed by 2 \(\hbox {VM}_2\) during 10 time units (100 tasks). The constraints on the number of resources are satisfied since 4 \(\hbox {VM}_1\), 4 \(\hbox {VM}_2\) and 2 \(\hbox {VM}_3\) have been rented.

Finally by solving model MCBS, the smallest feasible deadline within which the related assignment problem is feasible given the upper bounds: \(U_1=U_2=U_3=10\) and \(U=20\) is \(D=5\) with a cost of 42.8$. For level constraints of the provider more restrictive, \(U_1=U_2=4\) and \(U_3=2\), the smallest deadline threshold is 10 time units. If the constraints on the number of resources are \(U_1=4,\,U_2=7\) and \(U_3=4\) and the deadline is \(D=10\), model IP-NDS is not feasible any more and the optimal solutions for the models defined in this section are displayed in Table 5. Note that the lexicographic approach provides two efficient points with both objective function values, the cost and the makespan, equal to \((41.6,\,9)\) and \((43,\,8)\) by solving models MCBS-1 and MCBS-2, respectively.

Table 5 Optimal solutions and optimal values for Example 1 with upper bounds \(U_1=4,\,U_2=7\) and \(U_3=4\) and deadline \(D=10\)
Full size table

4 Global and instantaneous capacity constraints

From a practical point of view cloud providers limit the number of VMs available to a user at a given time even if from a theoretical point of view clouds could be able to provision an unlimited number of resources. Of course defining bounds on the number of resources strongly impacts the total cost and the feasible makespan since it directly affects the set of feasible allocations. The limits can be defined globally or instantaneously.

For example a global limit on the number of instances, independently of their types (50 instances per cloud service) is required by cloud Microsoft Azure. In the case of Amazon EC2, until September 2019 there were a limit of 20 VMs. Since October 2019, EC2 defines vCPU-based (virtual CPU) instance limits. Each instance has a (different) number of vCPUs, depending on its type. The total number of instances depends on their types, and any combination is allowed while the total number of vCPUs does not exceed the limit. This is also the case of cloud Google Cloud (GC). The bound on the number of CPUs allowed refers to the total number of virtual CPUs in all VM instances in a region. Although global capacity constraints are common in the literature, the attention has been focused more recently on instantaneous bounds limiting the number of resources that can be used at a given time. Capacity constraints can be more constraining in the case of using private clouds, which usually consist of a more restrictive set of resources. Also the fact of having a set of previously hired instances that could be used for solving the problem along the whole makespan is a source for constraints.


In this section we extend models MCBS and MCBS-M to integrate instantaneous resource constraints in place of global resource ones. The decision variable \(N_{ibkj}^d\) is replaced by \(N_{ibkj}^{dt_0}\), \(k\in K\), \(j\in J_k\), \(i\in I\), \(b\in \mathcal {B}_i\), \(d\in T\), \(t_0\in T\) to point out the starting time \(t_0\) of the VMs of instance type j in cloud k for execution of tasks of BoT b during exactly d time units. The new model MCBS-ins formulation is:

$$\begin{aligned} \min _{y,N}&\displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{d,\,t_0\in T} (P_{kj}\times d)N_{ibkj}^{dt_0} \end{aligned}$$
(4a)
$$\begin{aligned} \hbox {s.t.:}&\displaystyle \sum _{k\in K} y_{ik}=1,\quad i\in I&\end{aligned}$$
(4b)
$$\begin{aligned}&N_{ibkj}^{dt_0}\le U_{kj}y_{ik},\quad i\in I,\,B\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d,t_0\in T&\end{aligned}$$
(4c)
$$\begin{aligned}&\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{t_0=1}^t\displaystyle \sum _{d=t+1-t_0}^D N_{ibkj}^{dt_0}\le U_k,\quad k\in K,\,t\in T&\end{aligned}$$
(4d)
$$\begin{aligned}&\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{t_0=1}^t\displaystyle \sum _{d=t+1-t_0}^D N_{ibkj}^{dt_0}\le U_{kj}, \quad k\in K,\,j\in J_k,\,t\in T&\end{aligned}$$
(4e)
$$\begin{aligned}&\displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d\in T}\displaystyle \sum _{t0=1}^{D-d+1} w_{ibkj}^{d} N_{ibkj}^{dt_0}\ge \delta _{ib}, \quad i\in I,\,b\in \mathcal {B}_i&\end{aligned}$$
(4f)
$$\begin{aligned}&y_{ik}\in \{0,1\}\quad i\in I,\,k\in K{} & {} \end{aligned}$$
(4g)
$$\begin{aligned}&N_{ibkj}^{dt_0}\ge 0\,\hbox { integer} \quad i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d,t_0\in T&\end{aligned}$$
(4h)

Several constraints from MCBS have been adapted. Constraints (4d) and (4e) ensure that for each time period \(t\in T\), the number of active VMs is lower or equal to the corresponding upper bound. Note that a VM started at time \(t_0\) for d time units is running at time t if and only if \(t_0+d-1\ge t\). An instance \(j\in J_k,\,k\in K\) executes \(w_{ibkj}^d\) tasks of BoT \(b\in \mathcal {B}_i\) if and only if \(t_0+d-1\le D\). Hence the index \(t_0\) of the last sum in constraints (4f) goes from 1 to \(D-d+1\). Since an instance running beyond D does not make sense, we can add the following set of constraints:

$$\begin{aligned}&N_{ibkj}^{dt_0}=0,\, i\in I,\,B\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d\in T,t_0\ge D-d+2&\end{aligned}$$
(4i)

Model MCBS-ins is less restrictive than MCBS and can result in an optimal solution with lower cost since the upper bound is applied to each time period.

Let \(v({\textbf {M}})\) denote the optimal value of a model M.

Proposition 4.1

\(v({\textbf {MCBS-ins}})\le v({\textbf {MCBS}})\) and the inequality can be strict.

Proof

Let \(({\overline{y}},{\overline{N}})\) be an optimal solution of model MCBS. We define:

$$\begin{aligned} \begin{array}{ll} {\widehat{y}}_{ik}={\overline{y}}_{ik},&{}i\in I,\,k\in K\\ {\widehat{N}}_{ibkj}^{dt_0}=\left\{ \begin{array}{ll} {\overline{N}}_{ibkj}^{d}&{}\hbox {if }t_0=1\\ 0&{}\hbox {otherwise} \end{array} \right. &{}i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d,t_0\in T\\ \end{array} \end{aligned}$$

It easy to see that \({\widehat{y}}\) satisfies constraints (4b) and (4g). Since \({\overline{N}}\) meets constraints (1c) and (1h), then \({\widehat{N}}\) satisfies constraints (4c) and (4h). Moreover, for each \(k\in K\) and \(t\in T\):

$$\begin{aligned} \displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{t_0=1}^t\displaystyle \sum _{d=t+1-t_0}^D {\widehat{N}}_{ibkj}^{dt_0}=\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d=t}^D {\overline{N}}_{ibkj}^{d} \le \displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d=1}^D {\overline{N}}_{ibkj}^{d}\le U_k \end{aligned}$$

on the basis of constraints (1d). Therefore, constraints (4d) are satisfied. In a similar way, constraints (1e) ensure that \({\widehat{N}}\) satisfies constraints (4e). Constraint (4f) for \(i\in I\) and \(b\in \mathcal {B}_i\):

$$\begin{aligned} \displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d\in T}\displaystyle \sum _{t0=1}^{D-d+1} w_{ibkj}^{d} {\widehat{N}}_{ibkj}^{dt_0}=\displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{d\in T} w_{ibkj}^{d} {\overline{N}}_{ibkj}^{d}\ge \delta _{ib} \end{aligned}$$

where the constraints (1f) have been applied in the last inequality.

In conclusion, any feasible solution for MCBS provides a feasible solution for model MCBS-ins. Moreover, since the objective function (4a) only depends on the duration while the machine is working on some BoT:

$$\begin{aligned} \displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{d,\,t_0\in T} (P_{kj}\cdot d){\widehat{N}}_{ibkj}^{dt_0}=\displaystyle \sum _{k\in K}\displaystyle \sum _{j\in J_k}\displaystyle \sum _{i\in I}\displaystyle \sum _{b\in \mathcal {B}_i}\displaystyle \sum _{d\in T} (P_{kj}\cdot d){\overline{N}}_{ibkj}^{d} \end{aligned}$$

Hence, \(v({\textbf {MCBS-ins}})\le v({\textbf {MCBS}})\). Example 1 (see below) provides an example for which the inequality is strict. \(\square\)

Example 1 (continued). For MCBS-2 model, the optimal resources allocation, where 4 \(\hbox {VM}_1\), 7 \(\hbox {VM}_2\) and 2 \(\hbox {VM}_3\) are used, is displayed in Figure (1a). To reach this solution, first, the model MCBS-M with \(U_1=4,\,U_2=7,\,U_3=4,\,U=20\) is solved to determine the minimum makespan, \(\theta ^*=8\). Then the model MCBS for \(D=8\) provides the solution with the lowest cost, 43$ , displayed in Table 5.

Fig. 1
figure 1

Optimal resources allocation for \(D=8\) time units and \(U_1=4,\,U_2=7,\,U_3=4,\,U=20\) VMs

Full size image

The optimal solution of model MCBS-ins has a cost equal to 42.4$ which is strictly lower than \(v({\textbf {MCBS}})\). Figure (1b) displays the optimal resources allocation. This solution, where 9 \(\hbox {VM}_1\), 14 \(\hbox {VM}_2\) and 2 \(\hbox {VM}_3\) are allocated, is not a feasible solution for MCBS. However, the number of instances working in any time period is at most 4 for \(\hbox {VM}_1\), 7 for \(\hbox {VM}_2\) and 2 for \(\hbox {VM}_3\).

The next proposition allows to fix N-variables to zero in order to reduce the size of the problem.

Proposition 4.2

Any optimal solution of MCBS-ins satisfies that for \(i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,t_0\in T\):

$$\begin{aligned} N_{ibkj}^{dt_0}=0, \,\,d\in T,\,d\ne \left\lceil m\cdot \tau _{ibkj}\right\rceil ,\,m=1,2,\dots \\ \end{aligned}$$

Proof

Let us assume that there exists an optimal solution \((\overline{y},{\overline{N}})\) of MCBS-ins and a variable \(\overline{N}_{ibkj}^{d_0t_0}>0\) with \(d_0\ne \left\lceil m\cdot \tau _{ibkj}\right\rceil\) for a \(m\in \mathbb {N}\). Suppose that \(\left\lceil m\cdot \tau _{ibkj}\right\rceil < d_0 < \left\lceil (m+1)\cdot \tau _{ibkj}\right\rceil\). We can build a feasible solution \(({\overline{y}},{\widehat{N}})\), with \({\widehat{N}}_{ibkj}^{dt_0}=\overline{N}_{ibkj}^{dt_0}\), except for \({\widehat{N}}_{ibkj}^{d_0t_0}=0\) and \({\widehat{N}}_{ibkj}^{\left\lceil m\cdot \tau _{ibkj}\right\rceil t_0}={\overline{N}}_{ibkj}^{d_0t_0}\). This feasible solution corresponds to a better objective function value:

$$\begin{aligned} w_{ibkj}^{d_0}=w_{ibkj}^{\left\lceil m\cdot \tau _{ibkj}\right\rceil }\hbox {and} (P_{kj}\cdot \left\lceil m\cdot \tau _{ibkj}\right\rceil ) {\widehat{N}}_{ibkj}^{\left\lceil m\cdot \tau _{ibkj}\right\rceil t_0}<(P_{kj}\cdot d_0 ){\overline{N}}_{ibkj}^{d_0t_0} \end{aligned}$$

This is in contradiction with the optimality of \((\overline{y},{\overline{N}})\). \(\square\)

In proposition 4.2, we consider tasks requiring more than one time period to be solved by an instance type. Any optimal solution will use that instance type during a time which is a multiple of the time required for a task. The same arguments hold for MCBS.

Lemma 4.3

Any optimal solution of MCBS satisfies that for \(i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,d\in T\,\):

$$\begin{aligned} N_{ibkj}^{d}=0, \, d\ne \left\lceil m\cdot \tau _{ibkj}\right\rceil ,\,m=1,2,\dots \\ \end{aligned}$$

The next proposition aims to reduce the symmetry introduced by model MCBS-ins. More precisely, each feasible solution leads to alternative feasible solutions with the same objective function value by splitting the time during a machine is working as long as the number of tasks executed is not improved when the length of working time is increased. This result is not valid for model MCBS since the upper bound refers to the number of instances.

Proposition 4.4

An optimal solution of MCBS-ins exists such that for \(i\in I,\,b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\,t_0\in T\) such that \(\tau _{ibkj}<1\):

$$\begin{aligned} N_{ibkj}^{dt_0}=0, d\in T\hbox { such that } d\ge 2 \hbox { and } w_{ibkj}^d-w_{ibkj}^{d-1}\le w_{ibkj}^1\\ \end{aligned}$$

Proof

Suppose there exists an optimal solution \(({\overline{y}},{\overline{N}})\) of MCBS-ins which does not satisfy the statement conditions.

Since \(({\overline{y}},{\overline{N}})\) does not satisfy the statement conditions, there exists a variable \({\widehat{N}}_{ibkj}^{d_0t_0}>0\) with \(d_0\ge 2\) and \(w_{ibkj}^{d_0}-w_{ibkj}^{d_0-1}\le w_{ibkj}^1\) for some \(d_0\). Notice that \(w_{ibkj}^d\) defines the number of tasks of BoT \(b\in \mathcal {B}_i\) executed by a instance type j during exactly d time units, \(k\in K,\,j\in J_k,\,i\in I,\,b\in \mathcal {B}_i,\,d\in T\). When \(\tau _{ibkj}<1\), the number of tasks of BoT \(b\in \mathcal {B}_i\) executed by an instance type j during one time unit is at least one:

$$\begin{aligned} w_{ibkj}^d=\left\lfloor \displaystyle \frac{d}{\tau _{ibkj}}\right\rfloor \Rightarrow d=w_{ibkj}^d\tau _{ibkj}+\epsilon _d\hbox { with }\epsilon _d<\tau _{ibkj} \end{aligned}$$

For \(d\ge 2\):

$$\begin{aligned} \begin{array}{ll} d&{}=(d-1)+1=w_{ibkj}^{d-1}\tau _{ibkj}+\epsilon _{d-1}+1= w_{ibkj}^{d-1}\tau _{ibkj}+\epsilon _{d-1}+w_{ibkj}^1\tau _{ibkj}+\epsilon _1\\[10pt] &{}=(w_{ibkj}^{d-1}+w_{ibkj}^1)\tau _{ibkj}+(\epsilon _{d-1}+\epsilon _1) \end{array} \end{aligned}$$

Moreover,

$$\begin{aligned} 0\le \epsilon _{d-1}+\epsilon _1< 2\tau _{ibkj} \end{aligned}$$

Therefore, for \(d\ge 2\):

$$\begin{aligned} w_{ibkj}^d=\left\{ \begin{array}{ll} w_{ibkj}^{d-1}+w_{ibkj}^1&{}\hbox { if }\epsilon _{d-1}+\epsilon _1<\tau _{ibkj}\\[5pt] w_{ibkj}^{d-1}+w_{ibkj}^1+1&{}\hbox { if }\epsilon _{d-1}+\epsilon _1\ge \tau _{ibkj} \end{array} \right. \end{aligned}$$

Since \(w_{ibkj}^{d_0}-w_{ibkj}^{d_0-1}\le w_{ibkj}^1\) and \(d_0\ge 2\), we conclude that \(w_{ibkj}^{d_0}=w_{ibkj}^{d_0-1}+w_{ibkj}^1\).

We can build a feasible solution \(({\overline{y}},{\widehat{N}})\), with \({\widehat{N}}_{ibkj}^{dt_0}={\overline{N}}_{ibkj}^{dt_0}\), except for:

$$\begin{aligned} \begin{array}{l} {\widehat{N}}_{ibkj}^{d_0t_0}=0\\ {\widehat{N}}_{ibkj}^{(d_0-1)t_0}={\overline{N}}_{ibkj}^{d_0t_0}+{\overline{N}}_{ibkj}^{(d_0-1)t_0}\\ {\widehat{N}}_{ibkj}^{1(t_0+d_0-1)}={\overline{N}}_{ibkj}^{d_0t_0} \end{array} \end{aligned}$$

The new feasible solution has the same objective value function than \(({\overline{y}},{\overline{N}})\). In the case of \(\widehat{N}_{ibkj}^{(d_0-1)t_0}>0\) such that \(d_0-1\ge 2\) and \(w_{ibkj}^{d_0-1}-w_{ibkj}^{d_0-2}\le w_{ibkj}^1\), we repeat the process for \(d_0-1\). \(\square\)

Model MCBS-M can be extended to the MCBS-Mins problem to include instantaneous resource constraints. The mathematical formulation is written as follows:

$$\begin{aligned} \min _{y,N,\theta ,z}&\theta \end{aligned}$$
(5a)
$$\begin{aligned} \hbox {s.t.:}&(4b)-(4h)\end{aligned}$$
(5b)
$$\begin{aligned}&N_{ibkj}^{dt_0}\le U_{kj}z_{t_0+d-1},i\in I,\,\,\quad b\in \mathcal {B}_i,\,k\in K,\,j\in J_k,\nonumber \\&d,\;t_0\in T,\,\quad t_0+d-1\le D\end{aligned}$$
(5c)
$$\begin{aligned}&\theta \ge dz^d,\,\quad d\in T\end{aligned}$$
(5d)
$$\begin{aligned}&z^d\in \{0,\,1\},\,\quad d\in T \end{aligned}$$
(5e)

5 Computational experiments

In this Section, we present the results of some experiments carried out on a synthetic dataset generated from real data. Table 6 provides a summary of the above proposed models. The goal is to analyse in a realistic case the differences between the models and discuss their advantages and disadvantages. All models have been solved by using CPLEX 12.9.0.0. in a Intel(R) Core(TM) i7-9700 CPU 3.00GHz 32.0 GB RAM with Windows 10.

Table 6 Summary of the proposed models for Multi-Cloud Bot Scheduling
Full size table

5.1 Dataset

The test instances have been defined in accordance with the features of instances defined by well-known public cloud providers. They usually offer some specific VMs better adapted for intensive computing. This is the case of the C4, C5 and C5n families for Amazon EC2, the Fsv2, Fs and the F series for MS-Azure and the C2 Compute Optimized family for Google Cloud. Table 7 shows the synthetic list of cloud providers and instances used for the experiments. We are considering three virtual cloud providers, namely \(\hbox {C}_1\), \(\hbox {C}_2\), \(\hbox {C}_3\), offering, respectively, 6, 4 and 5 VMs types. Let \(\hbox {VM}_{kj}\) denotes a VM of instance type j in cloud \(\hbox {C}_k\).

Table 7 Instance types specifications of the three clouds
Full size table

Like EC2, we define the resource limit constraints in terms of vCPUs. Let \(U_k\) be the CPU quota for cloud \(k\in K\) and \(vc_{kj}\) be the number of vCPU of the instance type \(j\in J_k\), \(k\in \{{\hbox {C}_1,\hbox {C}_2,\hbox {C}_3}\}\). Notice that, for a given upper bound \(U_k\), an upper bound on the number of VMs of instance type j is computed as follows:

$$\begin{aligned} U_{kj}=\left\lfloor \displaystyle \frac{U_k}{vc_{kj}}\right\rfloor ,k\in K,\,j\in J_k \end{aligned}$$

Therefore, the total number of VMs depends on their types and any combination is allowed while the total number of vCPUs does not exceed the limit. Last column of Table 7 shows the resource limit constraint for each instance type when the upper bound on the total number of vCPUs of each cloud is 100. Regarding the applications and BoTs, they have not established in an arbitrary way, but they have been chosen from the literature (Juve et al. 2013) so as to have a variety in terms of running times and number of tasks. We consider three applications \(I=\{A_1,\,A_2,\,A_3\}\) and three BoTs in each application. BoT in application \(A_i\) are denoted by \(B_{ib}\) with \(b\in \{1,2,3\}\). As in Abdi et al. (2017), we define the computational size of an application (CSA) as the execution time of its tasks on a VM with a performance of 1 CCU. The data are summarized in Table 8.

Table 8 Tasks in the BoT workflows of each application
Full size table

5.2 Computing the minimum makespan for a set of given resource limits

In the resource allocation problem, once the availability for each cloud provider is known, we compute the minimum makespan, that is, a lower bound on the deadline needed to deal with all BoTs. For the sake of clarity, we have run the experiments with the same resource limits for the three clouds, that is, \(U_k=U\), \(k\in \{{\hbox {C}_1,\hbox {C}_2,\hbox {C}_3}\}\). The experiments are carried out for different values of U \(\in \{25,\,50,,\dots ,200,\,225\}\). By solving MCBS-M and MCBS-Mins models we determine the minimum makespan ensuring that all the jobs could be finished.

Table 9 shows in the second and third columns, the minimum makespan \(\theta ^*\) and the CPU times required to solve model MCBS-M, respectively. The fourth and fifth columns include the same values for model MCBS-Mins.

Table 9 Minimum makespan in time units and CPU times in seconds for each upper bound on the number of instances
Full size table

Model MCBS-M requires a smaller computation time than MCBS-Mins. Since the feasible region of MCBS-Mins includes the feasible region of MCBS-M, this model may provide a worse solution than the one of MCBS-Mins model as for \(N=125\). This behaviour is described in Fig. 2. When the resource limits are lower, the results given by MCBS-Mins are sharply better than those of MCBS-M. However, the maximum difference is 3 time units. As the availability of resources increases, the reduction of the deadline is slower, resulting in the same solution for both models.

Fig. 2
figure 2

Minimum makespan in time units vs Upper resources limit on VMs

Full size image

5.3 Effect of the deadline in the cost

In Sect. 5.2 we have seen how an upper bound on the number of resources of each cloud provider allows to determine the minimum deadline to operate all the BoTs by solving models MCBS-M and MCBS-Mins. However, when the deadlines are extended, it is interesting to identify their impacts on the costs. In this subsection, we consider an upper bound \(U=50\) on the capacity of resources of the three cloud providers and compute the minimal cost for several values of D. We solve model MCBS, whose capacity constraint is globally imposed as the number of instances allowed along the deadline period, and model MCBS-ins in which the limits refer to the number of instances allowed to work in simultaneous way. We also compare these results with the value provided by model IP-NDS.

Table 10 displays the minimum cost and the computational time in seconds for the three models and four deadlines. IP-NDS model is a very simple allocation problem which is solved in less than one second. As in the previous subsection, model MCBS requires a much lower computation time than model MCBS-ins. The running time of CPLEX is limited to 300 s. When CPLEX is interrupted after this limit, the best integer solution is provided and we include between parenthesis the MIP relative gap which refers to the quotient of the best integer objective minus the objective of the best node remaining between the best integer objective. For instance, for a deadline \(D=25\) both models MCBS and MCBS-ins provide the same objective value, 4285.464, however only model MCBS guarantees that this value is the optimal one. In the case of MCBS-ins, the relative gap of 0.01 means that CPLEX has found a feasible integer solution in the 1 percent of the optimal one.

Table 10 Minimum cost ($) and CPU times (seconds) vs the deadline (\(U=50\) VMs)
Full size table
Fig. 3
figure 3

Minimum cost ($) vs deadline (time units) (\(U=50\) VMs)

Full size image

Figure 3 displays the minimum cost behaviour as the deadline increases. Model MCBS-ins results in a lower cost value than MCBS for some deadlines. As expected, with a global upper bound on the number of vCPUs (model MCBS), a higher deadline is required to generate a cost similar to the one provided by MCBS-ins with shorter deadlines. As shown in Fig. 3 both MCBS and MCBS-ins are non-increasing monotone functions. However, IP-NDS makes use of the full deadline to allocate the instances which may result in a non monotone cost function. For MCBS-ins, the optimal value displayed is guaranteed to be the optimal only for \(D\in \{10,11,12,13,23,27,28\}\). The remaining values refer to the best integer solution value. The difference between the costs of the solutions provided by MCBS and MCBS-ins ranges from 0 (\(D\ge 19\)) to 1.26% (\(D=10,\,11\)).

5.4 Results from a biobjective perspective

Models MCBS-1 and MCBS-2 integrate jointly the cost and the makespan. In Sect. 5.2, the minimum makespan has been computed for a given set of resource limits, while in Sect. 5.3 the minimum cost is obtained for several values of deadline. The solutions are optimal when only one of the criteria is considered. Following, models MCBS-1 and MCBS-2 are applied for computing the corresponding efficient solutions.

Table 11 shows in the first and second columns the bounds considered for the number of resources and the deadline, respectively. The third and fourth columns include the cost and the makespan for the efficient solution computed by model MCBS-1 which first optimizes the cost. Model MCBS-2 first optimizes the makespan which is displayed at the fifth column, while the sixth column shows the minimum cost given that optimal makespan. Each row provides two efficient solutions for a given resource limit and a deadline. For instance, for \(U=25\) and \(D=25\), the minimum cost is 4403.772$  and to achieve this cost at least 23 units of time are required. The only way to reduce the makespan and reach the minimum value of 20 time units is to increase the cost. The minimum cost required to perform BoTs in 20 time units is 4430.028$.

Table 11 Efficient solutions for the biobjective problem in which the cost and the makespan are minimized
Full size table
Fig. 4
figure 4

Cost ($) and makespan (time units) of efficient solutions for several values of U (limit on the number of VMs) and deadlines D (time units) obtained by solving models MCBS-1 and MCBS-2

Full size image

The values of the cost and the makespan of the efficient solutions are displayed in the scatterplot displayed in Fig. 4. With a deadline \(D=25\), increasing the resource limits above \(U=175\) does not provide better solutions neither in terms of the cost nor the makespan. Therefore the figure does not include the values corresponding to the \(U\in \{200, 225\}\). Model MCBS-1 first optimize the cost, therefore the corresponding efficient solutions provides lower values than the ones provided by model MCBS-2. On the contrary, the lower values of the makespan are obtained by solving model MCBS-2.

Notice that, for \(U=50\) and deadlines \(D\in \{10,\,15,\,20,\,25\}\), the minimum makespan is 10 time units and the minimum cost with \(D=10\) is 4443.108$. For this reason, model MCBS-2 provides the same values of both objectives for all deadlines. However, model MCBS-1 provides better cost values as the deadline increases, while the makespan gets worse. For deadline \(D=25\), the efficient solution obtained by solving MCBS-2 with \(U=100\) is better than the one with \(U=75\). In this case, increasing the upper resources limit allow to decrease the cost and the makespan simultaneously. However, the efficient solution obtained by solving MCBS-2 with \(U\in {100,\,150,\,175}\) are not comparable.

5.5 Analysis of the two approaches when introducing capacity constraints

The previous computational experience raises the question of the impact to consider instantaneous or global capacity constraints. Notice that, given a deadline D, to make the best possible use of resources, the instances will be hired as long as possible in order to reduce the idle time in each time unit. When the instances are used until the deadline is met, like master VM in IP-NDS model, the instantaneous and global capacity constraints would coincide.

From the above results, it is clear that the MCBS-ins and MCBS-Mins models can provide better results than the MCBS and MCBS-M models, respectively. Note that the feasibility sets of MCBS and MCBS-M are included in the corresponding ones of MCBS-ins and MCBS-Mins. This is in line with the intuitive idea that a capacity constraint on each time unit offers more flexibility than a global capacity constraint.

For illustrative purpose, we set \(U=50\) and \(D=17\) and we solve both models. Table 12 shows from the second to the fourth columns the optimal solution (\(S_1\)) provided by model MCBS with a cost 4335.576. The best integer solution (\(S_2\)) provided by model MCBS-ins when interrupted after 300 s with a cost 4310.52 is displayed from the fifth to eighth columns. As both solutions are very similar, we have highlighted the differences in bold. Solution \(S_1\) uses 48 and 6 vCPUs from clouds \(\hbox {C}_3\) and \(\hbox {C}_1\), respectively. For solution \(S_2\) the number of vCPUs from cloud \(\hbox {C}_3\) goes from 16 at the 17th time period to 48 vCPUs instantaneously used at 6 out of 17 times periods. Notice that \(S_2\) is not a feasible solution of MCBS since for cloud \(\hbox {C}_3\) the total number of vCPUs are 80. On the other hand, the instantaneous capacity constraint makes possible to execute all the tasks in bag \(B_{23}\) paying for 80 time units of instances \(\hbox {VM}_{31}\) in solution \(S_2\) instead of 81 time units needed in solution \(S_1\).

Table 12 Description of the solutions provided by models MCBS and MCBS-ins with \(D=17\) time units
Full size table

From a mathematical point of view, a global capacity constraint only relies on the duration of each machine, which reduces the number of integer variables in the MCBS and MCBS-M models. However, to consider instantaneous capacity constraint requires to define the starting time of each instance to be able to evaluate at each time period then number of instances used from each cloud. Propositions 4.2 and 4.3 allow to reduce the number of integer variables. For illustrative purposes, we consider a capacity \(U=50\) for all the clouds and compute the number of binary and integer variables for each model. Table 13 displays these numbers for D varying from 10 to 25. The seventh column shows the number of integer variables different from 0 after applying both Propositions. Even if the number of integer variables is sharply reduced by up to 5%, the computational time is not decreased.

Table 13 Number of binary and integer variables in the models
Full size table

Bearing in mind previous results, it may occur that the improvements achieved are not large enough to justify the computational effort required to solve the MCBS-Mins and MCBS-ins models. Notice that the instantaneous capacity constraints rely on estimation of the execution time required for a task. Pham et al. (2020) introduce an efficient method to estimate the required execution time with only a few real executions, obtaining estimation errors below 10% and show that alternative methods display similar or worse results. We next illustrate these estimation errors impacts on the optimal value of MCBS and MCBS-ins models.

The running times displayed in Table 8 are considered as the Set 0. We define a uniform random distribution with a lower (respectively an upper bound) equal to 90% (respectively 110% ) of the Set 0 running times. Thus, the average times of the uniform distribution are exactly the running times of the Set 0. The lower and upper bounds of the associated uniform distribution are shown in Table 14. From these distributions we generate 10 sets of running times for the nine BoTs. Table 15 displays for each BoT the value of the generated running time for each set.

Table 14 Parameters of the uniform random distribution for each BoT
Full size table
Table 15 Generated running times (time units) for each BoT and each Set
Full size table

MCBS and MCBS-ins models with \(U=50\) and \(D=15\) are solved for each set of running times.

Fig. 5
figure 5

Minimum cost ($) vs set of running times (\(U=50\) VMs, \(D=15\) time units)

Full size image

Figure 5 shows the minimum cost obtained for both models for each set. The Set 0 consists in the running times defined in Table 8. The difference between the minimum costs for each set ranges from 0.285% (Set 9) to 2.1% (Set 5). We also compare the minimum cost for each set with the cost of Set 0. For MCBS-ins model, the difference ranges from \(-\)6.93% (Set 8) to 5.08% (Set 1). Similar values for MCBS model, from \(-\)6.96% (Set 8) to 5.22% (Set 1).

We have performed a paired sample t-test to evaluate whether MCBS-ins model provides better solutions than MCBS in terms of cost. In this statistical test, each set of running times gives a pair of observations, one for each model. As Fig. 6 displays a p-value equals to 0, the minimum cost obtained by MCBS model is significantly greater than the one obtained by MCBS-ins. In fact, with a confidence level of 95% the mean of the differences will be at least equal to 35.49. As expected, the mean of the differences between the minimum cost obtained with each set of running times is not significantly different from the corresponding minimum cost obtained with Set 0 since these differences are positive and negative. All of these differences are shown in Fig. 7. In conclusion the range of the difference to the minimum cost corresponding to Set 0 due to the estimation error associated with the running times is much larger than the differences between solutions obtained by MCBS and MCBS-ins models. The results suggest that it is not worth the computational effort of solving model MCBS-ins compared to model MCBS in terms of cost improvement.

Fig. 6
figure 6

Paired T-Test of the minimum costs ($) obtained by MCBS and MCBS-ins models

Full size image
Fig. 7
figure 7

Differences among the minimum costs obtained by MCBS and MCBS-ins models for each set of running times

Full size image

6 Conclusions and future research

The paper defines new optimization models integrating the VM type selection, resource scaling and workload allocation in a multi-cloud environment. These models only deal with one objective, the minimization of cost considering a deadline and the minimization of the makespan to run all the BoTs, respectively. In addition, we propose two models which involve jointly the costs and makespan objective functions. Two efficient solutions of the biobjective problem are computed by using a lexicographic approach. A first efficient solution is computed by prioritizing cost over makespan and the second one takes the reverse order. Capacity constraints with respect to the number of allowed resources imposed by the cloud providers in the case of public clouds or the cloud system itself in the case of private clouds are also considered.

The limits in the number of VM instances are defined according to two different paradigms. First, a global upper bound is considered, regardless of the time the instances are used. This approach is in accordance with those models assuming that VMs work as long as possible to complete the deadline. In the literature, most of the mathematical models looking for exact solutions assume this constraint. In contrast, we consider in this work the time VMs are working for executing the BoTs, extending the models and increasing the feasibility space. A second approach establishes an instantaneous bound on the number of VMs. This approach increases the number of feasible solutions, leading to better solutions in terms of cost, but increasing significantly the computation time as shown in the computational experience.

Finally, experiments carried out on synthetic realistic data put into highlight the cost improvements of the proposed solutions compared to the solutions of previous models and the effect of the model parameters in the cost and the makespan. A first analysis shows how the minimum makespan required to execute all BoTs decreases as the number of available resources increases, and stabilizes when a certain value is reached. Secondly, we consider fixed the resource limit in order to compute the minimum cost. The minimum values obtained by solving the proposed models are non-increasing monotone functions of the deadline. In addition to the individual analysis of cost and makespan, efficient solutions have been calculated for a bi-objective problem using a lexicographic approach for several values of resource limits and deadline. There is a great impact on the computational time for solving the model when instantaneous capacity constraints are considered, which is not accompanied by a noticeable improvement in the values of the cost or the makespan. In fact, we have seen that the range of the differences in the minimum cost due to the estimation error associated with the running times is much larger than the differences between solutions obtained with instantaneous or global capacity constraints.

As future work we aim at extending the mathematical models to consider more general applications, such as acyclic graphs of BoTs or even more general workflow structures that besides the parallel execution of tasks in a BoT must also consider some precedence relations among the different BoTs. In addition, the proposed models only consider on–demand instances and it is also assumed that the execution time of each task of a BoT remains constant. Interesting future work will be to see how to include reserved VMs and how to remove the assumption on the running times. Finally, we also aim at reducing the computational cost of the considered solutions by strengthening the formulations and defining specific solution methods that could take advantage of the specific structure of the problem formulation. The use of fine grained time units may result in decreased savings and raises the question of whether the computation time required is justified. As a result, reducing computation time by utilizing the model’s specific properties, similar to those examined in Propositions 4.2, 4.4 and Lemma 4.3, is considered a crucial future task.