Energyaware autoscaling algorithms for Cassandra virtual data centers
 1.1k Downloads
 1 Citations
Abstract
Apache Cassandra is an highly scalable and available NoSql datastore, largely used by enterprises of each size and for application areas that range from entertainment to big data analytics. Managed Cassandra service providers are emerging to hide the complexity of the installation, fine tuning and operation of Cassandra virtual data centers (VDCs). This paper address the problem of energy efficient autoscaling of Cassandra VDC in managed Cassandra data centers. We propose three energyaware autoscaling algorithms: Opt, LocalOpt and LocalOptH. The first provides the optimal scaling decision orchestrating horizontal and vertical scaling and optimal placement. The other two are heuristics and provide suboptimal solutions. Both orchestrate horizontal scaling and optimal placement. LocalOpt consider also vertical scaling. In this paper: we provide an analysis of the computational complexity of the optimal and of the heuristic autoscaling algorithms; we discuss the issues in autoscaling Cassandra VDC and we provide best practice for using autoscaling algorithms; we evaluate the performance of the proposed algorithms under programmed SLA variation, surge of throughput (unexpected) and failures of physical nodes. We also compare the performance of energyaware autoscaling algorithms with the performance of two energyblind autoscaling algorithms, namely BestFit and BestFitH. The main findings are: VDC allocation aiming at reducing the energy consumption or resource usage in general can heavily reduce the reliability of Cassandra in term of the consistency level offered. Horizontal scaling of Cassandra is very slow and make hard to manage surge of throughput. Vertical scaling is a valid alternative, but it is not supported by all the cloud infrastructures.
Keywords
Autonomic computing Cloud computing Green computing Optimisation Selfadaptation Apache Cassandra Big data1 Introduction
Autoscaling does not only mean to automatically increase/decrease the amount of resources. Autoscaling implies to adapt, over time, the configuration of the Cassandra VDC and of the cloud infrastructure. To realize an optimal autoscaling, the service provider could adopt three strategies: Vertical scaling, which means to change the Cassandra virtual nodes (vnodes) capacity at runtime, e.g. adding computing power (e.g. virtual cpu) and/or memory; Horizontal scaling, which means to add/remove, at runtime, Cassandra vnodes to/from the Cassandra VDC; Optimal placement, which means to instantiate the vnodes on the physical nodes in a way such that the usage of resources is optimised with respect to some objective function. In our specific case the objective function is the energy consumed by the datacenter and should be minimized.
The Opt and LocalOpt autoscaling algorithms orchestrate those three adaptation strategies, while the LocalOptH does only horizontal scaling and optimal placement. The BestFit is based on the classical Best Fit decreasing algorithm to approximate the solution of the bin packing problem. The algorithm is capable to do both horizontal and vertical scaling. The BestFitH is a variant that does only horizontal scaling. All the algorithms are designed to be integrated in the planning phase of a MAPEK controller (cf. Fig. 1). The scaling decisions are based on three parameters that can be easily collected: the vnodes throughput, the CPU usage, and the memory usage.
The optimal energyaware autoscaling is an algorithm that does an overall system reconfiguration at each scaling action needed to accommodate the resources for a specific tenant. That allow to have always a system configuration that minimize the energy consumed by the datacenter. The rational to introduce energyaware heuristics is twofold: first, the heuristics are applied locally, for the specific tenant the need to scale, and that reduces the perturbation of the performance for the tenants that do not need to scale. Second, the Opt has a complexity of the order \(O((N\times H)^{3/2})\) for N tenants and H physical nodes, while the heuristics have a complexity of the order \(O(H^{3/2})\) and \(O(H^2)\) for (localOpt) and (BestFit) respectively (more details are provided in Sect. 6). The not optimised Matlab code implementing the heuristics finds the suboptimal solution in a range \(10^{1}\), 10 s (when running on an Intel Core i5). The average time to find the optimum using the Matlab MILP solver is about 50 s with a maximum of about \(2 \times 10^3\) s.
1.1 Research contribution

we compare the optimal energy aware allocation proposed in [4] with two new autoscaling heuristics BestFitH and LocalOptH

we provide a discussion on the issues related to autoscaling in Cassandra virtual data centers and we give guidelines on how to best use the proposed algorithms, i.e. for medium/long term capacity planning and at runtime

we provide a detailed evaluation of the computational cost of the optimal autoscaling algorithms and of all the heuristic algorithms.

we provide a simple model to asses how the consistency level of a Cassandra VDC is impacted by the autoscaling and specifically by the placement of vnodes on physical machines.

we analyse the performance of the proposed algorithms in case of surge of requests and failure of physical nodes
1.2 Paper organization
The paper is organised in the following way. The next section discusses related work. The reference scenario we consider is presented in Sect. 3. Section 4 introduces the system model and the optimal adaptation problem formulation. The autoscaling algorithms are presented and discussed in Sect. 5. In Sect. 6 we provide the computational cost analysis. Issues on Cassandra autoscaling and recommendations on the use of the algorithms are discussed in Sect. 7. The experimental methodology (analysis cases, metrics and experimental setup) is described in Sect. 8, while the experimental results are described in Sect. 9. Finally, Sect. 10 provides concluding remarks.
2 Related works
The problem we are addressing has been partially covered in literature by research paper in different fields: QoS and energyaware datacenter management; VM placement; autonomic adaptation of cloud infrastructures; performance evaluation, management and adaptation of cassandrabased systems.
Examples of research works on measuring and managing the performance of NoSql distributed data stores such as Cassandra are [9, 23]. Chalkiadaki and Magoutis [6], Dede et al. [11], Kuhlenkamp et al. [18], Rabl et al. [25], Shankaranarayanan et al. [28], Shi et al. [30] are studies focusing on the horizontal scalability feature offered by such databases. Few studies consider vertical scaling, e.g. [6, 18], and configuration tuning [6, 12, 22, 28]. While Horizontal scaling, vertical scaling and configutation tuning approaches are somentime mixed, optimal placement (e.g. [1, 5, 13, 14, 16]) is never considered in combination with the other adaptation strategies.
In [9] and [23] the authors presented YCSB and YCSB++, the reference benchmarking frameworks for facilitating the comparison of cloud based dataserving systems. YCSB allows to simulate five different workloads and is compliant with BigTable, HBase, Cassandra, MongoDB, DynamoDB and more. In our work we decided to not to use YCSB because we are mainly interested in working with Ericssonn datasets and applications. However, our solution is based on a heuristic throughput model that is independent from the specific type of query and application.
In [30] has been evaluated the horizontal scalability of Cassandra and Hbase for a mix of sequential and random read and write operations, scan operations and structured queries. No report and consideration are provided on how and if the Cassandra and Hbase configuration impact the performance.
In [25] the authors evaluate the performance of six SQL and noSQL databases under the pressure of 5 different workloads. These benchmarking experiments has been extended in [18] with a performance evaluation of Cassandra on different Amazon EC2 infrastructure configurations. In comparison with those researches we consider only read, write and read and write requests because of interest for our industrial case. However, our model is independent from the specific type of query. In [18] the authors explore both horizontal and vertical scalability. Their results confirms the experience we had with Cassandra performance on a virtualized environment. That is, a reduction of the Cassandra throughput up to 50% compared with Cassandra performance in non virtualized clusters.
Concerning self adaptation, few work has been presented. In [6] the authors propose a QoS controller for a Cassandra cluster that aims to guarantee system performance by means of coordinating horizontal scalability (bootstrap of new nodes) and cache size (i.e. configuration tuning). The proposed solution has been evaluated by means of YCSB benchmark. In [28] the authors consider the problem of optimizing geographically distributed cloud data stores with respect to latency under failure scenarios. The authors adapt the system tuning three main factors: R and W quorum, location of replicas and number of replicas. On the basis of experimental results the authors concludes that quorumbased data store could benefit from an adaptable and fine grain replica configuration. Indeed not only different applications could need different replication strategies, but also for the same application different group of object could need different replication strategies. This work motivates our assumption on the need for application specific Cassandra configurations. However, while [28] is mainly interested in the optimal configuration of the quorum mechanism and of the replication strategies, we are focused on the application specific scaling actions (Vertical and Horizontal) and on energyaware optimal placement. Like [28], CADRE [31] shows that carefully distinguishing R + W queries in geographically distributed setting affects response time and carbon footprint. They propose an online algorithm to reduce carbon footprint while keeping response time low. The online algorithm is similar to our BestFit approach. Katsak et al. modify Cassandra for time varying resources by sending writes to vNodes and carefully maintaining a “working” set of available nodes. The choice of working set site and placement policies affects performance.
In [22] the authors propose AutoPlacer a mechanisms to selftune the placement of replicas in distributed keyvalue stores. Their goal is to minimize the cost of replicas in term of overall latency. In [12] the authors propose a multidimensional indexing techniques for supporting complex queries using multiple object attributes. Such technique requires a complex system configuration and the authors propose a model and techniques to automatically and dynamically reconfigure the system in dynamic workload environments.
A model for provisioning multitier applications in a cloud environment has been proposed by [29]. The authors proposed a simple and effective approach for resource provisioning to achieve a percentile bound on the end to end response time of a multitier application. The authors find that fewer highcapacity servers are preferable for high percentile provisioning. We leverage and verified this finding, but the solution can not be applied as it is for a Cassandrabased systems.
In [8] that authors consider the placement problem of virtual machines (VMs) of applications with intense bandwidth requirements. The proposed model fit in centralized storage scenarios like storage area networks and not in distributed storage scenarios like Cassandra.
The agility issue in scaling distributed storage systems as been addressed in [7]. The authors propose an elastic storage system, called JackRabbit, that can quickly change its number of active servers. JackRabbit is based on HDFS. Out paper confirm the agility issue.
3 Reference scenario
We consider a provider of a managed Apache Cassandra service offered to support enterprise applications. There are many examples of CassandraasaService providers: Rackspace (http://rackspace.com), Instaclustr (http://instaclustr.com/) and Seastar (http://seastar.io/), just to mention a few.
The tenants of the service are independent applications each using its own Cassandra VDC (in what follow we will interchangeably use the terms application and tenant). A Cassandra VDC is a set of Cassandra virtual nodes (vnodes), i.e. an instance of Cassandra software running on a virtual machine (VM). All the Cassandra VDCs are tenants in a cloud infrastructure (no matter if on a public or private cloud), or data center in what follows.
Applications submit NoSql queries (called operations in what follows) at a specific rate. Each application requires a minimum throughput, a certain level of data replication to cope with node failures, and has a dataset of a specific size. To satisfy these customer’s requirements the service provider has to properly plan the capacity and the configuration of each Cassandra VDC. On the other side, the service provider wants to minimise its power consumption. The Cassandraasaservice provider has a typical scalability issue when: a new tenant subscribes to a service; and/or when existing tenants variate their requirements by modifying the target throughput, the data replication factor, and/or the dataset size; and/or there is a surge in the throughput.
The scenario is schematised in Fig. 1. The figure shows three applications, each with a data replication factor of three, that means each application has three copies of each data item. Applications could be served by Cassandra vnodes with diverse capacity in term of supported throughput. This can be achieved, for example, by running the Cassandra vnodes on VMs with different CPU power and memory size and allocating the proper number of Cassandra vnodes. To maximise the utilization, the provider decided to compact the Cassandra vnodes only on three out of four servers. The autoscaler module is as a MAPEK controller. The autoscaling actions are based on data collected from the cluster infrastructure (the physical nodes and the hypervisor), from the Cassandra VDCs and from the applications. The executor controls the VMs and the Cassandra configuration parameters, as well as start and stop VMs and add/remove to/from Cassandra VDC the Cassandra vnodes.
4 Adaptation model

the number of vnodes of the Cassandra VDC (horizontal scaling)

the configuration of vnodes, e.g. in terms of CPU capacity and memory (vertical scaling)

the placement of vnodes (of the VDCs) on the physical infrastructure (optimal placement)
4.1 Workload and SLA model
The workload of a Cassandra VDC can be characterised by the following features: the type of requests, e.g. read only, write only, read & write, scan, or a combination of those; the rate of the operation requests; the size of the dataset; and the data replication_factor. Depending on the size of the dataset managed, a Cassandra VDC is classified as diskbound if the dataset does not fit the memory offered by all the vnodes in the VDC. Otherwise, CPUbound (see Eq. 1). Diskbound installations have a performance degradation of two order of magnitude compared to CPU bound configurations [25].
Our workload model is based on the following assumptions.
Assumption 1
The system workload consist of a set \(\mathcal {L}\) of read (R), write (W) and read & write (RW) operation requests: \(\mathcal {L}=\{R, W, RW\}\). Such operation requests are generated by the N independent applications and we assume that application i generates only requests of type \(l_i \in \mathcal {L}\). If \(l_i=R\) or \(l_i=W\) we have 100% R or W requests. In case \(l_i=RW\) we have \(\alpha \%\) read requests and (\(100\alpha \)) write requests (for example in our experiments \(\alpha =75 \%\)).
Assumption 2
Requests of type \(l_i\) are generated at a given rate measured in operations per second.
Assumption 3
The dataset size for application i is \(r_i\) GByte and the data are replicated with a factor \(D_i\)
Assumption 4
The workload is only CPU bound, hence the memory requirements are met.
Assumption 5
The internal/external network latency does not impact the autoscaling decisions. Hence it is not considered in the SLA.
Concerning Assumption 1, we limit the study to the set \(\mathcal {L}=\{R, W, RW\}\). However, the model we propose can deal with any type of operation requests, as clarified later in Sect. 4.3. Assumption 4 implies that the service provider has to set up, during the application onboarding phase, and to maintain, at runtime, the right number of vnodes for tenant i. Dealing only with CPU bound workloads exempt us from considering the workload consolidation problem (e.g. [32]). Besides, it is of interest for the customer to have CPU bound VDC in order to achieve the desired performance.
4.2 Architecture model
\(t^0_{l_i,j}\) as function of \(c_j\) (virtual CPU), \(m_j\) (GByte), \(heapSize_j\) and \(l_i\)
VM type and configuration  Throughput for different workloads (ops/s)  

j  \(c_j\)  \(m_j\)  \(heapSize_j\)  R  W  RW 
1  8  32  8  16.6 \(\times 10^3\)  8.3 \(\times 10^3\)  13.3 \(\times 10^3 \) 
2  4  16  4  8.3 \(\times 10^3\)  8.3 \(\times 10^3\)  8.3 \(\times 10^3\) 
3  2  16  4  3.3 \(\times 10^3\)  3.3 \(\times 10^3\)  3.3 \(\times 10^3\) 
Memory available for the dataset in a Cassandra vnode (JVM Heap) as function of the VM memory size
\(m_j\) (RAM size in GB)  1  2  4  8  16  \(\ge \)32 

\(heapSize_j\) (max Heap size in GB)  0.5  1  1  2  4  8 
To model vertical scaling actions, that is a change from configuration \(j_1\) to \(j_2\), we replace a VM of type \(j_1\) with a VM of type \(j_2\). However, in a real setting, hypervisors (e.g. VMWare) make it possible to resize, at runtime, the number of cores associated to a VM and the size of memory used without the need to shut down the VM. We do not consider the case of overallocation, that is the maximum number of virtual cores allocated on PM h is equal to \(C_h\).
Finally we assume that the local network latency do not impact the performance of the VDC and the system reconfiguration (Assumption 5).
4.3 Throughput model
We model the actual throughput \(T_i\) offered by the provider to application i as a function of \(x_{i,j,h}\)
4.4 Power consumption model
As service provider utility we chose the power consumption which is directly related with the provider revenue (and with IT sustainability).
Many ways of reducing the power consumption in cloud systems have been proposed the literature; two interesting survey are [24] and [19]. Different approaches can be used for the sustainable operation of data centers. If we focus on cloud management systems the techniques typically used are: scheduling, placement, migration, and reconfiguration of virtual machines. The ultimate goal is to optimise the use of resources to reduce power consumption. Optimisation depends on the context, it could mean minimising PM utilisation or to balance the utilisation level of physical machine with the use of network devices for data transfer and storage. Independently from the configuration or adaptation policy adopted all these techniques are based on power and/or energy consumption models (in [24] a detailed surveys). Power consumption models usually define a linear relationship between the amount of power used by a system as function of the CPU utilisation (e.g. [2, 3, 10]), or processor frequency (e.g. [17]) or number of core used (e.g. [26]).
5 Autoscaling algorithms
5.1 The optimal autoscaling
The pseudo code is listed in Algorithm 1. Opt simply invokes the solver for the optimization problem and returns: the optimal configuration of the system \(\mathbf {x}_{opt}\), that inform about the scaling actions; the remaining CPU and memory capacity (\(C^a\) and \(M^a\)) available after the adaptation; the type \(j^*\) of VM selected by the algorithm. The parameter e is an exit code flag that is true if a solution exist and false otherwise. The optimal configuration \(\mathbf {x}_{opt}\) indicates the necessary actions to perform (c.f. beginning of Sect. 4): horizontal scaling, vertical scaling and optimal placement. \(\mathbf {x}_{opt}\) is the solution \(\mathbf {x}\) to the optimization problem defined in Fig. 2, where: the set of constraints defined by Eq. 11 guarantee that the SLA is satisfied in terms of minimum throughput for all the tenants. For the sake of clarity we keep these constraints non linear, but they can be linearised using standard techniques from operational research if the throughput is modelled using Eq. 6. Eq. 12 introduces a set of constraints to guarantee that the number of vnodes allocated is enough to guarantee that the portion of the dataset handled by each node fits in the main memory and that the replication factor \(D_i\) specified in the SLAs is implemented. Equations 13 and 14 model the assumption that homogeneous VMs must be allocated for each tenant. \(\Gamma \) is an extremely large positive number. Equation 15 controls that the maximum capacity of the physical machine is not exceeded. A relaxation of this constraint would make it possible to model overallocation. In the same way, 16 controls that the memory allocated for the vnodes do not exceed the main memory capacity of the physical nodes. Equation 17 guarantee that the Cassandra vnodes are instantiated on at least \(D_i\) different physical machines. Equations 18 and 19 force \(s_{i,h}\) to be equal to 1 if the physical machine h is used by application i and to be zero otherwise. In the same way, the set of constraints 20 and 21 force \(r_h\) to be equal to 1 if the physical machine is used and zero otherwise. Finally, expressions 22 and 23 are structural constraints of the problem.
5.2 Heuristics
In a real scenario it is reasonable that new tenants subscribe to a service and/or that existing tenants change their SLAs (for example requesting the support for an higher throughput, for a different replication factor or for a different dataset size). In such dynamic scenarios, in order to satisfy the SLAs, the autoscaler should perform adaptation actions without perturbing the performance of the other tenants, that is for example avoiding vnodes migration.
A limitation of the Opt algorithm is that the scaling of a virtual data center or the instantiation of a new one can lead to an uncontrolled number of adaptation actions that involve all the tenants’ VDC and that could hurt the performance of the whole data center [4]. To solve that issue we propose four heuristic autoscaling algorithms that work locally allocating/deallocating resources only for the specified Cassandra VDC without reconfiguring VDCs of other tenants. The first heuristic is called LocalOpt and is energyaware. It applies locally the optimisation problem listed in Fig. 2, that is solve the optimization problem for only the one tenant. This implies that the configurations of the other Cassandra VDCs are not changed.
The pseudocode for the BestFit heuristic is reported in Algorithm 3. As for LocalOpt it requires as input \(\mathcal {H}_{a}\), \(C^a\) \(M^a\), the SLA sla for a current or new tenant i (\(\mathcal {I} = \{ i \}\)) and \(\mathcal {J}\). The code on lines 28 evaluates the number of vnodes required to satisfy throughput, dataset size, and data replication constraints, for each VM type. Line 9 selects the VM type that maximises the ratio between the requested throughput and the throughput achievable with the number of vnodes instantiated. That is, we try to minimise the over provisioning effect due to dataset constraints and, as result, the energy consumption is minimised. Lines 1015 check if the selected VM type satisfies available CPU and memory constraints. Otherwise, the second VM type that minimises the over provisioning of resources is selected and so on, until all the VM types are analysed. Line 16 returns \(\mathbf {x}_{sub} = \emptyset \) because no feasible solutions were found. Lines 19  30 place the vnodes on the PMs minimising the number of PMs used, packing as many vnodes as possible in a PM, of course considering the \(D_i\) constraint. That also minimise the energy consumption. The function any(\(c_{j^*} \le C^a\)) compares \(c_{j^*}\) with all the element of \(C^a\) and it returns true if at least one element of \(C^a\) is greater than or equal to \(c_{j^*}\). Otherwise, if no PMs satisfy the constraint it returns false. The same behaviour is valid for any(\(m_{j^*} \le M^a\)). The function sortDescendent(\(\mathcal {H}_{a}\)) sorts the \(\mathcal {H}_{a}\) in descending order. The function popRR(\(\mathcal {H}_{a}\),\(D_i\)) extracts, in roundrobin order, a PM from the first \(D_i\) in \(\mathcal {H}_{a}\). At Line 28, if there is no more room in the selected PMs h the set \(\mathcal {H}_{a}\) is updated removing the PMs h. At line 32, if not all the \(n^*_{i,j^*}\) vnodes could be allocated the empty set is returned because no feasible solutions for the allocation could be found. Otherwise, the suboptimal solution \(\mathbf {x}_{sub}\) is returned.
6 Computational cost
There are several algorithms to solve LP problems, including the wellknown simplex and interior points algorithms [20]. Widely used software packages (CPLEX^{®}, MATLAB^{®}) adopt variants of the wellknown interior point Mehrothra’s predictorcorrector primaldual algorithm [21], which has \(O(n^{\frac{3}{2}}\log \frac{(x^0)^Ts^0}{\epsilon })\) worst case iterations, where \(\epsilon \) is the accuracy and \((x^0)^Ts^0\) the starting point for the Mehrothra algorithm, and such that \(\epsilon \ge x^Ts\), where \(x^Ts\) is the final point in the algorithm. Hence, for a fixed \(\epsilon \) the Mehrothra algorithm has a complexity of \(O(n^{\frac{3}{2}})\), where n is the number of variables of the LP problem [27]. The complexity in our problem arises from the potentially large value of n, corresponding to the number of variables that is given by the following expression: \(n=N\times V \times H + N\times V + 3\times N + H\). This means that the worstcase complexity of our LP problem using Mehrothra’s predictorcorrector primaldual algorithm is \(O((N\times V \times H)^{\frac{3}{2}})\).
The LocalOpt call of the optSol, which is solved with the Mehrothra algorithm. Because optSol is executed only for one tenant, the complexity of LocalOpt is \(O(( V \times H)^{\frac{3}{2}})\).
The complexity of the BestFit adaptation algorithm (Algorithm 3) can be determined in the following way. The first loop (lines 38) and the second loop (lines 1215) run at most V iterations each. This means that the computational complexity of lines 118 is O(V). We then need to sort the list of available PMs, which has complexity \(O(H \log H)\). The third loop (lines 1930) may run for at most H iterations. In each iteration the two functions any \((c_j* \le C_a)\) and any \((m_j* \le M_a)\) are called; these functions both have complexity O(H). As a consequence, the worstcase complexity of the third loop is \(O(H^2)\). The complexity of Algorithm 2 is thus \(O(V) + O(H^2)\). In real scenarios V is much less then H, therefore the complexity is \(O(H^2)\).
The variants BestFitH and LocalOptH have the same complexity of the BestFitH and LocalOptH respectively.
7 Recommendations on the use of the autoscaling algorithms
Use of the autoscaling algorithms
Use case  Opt  LocalOpt  BestFit  LocalOptH  BestFitH 

Capacity planning  X  
Data center consolidation  X  
VDC consolidation  X  X  
Runtime adaptation  X  X  X  X 
As mentioned before the Opt algorithm produces too many reconfigurations of the whole data center. Moreover, for largescale systems, the polynomial complexity of the Opt is a limitation, especially if the workload changes at high frequency. Hence, the optimal autoscaling algorithm is more suitable to support capacity planning decisions and for periodical mid term consolidation actions.
All the heuristic autoscaling algorithms proposed are suitable for runtime adaptation decisions. Although, there are two cases that should be carefully considered: the algorithm recommend horizontal scaling actions and the algorithms recommend vertical scaling actions.
Horizontal scaling is seamlessly supported by the whole cloud stack, from application level, Cassandra in our case, to the hypervisor. The only limitation is the responsiveness of the scaling actions, that is bounded by the time needed to start a VM (about 2min) and by the time needed to add a Cassandra vnode to an existing VDC, scaling delay hereafter. Best practices for Cassandra cluster management suggest that, to preserve data consistency, vnodes should be added sequentially (one at time) and that the scaling delay is at least 2 min. While the VMs activation delay can be eliminated using a pool of warm VMs, the second could not be eliminated. In Fig. 4 we show an example of horizontal scaling for Cassandra.
Vertical scaling is partially supported by the cloud stack. For example, Open Stack supports live instance resizing, but not all the hypervisors do: VMWare support seamless vertical scaling, but with Xen and KVM the vertical scaling implies to shutdown and to restart the VMs. As before mentioned and as practically shown in Sect. 9.2 vertical scaling can help in managing surges of throughput. Let us consider the example in Fig. 4: if at time \(t_1\), rather than starting the horizontal scaling sequence, we operate a vertical scaling of the running nodes, we can manage a throughput surge by the deadline of \(t=5\).
 1.
The workload must be carefully characterized to properly size the vnodes capacity, that is \(t_{i,j,h}\)
 2.
Workload prediction and proactive autoscaling should be combined. The forecasting windows should be at least scaling delay time units ahead
 3.
For horizontal scaling, the activation of the Cassandra vnodes should be pipelined (cf. Fig. 4) and maintaining a pool of warm VMs helps in reducing the scaling delay
 4.
Vertical scaling can help in managing throughput surges, reducing the time to scale the capacity of the cluster (versus the horizontal scaling).
 1.
To use Opt, LocalOpt or BestFit algorithms for the first VDC configuration
 2.
To run, at run time, LocalOptH or BestFitH
 3.
To run, periodically, LocalOpt or BestFit for VDC consolidation.
8 Performance evaluation methodology
In this section we describe the performance evaluation scenarios, the performance evaluation metrics and the setup of the experiments.
8.1 Scenarios

Increase of the throughput (SLA variation). Customer needs and service level objectives can change over time. This scenario considers a planned increase of the throughput demand.

Surge in the throughput. This scenario considers an unpredicted increase in the throughput demand of a specific tenant i.

Physical node failures. This scenario contemplate the failure of physical machines, that implies the loss of a given number of Cassandra vnodes. In this context, we analyze how the placement of the vnodes (operated by the autoscaling algorithms) impact the consistency level reliability.
8.2 Performance metrics

\(P(\mathbf {x})\) the overall power consumption defined by equation 10;
 The Scaling Index for application i \(SI(t_1,t_2)_{i,j}\) is defined as the variation in the number and type of Cassandra vnodes when the system change its configuration at time \(t_1\) (\(\mathbf {x}(t_1)\)) into a new configuration at time \(t_2\) (\(\mathbf {x}(t_2)\)).SI represents a gap and not an absolute value of the number of VMs used. Positive values for SI means that new VMs are allocated. Negative value represent the number of VMs deallocated. SI allows to quantify both vertical and horizontal scaling actions.$$\begin{aligned} SI(t_1,t_2)_{i,j}=\sum _{\mathcal {H}} \left( x(t_2)_{ijh}  x(t_1)_{ijh} \right) . \end{aligned}$$
 The Migration Index for application i. \(MI(t_1,t_2)_i\) is defined as the number of Cassandra vnodes migrations that application i experienced when the system change its configuration at time \(t_1\) into a new configuration at time \(t_2\).where \(\Delta _{i,h}=1\) if \((s(t_2)_{i,h}  s(t_1)_{i,h}) > 0\) and \(\Delta _{i,h}=0\) otherwise. \(s(t)_{i,h}\) is the value of \(s_{i,h}\) at time t.$$\begin{aligned} MI(t_1,t_2)_i=\sum _{\mathcal {H}} \Delta _{i,h} \end{aligned}$$
 Number of delayed requests \(Q_i( \tau )\) for tenant i in a time interval \(\tau = t_{end}  t_{start}\). Assuming that \(T_i (t)\) is the actual throughput observed and that \(T^{min}_i(t ) \ge T_i (t )\) \(\forall t\in \tau \) we define$$\begin{aligned} Q_i( \tau ) = \int _{t_{start}}^{t_{end}} \left( T^{min}_i(t)  T_i (t) \right) dt . \end{aligned}$$

Consistency level reliability \(\mathcal {R}\) defined as the probability that the number of healthy replicas in the Cassandra VDC is enough to guarantee a specific level of consistency over a fixed time interval (c.f. Sect. 9.3 for details). We recall that, assuming independence of failures in the components, the reliability of K nodes working in parallel is defined as \(\mathcal {R}=1(1\rho )^{K}\), where \(\rho \) is the reliability of a single node and \((1\rho )^{K}\) is the probability that K nodes fail.
8.3 Setup of the experiments
To measure the maximum Cassandra throughput achievable (\(t^0_{l_i,j}\)) for each type of workload and VM type and to compute also the values for \(\delta _{l_i,j}^k\) we use a real cluster and a workload generator provided by Ericsson to reproduce their application behaviour. The cluster is composed of nodes with 16 cores and 128 GB of memory (RAM). The nodes are connected with a high speed LAN. We run VMware ESXi 5.5.0 on top of Red Hat Enterprise Linux 6 (64bit) and we use Cassandra 2.1.5. We use VMs with three different configurations, as reported in Table 1. The values obtained for \(t^0_{l_i,j}\) are reported in Table 1, while the values for \(\delta _{l_i,j}^k\) are reported in Table 4.
The performance of the proposed adaptation algorithms are assessed using Monte Carlo simulation for the Physical node failure scenario, while numerical evaluation is used for the SLA variation and Throughput surge scenarios. Experiments have been carried out using Matlab R2015b 64bit for OSX running on a single Intel Core i5 processor with 16GB of main memory. The model parameters we used for simulation are reported in Table 4.
9 Experimental results
9.1 Increase of the throughput (SLA variation)
Model parameters used in the experiments
Parameter  Value  Description 

N  1–10  Number of tenants 
V  3  Number of VM types 
H  8  Number of PMs 
\(D_i\)  1–4  Replication factor for App. i 
\(r_i\)  5–50  Dataset size for App. i 
\(\mathcal {L}\)  \( \{ R, W, RW \}\)  Set of request types 
\(T_i^{min}\)  10,000–70,000 ops/s  Minimum throughput agreed in the SLA 
\(C_h\)  16  Number of cores for PM h 
\(c_j\)  2–8  Number of vcores used by VM type j 
\(M_h\)  128 GB  Memory size of PM h 
\(m_j\)  16–32 GB  Total memory used by VM type j 
\(heapSize_j\)  4–8 GB  Max heap size used by VM type j 
\(\forall l_i\): \(\delta ^1_{l_i}\)  1  \(1\le x_{i,j,h} \le 2\) 
\(\delta ^2_{l_i}\)  0.8  \(3\le x_{i,j,h} \le 7\) 
\(\delta ^3_{l_i}\)  0.66  \(x_{i,j,h} \ge 8\) 
\(P_h^{max}\)  500 Watt  Maximum power consumed by PM h if fully loaded 
\(k_h\)  0.7  Fraction of \(P_h^{max}\) consumed by PM h if idle 
The power consumption is plotted in Fig. 6. For throughput higher than \(40 \times 10^3\) ops/s, with the optimal scaling is possible to save about 50% of the energy consumed by the heuristic allocation. For low values of the throughput (\(10  20 \times 10^3\) ops/s) the BestFit and BestFitH show a very high energy consumption compared to the LocalOpt and LocalOptH. When the throughput increase, the LocalOptH behave as the BestFit. For high throughput (\(60  70 \times 10^3\) ops/s), the energy consumed by the LocalOpt is less than all the other heuristics.
Figure 7 shows the Migration Index. Each box plot is computed over the data collected for the three tenants. We can observe that each application experiences between 0 and 3 vnode migrations depending on the infrastructure load state.
Considering the low values for the migration index for the Opt allocation and the high saving in the energy consumed compared with the other algorithms, it makes sense to perform periodic VDC consolidation using the Opt policy, as recommended in Sect. 7.
9.2 Throughput surge
The first case (Case A) uses VMs with the capacity specified in Table 1). The Opt autoscaling starts allocating four vnodes of Type 3 (\(3.3 \times 10^3\) ops/s) and then five. At time \(t=8\) the algorithm did a vertical scaling action allocating five vnodes of Type 2 (\(8.3 \times 10^3\) ops/s). Than, at time \(t=10\), twelve vnodes are allocated. Considering the serialization of the horizontal scaling actions (cf. Sect. 7) the seven Cassandra vnodes are added in 14 min. The LocalOpt behaves like the Opt in terms of scaling decisions. The BestFit autoscaling start allocationg 4vnodes of Type 3, than it scaleup to seven vnodes (at time \(t=8\)) and finally it did two vertical scaling actions: the first from vnodes Type 3 to Type 2, and the second from Type 2to Type 1 vnodes.
The number of delayed requests \(Q_i\) and the percentage with respect the total number of received requests (tot.req.) are reported in Table 5. \(Q_i\) and tot.req. are computed over the time interval the requested throughput \(T_{RW}^{min}\) exceed the actual throughput.
Intuitively, with Cassandra vnodes capable to handle a higher workload it should be possible to better manage the surge in the throughput. Hence, we have analyzed a Case B where we configure three new types of Cassandra vnodes capable to handle the following RW throughput: type 4, \(20\times 10^3\) ops/s; type 5, \(15\times 10^3\) ops/s; and type 6, \(7\times 10^3\)ops/s. Fig. 8 shows the behaviour of the Opt and BestFit autoscaling algorithms. The LocalOpt behaves as the Opt. From the plots, it is evident that with more powerful vnodes the autoscaling algorithms are capable to satisfy the requested throughput with a delay of only 2 min. The Opt starts allocating 3 vnodes of type 6, at time \(t=4\) a new node of type 6 is added and at time \(t=6\) the algorithm did a vertical scaling allocating 4 vnodes of type 5. At \(t=8\) the algorithm decided to add 2 more nodes of type 5, that are allocated in the next two time slots. The BestFit starts allocating vnodes of type 6 and scales from 4 to 6 nodes. After that, at \(t=10\) it performs a vertical scaling from type 6 to type 5. The values for delayed requests are reported in Table 5.
The take home message is that having a pool of vnodes type capable to handle from low throughput to very high throughput allow to manage throughput surges.
The number of delayed requests \(Q_i\) and the percentage with respect the total number of received requests (tot.req.)
\(Q_i\) (\(\times 10^3\))  \(\frac{Q_i}{tot. req.}\) (%)  

Case A  
Opt  191.84  22.78 
LocalOpt  191.84  22.78 
BestFit  70.89  46.33 
Case B  
Opt  7.66  4 
LocalOpt  7.66  4 
BestFit  70.58  30.29 
Consistency reliability \(\mathcal {R}\) for the consistency level of ONE and QUORUM
Onetoone  ntoone  

Opt  LocalOpt  LocalOptH  BestFit  BestFitH  
\(\rho =0.9\)  
\(\mathcal {R}_O _ {D=3}\)  0.9999995  0.9995  0.9995  
\(\mathcal {R}_Q _ {D=3}\)  0.999995  0.995–0.9995  0.9995  
\(\mathcal {R}_O _ {D=5}\)  0.99999999995  0.999995  0.999995  
\(\mathcal {R}_Q _ {D=5}\)  0.999999995  0.9995–0.999995  0.99995  
\(\rho =0.8\)  
\(\mathcal {R}_O _ {D=3}\)  0.99996  0.996  0.996  
\(\mathcal {R}_Q _ {D=3}\)  0.99984  0.98–0.996  0.996  
\(\mathcal {R}_O _ {D=5}\)  0.999999948  0.99984  0.99984  
\(\mathcal {R}_Q _ {D=5}\)  0.9999987  0.996–0.99984  0.9992 
9.3 Physical node failures
Cassandra offers three main levels of consistency (both for Read and Write): ONE, QUORUM and ALL. Consistency level of ONE means that only one replica node is required to reply correctly, that is it contains the replica of the portion of the dataset needed to answer the query. Consistency level QUORUM means that \(Q=\left\lfloor \frac{D}{2} \right\rfloor +1 \) replicas nodes are available to reply correctly (where D is the replication factor). Consistency level ALL means that all the replicas are available.
In the following, we assess the Consistency level reliability \(\mathcal {R}\) for a consistency level of ONE and of QUORUM when the replication factor is \(D=3\) and \(D=5\) for all \(i\in \mathcal {I}\) and when the different autoscaling algorithms are applied.
Table 6 reports the values of \(\mathcal {R}_{O}\) and \(\mathcal {R}_{Q}\) for \(D=3\) and \(D=5\), \(\rho =0.9\). The evaluation is done in the following way. We consider 5 customers that request for a randomly generated SLA: \(l_i\) is distributed as 10% RW, 15%W and 75% R; \(T_{min}\) is uniformly distributed in the interval [10.000, 18.000] ops/s; \(D_i= 3\) (5) and \(r_i\) is constant (8GB). The number of vnodes used by each tenant is \(n=6\) for the case \(D=3\) and \(n=10\) for the case \(D=5\). We run 10 experiments and we assess the best and worst case over all the allocated VDC.
In the first set of experiments we consider \(\rho =0.9\). The onetoone mapping offers a consistency level of ONE and QUORUM with a very high reliability of six 9 s and five 9 s respectively if \(D=3\) and if the replication factor increase to 5 the reliability increases to ten 9 s and eight 9 s for consistency ONE and QUORUM respectively.
Unfortunately, when a ntoone mapping is adopted the reliability of the consistency level drops down 3–4 orders of magnitude. In the case \(D=3\), we observed that all the autoscaling policies offer a consistency level ONE with a reliability of three 9 s. If the replication factor increases to 5 the reliability increase to five 9 s. For the consistency level QUORUM, we have a dependency on both the replication factor and the autoscaling policy. When the replication factor is three, Opt, LocalOpt and LocalOptH offer a consistency level of QUORUM with a reliability that ranges between two 9 s and three 9 s, while the BestFit and BestFitH offer a reliability level of three 9 s. When the replication factor increase to five, the reliability level offered by Opt, LocalOpt and LocalOptH increase to three 9 s in the worst case and five 9 s in the best case. The BestFit and BestFitH provide a reliability of four 9 s.
When we decrease the reliability of the physical machines to \(\rho =0.8\) the reliability decreases of two or three order of magnitude for the onetoone mapping and of one or two order of magnitude for the heuristics.
10 Concluding remarks
In this paper we explored the problem of energyaware autoscaling of Cassandra VDC in a multitenants environment. We presented an optimisation model that find the optimal autoscaling actions. The system model we propose relies only on the measure or estimate of the relationship between the throughput achievable and the number of Cassandra vnodes. This information is easy to be collected and maintained up to date at execution time. The performance of the optimal autoscaling is compared against two energyaware heuristics and two energyblind heuristics. The advantage of using heuristics is twofold: first, the heuristics are applied locally, and that reduces the perturbation of the performance of the tenants that do not need to scale. Second, the Opt has a complexity of the order \(O((V\times N\times H)^{3/2})\) for N tenants, H physical nodes and V Cassandra vnode Types, while the heuristics have a complexity of the order \(O((V\times H)^{3/2})\) and \(O(H^2)\) for (localOpt) and (BestFit) respectively (a details analysis has been provided in Sect. 6).
The lesson learned from that study is the following.
Planned variations of the throughput (increase and/or decrease) can be managed in an energy efficient way by the LocalOpt. For intense workload and for high utilized data centers the energy consumed by a LocalOpt allocation is comparable with the allocation determined by the BestFit. But for low throughput and/or low utilized data centers, the BestFit produces allocations that consume between the 48% and the 100% more energy than the LocalOpt.
The agility of Cassandra in scaling up is limited by the need to serialize the allocation of vnodes and by the scaling delay. Our experiments show that vertical scaling of vnodes is the only adaptation action capable to manage surges in the throughput.
Finally, a Cassandra cluster that use a onetoone mapping of vnodes on physical nodes offer the consistency level of ONE and of QUORUM with a very high level of reliability (between five and ten 9 s) in case of physical node failures and a node reliability of 0.9. On the contraty, when a ntoone mapping is implemented, the reliability for consistency level of ONE and QUORUM drop down of three  four order of magnitudes.
Hence, we have identified two open challenges. First, there is the need for energyefficient autoscaling algorithms that hide the structural limitation of Cassandra to scale fast. Such algorithms should prioritize vertical scaling actions and should take into consideration the state of the system (the algorithm we have proposed are memoryless). Second, there is the need for energyefficient and consistencyaware algorithms that do not impact the reliability of the consistency level offered by Cassandra when configured using a onetoone mapping.
Notes
Acknowledgements
This work is funded by the research project “Scalable resourceefficient systems for big data analytics”—Knowledge Foundation Grant No. 20140032, Sweden.
References
 1.Almeida Morais, F., Vilar Brasileiro, F., Vigolvino Lopes, R., Araujo Santos, R., Satterfield, W., Rosa, L.: Autoflex: service agnostic autoscaling framework for IaaS deployment models. In: 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), pp. 42–49 (2013). doi: 10.1109/CCGrid.2013.74
 2.Borgetto, D., Maurer, M., DaCosta, G., Pierson, J., Brandic, I.: Energyefficient and SLAaware management of IaaS clouds. In: 2012 Third International Conference on Future Energy Systems: Where Energy, Computing and Communication Meet (eEnergy), pp. 1–10 (2012)Google Scholar
 3.Buyya, R., Beloglazov, A., Abawajy, J.H.: Energyefficient management of data center resources for cloud computing: a vision, architectural elements, and open challenges. CoRR (2010). http://arxiv.org/abs/1006.0308
 4.Casalicchio, E., Lundberg, L., Shirinbab, S.: Energyaware adaptation in managed Cassandra datacenters. In: 2016 International Conference on Cloud and Autonomic Computing (ICCAC), pp. 60–71 (2016). doi: 10.1109/ICCAC.2016.12
 5.Casalicchio, E., Silvestri, L.: Mechanisms for SLA provisioning in cloudbased service providers. Computer Networks 57(3), 795–810 (2013). http://dx.doi.org/10.1016/j.comnet.2012.10.020. http://www.sciencedirect.com/science/article/pii/S1389128612003763
 6.Chalkiadaki, M., Magoutis, K.: Managing service performance in NoSQL distributed storage systems. In: Proceedings of the 7th Workshop on Middleware for Next Generation Internet Computing, MW4NG ’12, pp. 5:1–5:6. ACM, New York, NY, USA (2012). doi: 10.1145/2405178.2405183. http://doi.acm.org/10.1145/2405178.2405183
 7.Cipar, J., Xu, L., Krevat, E., Tumanov, A., Gupta, N., Kozuch, M.A., Ganger, G.R.: Jackrabbit: improved agility in elastic distributed storage (2012)Google Scholar
 8.Cohen, R., LewinEytan, L., Naor, J.S., Raz, D.: Almost optimal virtual machine placement for traffic intense data centers. In: 2013 Proceedings IEEE INFOCOM, pp. 355–359 (2013). doi: 10.1109/INFCOM.2013.6566794
 9.Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: Proceedings of the 1st ACM Symposium on Cloud Computing, SoCC ’10, pp. 143–154. ACM, New York, NY, USA (2010). doi: 10.1145/1807128.1807152. http://doi.acm.org/10.1145/1807128.1807152
 10.Dalvandi, A., Gurusamy, M., Chua, K.C.: Timeaware vmflow placement, routing, and migration for power efficiency in data centers. IEEE Trans. Netw. Serv. Manage. 12(3), 349–362 (2015). doi: 10.1109/TNSM.2015.2443838 CrossRefGoogle Scholar
 11.Dede, E., Govindaraju, M., Gunter, D., Canon, R.S., Ramakrishnan, L.: Performance evaluation of a MongoDB and Hadoop platform for scientific data analysis. In: Proceedings of the 4th ACM Workshop on Scientific Cloud Computing, Science Cloud ’13, pp. 13–20. ACM, New York, NY, USA (2013). doi: 10.1145/2465848.2465849. http://doi.acm.org/10.1145/2465848.2465849
 12.Diegues, N., Orazov, M., Paiva, J.A., Rodrigues, L., Romano, P.: Optimizing hyperspace hashing via analytical modelling and adaptation. SIGAPP Appl. Comput. Rev. 14(2), 23–35 (2014). doi: 10.1145/2656864.2656866. http://doi.acm.org/10.1145/2656864.2656866
 13.Ghanbari, H., Simmons, B., Litoiu, M., Barna, C., Iszlai, G.: Optimal autoscaling in a IaaS cloud. In: Proceedings of the 9th international conference on Autonomic computing, ICAC ’12, pp. 173–178. ACM, New York, NY, USA (2012). doi: 10.1145/2371536.2371567. http://doi.acm.org/10.1145/2371536.2371567
 14.Grozev, N., Buyya, R.: Multicloud provisioning and load distribution for threetier applications. ACM Trans. Auton. Adapt. Syst. (TAAS) 9(3), 13 (2014)Google Scholar
 15.Intel: Configuration and deployment guide for the Cassandra NoSQL data store on Intel architecture. Tech. rep., Intel Corporation (2013)Google Scholar
 16.Jiang, Y., Perng, C.S., Li, T., Chang, R.N.: Cloud analytics for capacity planning and instant vm provisioning. IEEE Trans. Netw. Serv. Manage. 10(3), 312–325 (2013). doi: 10.1109/TNSM.2013.051913.120278 CrossRefGoogle Scholar
 17.Kliazovich, D., Bouvry, P., Audzevich, Y., Khan, S.: Greencloud: a packetlevel simulator of energyaware cloud computing data centers. In: Global Telecommunications Conference (GLOBECOM 2010), 2010 IEEE, pp. 1–5 (2010). doi: 10.1109/GLOCOM.2010.5683561
 18.Kuhlenkamp, J., Klems, M., Röss, O.: Benchmarking scalability and elasticity of distributed database systems. Proc. VLDB Endow. 7(12), 1219–1230 (2014). doi: 10.14778/2732977.2732995. http://dx.doi.org/10.14778/2732977.2732995
 19.Mastelic, T., Oleksiak, A., Claussen, H., Brandic, I., Pierson, J.M., Vasilakos, A.V.: Cloud computing: Survey on energy efficiency. ACM Comput. Surv. 47(2), 33:1–33:36 (2014). doi: 10.1145/2656204. http://doi.acm.org/10.1145/2656204
 20.Megiddo, N.: On the complexity of linear programming. In: Bewley, T (ed.) Advances in Economic Theory: 5th World Congress, pp. 225–268. Cambridge University Press, Cambridge (1987)Google Scholar
 21.Mehrotra, S.: On the implementation of a (primaldual) interior point method. SIAM J. Optim. 2, 575–601 (1992)MathSciNetCrossRefzbMATHGoogle Scholar
 22.Paiva, J.A., Ruivo, P., Romano, P., Rodrigues, L.: Autoplacer: Scalable selftuning data placement in distributed keyvalue stores. ACM Trans. Auton. Adapt. Syst. 9(4), 19:1–19:30 (2014). doi: 10.1145/2641573. http://doi.acm.org/10.1145/2641573
 23.Patil, S., Polte, M., Ren, K., Tantisiriroj, W., Xiao, L., López, J., Gibson, G., Fuchs, A., Rinaldi, B.: Ycsb++: benchmarking and performance debugging advanced features in scalable table stores. In: Proceedings of the 2nd ACM Symposium on Cloud Computing, SOCC ’11, pp. 9:1–9:14. ACM, New York, NY, USA (2011). doi: 10.1145/2038916.2038925. http://doi.acm.org/10.1145/2038916.2038925
 24.Priya, B., Pilli, E., Joshi, R.: A survey on energy and power consumption models for greener cloud. In: 2013 IEEE 3rd International Advance Computing Conference (IACC), pp. 76–82 (2013). doi: 10.1109/IAdCC.2013.6514198
 25.Rabl, T., GómezVillamor, S., Sadoghi, M., MuntésMulero, V., Jacobsen, H.A., Mankovskii, S.: Solving big data challenges for enterprise application performance management. Proc. VLDB Endow. 5(12), 1724–1735 (2012). doi: 10.14778/2367502.2367512. http://dx.doi.org/10.14778/2367502.2367512
 26.Rocha, L.A., Cardozo, E.: A hybrid optimization model for green cloud computing. In: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing, UCC ’14, pp. 11–20. IEEE Computer Society, Washington, DC, USA (2014). doi: 10.1109/UCC.2014.9. http://dx.doi.org/10.1109/UCC.2014.9
 27.Salahi, M., Terlaky, T.: Mehrotratype predictorcorrector algorithm revisited. Optimization Methods Software 23(2) (2008)Google Scholar
 28.Shankaranarayanan, P., Sivakumar, A., Rao, S., Tawarmalani, M.: Performance sensitive replication in geodistributed cloud datastores. In: 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), pp. 240–251 (2014). doi: 10.1109/DSN.2014.34
 29.Sharma, U., Shenoy, P., Towsley, D.F.: Provisioning multitier cloud applications using statistical bounds on sojourn time. In: Proceedings of the 9th International Conference on Autonomic Computing, ICAC ’12, pp. 43–52. ACM, New York, NY, USA (2012). doi: 10.1145/2371536.2371545. http://doi.acm.org.miman.bib.bth.se/10.1145/2371536.2371545
 30.Shi, Y., Meng, X., Zhao, J., Hu, X., Liu, B., Wang, H.: Benchmarking cloudbased data management systems. In: Proceedings of the Second International Workshop on Cloud Data Management, CloudDB ’10, pp. 47–54. ACM, New York, NY, USA (2010). doi: 10.1145/1871929.1871938. http://doi.acm.org/10.1145/1871929.1871938
 31.Xu, Z., Deng, N., Stewart, C., Wang, X.: Cadre: carbonaware data replication for geodiverse services. In: 2015 IEEE International Conference on Autonomic Computing (ICAC), pp. 177–186 (2015). doi: 10.1109/ICAC.2015.15
 32.Ye, K., Wu, Z., Wang, C., Zhou, B.B., Si, W., Jiang, X., Zomaya, A.Y.: Profilingbased workload consolidation and migration in virtualized data centers. IEEE Trans. Parallel Distrib. Syst. 26(3), 878–890 (2015). doi: 10.1109/TPDS.2014.2313335 CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.