Keywords

1 Introduction

With the rapid development of the Internet [1], the data scale of the data center has developed rapidly. When the amount of data in the data center is increasing rapidly, the utilization of resources has become an issue of widespread concern in the industry [2]. To improve the utilization of resources, Co-allocated data centers have become an option for many companies. It is to deploy online services and offline workloads on the same cluster and share the data resources of the cluster to improve resource utilization.

There are new deadline requirements in offline applications in many enterprises [3]. For example, a shopping platform recommendation system has a data dependency relationship between offline workloads and online services. Offline workloads need to process intermediate data generated in real-time and provide timely feedback to users, guaranteeing the timeliness of the result data. Spark application is a typical offline batch workload, in traditional resource scheduling; it can’t solve the problems encountered in this scenario. In the current scenario, the input data of Spark applications, which is generated from online services can be partitioned and processed in a few phases on demand. The goal of Spark applications is to improve the throughput of data processing while ensuring the deadline requirement. Multiple Spark applications are executed at the same time in the co-allocated data center. How to partition the data and allocate resources among multiple applications has become a big challenge. This paper proposes a resource-scheduling model for Spark in co-allocated data centers, which can reasonably provide data-resource allocation for Spark applications and process more data while meeting the deadline requirement.

The rest of the paper is organized as follows. Section 2 introduces the related work of this article. Section 3 introduces the detailed design of time prediction modeling and the data-away resource scheduling strategy of the Spark application. Section 4 conducts experimental evaluation and analysis. Section 5 summarizes the main contributions of this paper.

2 Related Work

Resource scheduling of applications has been a major research direction in recent years. In the previous resource scheduling research, Kewen Wang and Mohammad Khan Divide a single application into multiple intervals to dynamically, allocate resources to save more resources and improve the utilization of resources [4]. Zhiyao Hu et al. optimized the Shortest Job First Scheduling, by fine-tuning the resources of one job for another job, until the predicted completion time of the job stops decreasing, reducing the overall running time [5].

However, more and more applications have new requirements for the deadline, which has not been considered in previous studies; Guolu Wang et al. proposed a hard real-time algorithm DVDA [6]. Compared with the traditional EDF algorithm, it not only considers the deadline of the application, but also considers the value density, resets the value weight function, and allocates resources to the highest weighted application by priority. With the advent of the data center, there is a dynamic change of available resources, Dazhao Cheng et al. propose a resource and deadline-aware Hadoop job scheduler RDS [7]. The resource allocation is adjusted in time through time prediction, Each job is divided into ten intervals, the resource allocation is adjusted through the execution time and forecast time of each interval, and a simple and effective model is also proposed to predict future resource availability through the recent historical available resources.

With the rapid increase of job scale, many parallel jobs are limited by the network that the cluster is difficult to expand. It is necessary to reduce the cross-rack network traffic by improving the locality of rack data. Faraz and Srimat proposed that ShufflerWatcher [8] tried to arrange the Reducer on the same rack as most M appers to localize the Shuffle stage, but only considering the situation of a single job for independent scheduling, Shaoqi Wang et al. found that there are data dependencies between many jobs in reality [9], and proposed Dawn composed of the online plan and network adaptive scheduler. The online plan determines the preferred rack according to the input data position of the task and the task relevance. After the network adaptive scheduler finds the idle resources of the rack, it selects the appropriate job to schedule on the rack according to the current network status.

3 Model Design

This chapter first introduces the framework overview, then it introduces the design scheme of the time prediction modeling and the data-aware resource scheduling.

The scheduling goal of this paper is the proportion of meeting deadline requirements and the throughput of data processing. The expression is as follows:

$$ {\text{DAR}} = \frac{1}{n}\mathop \sum \limits_{i = 1}^n f(y_i {,}y_i^{\wedge}),f(y_i {,}y_i^{\wedge}) = \left\{ {\begin{array}{*{20}l} {0{,}} \hfill & {y_i > y_i^{\wedge}} \hfill \\ {1{,}} \hfill & {y_i \le y_i^{\wedge}} \hfill \\ \end{array} } \right.{,} $$
(1)
$$ {\text{DTR}} = \mathop \sum \limits_{i = 1}^n D_i . $$
(2)

\(y_i\) and \(y_i^{\wedge}\) represent the actual execution time and deadline time of application i respectively, and the function \(f(y_i {,}y_i^{\wedge})\) represents whether application i is completed before the deadline. \(D_i\) represents the throughput of data processing for application i.

3.1 Framework Overview

This paper proposes a data-away resource scheduling model based on time prediction. The model is mainly divided into two parts, the first part is to perform time prediction modeling for each Spark application separately to ensure that it can be completed while meeting deadline requirements. The second part is the resource scheduling optimization algorithm, which uses the heuristic algorithm to select the best data-resource allocation plan to ensure that each application is completed while meeting deadline requirements and maximizing the data processing capacity of the spark application. The overall design framework is as follows (Fig. 1):

Fig. 1.
figure 1

Framework of the model

Full size image

3.2 Prediction of Spark Application Execution

This paper selects SVM as a time predictive modeling tool [10, 11]. SVM is a machine learning method developed in the mid-1990s, mainly to minimize the experience risk and confidence range to improve the generalization ability of the learning machine so that better statistics can be obtained in a small sample. Our goal is to predict the execution time of the Spark application, so we need to select the key factors that affect the time prediction. Since this paper models each application separately, internal factors such as the number relationship between the action operator and the transformation operator of the application, the number of shuffles, etc. are not included in the influencing factors. The main influencing factors selected in this paper are input data scale, core and memory resources.

Support vector regression is to transform the original input data x through a non-linear mapping into the corresponding high-dimensional feature space. The linear representation is \({{\upvarphi }}({\text{x}})\), and the linear regression is completed. SVR is the method of regression prediction [12, 13]. Through the Lagrangian multiplier method and KKT condition, the SVR can be expressed as:

$$ f(x) = \mathop \sum \limits_{i = 1}^m (\alpha^{\prime}_i - \alpha_i )k(x{,}x_i ) + b . $$
(3)

where \(k(x,x_i ) = \emptyset (x_i )^T \emptyset (x_j )\) is the kernel function. Commonly used kernel functions are linear kernel function, polynomial kernel function, and radial basis kernel function. Select 75% of the samples as the training data, and select the best kernel function through experiments to construct the prediction model.

PSO-Based Resource Scheduling Strategy.

The Particle Swarm Optimization (PSO) is a search optimization algorithm with simple operation and fast convergence speed [14, 15]. Each particle in PSO represents a feasible solution to the target problem; each particle mainly contains two attributes: position and velocity. The position represents a feasible solution, the velocity represents the moving speed and direction of the particle, and the movement process of the particle is called the search process of the particle. The update formula for the velocity and position of each particle is as follows:

$$ \begin{gathered} \quad \quad V_i (t + 1) = \omega *V_i (t) + \rm{\complement }_1 *rand*(Pb_i (t) - X_i (t)) \hfill \\ + \rm{\complement }_2 *ran*(gb(t) - X_i (t)){,} \hfill \\ \end{gathered} $$
(4)
$$ X_i (t + 1) = X_i (t) + V_i (t) . $$
(5)

In Eq. (4), t represents the number of iterations. \(Pb_i\) and \(gb\) respectively represent the optimal position of the ith particle and the global optimal position. \(\omega\) is the inertia factor, \({ }\rm{\complement }_1\) represents the cognitive ability of the particle, and \(\rm{\complement }_2\) represents the learning ability of the particle swarm. \(rand\) represents a uniform function in [0,1].

Definition of Particles.

In the PSO, the definition of particle swarm P is expressed as follows:

$$ {\text{P}} = \{ P_q {|}1 \le {\text{q}} \le {\text{pNumber}}\} . $$
(6)

\({\text{pNumber}}\) represents the number of particle swarms, and \(P_q\) represents particles. The formula of \(P_q\) is as follows:

$$ P_q = \{ (d_i {,}c_i {,}m_i ){|}1 \le i \le n\} . $$
(7)

In Eq. (7), n represents the number of spark applications, \((d_i {,}c_i {,}m_i )\) represents a data-resource scheduling solution of the ith spark application, \(d_i\) represents the throughput of data processed by the ith spark application, \(c_i\) and \(m_i\) respectively represent the core and memory resources allocated by the cluster.

Definition of Particle Fitness.

Each particle represents a data-resource scheduling solution between spark applications, and the fitness function of the particle represents the revenue that each particle can bring. The scheduling goal of this paper is that Spark applications can improve the throughput of data processing while ensuring the deadline requirement. Therefore, this paper sets the particle fitness as the sum of the data processed by each application, the fitness expression of particles is as follows:

$$ {\text{E}} = d_1 + d_2 + \ldots + d_n {,} $$
(8)
$$ {\text{s}}.{\text{t}}.{ }y_i \le deadline{,} $$
(9)
$$ \mathop \sum \limits_{i = 1}^n c_i \le C,\mathop \sum \limits_{i = 1}^n m_i \le M,\mathop \sum \limits_{i = 1}^n d_i \le D, $$
(10)
$$ c_i \ge 0,m_i \ge 0,d_i \ge 0 . $$
(11)

The constraints of Eqs. (10) and (11) respectively indicate that each application needs to be completed before the deadline, and the allocated data-resources are less than the currently available data-resources.

4 Experimental Results and Analysis

4.1 Experimental Setup

The selection of the experimental environment in this paper is a Spark cluster composed of 15 nodes, including 1 master node and 14 worker nodes. The detailed configuration of each Spark node is shown in Table 1.

The experiment in this paper is divided into two parts, one is the experiment of time prediction model accuracy, and the other is the experiment of resource scheduling strategy performance comparison. In the experiment, Wordcount, Sort, and Pagerank in Hibench are selected as the experimental workload.

Table 1. Experimental environment configuration
Full size table

4.2 Accuracy of Spark Application Execution Time Prediction

We use different kernel functions to model the time prediction of each application, and evaluate the accuracy of each model through RMSE and MAPE, and select the best time prediction model. The results of the prediction model for different workloads are shown in Fig. 2 and Fig. 3.

Through the comparison of time prediction accuracy under different kernel functions in Fig. 2, we can see that different applications can get a better prediction effect when using linear kernel function for time prediction. The time prediction results obtained by using the linear kernel function are more similar to the real execution time.

Fig. 2.
figure 2

Time prediction accuracy of different kernel functions

Full size image

Figure 3 evaluates the large error and relative error in the prediction results by using the evaluation indexes RMSE and MAPE, it can be obtained that when the linear kernel function is used for time prediction, the RMSE is reduced by an average of 27%, and the MAPE is reduced by an average of 1.9%. Therefore, the linear kernel function is selected as the kernel function for time prediction modeling.

Fig. 3.
figure 3

Experimental evaluation of different kernel functions

Full size image

4.3 Performance of The Resource Scheduling Strategy

Our resource scheduling strategy is compared with the conservative resource scheduling strategy and the radical resource scheduling strategy, using the DAR and TAR in Sect. 3 as the evaluation indicators of the experiment. The experiment is carried out in the cluster with variable resources, and the following experimental results are obtained.

Fig. 4.
figure 4

Performance of different resource scheduling strategies

Full size image

It can be seen from Fig. 4 that our method can bring about an increase in the throughput of data processing and the proportion of meeting deadline requirements. Compared with conservative and radical scheduling strategies, our resource scheduling strategy increases the throughput of application processing data by an average of 12% and 50%, respectively. The proportion of applications that meet deadline requirements has increased by 20% and 50%, respectively. Although the conservative resource scheduling strategy can ensure that there is output in the deadline demand, it cannot ensure that more data is processed before the deadline; Although the radical resource scheduling strategy can guarantee the processing of as much data as possible, it cannot guarantee the deadline requirements of Spark application, so there will be less effective data processing; the scheduling strategy in this paper takes into account both the demand for the deadline requirements and the demand for the throughput of data processing, and achieves a good result.

5 Conclusions

This paper proposes a resource-scheduling model for Spark in co-allocated data centers. This method is based on a time prediction model, which increases the throughput of data processing while meeting the deadline requirement, and it solves the new requirements of Spark applications. In the future, we intend to improve the performance of the data-away resource scheduling strategy by increasing the accuracy of time prediction and refining the conditions of scheduling policy.