1 Introduction

Smart substation acts as an important foundation and pillar of strong smart grid, which has characteristics of information digitization, networking communication platform and information sharing standardization [1, 2], and completes some functions of system monitoring, controlling and protection, etc. A trend of huge amount of data with characteristics of large scale, complex types, and wide area distribution produced by smart substation makes the traditional relational database more and more difficult to adapt to the requirements of large scale data processing from power enterprises [3, 4]. Presently, big data storage and processing are mostly based on large scale servers with relational database management systems, which need huge investment and have a shortage of low utilization ratio and poor scalability. Therefore, the design of power data center using traditional system is far from the requirements of big data storage, analysis and processing. Thus, how to process and analyze massive data produced by smart substation effectively becomes a great challenge. It is urgent to research on effective storage technology for big data [5].

Data warehouse using Hive is an infrastructure built on top of Hadoop cloud computing framework, with good scalability and fault tolerance [6, 7], which can integrate with lossless compression algorithms, such as BZip2, Deflate, Gzip and Lzo. Its underlying operations can be transformed into MapReduce parallel tasks [8–10], and its application interface uses HQL language, which provides the ability of quick development. Hive is different from the relational database. It has no special data formats, but it has three kinds of storage formats, including TextFile, SequenceFile and RCFile. Hive is designed towards the query and analysis of massive data, which can be used to build a data warehouse for processing big data of smart substation.

Considering the characteristics of big data of smart substation and Hive, a data compression processing method based on Hive is proposed to solve the mentioned problems. Experimental results show that it has a significant theoretical and practical value for processing big data of smart substation.

2 Hive and storage formats

2.1 Processing flow of Hive

Hive is introduced firstly in order to study the smart substation based on Hive. Hive is an open source data warehouse project with an extension based on Hadoop cloud computing platform published by apache software foundation, thus it supports a wide of data types, various kinds of structured and unstructured data with complex and heterogeneous storage formats [11]. Combined with the traditional structured query syntax, Hive itself defines Hive query language (HQL), through the analysis of HQL syntax by the driver. HQL tasks are transformed into MapReduce parallel tasks, thus they can take full advantage of the high performance and scalability of the cloud computing and realize complex processing for the big data. MapReduce parallel processing flow of Hive is shown in Fig. 1.

Fig. 1
figure 1

MapReduce parallel processing flow of Hive

Full size image

Hadoop distributed file system (HDFS) is the file management foundation of reading or writing data based on Hive. The unified management of distributed data is carried out by Namenode, Datanodes and client applications. Data processing flow based on Hive is shown in Fig. 2.

Fig. 2
figure 2

Data processing flow based on Hive

Full size image

Namenode acts as the management master of HDFS. Datanodes are responsible for the data blocks storage in HDFS, and reporting their status to Namenode with the heartbeat response periodically. If the Namenode does not obtain heartbeats from a Datanode, it will modify the configuration for Datanodes’ directory, and determine whether the Datanode appears fault. If so, it will not get the data operation request, then the client will read the same blocks from another Datanode, and the client applications access to the data in a streaming way in the HDFS system. Hive provides the applications with command line interface (CLI), client interface (Client) and web user interface (WUI). Attributes such as table name, column, and partition of Hive are stored in metadata database.

Reading request of data is sent to Namenode by the client process, and then the client reads the data in an FSInput streaming way, according to the distribution of data blocks stored in different Datanodes. Writing request of data is sent to Namenode by the client process, and then the client writes data in an FSOutput streaming way to different Datanodes specified by Namenode.

2.2 Compression storage formats of Hive

  1. 1)

    TextFile acts as the default storage format, which can be combined with the different lossless compression algorithms, as well as be detected and decompressed automatically by Hive.

  2. 2)

    SequenceFile is a kind of binary file which the Hadoop provides, the data will be serialized in files in the form of <key, value> pairs. SequenceFile of Hive inherits from the SequenceFile the Hadoop provides. SequenceFile format and its compression ways are shown in Fig. 3.

    Fig. 3
    figure 3

    SequenceFile format and its compression ways

    Full size image
  3. 3)

    RCFile is a special column oriented storage format, which skips the unrelated columns in query process. In fact, it does not really skip unwanted columns to jump to the target columns, but scan the stored metadata header of each row group to complete the above function. RCFile and its compression way are shown in Fig. 4.

    Fig. 4
    figure 4

    RCFile format and its compression way

    Full size image

This section introduces the principle of data processing and storage formats of Hive, which lays a theoretical foundation for the following sections.

3 Applications of substation based on Hive

Smart grid acts as the future development direction of the power grid, which includes power generation, transmission, distribution, conversion and dispatching, etc. Undoubtedly, smart substation is one of the most important links in the power gird [12–14], which is mainly composed of primary intelligent electronic device (IED) and secondary networking equipments. Monitoring and control systems play an important role in completing the ordinary operation of smart substation. Some main monitoring data of smart substation is listed in Table 1.

Table 1 Main monitoring data of smart substation
Full size table

In order to deal with the big data problems of smart substation, the applications of Hive can be integrated into the system of the smart substation, which is divided into three layers (processing layer, bay layer and substation layer). The applications of substation layer are based on bay layer and processing layer, include SCADA monitoring system and some other management systems.

The monitoring system and management system are integrated with Hive, not only can complete functions of automatic monitoring, automatic control, auxiliary decision and information sharing, but also can complete functions of big data mining and multidimensional data analysis, etc. The structure of smart substation system based on Hive is shown in Fig. 5.

Fig. 5
figure 5

Structure of smart substation system based on Hive

Full size image

The data processing flow of smart substation based on Hive can be logically divided into data source layer, computing layer, control layer and application layer. SCADA, data mining, auxiliary decision and multidimensional data analysis and other functions can be realized by using HQL interfaces. Four logical layers of data processing flow in the smart substation based on Hive are shown in Fig. 6.

Fig. 6
figure 6

Four logical layers of data processing flow based on Hive

Full size image

4 Analysis of results

Firstly, cloud computing cluster is built on Hadoop platform constructed in Ubuntu11.10 system, composed of a Namenode (Master) and three Datanodes (Data 1, Data 2 and Data 3). Hive data warehouse infrastructure is built on top of Hadoop. Distributed cloud computing cluster of Hive is shown in Fig. 7.

Fig. 7
figure 7

Distributed cloud computing cluster of Hive

Full size image

Secondly, load the massive substation data into Hive data warehouse. Take 15 monitoring simulation values of substation as an example, to study the data compression and storage.

4.1 Comparison of query time

The first experiment is carried out on three kinds of storage formats to study the query efficiency. Thirty million monitoring data records are stored in three kinds of storage formats, respectively. The query time of one field and eight fields is shown in Fig. 8.

Fig. 8
figure 8

Query time in three formats

Full size image

As shown in Fig. 8, comparing with the query time of one field and eight fields in three kinds of storage formats, the query time of RCFile is relatively less, the query time of TextFile is middle, while query time of SequenceFile is relatively more.

4.2 Lossless compression

Hive supports Bzip2, Deflate, Gzip and Lzo compression type. In order to verify the query efficiency after compression, the second experiment is carried out under condition of five million monitoring records, testing three kinds of storage formats, i.e., TextFile, SequenceFile (compressed in block way) and RCFile by using four kinds of lossless compression (BZip2, Gzip, Deflate and Lzo) [15–17], respectively.

The lossless compression ratios processed by different kinds of algorithms on three kinds of storage formats based on Hive are shown in Fig. 9.

Fig. 9
figure 9

Lossless compression ratios based on Hive

Full size image

As shown in Fig. 9, the BZip2 compression ratio is higher than those of the other three kinds of lossless compression algorithms. In condition of RCFile storage format, the compression ratio of RCFile reaches about 81.3 %, approximately 3.5 % higher than those of TextFile and SequenceFile. The lossless compression ratios of Deflate and Gzip algorithms reach about 73.4 %, while the Lzo compression ratio reaches about 56.8 %.

Query time with and without data compressions on three kinds of storage formats is shown in Fig. 10 (select V001 from table_name where Num = Num_max; select V001,…,V008 from table_name where Num = Num_max).

Fig. 10
figure 10

Comparison of query time

Full size image

Experimental results show that query time of BZip2 algorithm is relatively higher, and the efficiency is reduced by data compression.

Query time after Deflate, Gzip and Lzo become less than that without compression, which improves the query efficiency, at the same time, saving the storage capacity.

Although the BZip2 compression does not improve the query efficiency, when data stored in RCFile storage format, the query time of BZip2 almost equals to the efficiency without compression. It is showed that the RCFile improves the query efficiency to some extent.

Based on the above experimental results, big data of smart substation can be stored into Hive after compression according to actual demands.

4.3 Efficiency analysis of cluster

In Hive cluster system with p processors, if the parallel degree i satisfy \(i \le p\) (i = 1, 2, ···, n), without considering the parallel overhead, the adjacent speedup in the system can be defined simply as follows:

$$S_{(m,n)} (p) = \frac{{{{X_{n} } \mathord{\left/ {\vphantom {{X_{n} } {T({\text{X}}_{n} )}}} \right. \kern-0pt} {T({\text{X}}_{n} )}}}}{{{{X_{m} } \mathord{\left/ {\vphantom {{X_{m} } {T(X_{m} )}}} \right. \kern-0pt} {T(X_{m} )}}}} = \frac{{X_{n} T(X_{m} )}}{{X_{m} T(X_{n} )}}$$
(1)

where X m and X n are the work loads; T(X m ) and T(X n ) are the parallel running time.

Considering the parallel overhead, the adjacent speedup can be further described as:

$$\begin{aligned} S^{\prime}_{{(m,n)}} (p) &= \frac{{X_{n} /\left( {T(X_{n} ) + O({\text{X}}_{n} )} \right)}}{{X_{m} /\left( {T({\text{X}}_{m} ) + O(X_{m} )} \right)}} = \\ &\quad \frac{{\sum\limits_{{j = 1}}^{m} {X_{{m,j}} /V_{{m,j}} } + O(X_{m} )}}{{\sum\limits_{{i = 1}}^{n} {X_{{n,i}} /V_{{n,i}} } + O(X_{n} )}}\frac{{{\text{X}}_{n} }}{{X_{m} }} = \\ & \quad\frac{{X_{m} \sum\limits_{{j = 1}}^{m} {f_{{m,j}} /f_{{m,j}} V_{{m,j}} + O(X_{m} )} }}{{X_{n} \sum\limits_{{i = 1}}^{n} {f_{{n,i}} /f_{{n,i}} V_{{n,i}} } + O(X_{n} )}}\frac{{X_{n} }}{{X_{m} }} = \frac{{E_{n} }}{{E_{m} }} \hfill \\ \end{aligned}$$
(2)

For a system which parallel degree is i, \({{X}}_{n,i} = f_{n,i} {{X}}_{n}\), \({{X}}_{m,j} = f_{m,j} {{X}}_{m}\), \(i = 1,2,\cdot\cdot\cdot,n\), \(j = 1,2,\cdot\cdot\cdot,\;m\); \(f_{{n,i}}\) and \(f_{{m,j}}\) are the work load coefficients; \(V_{n,i}\) and \(V_{m,j}\) are running speed; O(X n ) and O(X m ) are parallel overhead time; \(\begin{aligned} E_{n} = X_{n} X_{m} \sum\limits_{j = 1}^{m} {{{f_{m,j} } \mathord{\left/ {\vphantom {{f_{m,j} } {V_{m,j} }}} \right. \kern-0pt} {V_{m,j} }}} + X_{n} O(X_{m} );\;E_{m} = X_{m} X_{n} \sum\limits_{i = 1}^{n} {{{f_{n,i} } \mathord{\left/ {\vphantom {{f_{n,i} } {V_{n,i} }}} \right. \kern-0pt} {V_{n,i} }}} + X_{m} O(X_{n} ). \hfill \\ \end{aligned}\)

Parallel computing should be executed as \(\left\lceil {i/p} \right\rceil\) times, the computing should be grouped by p to complete the computation of parallel degree i, when i is larger than p, at his time the adjacent speedup is described as:

$$S'_{(m,n)} (p) = \frac{{X_{m} \sum\limits_{j = 1}^{m} {{{ \, \left\lceil {j/p} \right\rceil f_{m,j} } \mathord{\left/ {\vphantom {{ \, \left\lceil {j/p} \right\rceil f_{m,j} } {V_{m,j} }}} \right. \kern-0pt} {V_{m,j} }} + O \, (X_{m} )} }}{{X_{n} \sum\limits_{i = 1}^{n} {{{ \, \left\lceil {i/p} \right\rceil f_{n,i} } \mathord{\left/ {\vphantom {{ \, \left\lceil {i/p} \right\rceil f_{n,i} } {V_{n,i} }}} \right. \kern-0pt} {V_{n,i} }}} + O \, (X_{n} )}}\frac{{{{X}}_{n} }}{{{{X}}_{m} }} = \frac{{E_{n}^{'} }}{{E_{m}^{'} }}$$
(3)

where the value of \(\left\lceil {i/p} \right\rceil\) is the minimum integer not less than \(i/p\). Parallel overhead \(O(x)\) which is a complicated function related with software and hardware and application including interactive, communicational and parallel overhead.

In fact, many factors impact on the parallel efficiency, therefore, the relative efficiency increment caused by the relative amount increment of data can be used to reflect the performance of the cluster comprehensively. Hence, the following mathematical formula can be obtained:

$$\begin{aligned} C_{{(m,n)}} (p) &= \left( {\mathop {\lim }\limits_{{\Delta x \to 0}} \frac{{E_{n} - E_{m} }}{{E_{m} }}\Bigg/\frac{{x_{n} - x_{m} }}{{x_{m} }}} \right) = \mathop {\lim }\limits_{{\Delta x \to 0}} \left( {\frac{{\Delta E/E}}{{(\Delta x/x)}}} \right) = \\ & \quad \mathop {\lim }\limits_{{\Delta p \to 0}} \frac{{\Delta E/\Delta x}}{{E/x}} = x\frac{{{\text{d}}E}}{{{\text{d}}x}}\Bigg/E = x\frac{{{\text{d}}\ln E}}{{{\text{d}}x}}\\ \end{aligned}$$
(4)

where variation C (m,n)(p) is a complex function which reflects the capability of running programs in parallel processing system, related with the workload X, the serial bottleneck, the load coefficient and some other factors.

Operations of HQL tasks are transformed into MapReduce parallel tasks, so the third experiment is carried out in order to test the parallel compression consuming time in three kinds of storage formats of Hive, by using BZip2, Deflate, Gzip, and Lzo four kinds of lossless compression algorithms, respectively.

Take one million, three million, five million, eight million, ten million, and twelve million monitoring data records as the data research object, record the parallel compression consuming time in different number of data records, then the curve of compression time is draw, as shown in Fig. 11.

Fig. 11
figure 11

Compression consuming time on three storage formats

Full size image

It is can be seen from Fig. 11 that the curve presents a convex trend, that is to say, the compression time of the more records is less than that of the less records. In order to quantitatively analyze the curve of the compression time in Fig. 11, S′(m,n)(p) and C (m,n)(p) are calculated with (2), (3) and (4). S′(m,n)(p) and C (m,n)(p) are shown in Table 2.

Table 2 S′(m,n)(p) and C (m,n)(p) of cloud cluster
Full size table

As shown in Table 2, when data records exceed three million, S′ (3,5), S′ (5,8), S′ (8,10), and S′ (10,12) are not less than one, which means that the processed data size in a unit of time increases, compression efficiency improves to some extent, as curves shown in Fig. 11 that, with data records increases, Hadoop cluster has a better compression executing efficiency, the compression efficiency increases to some extent.

C (m,n)(p) shows that different compression algorithms on different storage formats can provide detail information.

5 Conclusions

  1. 1)

    Storage format experiments verify that the query time of RCFile for big data is relatively less than that of TextFile and SequenceFile, and so big data of smart substation can be stored with RCFile format because of its better time response.

  2. 2)

    Lossless compression experiments verify that big data of smart substation can be stored into Hive after compression, and query efficiency of data compressed by Lzo is higher than that by Gzip, Deflate and BZip2, while BZip2 compression ratio of data is relatively higher.

  3. 3)

    Parallel compression experiments verify that with the data records increase in a certain range, the cluster has a better parallel processing efficiency, and S′(m,n)(p) and C (m,n)(p) of cloud cluster further prove that big data processing of smart substation based on Hive is feasible.