1 Introduction

Solar energy is an important renewable energy resource, but is intermittent in the short period owing to the fluctuations of solar radiation. Dramatic fluctuations cause the energy output of a solar power plant to rapidly decrease from hundreds of megawatts to zero output within a few minutes, and bring about huge risk to the stability of the electrical grid [1]. Therefore, an accurate forecast of solar power is the premise and key technology of the grid-connection for photovoltaic (PV) or concentrated solar thermal (CST) plants [2, 3]. For various power system operations, the geographical and temporal requirements differ in a solar power forecast [4, 5].

Solar power significantly depends on the solar irradiance, such as the global horizontal irradiance (GHI) for PV plants and the direct normal irradiance (DNI) for CST plants [6], and thus the solar power output forecast can be transformed into a solar irradiance (GHI or DNI) forecast [7,8,9]. In this study, we focus on a DNI forecast. The DNI is mainly affected by surface meteorological variables and column integrated variables, such as clouds and water vapor. Except for clouds, these atmospheric variables have been considered in most clear-sky models for a DNI estimation [10] or numerical weather prediction for a day-ahead solar irradiance forecast [11, 12] and long-term solar irradiance estimation [13]. However, for an inter-hour DNI forecast in all sky conditions, clouds have been the only factor considered in certain forecast models, and other atmospheric variables have been ignored [14,15,16,17,18,19]. Recently, the majority of all-sky DNI forecast models have employed two main parts: clear-sky DNI forecast and cloud fraction forecast [20,21,22]. In this way, except for clouds, the effectiveness of other atmospheric variables is considered in a clear-sky DNI estimation. However, the accuracy of clear-sky forecast is limited by the immeasurability of most clear-sky atmospheric variables under all-sky conditions, and thus most clear-sky models applied in all-sky forecast are empirical with few atmospheric variables [23,24,25].

To achieve a higher accuracy of an inter-hour DNI forecast, [26] modified a clear sky model by considering the variation of the atmospheric components. Reference [27] also developed a forecast method using adaptive clear-sky models. However, the improvement in accuracy was limited by adjusting these models. In this study, a framework of an inter-hour DNI forecast was developed by directly considering the atmospheric variables, including the historically measured relative humidity, wind speed and direction, DNI, and clouds, presented through a numerical time-series or images. To fuse different types of data and avoid some inputs overwhelmed by the large magnitude of other variables, all inputs were transformed into dimensionless variables. Firstly, a ground-based cloud image was in-painted and corrected to obtain more accurate all-sky information, and the cloud covers in six key areas were then extracted separately, each of which was set as the input of the forecast model. Secondly, the measured historical DNI was transformed into a clear-sky index using a clear-sky model, and was set as an input of the forecast model. Finally, a support vector regression (SVR) model was employed to predict the DNI for the following 10 minutes, with multiple inputs including the cloud cover, clear-sky index, wind speed, and relative humidity (instead of water vapor).

The remainder of this paper is constructed as follows. The instrument and data collection are introduced in Section 2. A forecast framework based on multiple data types and time-series is described in Section 3. Experiments carried out to evaluate the performance of the proposed method, along with the results and a discussion, are detailed in Section 4. Finally, some concluding remarks are provided in Section 5.

2 Data collection

All ground-based cloud (GBC) images and measured data employed were downloaded from the open database of the Solar Radiation Research Laboratory (39.74°N, 105.18°W, at 1828.8 m above sea level), which was provided by the National Renewable Energy Laboratory (NREL) [28].

2.1 GBC images

A Total Sky Imager (TSI-880, Yankee Co.) was used to automatically capture 24-bit color sky images every 30 s. Figure 1 shows two raw GBC images in which a valid circular sky region is distorted by a fisheye lens, and a part of the sky is obstructed by both the support arm and a shadow band, which are marked with dashed-line and solid-line rectangles, respectively.

Fig. 1
figure 1

Raw GBC images taken by TSI-880

Full size image

A total of 73 clear-sky GBC images were selected from 2013 to construct a clear-sky image dataset (CSID). All clear-sky images correspond to a solar zenith ranging from 89° to 17° (step size of 1°). The CSID set was constructed for the following preprocessing of the image inpainting to detect error pixels around the sun, as discussed in Section 3.1.

2.2 Measured numerical data

Some measured data were employed in the following proposed forecast framework, including the DNI, related humidity, wind speed, and wind direction. The DNI was measured using a pyrheliometer (Kipp & Zonen, CH1), mounted on a sun-following tracker and pointing precisely to the sun center within a 5° field of view, with a sampling interval of 1 min. An average of ten measured 1-min DNI values, as well as the wind speed and wind direction, were organized as the experimental data for the forecast model, contributing to the 10-min resolution dataset applied in this study. It is worth noting that the wind direction is from the north and ranges clockwise from 0° to 360°. The average of two wind directions is not always one-half of their arithmetic sum, but should be at the angular bisector of the smaller angle between the two directions. In other words, it is the opposite of their arithmetical average if their bias is greater than 180°. Considering the wind speed, the vector average method (VAM) [29] was used, and the average wind direction (Dave) during a 10-min period was calculated as follows:

$$D_{ave} = \left\{ {\begin{array}{ll} {D^{\prime}_{ave} \, } & {C_{y} > 0, C_{x} > 0} \\ {D^{\prime}_{ave} + 180^{ \circ } } & {C_{y} > 0, C_{x} < 0} \\ {D^{\prime}_{ave} + 180^{ \circ } } & {C_{y} < 0, C_{x} < 0} \\ {D^{\prime}_{ave} + 360^{ \circ } } & {C_{y} < 0, C_{x} > 0} \\ \end{array} } \right.$$
(1)
$$D^{\prime}_{ave} = \arctan \frac{{C_{y} }}{{C_{x} }}$$
(2)

where \(C_{y} = \sum\limits_{i = 1}^{10} {S_{i} \sin D_{i} }\) is the sum of y-components, Si is the wind speed, Di is the wind direction; and \(C_{x} = \sum\limits_{i = 1}^{10} {S_{i} \cos{D_{i}} }\) is the sum of x-components.

3 DNI forecast based on multiple data types and time-series

A new framework was developed for predicting the inter-hour DNI by combining GBC images and multiple numerical time-series. Figure 2 shows the structure of the forecast framework. To make full use of all inputs, the GBC images should be pre-processed to obtain more realistic sky conditions, and the inputs of the forecast model should be non-dimensionalized to avoid some inputs with a large magnitude overwhelming the others.

Fig. 2
figure 2

Structure chart of DNI forecast framework based on multiple time-series and GBC images

Full size image

3.1 GBC image processing

As shown in Fig. 3, the valid sky of a GBC image was first segmented into mathematical coordinates. The location of the sun (xs, ys) in the GBC image could then be calculated as follows:

$$r_{s} = r_{e} \cos z$$
(3)
$$x_{s} = x_{0} + r_{s} \sin \alpha$$
(4)
$$y_{s} = y_{0} + r_{s} \cos \alpha$$
(5)

where (x0, y0) is the center of the GBC image in Fig. 3a; rs is the amplitude of the sun in a GBC image; re is the amplitude of the sun at the 90° solar zenith; z is the solar zenith angle; and α is the solar azimuth angle. The sun was then recognized as a red point, as shown in Fig. 3d, and the shadow band was then recognized as a rectangle (white region) whose symmetry axis uses the line determined based on the center of the GBC image and the sun point. Finally, the support arm is fixed in a GBC image such that it could be artificially designated (green region).

Fig. 3
figure 3

GBC image segment

Full size image

Because of strong sunlight, the pixels around the sun are whiter and brighter in a clear sky and it is easy to detect these pixels as clouds without other specific information. Hence, the clear-sky index (k) is used to determine whether the sun is obscured by clouds, and is defined as follows:

$$k = \frac{{I_{m} }}{{I_{clr} }}$$
(6)

where Im is the measured DNI value; and Iclr is the clear-sky DNI value calculated using the clear-sky model developed in [26]. The clear-sky DNI indicates the theoretical maximum DNI under cloud-free conditions, and the clear-sky index (k) describes the attenuation of the DNI caused by clouds. Therefore, if k is bigger than the threshold, e.g. Tk = 0.9, the sun is viewed as unobstructed, and the white pixels around the sun are possible error pixels caused by an overexposure.

After determining the error pixels, distinguishing them from the surrounding clouds is the greatest difficulty. Various cloud detection algorithms have been proposed, each of which has shown a satisfactory performance under a specific situation. Among them, the clear-sky background different (CSBD) algorithm outperforms other methods in a circumsolar area when the sun is not obstructed by clouds, but it fails to detect cloud which blocked the sun or is optically thick and dark [30]. The red-blue ratio (R/B) algorithm achieves better results when the clouds are optically thick and dark, but it fails to distinguish clouds from clear sky when cirrus or stratus clouds are present [31]. Combining the advantages of the CSBD and R/B algorithms in cloud detection, the error pixels in a GBC image (It) are marked using the following steps.

Step 1: search a clear-sky GBC image Ic based on the closest solar zenith from the constructed CSID introduced in Section 2.1.

Step 2: rotate the image Ic to make its azimuth angle the same as the target image It.

Step 3: calculate the green-channel histograms of the two images (Ic and It).

Step 4: adjust the green channel of the clear-sky image (Ic) by multiplying a ratio to make its green distribution the same as that of It.

Step 5: mark the pixel whose difference between the two green channels is greater than threshold Tg as a cloud (M1).

Step 6: mark the pixel as a cloud (M2) using the R/B algorithm.

Step 7: remove the error pixels (M2⊕M1) around the position of the sun.

The mirror gradient algorithm [32] was used to fill in the occlusions in a GBC image by combining the symmetry of the inpainted region and the texture features of the cloud. Because the width of a shadow band is too large to ignore the distortion, the polar coordinates were used to interpolate values from the pixels at the same image zenith angle of the target pixel [33], which differs from the inpainting algorithm in [32]. As Fig. 3c shows, the point (x, y) is viewed as the target to be inpainted, and its symmetrical point (x′, y′) in an arc with radius r is over the line of point (x0, y0) and the sun; the symmetric point (x1, y1) is then obtained with respect to the right border, and the point (x2, y2) symmetric to point (x′, y′) is obtained with respect to the left border. The polar angle (γ) is changed from 0° to 120° to search the inpainted pixels and their symmetric pixels, and the range of radius (r) is changed from 0 to re. Hence, the mirror gradient algorithm was modified as follows:

$$\begin{aligned} P(x,y) & = \lambda_{1} (P(x_{1} ,y_{1} ) + \Delta P(x_{1} ,y_{1} )) \\ & \quad + (1 - \lambda_{1} )(P(x_{2} ,y_{2} ) - \Delta P(x_{2} ,y_{2} )) \\ \end{aligned}$$
(7)
$$\begin{aligned} P(x^{\prime},y^{\prime}) & = \lambda_{1} (P(x_{2} ,y_{2} ) - \Delta P(x_{2} ,y_{2} )) \\ & \quad + (1 - \lambda_{1} )(P(x_{1} ,y_{1} ){ + }\Delta P(x_{1} ,y_{1} )) \\ \end{aligned}$$
(8)

where λ1 is the weight and ∆P is the gradient of the pixel P along the arc determined by the four pixels. For the support arm shown in Fig. 3b, a horizontal line was used instead of an arc because its width is narrow. If the point (x, y) is the target to be inpainted, then we obtain its symmetric point (x1, y1) with respect to the left border, point (x′, y′) with respect to the y-axis, and point (x2, y2) symmetric to point (x′, y′) with respect to the right border. The horizontal range was from x = x0 − b1 to x = x0, and the vertical range was from y = y0 to y = y0 − re.

Finally, the distortion of the inpainted GBC image was corrected using the spherical orthogonal distortion correction method [34], and the pre-processing of the GBC image was completed.

3.2 Input nondimensionalization

The key point in predicting the inter-hour DNI is to predict the cloud motion, including the speed and direction. Because GBC images were sampled every 10 min, it is difficult to accurately identify the direction and speed of the cloud from two consecutive images. Considering the relationship between the cloud motion and wind, the wind speed and direction were used to estimate the cloud motion.

First, 30 groups of two consecutive GBC images were selected from January 2013, in which the cloud motion was very clear by the human eye. Then, the R/B algorithm was used to detect a cloud from the GBC image, and image enhancement and morphological methods were then used to find all cloud masses and their centroids. Finally, the cloud motion was calculated from two consecutive images comparing their centroids. It was found that a linear regression can be used to describe the relationship between the wind and cloud directions, as shown in Fig. 4 where the red line has a slope of 1, the blue points are the values of cloud direction calculated from the images and the corresponding wind direction, and their correlation coefficient is up to 0.74. Thus, the wind direction was defined as the cloud direction.

Fig. 4
figure 4

Relationship between wind and cloud directions

Full size image

However, the cloud speed does not have a clear relationship with the wind speed owing to the difficulty of cloud boundary detection and the lack of information regarding the cloud base height. Therefore, six cloud cover features were extracted from 20 × 20 pixel areas along the opposite direction of the wind (cloud), as shown in Fig. 2, to address the cloud effect on the DNI. The wind speed was set as one of the model inputs to adjust the weight of the cloud cover in the forecast model. Therefore, the GBC image was transformed to six cloud covers in 20 × 20 pixel areas dimensionlessly.

According to (6), the measured DNI can be transformed into clear-sky index k, thereby removing the influence of the sun’s position on the DNI. Here, k is a dimensionless variable with a scale of [0, 1]. The relative humidity is the ratio of the partial pressure of water vapor to the equilibrium vapor pressure of water at a given temperature, and thus the relative humidity was selected as one input variable instead of the water vapor, and ranges from 0 to 1 instead of a percentage. To allow the wind speed (S) to be within the same scale, it was transformed as follows:

$$S' = \frac{{S - S_{\hbox{min} } }}{{S_{\hbox{max} } - S_{\hbox{min} } }}$$
(9)

where Smin and Smax are the minimum and maximum wind speeds in the training set, respectively. Thus far, all inputs including the image and numerical time-series were transformed into dimensionless variables within the range of [0, 1].

3.3 DNI inter-hour forecast model

In view of the advantage of handling non-linear problems, SVR was applied as the DNI forecast model. The structure of the SVR model was introduced in detail in [35]. In this study, the kernel function of the SVR is the radial basis function. There are three model parameters: cost C, the tolerance of termination criterion ε and attenuation parameter ν. The three model parameters are adjusted using the cross-validation method [36] with the training and validation sets. The extensive grid search method was used to determine the three hyperparameters, where C was set in [1, 5] at a step size of 0.5, ε was set in [0.001, 0.1] at a step size of 0.001, and ν was set in [1, 3] at a step size of 0.2, respectively. The range of these hyperparameters was set based on the experimental results and previous experience.

To determine the input order of the clear-sky index, the Bayesian information criterion (BIC) was used and the results are as shown in Fig. 5 where p is the number of inputs. Thus, it is reasonable to select k(t − 3Δt), k(t − 2Δt), k(t − Δt) and k(t) as the forecast model inputs, where Δt is the forecast interval of 10 min. The same dimensions of relative humidity were selected during the same period. For the wind speed and GBC image, only S(t) and the cloud cover were selected concurrently. Finally, there were 15 inputs of the forecast model. The output of the SVR model was the next 10-min clear-sky index (k(t+Δt)), and the clear-sky index was then transformed into the DNI according to (6).

Fig. 5
figure 5

Results of BIC using clear-sky index in training set

Full size image

4 Results and discussion

80% of coupled data randomly selected from 2013 were set as the training set, and the remaining data (20%) were set as the validation set. Data from 2014 were set as the testing set. The accuracy of the forecast was assessed using three statistics, namely, the normalized mean bias error (nMBE), normalized mean absolute error (nMAE), and normalized root mean square error (nRMSE), which are defined as follows:

$$nMBE = \frac{1}{\bar{I}_{m} N}\left[ {\sum\limits_{i = 1}^{N} {(I_{fi} - I_{mi} )} } \right] \times 100\%$$
(10)
$$nMAE = \frac{1}{\bar{I}_{m} N}\left[ {\sum\limits_{i = 1}^{N} {\left| {I_{fi} - I_{mi} } \right|} } \right] \times 100\%$$
(11)
$$nRMSE = \frac{1}{\bar{I}_{m}}\sqrt {\frac{1}{N}\sum\limits_{i = 1}^{N} {(I_{fi} - I_{mi} )^{2} } } \times 100\%$$
(12)

where N is the number of samples; Ifi is the predicted value of the forecast model; Imi is the measured value; and \(\bar{I}_{m}\) is the average of all measured DNIs.

To evaluate the performance of the proposed forecast framework at predicting the DNI in the next 10 min, two groups of experiments were carried out: 1) using only the historical DNI time-series as inputs; 2) using both the numerical time-series and ground-based cloud images. Table 1 lists the four forecast models using only DNI time-series as inputs, including the persistent model (measured DNI as the presently predicted value), autoregressive (AR) model (with historical measured DNIs linearly fit as the predicted DNI), and artificial neural network (ANN) model (multilayer perceptron, with historical measured DNIs non-linearly fitted as the predicted DNI), which are introduced in Appendix A. Compared to the linear models (persistent and AR), the non-linear models (ANN and SVR) achieved better results in predicting the inter-hour DNI, and the SVR model performed slightly better than the ANN model, especially on nMAE.

Table 1 Performance of different models using only numerical DNI time-series as inputs for 10-min DNI forecast
Full size table

Table 2 lists the test results of different forecast models considering GBC images in predicting the DNI, where the AR-I, ANN-I and SVR-I models maintained the same structure as the AR, ANN and SVR models in Table 1, while adding six cloud covers (CCs) extracted from a GBC image as inputs. The ANN-I model was introduced in [19], and the forecast model based on digital image processing (FM-DIP) was developed in [15]. The forecast skill (Fs) is a criterion used to assess the performance of a forecast model over the persistent model, which is defined as follows:

$$Fs = \frac{{nRMSE_{per} - nRMSE_{f} }}{{nRMSE_{per} }} \times 100\%$$
(13)

where nRMSEper is the nRMSE of the persistent model; and nRMSEf is the nRMSE of the forecast model. Figure 6 shows the forecast skills of all models listed in Tables 1 and 2. Among these forecast models, the non-linear models performed better than the linear ones; in terms of input, the model with images used as inputs achieved better results with the same structure. For the three SVR models with different inputs, the proposed model achieved an optimal performance, with 34% improvement over the persistent model in terms of the nRMSE, which is mainly owing to the supplementary consideration of the water vapor effects, in addition to the historical DNI and cloud. It is therefore reasonable that the consideration of various atmospheric variables (including clouds and water vapor) in the DNI forecast model can improve the accuracy.

Table 2 Performance of different models with different inputs for 10-min DNI forecast
Full size table
Fig. 6
figure 6

Forecast skills of different models compared to persistent model

Full size image

To investigate the performance and fluctuations of the proposed method, the ramp rate (RR) was used to evaluate the fluctuations of the DNI time-series, and is defined as follows:

$$RR = \frac{{N_{\text{ramp}} }}{{N_{\text{total}} }} \times 100\%$$
(14)
$$N_{\text{ramp}} = \sum\limits_{t = 1}^{{N_{\text{total}} }} {\text{sgn} \bigg(\frac{{\left| {I_{m} (t) - I_{m} (t - 1)} \right|}}{{\hbox{max} (I_{m} (t),I_{m} (t - 1))}} > 0.5 \bigg)}$$
(15)

where Ntotal is the number of total testing sample couples and Nramp is the number of fluctuations defined as the bias between two consecutive DNIs greater than half of the larger DNI. In addition, Fig. 7a shows the nRMSE of the proposed method and the ramp rate of the predicted DNI for each month. The nRMSE agrees well with the ramp rate of the predicted DNI time-series, with a constantly increasing or decreasing trend. The nRMSE reached up to 31% in July, whereas the ramp rate was 35%. Throughout the year of the testing data (2014), the proposed method reached a steady improvement in the nRMSE of 34% on average over the persistent model, as shown in Fig. 7b.

Fig. 7
figure 7

Monthly distribution of the ramp rate of DNI time-series and nRMSE of proposed method for testing set (2014)

Full size image

5 Conclusion

A framework for predicting the inter-hour DNI was proposed, including the nondimensionalization of the input variables, employment of a forecast model based on the SVR, and transformation of outputs. Multiple input variables, involving the historical measured DNI, relative humidity, ground-based cloud image, and wind speed, were employed in the developed forecast model. The data from the NREL dataset for the entire year of 2013 were used to train the forecast model using a cross-validation method. The experimental results demonstrated that the proposed model was comparable to other models in terms of the forecast accuracy, and achieved a 34% improvement in the nRMSE over the persistent model.

The performance of the proposed forecast model is affected by the ramp rate of the DNI time-series, with a higher accuracy when the monthly DNI changes more gently, and vice versa. Therefore, considering the ramp rate and thereby constructing different forecast models will further improve the forecast accuracy, and should be investigated in the future studies.