Hourly solar irradiance forecasting based on statistical methods and a stochastic modeling approach for residual error compensation

Abstract

By reducing fossil fuel use, renewable energy improves the economy, quality of life, and environment. These impacts make renewable energy forecasting crucial for lowering fossil fuel utilization. This paper aims to mathematically improve time series forecasting literature by focusing on solar irradiance applications in Los Angeles, Denver, and Hawaii solar irradiance sites. A three-phased time series forecasting hybrid method is devised for this endeavor. The ARFIMA is used to forecast the original solar irradiance time series in phase I. Next, the dataset’s residuals, are retrieved by subtracting the phase I results from the observed time series to prepare the scenario for the following phase. A novel enhanced fractional Brownian motion is used for residual forecasting in phase II. The parameter estimation in phase II is implemented adaptively to capture the dynamic statistical characteristics of the time series efficiently. Finally, the phases I and II results are numerically conglomerated to form the final forecasting results in phase III. The residual forecasting part, in phase II, reveals a substantial superiority. Also, when comparing the proposed hybrid algorithm results to other existing cutting-edge algorithms applied to the same solar irradiance applications, the output demonstrates that the suggested algorithm has a significantly improved performance.

Appendices

Appendix A

Using various statistical performance evaluation metrics when forecasting time series is beneficial because different metrics assess different aspects of forecasting accuracy, such as magnitude errors, percentage errors, or scaling errors, and can provide a more comprehensive understanding of the model's performance. Using multiple metrics ensures that various dimensions of forecast quality are considered. For example, MAE focuses on magnitude errors, while MAPE emphasizes percentage errors. By examining multiple metrics, we gain a more complete picture of the model's performance. Additionally, different metrics can be sensitive to different types of errors. Some metrics, like MAE and RMSE, are sensitive to both overestimation and underestimation errors. On the other hand, metrics like MAPE or MASPE are more sensitive to percentage errors. Or for instance, Spearman's rho and Kendall's tau are robust to outliers and resistant to extreme values in the data. They are based on the ranks of the observations rather than their actual values. As a result, these measures are less influenced by a few extreme data points, which can be advantageous when working with time series that may contain outliers or data with heavy-tailed distributions. So, by using a combination of metrics, we can detect and analyze errors from multiple perspectives, allowing for a more nuanced assessment. Finally, no single metric is universally superior or immune to specific limitations. Each metric has its strengths and weaknesses. By employing multiple metrics, we mitigate the limitations of individual metrics and obtain a more robust evaluation of the forecasting model. It reduces the risk of relying solely on one metric that might overlook certain aspects of forecast accuracy or be influenced by outliers or specific characteristics of the data.

All the formulae regarding the metrics, which are employed in the assessment process in Sect. 2.1., are as the following:

Root Mean Square Error (RMSE):

$${\text{RMSE}} = \frac{{\sqrt {\mathop \sum \nolimits_{{{\text{i}} = 1}}^{{\text{N}}} \left( {{\text{Y}}_{{\text{i}}} - {\hat{\text{Y}}}_{{\text{i}}} } \right)\begin{array}{*{20}c} 2 \\ { } \\ \end{array} } }}{{{\text{N}}^{2} }}$$

(12)

in which N denotes the time series length, ${\mathrm{Y}}_{\mathrm{i}}$ is the original residuals time series and ${\widehat{\mathrm{Y}}}_{\mathrm{i}}$ is the forecasted residual time series.

Normalized Root Mean Square Error (nRMSE):

$${\text{nRMSE}} = \frac{{\sqrt {\frac{1}{{\text{N}}}\mathop \sum \nolimits_{{{\text{i}} = 1}}^{{\text{N}}} \left( {{\text{Y}}_{{\text{i}}} - {\hat{\text{Y}}}_{{\text{i}}} } \right)\begin{array}{*{20}c} 2 \\ { } \\ \end{array} } }}{{\frac{1}{{\text{N}}}\mathop \sum \nolimits_{{{\text{i}} = 1}}^{{\text{N}}} \left( {{\text{Y}}_{{\text{i}}} } \right)}}$$

(13)

Root Mean Squared Relative Error (RMSRE):

$${\text{ RMSRE}} = \sqrt {\frac{1}{N}\mathop \sum \limits_{i = 1}^{N} \left( {\frac{{\left( {{\text{Y}}_{i} - {\hat{\text{Y}}}_{i} } \right)}}{{{\text{Y}}_{i}^{ } }}} \right)^{2} }$$

(14)

Root Mean Square Difference (RMSD):

$${\text{RMSD}} = \frac{{\sqrt {\mathop \sum \nolimits_{{{\text{i}} = 1}}^{{\text{N}}} \left[ {\left\{ {({\text{Y}}_{{\text{i}}} - {\overline{\text{Y}}}_{{\text{i}}} ) - ({\hat{\text{Y}}}_{{\text{i}}} - \overline{{{\hat{\text{Y}}}}}_{{\text{i}}} )} \right\}\begin{array}{*{20}c} 2 \\ { } \\ \end{array} } \right]} }}{{{\text{N}}^{2} }}$$

(15)

where ${\overline{\mathrm{Y}} }$ and ${\overline{\widehat{\mathrm{Y}}} }$ are the mean of the original residual time series and forecasted ones, respectively.

Mean Absolute Error (MAE):

$${\text{ MAE}} = \frac{1}{{\text{N}}}{ }\mathop \sum \limits_{i = 1}^{N} \left| {\frac{{{\text{Y}}_{{\text{i}}} - {\hat{\text{Y}}}_{{\text{i}}} }}{{{\text{Y}}_{{\text{i}}} }}} \right|$$

(16)

Mean Absolute Percentage Error (MAPE):

$${\text{ MAPE}} = \frac{1}{{\text{N}}}{ }\mathop \sum \limits_{i = 1}^{N} \left| {\frac{{{\text{Y}}_{{\text{i}}} - {\hat{\text{Y}}}_{{\text{i}}} }}{{{\text{Y}}_{{\text{i}}} }}} \right| \times 100$$

(17)

Mean Absolute Scaled Percentage Err (MASPE):

$$\begin{aligned} {\text{MASPE}} & = \frac{{{\text{MAE}}_{{Target\,Algorithm\left( {TA} \right)}} }}{{{\text{MAE}}_{{Naive\,Algorithm\left( {NA} \right)}} }} \times 100 = \frac{{\frac{1}{{\text{N}}}{ }\mathop \sum \nolimits_{i = 1}^{N} \left| {\frac{{{\text{Y}}_{{\left( {TA} \right)i}} - {\hat{\text{Y}}}_{{\left( {TA} \right)i}} }}{{{\text{Y}}_{{\left( {TA} \right)i}} }}} \right|}}{{\frac{1}{{\text{N}}}{ }\mathop \sum \nolimits_{i = 1}^{N} \left| {\frac{{{\text{Y}}_{{\left( {NA} \right)i}} - {\hat{\text{Y}}}_{{\left( {NA} \right)i}} }}{{{\text{Y}}_{{\left( {NA} \right)i}} }}} \right|}} \times 100 \\ & = \frac{{\mathop \sum \nolimits_{i = 1}^{N} \left| {\frac{{{\text{Y}}_{{\left( {TA} \right)i}} - {\hat{\text{Y}}}_{{\left( {TA} \right)i}} }}{{{\text{Y}}_{{\left( {TA} \right)i}} }}} \right|}}{{\mathop \sum \nolimits_{i = 1}^{N} \left| {\frac{{{\text{Y}}_{{\left( {NA} \right)i}} - {\hat{\text{Y}}}_{{\left( {NA} \right)i}} }}{{{\text{Y}}_{{\left( {NA} \right)i}} }}} \right|}} \times 100 \\ \end{aligned}$$

(18)

R-squared (R²):

$${\text{R}}^{2} = 1 - \frac{{{\text{RSS}}}}{{{\text{TSS}}}} = 1 - \frac{{\mathop \sum \nolimits_{{{\text{i}} = 1}}^{{\text{N}}} \left( {{\text{Y}}_{{\text{i}}} - {\hat{\text{Y}}}_{{\text{i}}} } \right)\begin{array}{*{20}c} 2 \\ { } \\ \end{array} }}{{\mathop \sum \nolimits_{{{\text{i}} = 1}}^{{\text{N}}} \left( {{\text{Y}}_{{\text{i}}} - {\overline{\text{Y}}}} \right)\begin{array}{*{20}c} 2 \\ { } \\ \end{array} }}$$

(19)

In which RSS indicates the “sum of squares of residuals” and TSS stands for the “total sum of squares”.

Spearman’s rho and Kendall’s tau formula are as follows:

Spearman’s rho is equivalent to Pearson’s Linear Correlation Coefficient applied to the rankings of the columns ${xx}_{a}$ and ${yy}_{b}$. If all the ranks in each column are distinct, Spearman’s rho equation simplifies to:

$${\uprho }_{s} \left( {{\text{a}},{\text{b}}} \right) = 1 - \frac{{6\mathop \sum \nolimits_{ }^{ } {\mathcal{D}}^{2} }}{{N\left( {N^{2} - 1} \right)}}$$

(10)

where $\mathcal{D}$ is the difference between the ranks of the two columns, and $N$ is the length of each column.

Kendall’s tau is based on counting the number of $(i,j)$ pairs, for $i<j$, that are concordant—that is, for which ${x}_{a,i}-{x}_{a,j}$ and ${y}_{b,i}-{y}_{b,j}$ have the same sign. The equation for Kendall’s tau includes an adjustment for ties in the normalizing constant and is often referred to as tau-b.

For column ${x}_{a}$ in matrix $x$ and column ${y}_{b}$ in matrix $y$, Kendall’s tau coefficient is defined as:

$$\tau = \frac{2K}{{N\left( {N - 1} \right)}}$$

(11)

where $K=\sum_{i=1}^{n-1}\sum_{j=i+1}^{n}{\zeta }^{*}({x}_{a,i},{x}_{a,j},{y}_{b,i},{y}_{b,j})$, and

$$\zeta^{*} \left( {x_{a,i} ,x_{a,j} ,y_{b,i} ,y_{b,j} } \right) = \left\{ {\begin{array}{*{20}c} { 1: if \left( {x_{a,i} - x_{a,j} } \right)\left( {y_{b,i} - y_{b,j} } \right) > 0} \\ { 0: if \left( {x_{a,i} - x_{a,j} } \right)\left( {y_{b,i} - y_{b,j} } \right) = 0} \\ { - 1: if \left( {x_{a,i} - x_{a,j} } \right)\left( {y_{b,i} - y_{b,j} } \right) < 0} \\ \end{array} } \right.$$

The correlation coefficient might be anywhere between –1 and 1. A number of –1 indicates that one column’s rating is the inverse of the other, whereas a value of + 1 indicates that the two ranks are identical. A value of 0 indicates that the columns have no relationship.

Appendix B

2.1 The KC mathematical and complementary explanations

Let $X$ represent the solar irradiance time series, and $x$ denote a specific value within that series. The Kolmogorov complexity, denoted as ${K}_{c}(x)$, measures the length of the smallest program that, when run on a universal Turing machine $U$, produces the object $x$. Although ${K}_{c}(x)$ cannot be directly computed for arbitrary objects, it is approximated by the size of the most compressed version of $x$ (Mihailović et al. 2019a, 2023).

In practice, the calculation of the KC of a time series $X({x}_{1}, {x}_{2}, {x}_{3}, ..., {x}_{N})$ using the LZA algorithm involves several steps. Firstly, the time series is encoded by creating a binary sequence S using the characters 0 and 1 written as$s(i)$. Each element$s(i)$,$i=\mathrm{1,2},...,N$, of the sequence is set to 0 if ${x}_{i}<{x}_{t}$ or 1 if${x}_{i}>{x}_{t}$, where ${x}_{t}$ is a threshold typically chosen as the mean value of the time series (Mihailović et al. 2019a, 2023). Other encoding schemes are also available (Dingle et al. 2023).

The next step is to calculate the complexity counter $c(N)$, which represents the minimum number of distinct patterns present in the encoded sequence of length $N$. The complexity counter is a function of the sequence length $N$ and is bounded by $b(N)=N/{\mathit{log}}_{2}N$ as $N$ approaches infinity, denoted as $c(N)=O(b(N))$.

Finally, the normalized information measure ${C}_{k}(N)$ is computed as ${C}_{k}(N)=c(N)/b(N)$. For nonlinear time series, ${C}_{k}(N)$ varies between 0 and 1, although it may exceed 1 for finite-size random sequences (Mihailović et al. 2019a, 2023). It is worth noting that a pattern refers to a unique and non-repetitive sequence within the encoded time series. A flow chart illustrating the calculation of the KC of a time series using the LZA algorithm is provided in (Mihailović et al. 2019a, 2023).

2.2 The LE mathematical and complementary explanations

Quantitatively, the divergence of two trajectories in phase space with an initial separation $\delta X$ is approximately given by $\left|\delta X(t)\right|\approx {e}^{\lambda t}\left|\delta X(0)\right|$, where $\lambda$ represents the LE. It should be noted that this approximation assumes a linear relationship in the divergence. Since the rate of separation can vary depending on the orientation of the initial separation vector, there exists a spectrum of LEs, with the largest value commonly referred to as the LE. A positive value of the LE typically indicates chaotic behavior in the system. In our study, we calculated the LE for the solar irradiance time series using the Rosenstein algorithm (Rosenstein et al. 1993) implemented in the MATLAB program (Mohammadi 2009). The LE, denoted as $\lambda$, is obtained as the limit of the average logarithmic divergence rate as the time delay $\tau$ approaches infinity and the separation $\varepsilon$ approaches zero, given by $\lambda =\underset{\tau \to \infty }{\mathit{lim}}\underset{\varepsilon \to 0}{\mathit{lim}}\frac{1}{\tau }ln(\frac{\left|x(\tau )-{x}_{\varepsilon }(\tau )\right|}{\varepsilon })$ where $\left|x(0)-{x}_{\varepsilon }(0)\right|=\varepsilon$.

The Rosenstein algorithm is known for its efficiency, ease of application, and robustness to variations in embedding dimension, reconstruction delay, length of the time series, and noise level.

Appendix C

In our analysis, we estimated the most reliable prediction horizon (LT) following the procedure described in (Mihailović et al. 2019a). However, since the complexity of the solar irradiance datasets used in this study is low, we used the simplified version of LT as explained in Sect. 2.2.3, for predicting the time series. It is important to note that solar irradiance time series from other stations and countries may exhibit higher complexity levels (Mihailović et al. 2018, 2021).

In cases where solar irradiance time series exhibit higher complexity levels, the predictability can be influenced by randomness. To account for this, a randomization time (${\Delta t}_{rand}$) is introduced as ${\Delta t}_{rand}=1/{K}_{c}(x)$. This randomization time, denoted as KT, quantifies the time span beyond which randomness significantly affects predictability. The predictability horizon, LT, is then corrected for randomness and defined as the intersection of [0, ${\Delta t}_{lyap}$] and [0, ${\Delta t}_{rand}$]. This corrected LT takes into account the time window within the time series where complexity remains relatively unchanged. Figure 

Fig. 14

Predictability of the solar irradiance datasets given by the LT corrected by randomness (in hours)

14 illustrates the predictability of solar irradiance data in hours, considering the corrected LT for randomness. The corrected LT is inversely proportional to the KC. The figure shows that for KCH, the prediction horizon ${\Delta t}_{rand}$ falls between two and four hours. Figure 

Fig. 15

LT versus KT (in hours)

15 displays the relationship between LT and KT. The impact of randomness on reducing the LT can be observed by comparing Figs. 5, 14, and 15. According to Fig. 5, the longest predictability in LT units for all datasets is around 10 h. However, considering the presence of randomness as shown in Fig. 14, this predictability is reduced to 4 h.