Forecasting Solar Flares Using Magnetogram-based Predictors and Machine Learning

Abstract

We propose a forecasting approach for solar flares based on data from Solar Cycle 24, taken by the Helioseismic and Magnetic Imager (HMI) on board the Solar Dynamics Observatory (SDO) mission. In particular, we use the Space-weather HMI Active Region Patches (SHARP) product that facilitates cut-out magnetograms of solar active regions (AR) in the Sun in near-realtime (NRT), taken over a five-year interval (2012 – 2016). Our approach utilizes a set of thirteen predictors, which are not included in the SHARP metadata, extracted from line-of-sight and vector photospheric magnetograms. We exploit several machine learning (ML) and conventional statistics techniques to predict flares of peak magnitude ${>}\,\mbox{M1}$ and ${>}\,\mbox{C1}$ within a 24 h forecast window. The ML methods used are multi-layer perceptrons (MLP), support vector machines (SVM), and random forests (RF). We conclude that random forests could be the prediction technique of choice for our sample, with the second-best method being multi-layer perceptrons, subject to an entropy objective function. A Monte Carlo simulation showed that the best-performing method gives accuracy $\mathrm{ACC}=0.93(0.00)$, true skill statistic $\mathrm{TSS}=0.74(0.02)$, and Heidke skill score $\mathrm{HSS}=0.49(0.01)$ for ${>}\,\mbox{M1}$ flare prediction with probability threshold 15% and $\mathrm{ACC}=0.84(0.00)$, $\mathrm{TSS}=0.60(0.01)$, and $\mathrm{HSS}=0.59(0.01)$ for ${>}\,\mbox{C1}$ flare prediction with probability threshold 35%.

Appendices

Appendix A: Importance of Predictors for Flare Prediction

We computed the Fisher score (Bobra and Couvidat, 2015; Chang and Lin, 2008; Chen and Lin, 2006) and the Gini importance (Breiman, 2001) for every predictor in the case of ${>}\,\mbox{M1}$ and ${>}\,\mbox{C1}$ flares. The obtained values for the importance of several predictors are presented in Figures 7 and 8 for ${>}\,\mbox{C1}$ and ${>}\,\mbox{M1}$ flare prediction, respectively. The Fisher score, $F$, is defined for the $j$th predictor as

$$ F(j) = \frac{ (\bar{x}^{(+)}_{j} - \bar{x}_{j})^{2} + (\bar{x}^{(-)}_{j} - \bar{x}_{j})^{2} }{ \frac{1}{n^{+} - 1} \sum_{k=1}^{n^{+}} ( x^{(+)}_{k,j} - \bar{x}^{(+)}_{j})^{2} + \frac{1}{n^{-} - 1} \sum _{k=1}^{n^{-}} ( x^{(-)}_{k,j} - \bar{x}^{(-)}_{j})^{2} }. $$

(A.1)

In Equation A.1, $\bar{x}_{j}$, $\bar{x}^{(+)}_{j}$, and $\bar{x}^{(-)}_{j}$ are the mean values for the $j$th predictor over the entire sample, the positive class, and the negative class, respectively. Furthermore, $n^{+}$ ($n^{-}$) are the number of positive (negative) class observations. In addition, $x^{(+)}_{k,j}$ $(x^{(-)}_{k,j})$ are the values for the $k$th observation of the $j$th predictor belonging in the positive (negative) class. The higher the value of $F(j)$, the more important the $j$th predictor.

Figure 7

Importance of several predictors while predicting ${>}\,\mbox{C1}$ flares.

Figure 8

Importance of several predictors while predicting ${>}\,\mbox{M1}$ flares.

The Gini importance is returned with the randomForest function of the randomForest package in R. The higher the Gini importance of the $j$th predictor, the more important this predictor.

We note that the correlation between the two quantities (Fisher score and Gini importance) is $r = 0.7441$ for ${>}\,\mbox{C1}$ flares and $r = 0.7535$ for ${>}\,\mbox{M1}$ flares, respectively. This shows that the two methods qualitatively agree on describing which predictors are the most important regarding flare prediction, in both classes of flare prediction. Figure 7 also shows that for ${>}\,\mbox{C1}$ flares, the top three ranked predictors for both Fisher score and Gini importance are the two versions of Schrijver’s $R$ and $\mathrm{WL}_{\mathrm{SG}}$. For the ${>}\,\mbox{M1}$ flares, from Figure 8 the top four ranked predictors for either Fisher score or Gini importance are the two versions of Schrijver’s $R$, $\mathrm{WL}_{\mathrm{SG}}$, and $\mathrm{TLMPIL}_{\mathrm{Br}}$. In Appendix A the terminology for every predictor is explained in Table 7.

Table 7 Abbreviations for predictors used in the main text (Symbol1) and in Figures 7 and 8 (Symbol2).

Appendix B: Prediction Models Resulting from Ranking the Predictors

We employed a backward elimination procedure, eliminating gradually predictors according to their Fisher score rank, starting form the model with all $K=13$ predictors included. In every step, we eliminated the least important predictor from the set of currently included predictors. In this way, we obtained prediction results for models with 2, 3, …, 11, 12 predictors included for the ML methods, RF, SVM, and MLP and the conventional statistics methods LM, PR, and LG. The results of this iterative procedure for flares ${>}\,\mbox{C1}$ and ${>}\,\mbox{M1}$ are presented in Figures 9 and 10.

Figure 9

Out-of-sample skill scores (TSS and HSS) for the three ML prediction methods and the three statistics methods during the ranking procedure for ${>}\,\mbox{C1}$ flares. The thick continuous lines depict the averages of the skill scores over 30 randomized runs.

Figure 10

Same as in Figure 9, but for ${>}\,\mbox{M1}$ flares.

Figure 9 shows that there is a cut-off for the number of parameters included in the RF equal to the 6 most important ones (according to the Fisher score in Equation A.1) above which the RF is advantageous over the other two ML algorithms. For low-dimensional prediction models (e.g. fewer than 6 included parameters) there is no special advantage in using RF, and MLP or SVM seem a better choice then. This finding shows that of the highly correlated set of predictors, the MLP and SVM perform well using only a handful of them (fewer than 6), but the RF continues to improve its performance in higher-dimension settings, when the prediction model includes all 12 most important predictors. There is interest in investigating the performance of RF when the number of (correlated) predictors would be twice or three times that of the present study (24 – 36 predictors). Would the upward trend in Figure 9a continue to hold when the number of included parameters increased to 24 or 36? We note that RF is the only ML algorithm in the present study that belongs in the category of “ensemble” methods. Moreover, in Figure 9, the performance of the three conventional statistics methods LM, PR, and LG is presented. Clearly, the LM presents the worst forecasting ability, and we also note that the other two methods, PR and LG, score similar values for the TSS and HSS in general. It is also noteworthy that the profiles of PR and LG are very flat as a function of the number of included predictors, even flatter than the profiles from SVM and MLP.

Likewise, Figure 10 shows that for low-dimensional settings, RF is worse than MLP. The cut-off seems again to be six included parameters. Above this value, the RF provides better out-of-sample TSS and HSS than MLP. There seems to be a problematic region between three and six parameters included for the SVM, where adding more parameters to the SVM degrades its performance. With more than six parameters, the SVM performance improves again. Similarly to the ${>}\,\mbox{C1}$-class flares case, we again note in Figure 10 rather flat profiles for the TSS and HSS for the conventional statistics methods, with PR and LG showing better behavior than LM.

One general conclusion is that for very few predictors $K<6$, all methods work the same, so for parsimony, the conventional statistics methods could be preferred. This is also true for very small samples $N<2{,}000$ (results available upon request). Conversely, when $K > 6$ and $N \geq10{,}000$, the ML methods and especially the RF are better.

We note that in Appendix B, the MLP always has four hidden nodes and the SVM has $\gamma$ and cost parameters analogously to the full $K=13$ SVM model for ${>}\,\mbox{C1}$ and ${>}\,\mbox{M1}$ flares cases.

Appendix C: Validation Results when Predictions Are Issued Only Once a Day (at Midnight)

We present here forecasting results in the following scenario:

• i) The training is performed as in the main scenario.

• ii) The testing is performed only for the observations in the testing set of the main scenario, which correspond to a time of 00:00 UT. To achieve this, we filter for the observations in the previous testing set with midnightStatus = TRUE.

This method of training-testing is called the “hybrid method”, where training is done with a cadence of 3 h and a forecast window of 24 h, and testing is done with a cadence of 24 h and a forecast window of 24 h. The hybrid method is preferable over using a training phase with a cadence of 24 h, which would result in undertrained models because of the limited sample size during training.

Tables 8 and 9 are analogous to Tables 5 and 6 of the main scenario, but for midnight-only (so once a day) predictions. For completeness, we recall that Table 8 is for ${>}\,\mbox{M1}$ flare prediction and Table 9 is for ${>}\,\mbox{C1}$ flare prediction.

Table 8 Same as Table 5, but for predictions issued only at midnight.

Table 9 Same as Table 6, but for predictions issued only at midnight.

By comparing Table 8 to Table 5, we see that BS and AUC do not change much on average when we move from the baseline scenario to the midnight prediction scenario. Nevertheless, the associated uncertainty increases in the case of midnight-only predictions, since the size of the testing set is smaller (only one, rather than eight, predictions per day). More significant differences are observed for BSS since the associated climatology is also different. Nevertheless, the finding that RF is the best overall method continues to hold.

Similar conclusions can be drawn for the ${>}\,\mbox{C1}$ flare prediction case, that is, through Tables 9 and 6. Here, noticeably, not even the BSS changes significantly, since the underlying climatology seems similar in both cases. This is because in contrast to ${>}\,\mbox{M1}$ class flares with a mean frequency of ${\approx}\,5\%$, ${>}\,\mbox{C1}$ class flares show a mean frequency of ${\approx}\,25\%$.

Finally, Tables 10 and 11 present the skill scores ACC, TSS, and HSS for the midnight prediction scenario analogously to Tables 2 and 3 for the baseline scenario. For completeness, we note that Table 10 pertains to ${>}\,\mbox{M1}$ flare prediction and Table 11 to ${>}\,\mbox{C1}$ flare prediction. We see that on average, the issuing of midnight-only predictions does not change the ACC, TSS, and HSS much with respect to the probability threshold. For example, on ${>}\,\mbox{M1}$ flare midnight-only predictions, the RF provides $\mbox{ACC}=0.93\pm0.01$, $\mbox{TSS}=0.73\pm0.04$, and $\mbox{HSS}=0.47\pm 0.03$ for a probability threshold of 15%. Furthermore, for ${>}\,\mbox{C1}$ flare midnight-only predictions, the RF yields $\mbox{ACC}=0.85\pm0.01$, $\mbox{TSS}=0.63\pm0.02$, and $\mbox{HSS}=0.61\pm0.02$ for a probability threshold of 35%.

Table 10 Same as Table 2, but for predictions issued only at midnight.

Table 11 Same as Table 3, but for predictions issued only at midnight.

Appendix D: Concluding Remarks on ML Versus Statistical Methods for Flare Forecasting

In order to assess the overall forecasting ability of ML versus statistical approaches in our dataset and problem definition, we employed the weighted-sum (WS) multicriteria ranking approach (Greco, Figueira, and Ehrgott, 2016), using a composite index (CI) defined in Equation D.1:

$$ {\mathrm{CI}}=\frac{1}{3}\biggl(\mathrm{\frac{ACC - ACC_{min}}{ACC_{max} - ACC_{min}}}+\mathrm{ \frac{TSS - TSS_{min}}{TSS_{max} - TSS_{min}}}+\mathrm{\frac{HSS - HSS_{min}}{HSS_{max} - HSS_{min}}}\biggr). $$

(D.1)

The CI value was computed for $6 \times21 = 126$ probabilistic classifiers using the set of methods {MLP, LM, PR, LG, RF, SVM} and a probability threshold grid of 5%. Then, the 126 probabilistic classifiers were ranked in non-increasing values of the CI index. A normalization was made for ACC, TSS, and HSS, so that each metric over the set of alternatives took values in the range $[0,1]$. The normalization is useful because the range of values for ACC is different from the range of values for TSS and HSS. Moreover, $\mathrm{{ACC}_{min}}$ is the minimum of ACC over all 126 alternative models. Likewise, $\mathrm{{ACC}_{max}}$ is the maximum ACC obtained over all 126 alternative models. Similar facts hold for $\mathrm{{TSS}_{min}}$, $\mathrm{{TSS}_{max}}$, $\mathrm{{HSS}_{min}}$, and $\mathrm{{HSS}_{max}}$. Analytically, Table 12 presents the results of the multicriteria ranking approach for all methods we used with various probability thresholds, especially for the ${>}\,\mbox{C1}$ flare forecasting case. Table 13 conveys a similar ranking of all methods developed in this paper, but for the ${>}\,\mbox{M1}$ flare prediction.

Table 12 Ranking of all models with ML and statistical methods with the multicriteria WS method with respect to the three criteria ACC, TSS, and HSS and using a weight vector equal to $w=[1/3, 1/3, 1/3]$ for ${>}\,\mbox{C1}$ flare forecasting. The methods are ranked in decreasing order of CI for varying levels of probability thresholds used, with a grid of 5% for the probability thresholds. The six best positions of the ranking are covered by the RF method for various probability thresholds.

Table 13 Same as Table 12, but for ${>}\,\mbox{M1}$ flare forecasting.

Figure 11 summarizes the results shown in Tables 12 and 13, so that the differences between ML and statistical methods are highlighted (e.g. see Figures 11b and 11d). Similarly, conclusions for the merit of all methods developed in this paper can be drawn in Figures 11a and 11c. The top $100\tau$ percentile methods are those ranked in the corresponding positions of Tables 12 and 13. For example, the top $16.6\%(1/6)$ methods are those ranked in positions 1 – 21. For low values of $\tau$, we obtain the best methods designated as the top $100\tau\%$ methods. From Figure 11 we see that both for ${>}\,\mbox{C1}$ and ${>}\,\mbox{M1}$ flares, the RF has the greatest frequency in the top 16.6% percentile of methods, with a frequency of 33.3%. This means that in Tables 12 and 13, in positions 1 – 21, the RF method appears seven times in each table. We also see in Figure 11b that for ${>}\,\mbox{C1}$ flares, the top 16.6% methods are of type ML with a frequency 71% (versus 29% for statistical methods). Similarly, in Figure 11d, ML dominates in the top 16.6% methods with a frequency of 62% (versus 38% for statistical methods).

Figure 11

Descriptive statistics on the frequency with which every forecasting method for any probability threshold presents itself in the top $100\tau\%$ percentile of the CI distribution. Panels (a) and (c) describe frequencies for all methods, and panels (b) and (d) group the results by category of methods (e.g. ML vs. statistical methods). For example, for ${>}\,\mbox{C1}$ flares in panel (a), the top 16.6% methods are dominated by RF with a frequency of $7/21=33\%$. Likewise, for ${>}\,\mbox{M1}$ flares in panel (c), the top 16.6% methods are again dominated by RF with a frequency of $7/21=33\%$.