Interpretable ML-Based Forecasting of CMEs Associated with Flares

Abstract

Coronal mass ejections (CMEs) that cause geomagnetic disturbances on the Earth can be found in conjunction with flares, filament eruptions, or independently. Though flares and CMEs are understood as triggered by the common physical process of magnetic reconnection, the degree of association is challenging to predict. From the vector magnetic field data captured by the Helioseismic and Magnetic Imager (HMI) onboard the Solar Dynamics Observatory (SDO), active regions are identified and tracked in what is known as Space Weather HMI Active Region Patches (SHARPs). Eighteen magnetic field features are derived from the SHARP data and fed as input for the machine-learning models to classify whether a flare will be accompanied by a CME (positive class) or not (negative class). Since the frequency of flare accompanied by CME occurrence is less than flare alone events, to address the class imbalance, we have explored the approaches such as undersampling the majority class, oversampling the minority class, and synthetic minority oversampling technique (SMOTE) on the training data. We compare the performance of eight machine-learning models, among which the Support Vector Machine (SVM) and Linear Discriminant Analysis (LDA) model perform best with True Skill Score (TSS) around 0.78 ± 0.09 and 0.8 ± 0.05, respectively. To improve the predictions, we attempt to incorporate the temporal information as an additional input parameter, resulting in LDA achieving an improved TSS of 0.92 ± 0.04. We utilize the wrapper technique and permutation-based model interpretation methods to study the significant SHARP parameters responsible for the predictions made by SVM and LDA models. This study will help develop a real-time prediction of CME events and better understand the underlying physical processes behind the occurrence.

Appendices

Appendix A: Machine Learning Models

• Linear models

  • Support Vector Machine (SVM) (Cortes and Vapnik, 1995) can be used effectively for classification and regression. In a high-dimensional space, SVM sets a hyperplane for binary classification. This is achieved by transforming a nonlinear space to a linear space using a kernel at the maximized distance between the classes from the nearest training data points. As the margin between two classes becomes larger, SVM classifies effectively in an ideal scenario. However, in a real case, there will always be a few cases on the other side that cannot be separable, so support vectors can be there on either side of the class. Finding an optimized method between minimizing the support vectors and maximizing the separable margin allows defining a good model leading to a reduced misclassification rate. Support vector machine can handle both linear and nonlinear data using kernels. In this article, we use a radial basis function (RBF) kernel similar to Bobra and Ilonidis (2016).

  • Logistic regression is a statistical method that is used to estimate the probability of a data point belonging to either class. This is achieved using a sigmoid function in which, above a particular threshold, the data point belongs to one class, and below the threshold, it belongs to the other class.

• Discriminant model

  • Linear discriminant analysis (LDA) (Tharwat et al., 2017) assumes that the input data is Gaussian. For each class, LDA estimates the mean and variance from the given data. During prediction, LDA tries to get the probability of the new input belonging to a particular distribution of a class using the Bayes theorem. The class which has a higher probability is assigned as the respective prediction for the input. It further acts as a dimensionality reduction technique by projecting features from a higher-dimension to a lower-dimension space.

• Decision based models

  • A decision tree classifies data according to the attribute associated with the root node and later moves down to the leaf node by making certain decisions. Decision trees are naturally immune to a correlation between variables and resistant to outliers. However, decision trees are prone to overfitting, producing biased trees based on a particular majority class.

• Ensembling models

  • As mentioned above, decision trees are a bit unstable and suffer from generalization issues. Ensemble techniques in the form of bootstrapping or bagging that draw several different decisions from the dataset can be helpful in such cases. The prediction, thereby using majority voting or average outcomes, makes the structure more stable, called as Random Forest (RF) (Breiman, 2001). RF is more stable, accurate, and adaptable to both classification and regression. Random Forest helps in generalization and reduces overfitting by changing the algorithms that are learned by subtrees each time. Random Forest may suffer the worst performance if there are more correlated variables.

  • Boosting (Freund, 1995) is a sequential process where instead of averaging the outcomes, the decisions are weighted and adjusted in sequence models according to the errors obtained from the previous model. The Adaboost model is an ensembling model of weak learners where each model is boosted by the performance of the ensembles to obtain a good classifier with good accuracy.

  • Extreme Gradient Boosting (XGBoost) (Chen and Guestrin, 2016) is an ensembling algorithm enclosing groups of individual learners; it uses gradient descent to minimize the loss function. Further modeling is designed using a regularization parameter and inbuilt cross validation; thus, reducing overfitting and generalization error. Tuning and visualization of Xgboost is a bit complex compared to Adaboost and Random Forest.

Appendix B: Wrapper Features

In Table 5 and 6, the first column represents the number of features followed by the performance metrics and the features that contributed to the performance of SVM. The features that contribute to the performance of LDA are shown in Table 6.

Table 5 Wrapper features for SVM.

Table 6 Wrapper features for LDA.

Appendix C: Metrics

A True Positive (TP) case is defined here as when both the actual and predicted classes are eruptive flares. A False Positive (FP) is when the actual is a confined flare, whereas the predicted class is an eruptive flare, thus leading to false alarms. When the actual event is an eruptive flare, whereas the predicted is falsely classified as a confined flare, it is termed as False Negative (FN). True Negative (TN) refers to when both actual and prediction classes are confined flares. Thus, from the confusion matrix, it is understood that a good model should have fewer False Positives and also False Negatives. Performance metrics should be carefully chosen before assessing the performance of the model with class imbalance. Metrics such as accuracy show higher performance due to better classification of the majority class than the minority class. Since we have more confined flare events, using accuracy as a metric will lead to a biased model evaluation. Hence, we evaluate a few other performance metrics based on TP, FP, TN, and FN obtained from the confusion matrix. True Skill Score, Probability of Detection (PoD), False Alarm Rates (FAR), and False Detection Rate (FDR) are a few metrics that are described in detail below.

• Probability of Detection (PoD): The PoD is described as the probability that our model will detect the occurrence of eruptive flare events. Thus, we calculate the ratio between true positive events and total actual eruptive flares. It is also known as True Positive Rate (TPR). It can be described by Equation 1:

  $$ PoD=\frac{TP}{TP+FN}\cdot $$

  (1)

• False Alarm Rate (FAR): The performance of a model is considered good when there are fewer false alarms. Thus, minimizing the ratio of wrongly categorized eruptive flare events to the total number of actual confined flare events should be considered, which is denoted by False Alarm Rate (FAR), otherwise known as False Positive Rate (FPR). FAR is calculated by Equation 2:

  $$ FAR=\frac{FP}{FP+TN}\cdot $$

  (2)

  The good model will maximize the performance of PoD, but there exists a trade-off while trying to minimize FAR; thus, an optimum balance should be found between both metrics.

• False Discovery Rate (FDR): FDR can be defined as the rate at which false alarms occur among all predicted eruptive flare events. FDR is given as in Equation 3:

  $$ FDR=\frac{FP}{FP+TP}\cdot $$

  (3)

• True Skill Score (TSS): In order to avoid bias towards the majority class events, we try to evaluate the model based on the metric TSS. TSS can be seen as the difference between PoD and FAR. Thus, a good model with a high TSS score will have more PoD of eruptive flare events while minimizing the FAR. However, maximizing the only metric TSS may lead to more False Positives. TSS ranges from −1 to +1, with +1 as the best measure for our desired class. TSS can be evaluated as in Equation 4. TSS is not affected by the class imbalance; hence, we have considered it as our base metric to evaluate our performance of the model.

  $$ TSS=\frac{TP}{TP+FN}-\frac{FP}{FP+TN}\cdot $$

  (4)