Solar Energetic Particle Forecasting Algorithms and Associated False Alarms

Abstract

Solar energetic particle (SEP) events are known to occur following solar flares and coronal mass ejections (CMEs). However, some high-energy solar events do not result in SEPs being detected at Earth, and it is these types of event which may be termed “false alarms”.

We define two simple SEP forecasting algorithms based upon the occurrence of a magnetically well-connected CME with a speed in excess of $1500~\mbox{km}\,\mbox{s}^{-1}$ (a “fast” CME) or a well-connected X-class flare and analyse them with respect to historical datasets. We compare the parameters of those solar events which produced an enhancement of ${>}\,40~\mbox{MeV}$ protons at Earth (an “SEP event”) and the parameters of false alarms.

We find that an SEP forecasting algorithm based solely upon the occurrence of a well-connected fast CME produces fewer false alarms (28.8%) than an algorithm which is based solely upon a well-connected X-class flare (50.6%). Both algorithms fail to forecast a relatively high percentage of SEP events (53.2% and 50.6%, respectively).

Our analysis of the historical datasets shows that false-alarm X-class flares were either not associated with any CME, or were associated with a CME slower than $500~\mbox{km}\,\mbox{s}^{-1}$; false-alarm fast CMEs tended to be associated with flare classes lower than M3.

A better approach to forecasting would be an algorithm which takes as its base the occurrence of both CMEs and flares. We define a new forecasting algorithm which uses a combination of CME and flare parameters, and we show that the false-alarm ratio is similar to that for the algorithm based upon fast CMEs (29.6%), but the percentage of SEP events not forecast is reduced to 32.4%.

Lists of the solar events which gave rise to ${>}\,40~\mbox{MeV}$ protons and the false alarms have been derived and are made available to aid further study.

1 Introduction

Solar energetic particles (SEPs) pose a significant radiation hazard to humans in space (Hoff, Townsend, and Zapp, 2004) and in high-flying aircraft, particularly at high latitudes (Beck et al., 2005). They may also cause serious damage to satellites (Feynman and Gabriel, 2000) and make high-frequency radio communications either difficult or impossible (Hargreaves, 2005). Accurate forecasting of the arrival of SEPs at locations near Earth is consequently vital.

SEPs are known to be energised by flares and coronal mass ejections (CMEs), processes which can take place within the same active region on the Sun in close temporal association. Flares exhibiting high levels of energy emission in soft X-rays (SXR) and CMEs with high speeds have long been associated with a high likelihood of SEPs being detected at Earth (see e.g. Dierckxsens et al., 2015). The bases for making such associations are studies of large numbers of events which are directed towards demonstrating the connection between flare and CME properties, and SEP events. These studies proceed to look for correlations between event parameters and the proportion of associated solar event SEPs (e.g. Belov et al., 2005; Cliver et al., 2012).

Whether SEPs are actually detected at Earth, however, may depend upon many different factors: the mechanism behind their acceleration, the energy and efficiency of that acceleration, the location of the acceleration site, whether the particles can escape into the interplanetary medium, and how they travel through it.

It is not the case that SEPs are detected at Earth following all large flares and fast CMEs (e.g. Klein et al., 2011). Solar events of this type, which might reasonably be expected to produce SEPs at Earth but which do not, may be termed “false alarms”. Furthermore, some SEP events may follow smaller solar events, so that they are “missed events” for SEP forecasting algorithms based on intense flares and/or fast CMEs.

Many SEP forecasting tools base their prediction upon the observation of intense solar flares and/or radio bursts. For example, the proton prediction system proposed by Smart and Shea (1989) makes a forecast based upon flare intensity and position. It produces almost equal numbers of correct forecasts, false alarms, and missed events (Kahler, Cliver, and Ling, 2007).

The National Oceanic and Atmospheric Administration (NOAA) Space Weather Prediction Center (SPWC) uses a system named “Protons” which is described by Balch (1999). The tool aims to forecast the arrival of SEPs near Earth following the detection of solar flares and radio bursts. Balch (2008) validated the system over a period between 1986 and 2004, and found that its false-alarm rate was 55%. The tool, however, is only used as a decision aid, and the actual forecasts issued by SWPC have improved over time.¹ Kahler and Ling (2015) combined SEP event statistics with real-time SEP observations to produce a forecast which changes dynamically.

Laurenza et al. (2009) developed the empirical model for solar proton events real-time alert (ESPERTA) method of SEP forecasting based upon flare size, flare location, and evidence of particle acceleration and escape. Their emphasis was to maximise the time between the issue of an SEP event warning and the arrival of the particles, and their aim was to produce an automated forecasting tool with a view to issuing warnings of SEP events without human intervention. Whilst it is a significant improvement over the Protons tool, the false-alarm rate was, nevertheless, between 30% and 42% (Alberti et al., 2017). The forcasting solar particle events and flares (FORSPEF) model, proposed by Papaioannou et al. (2015), aims to make forecasts of both flares and SEPs. Its SEP forecasting algorithm is based upon a purely statistical approach and has not yet been validated.

Other forecasting tools use different methods. It has also been shown that type II radio bursts at decametric–hectometric (DH) wavelengths may be used to aid the forecasting of SEP events. Winter and Ledbetter (2015) have described a statistical relationship between DH type II radio bursts, the properties of the associated type III burst, and peak proton flux. During the period they analysed (2010 to 2013), they were able to make predictions of an SEP event with a false-alarm rate of 22%.

The relativistic electron alert system for exploration (REleASE) SEP forecasting tool (Posner, 2007) relies upon the fact that electrons will travel faster than protons and will therefore arrive at 1 AU first. A forecast of expected proton flux is made based upon the real-time electron flux measurements.

Although the majority of currently operational data-based forecasting schemes make use of flare information, it is widely thought that the use of CME information would substantially improve algorithm performance. While from an operational point of view it is currently not trivial to obtain CME parameters in real time, it is important to compare the performance of flare-based versus CME-based algorithms and determine whether a combination of flare and CME parameters within a forecasting tool may be beneficial.

Along with empirical forecasting algorithms which are based upon solar observations, several physics-based space weather forecasting tools have recently been developed, e.g. the SOlar Particle ENgineering COde (SOLPENCO) (Aran, Sanahuja, and Lario, 2006), a solar wind simulation including a cone model of CMEs (Luhmann et al., 2010), and the Solar Particle Radiation SWx (SPARX) model (Marsh et al., 2015).

A catalogue of 314 SEP events and their parent solar events between 1984 and 2013 has been produced by Papaioannou et al. (2016). It is expected that this database will provide a solid basis for the analysis of SEP events and the characteristics of their parent solar event. The catalogue does not, however, include information on solar events which were false alarms. In order to improve SEP forecasting tools for space weather applications, an analysis of the characteristics of false-alarm events should be carried out with a view to gaining an understanding of why SEPs were not observed.

Some statistical studies of SEP events and false alarms have been undertaken. Most take the same approach as Papaioannou et al. and Laurenza et al., starting by considering the SEP events and then looking for the possible parent solar events. Gopalswamy et al. (2014) examined solar events during the early part of Solar Cycle 24, and considered why some which had very fast CMEs and large flares did not produce ground-level enhancements of energetic particles, as might have been expected. They suggested that poor latitudinal magnetic connectivity between the solar event and Earth may have been an important factor.

Marqué, Posner, and Klein (2006) examined a small number of CMEs with a speed greater than $900~\mbox{km}\,\mbox{s}^{-1}$ which had no radio signature of flare-related acceleration, and found that none produced conspicuous SEP events at Earth. These authors argued therefore that a CME shock without an associated flare is not sufficient to produce SEPs.

Wang and Zhang (2007) suggested that X-class flares not associated with any CME may occur closer to the magnetic centre of their source active region and may therefore be confined by overlying arcade magnetic fields. Klein, Trottet, and Klassen (2010) investigated a small number of these “CME-less” flares further and argued that no SEP event might be expected following a flare which shows high peak emission in soft X-rays but does not exhibit radio emission at decimetre and longer wavelengths.

Most of the large sample studies described above started by considering SEP events and then looked for possible parent solar events. In this article, we take a different approach. We start our analysis by considering solar events and determining whether an SEP event was measured at Earth a short time thereafter. We focus on intense flares and fast CMEs and define two possible forecasting algorithms, the first based solely on the occurrence of an intense flare, and the second on that of a fast CME. The performance of the algorithms is quantified by evaluating them over historical datasets and the characteristics of the false alarms which were studied. In addition, missed events, i.e. SEP events which were not forecast, are also identified and studied. Finally, we discuss how a new algorithm which combines flare and CME properties may be introduced to achieve a better performance.

We provide lists of false alarms based upon the forecasting algorithms in order that they may form the basis of future studies and comparisons, together with a list of the solar events which produced ${>}\,40~\mbox{MeV}$ protons. We analyse the properties of the false-alarm events to determine whether reasons can be identified that would explain why they did not produce SEPs at Earth.

2 False Alarms and Forecasting Algorithms

A false alarm may simply be defined as a solar event which is predicted by a forecasting algorithm to produce SEPs at Earth, but which fails to do so. Specification of a forecasting algorithm and determination of its associated false alarms requires identification of the following three points:

• 1. The criteria and observational data sets by which a solar event is assigned a high likelihood of producing SEPs at Earth. Typically, this will include identification of the type of solar event (e.g. flare or CME) expected to produce SEPs, of a requirement on the intensity of the event (e.g. a flare with a peak SXR flux, $f_{\mathrm{{sxr}}}$, which exceeds a specified threshold intensity, $f_{\mathrm{{thr}}}$, or a CME with a speed $v_{\mathrm{{CME}}}$ which is faster than a threshold speed $v_{\mathrm{{thr}}}$), of a positional requirement (e.g. an event with a source region West of a given longitude), and possibly of other parameters.

• 2. The criteria by which it is determined that an SEP event has occurred or not. These will typically include specification of the instrument being used to measure particle flux intensity, of the species of particle examined and its energy range, and of the SEP intensity threshold, $I_{\mathrm{{thr}}}$, used to establish whether an SEP event was detected following a particular solar event.

• 3. The method by which the solar event is associated with the SEP event.

We discuss each of these requirements in Sections 2.1, 2.4, and 2.5, respectively.

2.1 Solar Event Parameters

As our source for CME data we have used the CME catalogue of the Co-ordinated Data Analysis Workshop (CDAW)² (Gopalswamy et al., 2009). This catalogue is produced manually, CMEs being identified visually from images obtained by the C2 and C3 coronagraphs of the Large Angle and Spectrometric Coronagraph Experiment (LASCO) (Brueckner et al., 1995) on board the Solar and Heliospheric Observatory (SOHO) spacecraft.

Information is published in the catalogue on various CME parameters including, inter alia, the time it is first seen in the LASCO images, its width, and its position angle. CDAW publishes three values for the speed of CMEs in its catalogue, each calculated by different means: we use the first, the “linear” speed, which is obtained simply by fitting a straight line to the height–time measurements. Importantly, there is no information directly available from the catalogue as to whether the CME is Earth directed, or from where on the solar disc it originated. This imposes serious limitations in analysing whether a particular CME is likely to produce SEPs at Earth.

Solar flares are classified by their peak SXR emission as measured in the 1 – 8 Å channel of the Geostationary Observational Environmental Satellites (GOES) (Grubb, 1975) X-ray Sensor (XRS) instruments. Flares with a peak flux in this energy channel above $10^{-4}~\mbox{W}\,\mbox{m}^{-2}$ are designated to be of class X; those with a peak flux between $10^{-5}$ and $10^{-4}~\mbox{W}\,\mbox{m}^{-2}$ are of class M; and classes C, B, and A are defined in a similar fashion. No single instrument has been in continuous operation since 1975, although the design has changed little over the years (Garcia, 1994).

As our source for solar flare data, we have used the GOES SXR Flare List, which has been continuously maintained since 1975, and which may be downloaded from the website³ of the Heliophysics Integrated Observatory (Bentley et al., 2011).

In addition to reporting the maximum SXR intensity and the time of the start, peak, and end of the flare, the GOES SXR Flare List also usually reports its heliographic co-ordinates. However, there is a significant number of flares for which the list does not provide this information. In these cases, we have used values for co-ordinates from the following sources:

• 1. Co-ordinates reported in the SolarSoft Latest Events Flares List (gevloc) (which may also be obtained through Helio).

• 2. The reported co-ordinates of the active region (AR) from which the flare originated according to the GOES SXR flare list.

• 3. Making our own estimate of co-ordinates by watching movies of 195 Å images taken by the Extreme ultraviolet Imaging Telescope (EIT) on board the SOHO spacecraft or of 195 Å images taken by the Atmospheric Imaging Assembly (AIA) on board the Solar Dynamics Observatory (SDO).

CMEs and solar flares, particularly high-energy events, often occur within a short time of each other from the same solar active region. Making associations between these solar events is required so as to gain an understanding of the type of event which did or did not produce SEPs at Earth: it also allows an estimate to be made of the site of origin of the CME from the reported heliographic coordinates of its associated flare.

We developed a method of making associations between CMEs and flares automatically, which we set out in Appendix A. Whilst we are confident that the method produces correct associations in over 90% of cases, to be sure, we also viewed 195 Å (obtained by the EIT on board SOHO) and 193 Å (obtained by the AIA on board SDO) movies of each solar event. We confirmed the associations made by the automatic method in 156 cases, changed them in six cases, and were unable to confirm the associations in a further 17 cases because EIT or AIA images were not available.

2.2 Location Criterion for Solar Events

It is well known that solar events with origin in the West of the Sun as observed by an observer on Earth are more likely to produce SEPs than those originating in the East. Therefore it is common to introduce a positional criterion within SEP forecasting algorithms. Figure 1 shows the heliographic longitude of the 171 SEP-producing events between 1 April 1980 and 31 March 2013 for which we were able to determine coordinates. Of these, 86.5% (148/171) had their origin in a solar event which occurred at a site West of E20, hence our choice of positional requirement in the forecasting algorithms. We call solar events which have their origin West of E20 “western events”.

Figure 1

Heliographic longitude and latitude of solar events which produced an SEP event according to the criteria defined in Section 2.4 between 1 April 1980 and 31 March 2013.

2.3 The Forecasting Algorithms

The two forecasting algorithms we investigate in this work are based upon the fact that the more energetic the solar event, the greater the likelihood of that event producing SEPs at Earth, particularly if it was magnetically well connected (e.g. Dierckxsens et al., 2015). The algorithms are:

• A.1 A frontside CME with a reported speed of $1500~\mbox{km}\,\mbox{s}^{-1}$ or greater (a “fast” CME) occurring West of E20 on the solar disc will result in an SEP event being detected at Earth.

• A.2 An X-class flare occurring West of E20 on the solar disc will result in an SEP event being detected at Earth.

We evaluate both the forecasting algorithms over the time range from 11 January 1996 until 31 March 2013 (“time range 1”); for algorithm A.2 we are also able to examine a longer period, between 1 April 1980 and 31 March 2013 (“time range 2”). In time range 1, there were 143 fast CMEs (according to our definition set out in A.1) reported by CDAW and 140 X-class flares. In time range 2, there were 403 X-class flares.

Table 1 sets out the numbers of solar events which we have examined in this study. A number of solar events have had to be excluded from our analysis because of data gaps, the saturation of detectors, or another cause, or because it was not possible to determine the heliographic co-ordinates.

Table 1 Numbers of solar events we studied. Column 1 shows the time range over which data have been analysed, column 2 the type of solar event considered, column 3 the total number of solar events within the period investigated, column 4 the number of events for which we were able to determine coordinates (after removal of events which were discarded because of data gaps, saturation of detectors, or other reasons), and column 5 the number of events which occurred West of E20.

2.4 SEP Event Parameters

The definition of an SEP event typically includes a specification of the instrument being used to measure the particle flux, of the species of particle which is examined and its energy range, and of the SEP intensity threshold, $I_{\mathrm{{thr}}}$, which is used to establish whether an SEP event was detected following a particular solar event.

Particles accelerated by solar events include electrons, protons, and heavier ions, but we have chosen to analyse high-energy (${>}\,40~\mbox{MeV}$) protons. The threshold considered is slightly higher than the ${>}\,10~\mbox{MeV}$ threshold used by NOAA, making our event list less biased towards interplanetary shock-accelerated events. This choice also avoids proton enhancements caused by magnetospheric effects.

Because our threshold energy for protons is higher than that used by NOAA, we compared peak ${>}\,40~\mbox{MeV}$ fluxes for our event sample with the peak ${>}\,10~\mbox{MeV}$ fluxes for the same events. For each of our events a value for ${>}\,10~\mbox{MeV}$ flux was obtained from the NOAA SEP list.⁴. Eleven of the SEP events at ${>}\,40~\mbox{MeV}$ did not reach the NOAA threshold of 10 pfu at ${>}\,10~\mbox{MeV}$, and for these we estimated the peak flux by visual analysis of the plots of each event.⁵ Figure 2 is a plot of the peak flux of ${>}\,10~\mbox{MeV}$ protons plotted against the peak proton flux in the ${\sim}\,40\,\mbox{--}\,80$ MeV energy channel of the GOES EPS instruments for the SEP events in time range 1. The dotted horizontal line is at the NOAA threshold of ten particles $\mbox{cm}^{-2}\,\mbox{s}^{-1}\,\mbox{sr}^{-1}$ (pfu). The highest value for the maximum peak flux at ${>}\,40~\mbox{MeV}$ in time range 1 was approximately 100 pfu – the same event at ${>}\,10~\mbox{MeV}$ produced 31700 pfu according to NOAA.

Figure 2

Peak flux of ${>}\,10~\mbox{MeV}$ protons as reported by NOAA plotted against the peak proton flux in the GOES ${\sim}\,40\,\mbox{--}\,80~\mbox{MeV}$ energy channel in time range 1. The dotted horizontal line is at the NOAA threshold of 10 pfu.

All instruments which detect proton intensities are subject to slight fluctuations, and not all of these can properly be said to be SEP events. The definition of the intensity threshold, $I_{\mathrm{{thr}}}$, must be high enough so as to exclude the normal fluctuations in measurements, but low enough to ensure that rises which are genuinely due to solar events are included. We set $I_{\mathrm{{thr}}}$ to be a 2.5-fold increase in proton intensity over the quiet-time background level.

For this study we have used GOES SEP data because they allow us to study SEP events over a time period of more than 30 years. No one instrument has been in continuous operation during that time, and so we have had to use data from a number of different GOES satellites. Table 2 sets out which spacecraft we have used and the energy channel we considered to establish the occurrence of an SEP event. There are slight differences in the energy channels, particularly in the case of GOES 2, but we take the view that the differences are so small as to have a negligible effect upon our results. We downloaded data from the European Space Agency’s Solar Energetic Particle Environment Monitor (SEPEM) website (Crosby et al., 2010).⁶ Data from 1 April 1987 onwards had been cleaned and intercalibrated by the SEPEM team; prior to that date, we used their raw data.

Table 2 Instruments used to obtain data on proton intensity, the dates between which data from that instrument were used, and the energy channels which were analysed. Column 1 gives the name of the spacecraft from which the data were taken, column 2 the date from which we began to use these data, and column 3 the date when we ceased using these data. Column 4 shows the range of proton energies measured by the instrument we used, and column 5 whether the data were raw or had been cleaned by the SEPEM team.

It is not always easy to determine whether an SEP event had occurred if the instrument were still recording high-energy protons from a previous event. When the intensity level had not returned to within 2.5 times the quiet-time background level by the time of the start of the solar event we investigated, that solar event was disregarded – we were unable to determine whether that event produced SEPs at Earth. The only exceptions were those cases where there was a clear increase in proton intensity which could only be attributed to the solar event in question, in which case it was treated as an SEP event.

We determined that during time range 2, there had been 221 flux enhancements in the GOES ${>}\,40~\mbox{MeV}$ proton channel which satisfied our definition of an SEP event.

2.5 Association of Solar Events and SEP Events

A criterion for associating solar events and SEP enhancements is necessary. First we took the start time of the solar event. For CMEs not associated with a flare, we used the time at which the CME was first reported in the CDAW catalogue; for CMEs which were associated with a flare and for all flares, we used the reported start time of the flare.

We then searched the GOES proton data for a subsequent SEP event. In most cases, the SEP enhancement began before another solar event was reported, in which case the association between the solar event and the SEP enhancement was made. In some instances, however, another solar event was reported before the SEP enhancement commenced. For these cases, it was assumed that this new solar event accelerated the particles, unless that event was so close in time to the arrival of the SEPs (${\sim}\,20~\mbox{minutes}$) that it was unlikely that the new event could have been the cause. None of our confirmed solar event – SEP association time differences was as short as 20 minutes.

A number of solar events had to be discarded because they coincided with gaps in SEP data, meaning that we were unable to determine whether they had produced an SEP enhancement. However, if there had been short outages (${\sim}\,3~\mbox{hours}$), and there was no evidence of an SEP event either side of the outage, the solar event was counted as a false alarm.

We also associated solar events with all of the 221 proton events we identified. In some cases, the associated flare was of a class smaller than X and/or the associated CME was not a fast one according to our definition. Of these 221 events, we were not able to determine co-ordinates of the parent solar event for 50. The event was a western one in 148 of the remaining 171 cases.

3 Identification of False Alarms and Evaluation of the Forecasting Algorithms

We applied the forecasting algorithms described in Section 2.3 to the historical datasets we collected. We evaluated both algorithms over time range 1 (1996 to 2013), and in addition, we evaluated algorithm A.2 over the longer time range 2 (1980 to 2013).

3.1 Algorithms A.1 and A.2 over Time Range 1

Figure 3 shows the results of applying the two SEP forecasting algorithms to the data set for time range 1. The number of correctly forecast SEP events is shown by the blue bar and named $\alpha$; the number of false alarms is represented by the red bar and named $\beta$; and the number of SEP events which occurred but were not forecast by the algorithm (the “missed events”) is shown as the green bar and named $\gamma$. There was a total of 107 SEP events in time range 1. Of the 86 SEP events for which we were able to determine the coordinates of the parent solar event, 91.9% (79/86) were western events.

Figure 3

Numbers of correctly forecast SEP events, false alarms, and SEP events which were not forecast for the two forecasting algorithms during time range 1.

Algorithm A.1 considers western fast CMEs. There were 52 such events during the period in question, and 71.2% (37/52) produced SEPs at Earth. Thus the false-alarm rate was 28.8% (15/52), but the algorithm failed to forecast 53.2% (42/79) of the SEP events for which the parent solar event was a western one. Of all the SEP events for which coordinates could be determined, it missed 57.0% (49/86).

Algorithm A.2 uses western X-class flares as the basis for the forecast. There were 79 such flares in time range 1, and 49.4% (39/79) produced SEPs at Earth. The false-alarm rate was therefore 50.6% (40/79), and the algorithm failed to forecast 50.6% (40/79) of the SEP events for which the parent solar event was a western one. Of all the SEP events for which coordinates could be determined, it missed 54.7% (47/86).

Table 5 in Appendix B provides the list of false alarms for algorithm A.1 and Table 6 in Appendix C contains the false alarms for A.2; the same lists are available electronically as supplementary material.

As well as reaching for an understanding of the underlying physical differences between those solar events which produced SEPs at Earth and the false alarms, we also aim to measure the efficacy of the forecasting algorithms. A high percentage of correctly forecast SEP events ($\alpha$) coupled with a low number of false alarms ($\beta$) is desirable, but not at the expense of failing to forecast a large number of the SEP events which did occur ($\gamma$). We used two ratios in our evaluation:

• 1. The “false-alarm ratio” (FAR) gives the fraction of forecast events which actually did occur. It is defined as

  $$ \mathrm{FAR} = \frac{\beta}{\alpha + \beta}. $$

  (1)

  The FAR is sensitive to the number of false alarms, but takes no account of missed events. Possible scores range from 0 to 1, with the “perfect” score being 0.

• 2. The “critical success index” (CSI) is a measure of how well the forecast events correspond to the observed events. It is defined as

  $$ \mathrm{CSI} = \frac{\alpha}{\alpha + \beta + \gamma}. $$

  (2)

  Possible scores range from 0 to 1, with the “perfect” score being 1.

3.2 Forecasting Algorithm A.1: Fast CMEs

All the CMEs in our sample were from the front-side of the Sun and had an associated flare which was used to determine the coordinates. The FAR for algorithm A.1 is 0.29 and the CSI, not taking account of the missed eastern events, is 0.39. If the eastern events were to be included within the calculation for the CSI, its value would be reduced to 0.37. The evaluation scores for this algorithm over time range 1, and for algorithm A.2 over both time ranges, are summarised in Table 3. It is not clear whether the high number of missed events is due to the fact that the measured velocity of the CME, $v_{\mathrm{{CME}}}$, is the plane-of-the-sky speed, if in general the speeds measured by examination of coronagraph images are not sufficiently accurate, or if more physics need to be included in the analysis.

Table 3 Summary of the evaluation scores for the two forecasting algorithms: the “false-alarm ratio” (FAR) and the “critical success index” (CSI) over time range 1. Algorithm A.2 is also evaluated over time range 2. Column 1 shows the forecasting algorithm being considered, column 2 the time range over which the analysis was made, column 3 lists the FAR for that algorithm, column 4 the critical success index (CSI) not taking into account the missed eastern events, and column 5 the CSI were these additional missed events to be included.

In Figure 4 we plot the peak SXR intensity of the CME associated flare against its speed for those solar events in time range 1 which produced SEPs at Earth (top left, blue circles); for those events in the same period which were false alarms according to algorithm A.1 (top right, red squares); for SEP events missed by algorithm A.1 (bottom left, green diamonds); and for all events together (bottom right). This shows that many of the fast CME false alarms occur close to the threshold speed, $v_{\mathrm{ {thr}}}$, which means that increasing the threshold would reduce the number of false alarms, although it would also increase the number of missed events. A significant fraction of SEP events were associated with CMEs of a reported speed much slower than $1500~\mbox{km}\,\mbox{s}^{-1}$. It is also clear that many of the false alarms have a flare intensity ${<}\,\mbox{M3}$.

Figure 4

Flare class versus associated CME speed for the solar events which produced SEPs ${>}\,40~\mbox{MeV}$ at Earth in time range 1 (top left, blue circles); for fast CMEs which were false alarms according to forecasting algorithm A.1 (top right, red squares); for SEP events missed by algorithm A.1 (bottom left, green diamonds); and for all events together (bottom right).

Gopalswamy et al. (2014) studied major solar eruptions during the first 62 months of Solar Cycle 24 and suggested that among other parameters, the separation in latitude between the flare and the footpoint to Earth may be an important factor in determining whether high-energy particle events are detected. Therefore we define a parameter, $\Delta\delta$, the difference between the latitude of the flare, $\delta_{\mathrm{flare}}$, and the latitude of the Earth’s footpoint, $\delta_{\mathrm{Earth}}$, i.e. the parameter $\Delta\delta$ takes into account the inclination of Earth’s orbit. In Figure 5 we plot $\Delta\delta$ against time for algorithm A.1, together with histograms for $\Delta\delta$. The events correctly forecast to produce SEPs are presented in the top plots (shown in blue), and the false alarms in the bottom plots (shown in red). Of the fast CMEs which had their origin within ${\pm}\,10~\mbox{degrees}$ of the Earth’s footpoint, 64.7% (11/17) produced SEPs; of those which had their origin outside this range, 74.3% (26/35) produced SEPs. Overall, there does not appear to be a significant difference between the distribution in $\Delta\delta$ for SEP events and false alarms.

Figure 5

Plots of $\Delta\delta$ against time for algorithm A.1, together with histograms of $\Delta\delta$. The top plots present the results for the solar events which were correctly forecast to produce SEPs at Earth (shown in blue); the bottom plots show the false alarms (shown in red).

Figure 6 shows histograms of the heliographic longitude of solar events in time range 1 correctly forecast by algorithm A.1 to produce an SEP event (top left), of algorithm A.1 false alarms (top right), of SEP events missed by algorithm A.1 (bottom left), and of all SEP events (bottom right). There is a peak of SEP-producing fast CMEs between W50 and W90. The false alarms for algorithm A.1 are relatively evenly distributed, as are the SEP events not forecast by A.1.

Figure 6

Histograms of the heliographic longitude of solar events in time range 1 of algorithm A.1 SEP events (top left); of algorithm A.1 false alarms (top right); of SEP events missed by algorithm A.1 (bottom left); and of all SEP events (bottom right).

In Figure 7 we plot $\Delta\delta$ against the longitude of the 37 western fast CMEs which produced an SEP event in time range 1. The size of the marker reflects the peak SXR intensity of the associated flare, and its colour is representative of the width of the CME. The bottom plot gives the same information, but for the false alarms according to algorithm A.1. On average, the size of the markers in the middle plot is smaller than the size of those in for the SEP-producing events. Thus, the peak SXR intensity of a flare associated with a fast CME is relevant to the question as to whether SEPs will arrive at Earth.

Figure 7

$\Delta\delta$ versus heliographic longitude for the western fast CMEs which produced SEPs at Earth in time range 1 (top plot); and for those which were false alarms according to algorithm A.1 (bottom plot). The size of the marker represents the peak SXR intensity of the flare: for example, the point at S20W95 in the top plot was an M1.8 flare, whereas the point at S21E08 in the same plot was an X17.2 flare. The colour of the marker represents the CME width.

Also apparent from Figure 7 is that the CME width is an important parameter. Of the 37 SEP-producing CMEs, 86.5% (32/37) were reported to be haloes by the CDAW catalogue. By contrast, for the false alarms of algorithm A.1, only 46.7% (7/15) were haloes. Therefore we find that halo CMEs are more likely to produce SEPs than non-haloes. This result is consistent with the findings of Park, Moon, and Gopalswamy (2012), who found that solar events which had the highest probability of producing 10 MeV protons were full-halo CMEs with a speed exceeding $1500~\mbox{km}\,\mbox{s}^{-1}$.

Kwon, Zhang, and Vourlidas (2015) examined 62 halo CMEs (as reported by the CDAW catalogue) which occurred between 2010 and 2012 and were observed by three spacecraft separated in longitude by nearly $180^{\circ}$. They found that 42 were observed to be haloes by all three spacecraft. They concluded that a CME may appear to be a halo as a result of fast magnetosonic waves or shocks, and that apparent width does not represent an accurate measure of CME ejecta size.

3.3 Forecasting Algorithm A.2: X-Class Flares

Algorithm A.2 has an FAR of 0.51. Whilst it makes almost exactly the same number of correct forecasts as algorithm A.1, the percentage of correct forecasts is lower. The proportion of missed SEP events is also relatively high, leading to a CSI of 0.33 without accounting for the missed eastern events, or of 0.31 if the missed eastern events were to be included.

In Figure 8 we plot the SXR intensity for the solar flares above the threshold of A.2 against associated CME speed, and for SEP events missed by algorithm A.2 in the same format as in Figure 4. There is some symmetry with Figure 4 in that many of the false alarms fall close to the chosen threshold. Not all events above the A.2 threshold have an associated CME. Of the 122 X-class flares which occurred in time range 1 (and which did not coincide with a LASCO data gap), 14.8% (18/122) had no associated CME. However, the percentage of A.2 false alarms which did not coincide with a LASCO data gap and which did not have an associated CME is 26.5% (9/34).

Figure 8

Flare class versus associated CME speed for the solar events which produced SEPs ${>}\,40~\mbox{MeV}$ at Earth in time range 1 (top left, blue circles); for X-class flares which were false alarms according to forecasting algorithm A.2 (top right, red squares); for SEP events missed by algorithm A.2 (bottom left, green diamonds); and for all events together (bottom right).

In Figure 9 we show histograms of the heliographic longitude of solar events in time range 1 for algorithm A.2 in the same format as Figure 6. There appears to be no significant difference in the longitudinal distribution of western X-class flares which produced an SEP event and those which were false alarms, but in this case, the SEP events which were not forecast by algorithm A.2 do have a clear peak between W20 and W80.

Figure 9

Histograms of the heliographic longitude of solar events in time range 1 of algorithm A.2 SEP events (top left); of algorithm A.2 false alarms (top right); of SEP events missed by algorithm A.2 (bottom left); and of all SEP events (bottom right).

In the top plot of Figure 10 we show $\Delta\delta$ against the longitude of the 39 western X-class flares which produced an SEP event in time range 1. As in Figure 7, the colour of the marker is representative of the width of the flare’s associated CME as reported by CDAW, but in the case of Figure 10, the size of the marker reflects the duration of the flare itself. The bottom plot gives the same information, but for the false alarms according to algorithm A.2.

Figure 10

$\Delta\delta$ versus heliographic longitude for the western X-class flares which produced SEPs at Earth in time range 1 (top plot); and for those which were false alarms according to algorithm A.2 (bottom plot). The size of the marker represents the relative duration of the flare: for example, the flare marked at S18W33 in the top plot had a duration of 10 minutes, whereas the flare at S03W38 in the same plot lasted 120 minutes. The colour of the marker represents CME width.

X-class flares which were false alarms tended to be shorter than those which produced SEPs. The average flare duration for the SEP-producing X-class flares was 46.3 minutes, and 25.6% were longer than 60 minutes (“long-duration flares”). For the false alarms, the average flare duration was 24.9 minutes, and only 5.0% (2/40) were long-duration flares. It has previously been shown that there is an association between long-duration flares and CMEs (Yashiro et al., 2006), therefore the trend with duration may be connected with the fact that large flares without CMEs are more likely to be false alarms.

In this case, the width of the associated CME is also an important parameter. Of the 39 western X-class flares which produced SEPs at Earth, we were able definitively to associate 37 with a CME (the other two occurring during times when LASCO did not produce any data). Of those 37, 86.5% (32/37) were halo CMEs. In contrast, for the false alarms, we were able to confirm associations with CMEs in 25 cases. Of these 25, only 44.0% (11/25) were haloes.

3.4 Algorithm A.2 over Time Range 2

Over the longer period of time range 2, we analysed 197 western X-class flares, and 39.1% (77/197) produced SEPs at Earth. The false-alarm rate was thus 60.9% (120/197), and the algorithm failed to forecast 47.8% (71/148) of SEP events. Of all the SEP events for which coordinates could be determined, it missed 55.0% (94/171). Therefore the FAR was 0.61, and the CSI was 0.29 without the missed eastern events and 0.26 with them. The FAR is higher for this longer time period than that for time range 1. Table 7 in Appendix D provides the list of false alarms for algorithm A.2 over time range 2.

In Figure 11 we plot $\Delta\delta$ against date for this longer time period together with histograms for $\Delta\delta$. In the left-hand plots the duration of the flare is denoted by the size of the marker. Figure 11 shows a significant difference in the $\Delta\delta$ distribution for events which produced SEPs and false alarms. For the former, the distribution is rather flat, whereas for the latter, very many events are characterised by large $\Delta\delta$.

Figure 11

Plots of $\Delta\delta$ against time for algorithm A.2 over time range 2, together with histograms of $\Delta\delta$. The top plots present the results for the solar events which were correctly forecast to produce SEPs at Earth (shown in blue); the bottom plots the false alarms (shown in red). The size of the marker in the left-hand plots is representative of the duration of the flare: for example, the flare in October 1989 shown at S35 in the top plot lasted 8 hours 48 minutes, whereas the flare in November 1998 shown at S29 in the same plot lasted 19 minutes.

A significantly higher number of false alarms originated in the southern solar hemisphere during Solar Cycle 22 (taken to be 1 January 1987 until 31 December 1995; 80% or 40/50) than from the north (20% or 10/50). Furthermore, in Solar Cycle 24 (taken to be from 1 January 2010 onwards), only two western X-class flares were false alarms.

Moreover, X-class flares between 1980 and 1995 were on average longer than those post 1995. Table 2 shows that we have taken data from GOES 7 and its predecessors for dates before 1 March 1995, and from GOES 8 and its successors after that date. We are not aware of any reason why a change of instrument should produce such a result, nor are we aware of any change in the way the flare duration has been measured.

4 Improvement of the Forecasting Algorithms

We examined ways in which the performance of the forecasting algorithms might be improved. We note in particular

• 1. that algorithm A.1 produced the lowest number of false alarms, and that many of these had an associated flare intensity ${<}\,\mbox{M3}$; and

• 2. that X-class flares without an associated CME, or associated with a CME of speed less than $500~\mbox{km}\,\mbox{s}^{-1}$, did not produce SEPs.

We therefore define a third forecasting algorithm as follows:

• A.3 A front-side CME with a reported speed of $1500~\mbox{km}\,\mbox{s}^{-1}$ or greater occurring West of E20 on the solar disc which is associated with a flare of class M3 or greater, or a solar flare of class X or greater which occurs West of E 20 on the solar disc and is associated with a CME of speed greater than $500~\mbox{km}\,\mbox{s}^{-1}$ will result in an SEP event being detected at Earth.

There were 71 such events in time range 1 and 70.4% (50/71) produced SEPs at Earth. For this algorithm, we have had to discard five of the SEP events which occurred during a time when there were no data from the LASCO coronagraph. Thus the false-alarm rate was 29.6% (21/71) and the algorithm missed 32.4% (24/74) of the SEP events for which the parent solar event was a western one, or 38.3% (31/81) of all SEP events. The false-alarm ratio is thus comparable to that produced by algorithm A.1, but A.3 misses far fewer SEP events, and consequently, the CSI is significantly higher at 0.53, not including the missed eastern events, or 0.49 were they to be included. The result is summarised in Table 4. We also show the result graphically in Figure 12, which is in the same format as Figure 3. It may be possible to formulate better forecasting algorithms, but we suggest that increased forecasting accuracy will only come if the properties of both flares and CMEs are taken into account.

Figure 12

Number of correctly forecast SEP events, false alarms, and SEP events which were not forecast for the three forecasting algorithms during time range 1.

Table 4 Summary of the evaluation scores for algorithm A.3 in the same format as Table 3.

5 Summary and Conclusions

We have used historical datasets in order to assess the efficacy of two simple SEP forecasting algorithms which were based upon the occurrence of magnetically well-connected energetic solar events: western fast CMEs and X-class flares. We used in our definition of SEP event a threshold value for a proton energy of ${>}\,40~\mbox{MeV}$.

An algorithm purely based on the detection of a fast CME (A.1) performs reasonably well in terms of false alarms (with a false-alarm ratio of 28.8%), but misses a significant fraction of actual SEP events (53.1%). It is unclear whether this is due to experimental limitations in the determination of the CME speed, or if there are other physical properties which would need to be measured and included in the algorithm to assess the SEP producing potential of a CME more accurately. False alarms for this type of algorithm tend to be associated with flares of magnitude smaller than M3. There does not seem to be any positional trend in the source location of the false alarms.

An algorithm purely based on the detection of an intense flare (A.2) correctly forecasts almost the same number of SEP events as A.1, but has a much higher false-alarm rate (50.6%). Like A.1, it misses a significant fraction of SEP events (also 50.6%). We found that false alarms for this algorithm tend to be flare events of shorter duration, compared to those that did produce SEPs. Of these false alarms, 37% were not associated with a CME. An earlier study has analysed confined flares (CME-less flares) and emphasised that this type of event tends not to produce SEPs (Klein, Trottet, and Klassen, 2010). In terms of their longitudinal location, A.2 false-alarm events were quite uniformly distributed. We also determined that SEP events not forecast by algorithm A.2 were preferentially located in the well-connected region (between W20 and W80), suggesting that for this region, a lower flare magnitude threshold may need to be used.

When evaluated over a longer time range which includes Solar Cycle 21 (time range 2), algorithm A.2 performs less well than over time range 1. This may be due to instrumental effects associated with different GOES detectors being employed at different times, or it may be a real physical effect. We found that there is a systematic trend for flare durations to be longer in Cycle 22 than in Cycle 23, and this may be an instrumental effect.

It has previously been suggested that the latitudinal separation, $\Delta\delta$, between the flare location and the footpoint of the observing spacecraft plays a role in whether high-energy particles are detected (Gopalswamy et al., 2014). In our analysis, carried out over a wider time range, we found that false alarms for algorithm A.2 tended to be associated with a large latitudinal separation $\Delta\delta$, whilst this was not the case for algorithm A.1.

We defined a new forecasting algorithm, A.3, based upon the parameters of both flares and CMEs. This algorithm performed better than the algorithms based solely upon one type of solar event: it correctly forecast 70.4% of SEP events during time range 1 and thus had a false-alarm rate comparable to that of algorithm A.1 (29.6%). It also missed far fewer SEP events (32.4%, or 38.3% if eastern events were to be included) than both algorithms A.1 and A.2.

In test particle simulations it has been shown that SEPs may exhibit significant cross-field drift velocities depending on the configuration of the interplanetary magnetic field (Dalla et al., 2013; Marsh et al., 2013). Future work will assess whether the specific polarity of the magnetic field may influence the detection or non-detection of SEPs at a given location.

We have made available, in electronic form as supplementary material, lists of the ${>}\,40~\mbox{MeV}$ proton false alarms according to each of the algorithms we analysed, together with a list of the solar events which produced the ${>}\,40~\mbox{MeV}$ SEP events. We hope that these lists can be used as the basis for further studies and comparisons.

Appendices

Appendix A: Association of Solar Flares and CMEs

It has long been accepted that solar flares and CMEs, particularly energetic events, often occur within a short time of each other from the same solar active region, but making associations between them is no trivial exercise. There is no standard approach: for example, Reinard and Andrews (2006) associated a flare with a CME if the CME occurred within a two-hour window centred on the time of the peak of the flare; others made associations using both temporal and spatial criteria (Vršnak, Sudar, and Ruždjak, 2005; Dumbović et al., 2015). Below we describe a method of making associations between CMEs and flares automatically, and evaluate its accuracy.

In the light of the connection between high-energy eruptive events and SEPs, we decided to look for associations involving CMEs reported by CDAW to have a speed of $1000~\mbox{km}\,\mbox{s}^{-1}$ or faster (“rapid CMEs”), and flares reported in the GOES SXR list to be of class M5 or greater (“intense flares”). We examined all such events between 1 July 2011 and 31 August 2012, this period being chosen solely because it provided a dataset which was small enough to allow individual observation of each event, yet large enough to allow wider conclusions to be drawn.

There were 55 rapid CMEs and 32 intense flares reported in the 13-month period under investigation. Of these, we did not study further 3 of the rapid CMEs and 1 of the intense flares because they coincided with data gaps. Hence there were 83 events which formed the basis of our study of flare–CME associations.

In order to set a benchmark against which any automated method of associating CMEs and flares could be judged, we needed to know unambiguously whether any of the 83 energetic events were associated with another solar event. Consequently, we watched movies at 193 Å of each one of these events, each movie having been created from data obtained by the Atmospheric Imaging Assembly (AIA) on board the Solar Dynamics Observatory (SDO) spacecraft.

For each of the intense flares, identification was made visually from the AIA/SDO movies. We looked for increases in intensity on the solar surface at the time of the flare specified by the GOES SXR list, but when the site of the flare was not obvious, we accepted the reported coordinates. Whilst watching the movies of the intense flares, we also searched for evidence of an associated CME. If we were able to see any ejected material, any loop distortion, or any coronal dimming consistent with the flare site within one hour either side of the reported time of the flare, we associated that flare with a CME (regardless of whether this was a rapid CME).

Rapid CMEs were identified by searching visually for evidence of any ejected material, any loop distortion, or any coronal dimming at the time reported by CDAW. If such evidence was present (and was consistent with a front-side event), the CME was regarded as having occurred on the face of the disc; if there was no such evidence, the CME was regarded as a back-side event. Associations were made between a rapid CME and a flare (regardless of whether this was an intense flare) if the reported time of the CME (i.e. the time the CME was first seen in the LASCO C2 images) fell between one hour before the reported start of the flare and one hour after its reported end, and the evidence of the CME was consistent with the flare site.

As a result of making the associations manually, we found that 35 of the 52 fast CMEs were on the face of the disc. This proportion is slightly higher than might have been expected (given that we can only see one side of the Sun at any one time, we might expect that only half of the CMEs we see would be from the face of the disc), but can be explained by two factors: first, there were large numbers of CMEs from the same active regions (two active regions produced five each, and one other eight), and this may slightly distort the figures; secondly, 17 of the 52 events were reported to occur very close to the limb, meaning that we may have seen a CME which originated from just behind the limb.

Of the 35 rapid CMEs which occurred on the face of the disc, all were associated with a flare of some kind; 46% (16/35) were associated with an intense flare. Of the 31 intense flares, 84% (26/31) were associated with a CME.

In every instance where we had associated a solar flare with a CME, the flare was reported in the GOES SXR list as having commenced before the CME was first reported in the CDAW catalogue. It should be noted that this is not an indication of actual chronology – as an example of where there is evidence of a CME lifting off before its associated flare, see Harrison and Bewsher (2007) – but it is of significance when devising a method of automatically making associations between flares and CMEs.

CDAW reports the time of a CME as being when it is first seen in images produced by the LASCO C2 coronagraph. This instrument, however, has a field of view between about two and six solar radii (as measured from the Sun’s centre), and the images used by CDAW have a cadence of, at best, 12 minutes and sometimes much longer. The combination of these factors means that the reported time of the CME may be many minutes after is actual “lift-off” time, $t_{o}$.

Any attempt to make an estimate of $t_{o}$ faces a number of difficulties: there is no information as to the height of the CME when it was first ejected; no information as to whether it has accelerated or decelerated before its first appearance in the C2 images; and no information as to the direction of the CME. Nevertheless, finding a first approximation of $t_{o}$ is more likely to result in accurate associations between CMEs and flares than using the time of the CME as reported by CDAW.

We make the simple assumptions that by the time the CME reaches the field of view of the C2 coronagraph, it has travelled (at least) one solar radius and has undergone neither significant acceleration nor deceleration. An estimate for $t_{o}$ is then obtained by using the reported speed of the CME.

In order to take into account of the difficulties caused by the cadence of the images, we define $\Delta t$ as a number of minutes both before and after a flare. For example, if we take $\Delta t = 12$, we compare $t_{o}$ with a time window opening 12 minutes before the flare began and closing 12 minutes after it ended. Plainly, the greater $\Delta t$, the more likely it is that $t_{o}$ will fall within the window, and hence the greater the likelihood of false associations being made.

A good correlation could be found between those flare–CME associations which had been made manually and those using a value of $\Delta t$ of just 30 minutes. We did investigate whether it may be possible to improve the accuracy of the method by imposing a spatial criterion, for example by requiring the position angle of the CME to agree with the latitude and longitude of the flare to within a particular number of degrees. We found that the overall accuracy was not improved by the imposition of such a criterion.

There will, of course, always be a small number of (usually) false associations when using this automatic method given that occasionally apparently unconnected solar events sometimes occur almost simultaneously. Nevertheless, in our sample the method correctly identified 98% (60/61) associations and correctly identified 86% (19/22) non-associations, an overall success rate in 95% (79/83) of cases.

Appendix B: Algorithm 1: False Alarms in Time Range 1

Table 5 List of fast CMEs between 1 January 1996 and 31 March 2013 which were false alarms. Column 1 gives the time the CME was first reported, column 2 its speed, and column 3 its acceleration all as reported by CDAW. Column 4 is the position angle at which the height–time measurements had been made (called by CDAW the “measurement position angle”). Column 5 is the CME width. Columns 6 to 10 give the parameters of the associated flare: its start time, heliographic latitude, heliographic longitude, class, and duration.

Appendix C: Algorithm 2: False Alarms in Time Range 1

Table 6 List of X-class flares between 1 January 1996 and 31 March 2013 which were false alarms. Column 1 gives the time the CME was first reported, column 2 its speed, and column 3 its acceleration all as reported by CDAW. Column 4 is the position angle at which the height–time measurements had been made (called by CDAW the “measurement position angle”). Column 5 is the CME width. Columns 6 to 10 give the parameters of the associated flare: its start time, heliographic latitude, heliographic longitude, class, and duration.

Appendix D: Algorithm 2: False Alarms for Time Range 2

Table 7 List of X-class flares between 1 April 1980 and 31 December 1995 which were false alarms. Column 1 gives the start time of the flare, column 2 its heliographic latitude, and column 3 its heliographic longitude. Column 4 is the class of the flare, and column 5 its duration.