Inclusion of Real-Time In-Situ Measurements into the UCSD Time-Dependent Tomography and Its Use as a Forecast Algorithm
- First Online:
- Cite this article as:
- Jackson, B.V., Clover, J.M., Hick, P.P. et al. Sol Phys (2013) 285: 151. doi:10.1007/s11207-012-0102-x
- 532 Downloads
The University of California, San Diego (UCSD) three-dimensional (3D) time-dependent tomography program, used for over a decade to reconstruct and forecast coronal mass ejections (CMEs), does so from observations of interplanetary scintillation (IPS) taken using the Solar-Terrestrial Environment Laboratory (STELab) radio arrays in Japan. An earlier article (Jackson et al. in Solar Phys.265, 245, 2010) demonstrated how in-situ velocity measurements from the Advanced Composition Explorer (ACE) space-borne instrumentation can be used in addition to remote-sensing data to constrain a time-dependent tomographic velocity solution. Here we extend this in-situ inclusion to density measurements, and show how this constrains the tomographic density solution. Supplementing remote-sensing observations with in-situ measurements provides additional information to construct an iterated solar-wind parameter that is propagated outward from near the solar surface past the measurement location, and throughout the volume. As in the case of velocity when this is done, the largest changes within the volume are close to the radial directions around Earth that incorporate the in-situ measurements; the inclusion significantly reduces the uncertainty in extending these measurements to global 3D reconstructions that are distant in time and space from the spacecraft. At Earth, this analysis provides a finely tuned real-time result up to the latest time for which in-situ measurements are available, and enables more-accurate extension of these results near Earth to those remotely sensed. We show examples of this new algorithm using real-time STELab IPS data that were used in our forecasts throughout Carrington rotations 2010 through 2016, and we provide one metric prescription that we have used to determine the forecasting accuracy one, two, and three days in advance of the time data become available to analyze from STELab. We show that the accuracy is considerably better than assuming persistence of the same signal over one to two days in advance of when the data are available.
Observations of interplanetary scintillation (IPS) of meter-wavelength intensity variations from point radio sources have long been a source of heliospheric remote-sensing information. IPS measures small-scale (∼ 150 km) density variations along the line of sight (LOS) to a radio source (e.g., Hewish, Scott, and Wills 1964; Ananthakrishnan, Coles, and Kaufman 1980).
In Jackson et al. (2010a) (hereafter Paper 1) we describe some of the background analyses that have been used at the University of California, San Diego (UCSD) to provide three-dimensional (3D) depictions of heliospheric structures using IPS techniques. The reader is asked to refer to this article (and references therein) for more details introducing these analyses. Over the years a Computer Assisted Tomography (CAT) program has been developed (Jackson et al.1998, 2003, 2010d; Kojima et al.1998; Jackson, Hick, and Buffington 2002; Hick and Jackson 2004; Jackson and Hick 2005) that iteratively fits remotely-sensed observations to a kinematic solar-wind model. Other articles detail how the time-dependent tomographic analyses can be used to map to velocities and densities to near Earth (e.g., Jackson et al., 2006, 2008, 2010c; Bisi et al., 2007, 2009a, 2010a), at Mars (Jackson et al.2007), and at the Solar TErrestrial RElations Observatory (STEREO) spacecraft (Kaiser et al.2008). See Bisi et al. (2009b) and Jackson et al. (2011), for the STEREO analyses. This technique has performed well in retrospective analyses using a variety of remote-sensing data sets.
Using our tomographic technique we have fit data from the Solar-Terrestrial Environment Laboratory (STELab; Nagoya University, Japan) (Kojima and Kakinuma 1987), and in the past operated a real-time forecasting system during the nine-month period each year when the STELab IPS arrays were operational. From early in the last decade up until the beginning of 2011, these IPS results were compared with velocities and densities measured in real time available from the Advanced Composition Explorer (ACE) (Stone et al.1998) Solar Wind Electron Proton Alpha Monitor (SWEPAM) (McComas et al.1998) instrumentation. In the spring of 2011, the time-dependent forecasts at UCSD have been carried out in a way that is different from the original technique.
Forecasts of density and velocity prior to 2011 have often not been as accurate as possible, and in Paper 1 we describe a way to rectify this by including available in-situ velocity information near the Earth into the forecast. This extension more accurately reproduces the ACE in-situ velocity measurements at Earth at the appropriate spatial and temporal resolutions of the 3D reconstructions. It also shows that with this extension the global tomographic solution is not significantly affected. Remaining differences between the in-situ densities and velocities at Earth and their remotely-sensed counterpart can provide insight into the 3D structures of which they are a part, and can also be used to more accurately extend spacecraft in-situ measurements to locations only accessed remotely. Since the spring of 2011, we have extended our analyses to include in-situ measurements of density as well as velocity, and in this paper we describe how this density inclusion is accomplished with the STELab IPS data. Unlike the analysis shown in Paper 1 both in-situ velocities and densities are used to provide accurate weighting along each near-Earth LOS portion. This new technique provides a match of velocity and now density at Earth superior to the former technique. In this paper, we provide a metric that describes the extent to which this new technique can be judged to fit the ACE in-situ velocities and densities that are also compared with the remotely-sensed forecast values.
Section 2 describes the tomographic program developed to fit IPS data that has been extended to now include in-situ density measurements. Section 3 compares results for a portion of Carrington rotation (CR) using a combination of remote-sensing observations and in-situ density and velocity measurements. Section 4 provides a forecast metric that is used to determine how well the forecast works when including available in-situ density and velocity information. Section 5 presents a summary and conclusions.
2 3D Reconstructions Using In-Situ Density Measurements
The UCSD remote-sensing reconstruction technique provides 3D solar-wind density and velocity by applying an inversion technique to the IPS data. For a more complete description of the time-dependent analysis using either IPS or Thomson-scattering data see Jackson et al. (2010d) and references therein. When a transient structure such as a heliospheric response to a coronal mass ejection (CME) crosses a large range of solar elongations, it is viewed from widely different directions. Both this changing perspective and the changing LOS weighting for the outwardly expanding solar-wind material are exploited to reconstruct a 3D time-dependent solar-wind model, as explained more fully in Paper 1.
In addition to its use at UCSD this same time-dependent analysis system is extant at the NASA Goddard Community Coordinated Modeling Center (CCMC) for utilizing STELab IPS data. This analysis maps the heliosphere using all available data covering an interesting period of observation, including data both prior to and well after the ejected solar material moves past 1 AU. These compare favorably with other techniques used to invert the IPS data to show the extent of CME structures (e.g., Tokumaru et al., 2005, 2007) using archival data sets. The program can be operated in two different modes. The second mode of operation of the UCSD program uses data up to the present, and allows a forecast of the arrival of transient heliospheric structures. Since the summer of 2011 this analysis system now also provides an option to include in-situ measurements of velocity and density as described in this article.
The in-situ measurements are integrated into the reconstruction as a “line of sight” with a single LOS segment close to Earth. Because global density measurements strongly depend on the LOS scintillation weighting, an alternate method such as the in-situ inclusion that provides a proper analysis at this LOS location is of high significance. The in-situ inclusion supersedes this weighting by imposing more significance on the in-situ value (at the location of the arrow in Figure 1). Weighting of in-situ measurements relative to the remotely-sensed data set is somewhat arbitrary. We chose a weighting for each in-situ one-hour average measurement that is ten times the total weight of each LOS observation. Since there are on average approximately 600 LOS g-level observations used during the 27-day CR periods in 2011 (thus approximately 20 lines of sight per day), the in-situ weighting dominates that of the remote-sensing value total by slightly over a factor of 10. This weighting for the in-situ data measurement ensures that in-situ densities are accommodated in the analysis as well as the reconstruction resolutions can provide. However, this arbitrarily high weighting leaves little ability for the least squares fit to depart from the in-situ density values which may themselves have significant error. For our forecast analysis, we attempt to match the ACE Level-0 density data as well as possible (even though, as noted, these data may have systematic error with a mean value somewhat lower than other in-situ spacecraft densities measured near the L1 Lagrangian point). The effect of the inclusion of in-situ density or this weighting into the time-dependent tomography leaves the remotely-sensed IPS result beyond the immediate vicinity of Earth relatively unchanged as shown for velocity in Paper 1.
The ACE Level-0 density data generally appear to have a lower mean base value than other in-situ data sets near Earth, or for that matter than the more-refined ACE Level-2 data. For IPS g-level values, this presents a problem since a value of g=1.0 is supposed to represent a mean ambient solar-wind value. To match densities that have a lower than normal mean requires that a mean g-level fit to abnormally low in-situ densities would drive some volumetric density excursions non-physically below zero. To counter this for these ACE Level-0 density data, all scintillation levels below g=1.0 are scaled by a factor of the ratio of the mean value determined from the ACE data during the interval relative to an assumed mean level of 5.0 Np. The new mean g-level is subtracted from the old g-level for those LOS above this value. This somewhat arbitrary remedy may not prove appropriate for all time intervals, but it seems to suffice for the 2011 time interval presented here.
3 Comparison of Remote-Sensing and Remote-Sensing Plus In-Situ Inclusion
4 Real Time Density Forecasts
Since early May 2011 (CR2010) we have been operating our time-dependent tomography routine daily at UCSD and forecasting time series values of density and velocity ∼ three days into the future, and the study given below describes and shows how well we do in practice from the IPS arrays in Japan. In this case the real-time forecasts from the website-saved files check how well the system performs over an extended time period.
The radio arrays of the STELab system, and in particular for observing IPS g-level at Toyokawa, Japan, can only view radio sources crossing the local meridian; the last radio sources observed in Japan are observed at approximately 18:00 h local time, or about 06:00 UT. Data from the day’s analysis are downloaded from the STELab arrays, processed automatically at STELab (usually by about 04:00 h local time), and left on a STELab ftp website for public use. An hourly query of this website from UCSD approximately 10 hours after the day’s data are obtained in Japan (16:00 UT) generally indicates a new file with analyzed data are available at STELab, and these are then downloaded beginning a series of programming steps. For the analysis that includes the real-time in-situ data, this requires that an additional program be run which queries the National Oceanic and Atmospheric Administration (NOAA) computer every hour for real-time ACE data.
It takes approximately 6 minutes to complete the IPS tomography program forecast and to output the associated volumetric files. Once the tomography analysis completes, a further procedure provides visualizations for the website. First, images are made from the current tomography analysis that are presented immediately (within 18 minutes) on the website from the latest tomography run. Following are animations of the original tomographic analysis for use throughout the daily period before the next tomography run is available. The image data sets require updates that change time markers and updated in-situ measurements. These are refreshed hourly from the earlier-run tomography until there are no longer forecast volumetric files resident on the computer.
Correlation of IPS change from the change measured by ACE SWEPAM.
Important in the correlation is not only how well we forecast the future measurement change, but also the size of the future change. This change is not indicated by the correlation, and in a few instances such as the density change for CR2010 and the velocity change for CR2011, the amount of change present in the ACE data are less than 1.0 Np cm−3 and 100 km s−1, respectively. It is also clear that this type of correlation is only one metric that can be used. We summarize these results further in the final section, and describe how we expect to better these forecasts in the future.
5 Summary and Conclusions
The inclusion of in-situ measurements into the remote-sensing observations provides a better constraint on the iterated source surface solution, and by outward extension into the heliosphere, provides a positive way of extending current observations into the future as is indicated in the forecast analyses in Figure 5 and Table 1. Of course, an essential test of the 3D reconstruction technique requires that results incorporating the heavily weighted in-situ data are better fit when in-situ densities are incorporated and this is certified by this study. The in-situ input to the 3D reconstruction program is incorporated in the form of hourly averages, and the output of the reconstruction program is averaged spatially and temporally with the Gaussian filters noted in Section 2. For comparisons commensurate with the resolutions imposed by the tomographic analyses, the ACE data have been averaged using a 3/4-day temporal filter. For this reason alone the comparisons are not expected to be exact, but only close to what is measured in situ, and this is also shown in the current study (Figures 4 and 5).
This article shows for space weather forecast analyses that the present methods provide an improvement in the IPS 3D velocity reconstruction at Earth. In-situ data have been the primary measurements available for study of solar-wind plasma parameters near Earth, and are thus the standard available for comparison. Even so, we do not know a priori whether the ACE Level-0 in-situ density measurements are closer to the correct values than those measured remotely, and we simply assume they are, and we try to match them in the future. Other spacecraft near Earth may give better results and allow better density forecasts than ACE, and especially in the use of the mean level density fit to g-level that is mentioned in the last paragraph of Section 2. However, which of the other L1 spacecraft in-situ density measurements can be better forecast using this technique is beyond the scope of the current study, and presently their data are not available in real time to enable forecast analysis.
If the inclusion of in-situ measurements into the tomographic result were simply a way to smooth these so that they blend into the result determined by remote sensing alone, we would not expect much improvement in the remote-sensing forecast results beyond the immediate vicinity of the last in-situ measurements. This does not seem to be the case for velocities as shown by Paper 1, and we speculate that the inclusion of the in-situ measurements near the observing point significantly improves each LOS result. It probably does this by refining information close to the observer, where small amounts of noise might seriously alter the result along the whole LOS. The inclusion of the in-situ data can also eliminate source values that are erroneous, since it is possible that a few cause large deviations in our modeling and are not otherwise removed by the iterative-fitting criterion. Although a correct g-level analysis undoubtedly helps the analysis of velocity and vice versa, the two parameters of g-level (the density proxy) and velocity are nearly separable and the tomographic program converges well and approximates the same answer for the remaining parameter whether or not velocities or g-levels alone are available. Currently during the winter months in Japan, only g-level values are available from the new STELab Toyokawa array that operates year-round, and with no velocities to help in the analysis, the forecasts have worked satisfactorily throughout the 2011 – 2012 winter season. This operation of the tomographic program was explored more thoroughly by Jackson et al. (1998), using the archival IPS data sets available at that time; a more comprehensive analysis of this using current STELab Toyokawa g-level data, and with and without the inclusion of in-situ data is beyond the scope of the current investigation described here.
We would like to provide a better forecast analysis, and while the metric we have chosen to display is encouraging and shows that generally one or perhaps two days in the future we do better than a simple extension of the current non-inclusion results into the future, the time series changes and the forecast correlations shown are not exceptional. This is surely due in part because the correlations for the remotely-sensed density data comparison with the ACE Level-0 data during this period are not very good. Others plan to provide their own metrics for this same-type analysis (e.g., MacNeice et al.2011), and perhaps a better way to provide a different metric will be found to improve the forecasts, and consequently these analyses for research purposes using the archival data set.
Limits to the IPS density forecasting technique include the assumptions made to extend g-level variations to density forecasts. This is a non-linear process even in weak scattering (for 327 MHz IPS observations, this is generally beyond about 11.5∘ elongation), and it assumes that a relationship can be found between g-level determinations and density that is the same for all heliospheric structures (see Section 2). For instance, the possibility has often been expressed that the regions behind shocks could have fundamentally more turbulence than the increased density would indicate (e.g., following co-rotating structures – Ananthakrishnan, Coles, and Kaufman 1980). This has not been checked systematically in these analyses, but there is not much evidence for this so far in the forecast CME shock analyses. The 09 November 2011 halo CME is a good example of an event that provides a shock observed at Earth, and yet has no discernable additional turbulence that provides a higher than expected density. Other innovations are possible in the near future. The assumption is not optimal that all source sizes used in the IPS weighting are of the same value (see Bisi et al.2010b for how source size affects the LOS weighting). Inclusion of measured source sizes (Manoharan and Ananthakrishnan 1990), or using these same iterative techniques to modify source sizes to provide better 3D reconstruction fits, could in turn improve tomographic analyses both for research studies and forecasting.
Reducing other limits at STELab could provide forecasts with shorter latency. The actual analysis time required to process the IPS data generally amounts to about three hours each day. The 3D reconstruction analysis uses in-situ data available within an hour of the tomographic run. However, the remotely-sensed IPS observations are at least 10 hours old. Although this data latency could be shortened at STELab, it would require processing the IPS data before local midnight in Japan, and foregoing calibrations of the arrays following data recording. Thus, processing the IPS data to provide updates as sources are observed would help, but is not currently planned, because it is very labor intensive for an instrument where archival research is the primary focus rather than real-time operation.
There are also several limits to the technique to provide a better IPS density forecast that can be overcome by improved analysis systems. Primary limits to the IPS observations are the few radio sources that can be recorded by the current STELab system and the unavoidable daily cadence. The first limits the amount of sky coverage and 3D resolution, while the latter means that the fastest outward-moving events can go unseen before they arrive at Earth. These are well-known limits to a single IPS system, and while provision of more lines of sight can be made by using a larger array as at Ooty, India (e.g., Manoharan 2010), or at Pushchino, Russia (e.g., Chashei et al.2011), this enhancement quickly becomes an arduous undertaking, especially in Japan where buying parcels of land suitable for large radio arrays adds considerably to the cost of a new or larger array. While an IPS array capable of obtaining over 150,000 lines of sight daily as from the SMEI imager analyses (see Jackson et al.2010b) is probably not possible, a more modest ground-based venture would still prove to be far less expensive than a spacecraft mission, and could achieve somewhat similar results. The second limit, providing more frequent than once-a-day observations, has a solution by obtaining data for a forecast from different longitudes around the globe and integrating them into a single analysis. However, as yet no other existing IPS radio array operates in real time as does the system at STELab. While there has been an incentive to provide combined analysis using data from different world radio arrays using archival IPS data sets, there has so far been no combined forecast analysis attempted. Building, and especially provisioning, similar radio arrays around the world at different longitude locations is a far more expensive option than using the existing radio arrays. Finally, there are interference problems, some man-made and others caused by Earth-based weather, at the existing single sites and systems. In the past when all similar STELab radio arrays were used to determine scintillation level, the g-level determination could be mitigated by selecting the radio site that showed the lowest scintillation level for a given source on that day. The correlations shown in Figure 5 and Table 1 do not provide any means for this assessment, but they could, and it is expected that a quality criterion from each data set would substantially allow poor data to be monitored and flagged as suspect. For instance, a more careful look at the record shows that during CR2112 and CR2113 (July – August 2011) there were many several-day outages caused by lightning from thunderstorms and high-speed wind in part associated with the several typhoons that passed through central Japan during this period. Although such weather-related interference can be mitigated to some extent by careful source editing, there is no real recovery from data outages caused by the inability to operate the radio arrays during and following extreme weather events. For the entries in Table 1, the low-density correlation forecast in July and August is most-certainly affected by these observational outages.
In summary, we show a system that works reasonably well year round to determine the direction and propagation of heliospheric transient phenomena, and to forecast the arrival (or non-arrival) of these structures in the vicinity of the Earth when compared with simply assuming tomorrow’s measurements will be the same as today’s. We expect that these analyses can still be improved considerably, and that this will lead to better forecasts, and suggest better means of IPS data accrual and editing, and of making the global heliospheric research analysis possible from this unique data set.
B.V. Jackson, P.P. Hick, M.M. Bisi, J.M. Clover, and A. Buffington were supported at UCSD by NSF grants ATM-0852246, and AGS-1053766 and NASA grant NNX11AB50G. In addition for these analyses, M.M. Bisi was supported by NSF grant ATM-0925023 at UCSD and by a UK STFC Standard Grant to Aberystwyth University, and B.V. Jackson was also supported in part by AFOSR grant 11NE043. The authors wish to acknowledge and thank the group at STELab, Nagoya University not included in the author list (M. Kojima, K. Fujiki, and students) for their continued support, and for making IPS data sets available under the auspices of a joint collaborative agreement between the Center for Astrophysics and Space Sciences at UCSD and STELab, Nagoya University. We also wish to thank Kevin Schenk and the LASCO coronagraph group for the 09 November 2011 halo CME images taken using the C2 coronagraph, and also to the STEREO SECCHI group for making available their COR 2 images.
This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.