Recent Progress on Inverse and Data Assimilation Procedure for High-Latitude Ionospheric Electrodynamics

Polar ionospheric electrodynamics plays an important role in the Sun–Earth connection chain, acting as one of the major driving forces of the upper atmosphere and providing us with a means to probe physical processes in the distant magnetosphere. Accurate specification of the constantly changing conditions of high-latitude ionospheric electrodynamics has long been of paramount interest to the geospace science community. The Assimilative Mapping of Ionospheric Electrodynamics procedure, developed with an emphasis on inverting ground-based magnetometer observations for historical reasons, has long been used in the geospace science community as a way to obtain complete maps of high-latitude ionospheric electrodynamics by overcoming the limitations of a given geospace monitoring system. This Chapter presents recent technical progress on inverse and data assimilation procedures motivated primarily by availability of regular monitoring of high-latitude electrodynamics by space-borne instruments. The method overview describes how electrodynamic state variables are represented with polar-cap spherical harmonics and how coefficients are estimated from the point of view of the Bayesian inferential framework. Some examples of the recent applications to analysis of SuperDARN plasma drift, Iridium, and DMSP magnetic fields, as well as DMSP auroral particle precipitation data are included to demonstrate the method.


Introduction
The most dynamic electromagnetic energy and momentum exchange processes between the upper atmosphere and the magnetosphere take place in the polar ionosphere. Physical processes producing aurora involve ionization and excitation of atmospheric constituents due to energetic charged particles precipitating into the upper atmosphere from the magnetosphere along the geomagnetic field lines, which in turn modulates the ionosphere's ability to conduct electric currents. Polar ionospheric electrodynamics plays an important role in the Sun-Earth connection chain, acting as one of the major driving forces of the upper atmosphere and providing us with a means to probe physical processes in the distant magnetosphere. Accurate specification of the constantly changing conditions of high-latitude ionospheric electrodynamics has long been of paramount interest to the geospace science community.
Global monitoring of high-latitude geospace has dramatically improved thanks to a recent expansion of ground-based and space-based observing capability. International consortiums of ground-based instrumentation such as the Super Dual Auroral Radar Network (SuperDARN) (e.g., Greenwald et al. 1995), International Real-Time Magnetic Observatory Network (e.g., Love 2013) and SuperMAG (e.g., Gjerloev 2009) have made a large volume of quality-controlled, standardized data accessible to the public. Acquisition, processing, and distribution of engineering-grade magnetometer data from the Iridium satellite constellation for scientific purposes by the Active Magnetosphere and Polar Electrodynamics Response Experiment (AMPERE) program (Anderson et al. 2000) have been instrumental in making continuous, global monitoring of geomagnetic-field-aligned currents (FAC) possible. Defense Meteorological Satellite Program (DMSP) space environment instruments have long been providing valuable measurements of precipitating electron and ion particles, magnetic fields, and ultraviolet spectrographic images (e.g., Rich 1984;Hardy et al. 1984;Paxton et al. 2002). And the Swarm multi-satellite mission (Friis-Christensen et al. 2006) provides high precision measurements of magnetic fields that complement theses existing geospace observing systems.
Data assimilation techniques such as the Assimilative Mapping of Ionospheric Electrodynamics (AMIE) procedure of Richmond and Kamide (1988) have long been used in the geospace science community as a way to obtain complete maps of high-latitude ionospheric electrodynamics by overcoming the limitations of a given geospace monitoring system. The procedure combines a number of different types of space-based and ground-based observations with an empirical model of ionospheric electrodynamics to infer distributions of ionospheric electric fields and currents, FAC, associated geomagnetic perturbation fields at both ground and low-Earth-orbit altitudes, Hall and Pedersen conductance, and Joule heating. AMIE maps have yielded a number of important insights into the coupling of the magnetosphere, ionosphere, and thermosphere that takes place at high latitudes. Lu (2017) provides a comprehensive overview of AMIE applications. This paper presents an overview of the recent technical developments of the inverse and data assimilation procedure for high-latitude electrodynamics. Some of these developments are a consequence of a reformulation of the best linear unbiased estimation problem presented in Richmond and Kamide (1988) as a Bayesian estimation problem (Matsuo et al. 2005). Under the assumption that electrodynamic variables are Gaussian distributed, these two estimation problems are equivalent. A Bayesian perspective has helped to clarify the role of the prior model (background) error covariance as a key component in the modeling of Gaussian processes, and thus guided modeling and estimation of prior covariance functions from a large volume of SuperDARN data (Cousins et al. 2013a), DMSP particle precipitation data (McGranaghan et al. 2015(McGranaghan et al. , 2016, and Iridium magnetic perturbation data (Cousins et al. 2015b;Shi et al. 2019). Even though ionospheric conductivity serves as a critical linkage in electromagnetic energy and momentum exchange processes, direct monitoring of this conductivity is almost nonexistent. Another notable development led by McGranaghan et al. (2016) is an assimilative mapping of the conductance using the auroral ionization derived from DMSP electron energy flux spectra with help of the GLobal airglOW (GLOW) model (Solomon et al. 1988) without the assumption of Maxwellian distribution. Since the AMIE has been developed with an emphasis on inverting ground-based magnetometer observations for historical reasons (Kamide et al. 1981;Richmond and Kamide 1988), it is not tailored to analyses of space-based magnetometer data from DMSP, Iridium, and Swarm. In order to solve the optimization problem in terms of electrostatic potential, the space-based magnetometer data first need to be converted to electrostatic potential through the application of Ohm's law and current continuity. To minimize the impact of conductance on the inversion of space-based magnetometer data for FAC, the optimization problem is now being solved in terms of both magnetic potential and electrostatic potential Cousins et al. 2015a).

Representation of Electrodynamic State Variables Using Scalar and Vector Polar-Cap Spherical Harmonic Basis Functions
The ionosphere is treated as a thin conductive slab centered at a reference height h r = 110 km, and the current above the ionosphere is assumed to be strictly radial. The effect of the neutral wind dynamo is not considered. Electrodynamic variables analyzed here include the electrostatic potential , electric fields E, Pedersen and Hall conductance (height-integrated conductivity) p , h , height-integrated horizontal ionospheric current density J ⊥ , toroidal magnetic potential associated with field-aligned current density J , and equivalent current potential associated with ground-based magnetic fields. These variables are presumed to be related to each other as follows.
is the conductance tensor, ∇ 2 hor is the horizontal Laplacian, and μ o is permeability of free space. Equation (10.4) results from the assumption of strictly vertical J that allows equating the curls of J ⊥ and the equivalent current (i.e., Fukushima Theorem). If the Pedersen and Hall conductances are given, the relationship among all electrodynamic variables (10.1)-(10.5) becomes linear.
In the procedure, electrodynamic variables are expressed in terms of the polarcap spherical harmonic basis functions developed by Richmond and Kamide (1988). Suppose that represents a matrix of the polar-cap spherical harmonic basis functions evaluated at discrete grid locations specified by the Modified Magnetic Apex longitude φ m and latitude λ m at the altitude of h r (Richmond 1995) and that x denotes a vector of the coefficients. is furthermore given by a set of 244 polar-cap spherical harmonic basis functions up to order m = 12, with non-integer degrees n up to a maximum of n = 72.6 for m = 0, with a polar-cap co-latitude for the functions of 40 • . Therefore, x is a column vector of 244 elements and is an n × 244 matrix, where n is the number of grid points. Using the Nyquist sampling rate, the effective resolution is 15 • longitude and 2.5 • latitude. Let's suppose that the electrostatic potential at φ m and λ m is given by where t is the truncation error, and the electric fields E by where a (n × 244) matrix contains the gradients of the polar-cap spherical harmonic basis functions, which discretizes (10.1). The toroidal potential at φ m and λ m is then given by where t is the truncation error, and the FAC magnitude J by where a (n × 244) matrix contains a simplified evaluation of (10.5) using the analytical expression of the horizontal Laplacian of polar-cap spherical harmonic basis functions applicable to spherical coordinates, rather than the full expression applicable to M(110) coordinates. As explained in , this computational simplification introduces errors on the order of 10%. For a given p and h , x E and x M are related linearly through the current continuity and Ohm's law (10.2)-(10.3).

Bayesian State Estimation for Gaussian Processes
Suppose that y represents a vector of j observations that may consist of electric field, ground-based magnetic field, and/or space-based magnetic field measurements at discrete observation locations. By evaluating the polar-cap spherical harmonics and their derivatives at observation locations, y can be expressed as where H is a ( j × 244) matrix that contains the polar-cap spherical harmonic basis functions and their spatial derivatives with corresponding vector calculus operations as specified in (10.1)-(10.5), x denotes a vector of the 244 coefficients, and r is the sum of observational and truncation errors. The objective of the Bayesian state estimation is to infer the polar-cap spherical harmonics coefficients x given observations y according to Bayes rule: The vectors x and y are herein assumed to be distributed according to the multivariate normal distribution denoted by MN as x b is specified by using an empirical model. C b is described in the following section. The errors r are assumed to be uncorrelated, so C r is given by a diagonal matrix of the variance of observational error. The posterior distribution or the conditional distribution of x given observations y is given by the multivariate normal distribution as where x a is the posterior mean or the data assimilation analysis and C a is the analysis error covariance < (x a − x)(x a − x) T >. In the case of normally distributed x and y and linear H, there are closed formulae for x a and C a (e.g., Jazwinski 1970;Lorenc 1986): (10.15) By specifying C b , C r , H, and x b , the analysis x a and error covariance C a can be computed for given observations y. The prior model error covariance C b plays an important role here, not only balancing the weighting between observations and the prior model but also spreading the observation-model discrepancy information spatially according to the correlation represented in the covariance.

Nonstationary Covariance Modeling
Following the approach adopted in Matsuo et al. (2005) as a way to incorporate anisotropic and inhomogeneous characteristics of the prior (background) model errors into the analysis (10.14) in a computationally tractable manner, C b is modeled using the empirical orthogonal functions (EOFs, i.e., principal components). EOFs and their coefficients are estimated in advance of the data assimilation, for instance, from 50 million total SuperDARN plasma drift data points over January 2011 through August 2012 for electrostatic potential (Cousins et al. 2013a), from over 60 million DMSP electron energy flux spectra during the solar cycles 22 and 24 for conductance (McGranaghan et al. 2015), and from over 300 days of Iridium magnetic perturbation data from 2010 to 2015 for field-aligned currents (Shi et al.

2019).
Since observation sampling is often irregular and incomplete, a straightforward eigenvalue decomposition of sample covariance cannot be applied to the dataset. Instead, the nonlinear regression analysis of Matsuo et al. (2002) is used, wherein p principal components are expressed by a linear combination of the polar-cap spherical harmonic basis functions of Richmond and Kamide (1988), and each component is estimated sequentially by a back-fitting technique along with orthonormalization of the regression coefficients for each component. Each EOF can be expressed as β, where β is a 244 × p matrix. Then C b is given as where C γ is the covariance < γ γ T > of the EOF coefficients γ , where γ is a p × 1 column vector. EOFs estimated by the method of Matsuo et al. (2002) are equivalent to the eigenfunctions of a covariance matrix computed from observational data. As with other principal component analysis methods, a certain replication of data samples is required to estimate C γ and β from the observations. Figure 10.1 shows 2-dimensional correlation maps for electrostatic potential computed from the EOF-based covariance derived from SuperDARN data (Cousins et al. 2013a) where p is set to 30. It is evident that the correlation structures are highly anisotropic with a larger correlation length scales in the zonal direction in comparison to the meridional direction, and correlations vary depending on reference point locations. These are features of strong nonstationary correlation, which will enable the data assimilation procedure to spatially distribute the impact of observations with consideration of realistic location-specific correlation structures of SuperDARN plasma drifts or electric fields. Cousins et al. (2013b) presents an inverse and data assimilation procedure designed to specifically estimate x E as defined in (10.6) and (10.7) from SuperDARN data. A comprehensive cross-validation study (Cousins et al. 2013b) wherein observations are systematically set aside for validation and compared to predictions by data assimilation outperforms the standard SuperDARN mapping procedure (Ruohoniemi and Baker 1998;Shepherd and Ruohoniemi 2000). The inverse and data assimilation procedure is found to reduce median prediction errors by up to 43% as compared to the standard SuperDARN mapping procedure. The procedure is built using the prior covariance modeled with EOFs obtained by Cousins et al. (2013a) and the prior mean specified by the empirical plasma convection model of Cousins and Shepherd (2010). Figure 10.2 compares the maps of electrostatic potentials obtained by the standard SuperDARN mapping procedure (Ruohoniemi and Baker 1998;Shepherd and Ruohoniemi 2000 to the ones by Cousins et al. (2013b) along with maps of the uncertainty associated with assimilative mapping as given by the diagonal elements of C a (10.15). The uncertainty reflects the observation distributions with higher uncertainty found in the area of the SuperDARN data gap. The comparison also highlights the role of the nonstationary covariance in the inverse and data assimilation procedure that help regularize assimilative mapping analysis. Matsuo et al. (2015) presents an inverse and data assimilation analysis of spacebased magnetometer data that directly solves for x M as defined in (10.8) and (10.9) to circumvent the need to use conductance in analysis of space-based magnetometer data for FAC as has been originally done in Richmond and Kamide (1988). Note that the uncertainty associated with the magnetic potential analysis is shown in the black-and-white contour in the background, with darker shades indicating greater errors. For comparison, the bottom row shows maps of the FAC provided by the AMPERE program. The AMPERE data product obtained from the spherical harmonic fit has an effective resolution of 3 • latitude and 36 • longitude (Anderson et al. 2014). As discussed in Matsuo et al. (2015), the overall distribution of FAC The plots in the bottom row are maps of the FAC provided by the AMPERE program, estimated from the AMPERE data over a 10 min interval using Altitude Adjusted Corrected Geomagnetic Coordinates (Fig. 5 of Matsuo et al. 2015) is similar to the one obtained by the current procedure, except for a few notable differences in the detail, such as the absence of high-frequency features and more longitudinally continuous FAC spatial structures are seen in the present analysis. Thanks to the regularization through the use of the prior model error covariance in solving the inverse problem, there is no need to fill the data gap with synthetic data to make a regression analysis stable, as is required in the AMPERE inversion.

Dual Optimization Approach
The framework for the inverse and data assimilation procedure described in Sect. 10.2 has thus far been applied to assimilative analysis of individual electromagnetic variables. In this section, the same framework is applied to the analysis of multiple variables. The relationship among electrodynamic variables given in (10.1)-(10.5) is nonlinear, requiring a nonlinear optimization approach. As an intermediate step toward implementing a fully nonlinear solver, Cousins et al. (2015a) presents a dual optimization approach by combining the two linear optimization approaches presented in Sects. 10.3 and 10.4 but using both SuperDARN and Iridium magnetic perturbation data. For a given conductance p and h , optimal values for x E and x M are estimated independently. Specifically, the optimal interpolation (or Kalman filter update) Eqs. (10.14) and (10.15) are applied to x E with the prior error covariance for electrostatic potential estimated from the SuperDARN data (Cousins et al. 2013a) and with y being composed of SuperDARN plasma drifts and Iridium magnetic perturbation fields. For estimation of x M , (10.14) and (10.15) are applied with the prior error covariance for toroidal magnetic potential estimated from the Iridium magnetic perturbation data (Cousins et al. 2015b). Figure 10.4 demonstrates the benefit of incorporating both SuperDARN and Iridium magnetic perturbation observations into the estimation of both electrostatic and magnetic potential (Cousins et al. 2015a). For example, as shown in the orangeshaded background contour in Fig. 10.4a and d, the uncertainty for electrostatic potential distributions estimated from SuperDARN data alone is higher in comparison to the uncertainty when both data are assimilated. This is particularly evident in the dawn cell where there is no SuperDARN data but there is Iridium data. McGranaghan et al. (2016) have examined the effects of using different conductances in this dual optimization approach to assimilative mapping. When p and h are estimated by assimilation of the DMSP electron precipitation data (blue in Fig. 10.5) rather than specified by a climatological model (red in Fig. 10.5), the prediction of SuperDARN plasma drifts by assimilative analysis of Iridium magnetic perturbation data becomes more consistent with SuperDARN plasma drifts observations, as shown in Fig. 10.5c and f. Note that SuperDARN data are not used here for prediction of Iridium magnetic perturbation data, and vice versa.

Summary
This paper demonstrates that simultaneous analysis of multiple types of space-based and ground-based global geospace observations enabled by the inverse and data assimilation procedure provides a global perspective of high-latitude ionospheric electrodynamics. The paper summarizes important technical developments that have been made in response to the expansion of high-latitude geospace observing systems. The primary areas of the methodological extension to the AMIE (Richmond and Kamide 1988) are (a) the optimization in terms of both magnetic and electrostatic potential to minimize the impact of conductance on the inversion of space-based Iridium and DMSP magnetometer data for FAC mapping Cousins et al. 2015a); (b) the use of realistic prior error covariance estimated from a large data set of SuperDARN (Cousins et al. 2013a Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made. The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.