1 Introduction

The concept of space weather started to penetrate through the academic solar-terrestrial physics (STP) community in the early 1990s. Science behind, and technological consequences of, space weather have, of course, much longer history (see Lanzerotti’s history review, this issue). We do not need to go as far as Galileo’s observations of sunspots or the investigations of Anders Celcius and Olof Hjorter showing that the magnetic needle reacts on bright auroral displays, both of which still today are among the most readily accessible space weather phenomena. One of the most important early events was the large flare on the Sun on September 1, 1859, observed by Carrington and Hodgson. Only 17 hours later an exceptionally strong magnetic storm, perhaps the strongest ever recorded using scientific instruments, commenced on the Earth. However, it took a long time to understand what really had happened and how the effects of solar activity could be carried to the Earth.

Today we have good reasons to assume that the agent must have been a large and very fast coronal mass ejection (CME) released in association with the flare, but CMEs were not identified until 1972. Finding these giant magnetic and plasma ejecta was a major step forward in understanding Sun–Earth connections. CMEs not only turned out to be major drivers of strong magnetospheric storms but also to play a critical role in the acceleration of solar energetic particles to tens of MeV energies.

We have today a highly sophisticated understanding of many aspects of solar and solar-terrestrial physics, and the study of space weather phenomena form a subset of the field. Thus why are we concerned about the scientific foundations of space weather when we already have STP? We can try to answer the question from the viewpoint of the practical rationale behind space weather. Space weather emphasizes applications and societal relevance much more explicitly than STP, which can be seen as “pure” science. We can take magnetic reconnection as an example. In basic space plasma physics the question, what breaks the frozen-in condition for electrons, is highly relevant to ask whereas in space weather it is much more important to find out when and where the reconnection really takes place, and also to be able to forecast the event and its consequences in advance.

Space weather is science with applications. In the absence of direct applications, nobody would have ever introduced the concept of space weather. In space weather the applications pose the most important and hardest scientific questions. Particularly, the progress needed to improve both short- and long-term forecasting challenges our understanding of the scientific foundations. Sometimes the path from basic research to applications may be short but more often it is long. In this field the easy questions have already been answered and the difficult ones will be with us for a long time to come.

This is the lead-article to papers based on the one-week workshop on Scientific Foundations of Space Weather at the International Space Science Institute in Bern, Switzerland, in June 2016. During the workshop the participants looked for answers to questions:

  • What kind of progress have we made in the understanding of the scientific foundations of space weather during the past few decades?

  • What are the holes in our understanding that can be filled in the near future?

  • What is needed for long-term progress?

This article aims at setting the tone for the rest of the volume. We do not want to pre-empt the content of the other articles. Instead we wish to highlight some of the key achievements in the past and challenges for the future.

2 Space Weather Phenomena from the Sun to the Earth

In science we often tend to be reductionists, studying our pet corner of the universe, whether it is a certain decay channel of the Higgs boson or the very first microseconds after the Big Bang. The need to understand space weather motivates the STP community to form a more integrated view of the entire chain of processes from the solar activity to near-Earth space and the surface of our planet. Progress on this more basic level, beyond the phenomenological descriptions of space weather, has only begun. That is why workshops like this are needed.

2.1 Space Weather Chains Connecting the Sun to the Earth

There are three main routes along which the Sun affects the Earth’s space environment: the electromagnetic connection, the plasma chain mediated by the solar wind, and the energetic particle chain consisting of high-energy charged particles accelerated in the solar corona and along their way to the Earth.

The electromagnetic solar radiation does much more than transferring information of solar activity and eruptions. The solar EUV radiation ionizes the upper layers of the atmosphere forming the partially ionized ionosphere. Strong X-ray flares can also cause observable ionization events in the atmosphere. In fact, a 110-nT perturbation in the Greenwich magnetogram was observed at the time of the Carrington flare (e.g., Cliver and Svalgaard 2004), much before anybody knew of X-rays and how these can lead to generation of electric currents in the space environment that give rise to disturbances in the terrestrial magnetic field. In rare cases strong radio bursts associated with solar flares can also cause significant problems by directly introducing noise in the wireless communication systems (Lanzerotti et al. 1999). The most extreme case thus far observed at GPS frequencies on 6 December, 2006, will be discussed further in Sect. 2.9.

The understanding of the decisive role of the plasma chain developed gradually with the progress of solar-terrestrial physics. Already before the Carrington flare, Schwabe (1844) had found the quasi-periodic approximately 11-year cycle in the number of sunspots. Hale (1908) was the first to observe strong magnetic fields in the sunspots. By that time magnetic variations both on the Sun and on the Earth were connected to phenomena that we today understand as elements of space weather. But how sufficient amounts of energy could be transported from the Sun to the Earth to generate the global-scale disturbances, remained a mystery.

During the early 20th century some scientists (e.g., Chapman 1918; Lindeman 1919; Chapman and Ferraro 1931a,b) proposed that ejections from the Sun of quasi-neutral gas streams, or plasma, could affect the geomagnetic environment when hitting the Earth’s magnetic field. They were on the right track but they did not yet understand how the particles could escape from the gravitational pull of the Sun. The escape velocity from the solar surface is 618 km s−1, which is much larger than the thermal velocity of the gas on the solar surface at the black-body temperature of 5778 K. With improving spectroscopy in the 1940s the solar corona was found to be much hotter, up to a couple of million kelvin, the corresponding thermal speed of a proton thus being sufficient for escape from the Sun. How the corona is heated remains, despite much observational, theoretical and modelling progress, a largely unsolved problem. The reason why this question is still unanswered lies in the complexity of processes in the solar atmosphere that prevents the identification of unique and simple heating mechanisms (for a recent review, see Klimchuk 2015).

In the 1950s Biermann (1951, 1957) demonstrated the existence of a continuous interplanetary corpuscular flow from the Sun by a careful study of cometary tails. In addition to the slowly escaping dust tail lagging behind the comet, he recognized a much faster, almost anti-sunward, outflow from the comet, which must be swept by a fast flow originating from the Sun. Alfvén (1957) pointed out that this flow had to be in the form of magnetized plasma that was picking up charged particles from the cometary environment. In 1958 Parker published the first physics-based model for plasma escape from the Sun (Parker 1958). Soon thereafter the first spacecraft moving outside of the Earth’s magnetosphere directly proved the existence of continuously flowing magnetized solar wind (Neugebauer and Snyder 1962). The existence and basic properties of the plasma chain connecting the Sun and the Earth were firmly established.

However, it still remained unclear how the solar wind can shake the magnetosphere as strongly as the most severe magnetic storms indicated. There was also the enigma of solar energetic particles (SEPs). There was a correlation between solar flares and SEPs but the correlation was far from perfect and the strong SEP fluxes often lasted long after the flare activity on the Sun had subsided. That the magnetosphere was occasionally hit by interplanetary shock waves was known and these were understood to be potential particle accelerators, but the main drivers of the shocks were unknown. Another early observation was that the flux of galactic cosmic rays transiently decreased during strong solar magnetic activity (Forbush 1938); these became later known as Forbush decreases. Somehow the enhanced solar magnetism seemed to provide a strengthened magnetic shield against galactic cosmic ray particles.

The solution to these questions came with the first observations of coronal mass ejections in 1972 (Tousey 1973). In addition to the slow and fast solar wind, the Sun was found, from time to time, to be able to expel huge plasma and magnetic clouds to the interplanetary medium. A large fraction of these propagate with super-Alfvénic speed with respect to the background solar wind, producing shock waves that are efficient particle accelerators. A fast and large CME, in particular when its magnetic field has a strong southward component (\(B_{S}\)), is the most efficient driver of magnetospheric activity and such a structure lies behind all strongest magnetospheric storms.

2.2 Solar Drivers of Space Weather

The ultimate driver of space weather phenomena is the variable plasma flow in the convective zone of the Sun that shapes the structure and dynamics of the solar atmosphere. The variations in the plasma flow take place on many spatial and temporal scales leading to magnetically active regions and sunspots on the solar surface. On the largest, coronal, scales the solar magnetic field forms regions of open and closed magnetic flux. The fast solar wind is understood to escape from the open-flux coronal holes but the origin of slow solar wind is much more unclear.

On smaller scales the magnetic flux emerges as magnetic arcades above the solar surface in the form of flux ropes with both ends tied to the surface and below. The line-tying to the solar surface is strong and when the footpoints of a flux rope move, the rope becomes twisted. If the twisting continues long enough, a macroscopic plasma instability sets in and the flux rope may erupt and escape from the Sun as a CME. This is by no means the only scenario to explain the birth of a CME, but fundamentally it is an event where a flux rope is destabilized through some plasma physical process. It is worth noting that the eruptions take place in the corona without any significant changes on the surface.

The CMEs are often, but not always, associated with solar flares. On the other hand, there is much flaring activity on a very wide range of energy scales without any signs of CMEs. Large enough flares are observable at all electromagnetic wavelengths from radio waves to hard X-rays, even to \(\gamma \)-rays, and they accelerate electrons to relativistic energies and protons up to the MeV–GeV range. Until the discovery of CMEs solar flares were thought to be the main drivers of magnetospheric storms. This “solar flare myth” actually survived in parts of the magnetospheric research community until the early 1990s, when a landmark paper by Gosling (1993) shifted the emphasis from flares to CMEs. It is, however, important to recognize that solar eruptive phenomena are dynamically very complex and interrelated. Thus it would be misleading to discard the role of flares in solar phenomena affecting the Earth’s environment (Hudson et al. 1995). The origins of solar eruptions are discussed further by Green et al. (this issue).

While separate phases of the eruptions can be studied with sophisticated numerical models, here lies a serious modelling challenge. The plasma parameters from the subsurface convection zone to the hot and tenuous corona vary so largely that different approaches must be applied to describe the different regimes. Even with the present pace of computing power evolution and development of ingenious numerical schemes this state of matters will remain work-in-progress for the foreseeable future. The causal chains of physical processes that shape, from the photosphere to the outer corona, the structures and their dynamics are at the heart of the predictability of solar phenomena vital for quantitative space weather forecasting. Without significant progress in understanding these causal chains, advancing the science of space weather will remain a challenge.

Some immediate needs for modelling improvements were identified at this workshop. More parametric studies with simple and idealistic models are required to understand the basic physics of eruptions, for example, torus instability and flare reconnection. As the heat flux is an essential element in the solar environment, more efforts to develop thermodynamic MHD approach are called for.

2.3 Propagation of Solar Perturbations in the Inner Heliosphere

The solar wind is traditionally divided into the slow (around 300 to 550 km s−1) and the fast (around 650 to 900 km s−1) wind. At 1 AU the electron number density of the slow wind is typically about 10 electrons per cm3. The positive charges are almost all protons (\({\sim} 95\)%) with small fractions of helium nuclei (∼4%) and heavier elements (\({<}1\)%). Electron temperature of the slow wind is typically 1000 K and proton temperature \(3\times 10^{4}\) K, whereas the fast wind is more tenuous, some 3 electrons per cm3, and much hotter, electron temperature being typically \(10^{5}\) K, proton temperature \(2\times 10^{5}\) K.

There are large variations in these numbers, and it has recently become questioned whether the solar wind distribution really is bimodal or is there some kind of underlying dynamic continuum of states. This question is related to origins of the different types of solar wind (see Cranmer et al., this issue). It is fairly clear that the fast wind emerges from the coronal holes but there is no consensus about the coronal sources of the slow wind.

The Ulysses mission clearly demonstrated the out-of-ecliptic bimodality of the solar wind. Around solar minimum, the uniformly high-speed solar wind originated in the large solar polar coronal holes, while the slower, much more variable solar wind streams were associated with the more complex, magnetically closed corona (McComas et al. 1998). It was found that the charge state composition of the solar wind was an objective measure of the origin of individual streams. In particular, the ratio of O7+ to O6+ indicated the freezing-in temperature of the stream (where the plasma becomes collisionless). The temperature distribution was found to be strongly bipolar and is strongly anti-correlated to the solar wind speed (Geiss et al. 1995).

The bimodality is not as evident the in-ecliptic observations, and thus in the near-Earth space. Although bimodality can be observed, it is less clear-cut and more variable. Even if solar wind streams may be well differentiated at their coronal origin, the dynamic effects of the interactions in the non-uniform solar wind modify and mix the kinetic signatures, so that bimodality based on speed becomes obscured. Consequently, the kinetic parameters (speed, density and temperature) at 1 AU are not an unambiguous guide to the origin of the solar wind streams. Simultaneous measurements of ion charge composition would be necessary—unfortunately these are rarely available.

Both the fast wind and the stream interaction regions (SIR), often called corotating interaction regions (CIR), between the fast and slow wind are important drivers of weak and moderate magnetospheric storms. While fast and slow winds escape radially from adjacent regions, the solar rotation forming the Parker spiral leads to structures where either the fast wind pushes against the slower wind causing a fast forward compressional wave, or the slow wind lags behind the fast wind and an in the solar wind frame reverse rarefaction wave emerges. The fast forward wave gradually evolves toward a shock. In most cases the wave does not steepen to a fully developed shock until beyond the Earth’s orbit, but it can already be sufficiently strongly compressed to drive a moderate storm in the magnetosphere, in particular if the compression has led to sufficiently enhanced southward component of the magnetic field.

But where do the interplanetary propagating CMEs, often called ICMEs, belong in this scheme of fast and slow wind? At and after the eruption, CMEs are accelerated to very different speeds (e.g., Yurchyshyn et al. 2005). Sometimes the CME is slower than the solar wind in which it is embedded. In such cases the ICME is gradually dragged close to the speed of the ambient wind. The process can be described in terms of anomalous viscosity, i.e., wave–particle interactions. Similar interaction decelerates an ICME that is just slightly faster than the ambient wind. The relative velocity between these ICMEs and the ambient wind is too small to drive a bow shock ahead of the ICME. Nevertheless, the magnetic field may have fairly large southward component when the structure hits the magnetosphere and even a strong storm can result.

Fast CMEs causing the strongest storms can erupt with very high speeds. The acceleration must come from a rapid release of magnetic tension in the erupting structure and can be expressed in terms of the Ampère force (\(\mathbf{J}\times \mathbf{B}\)). The famous extreme event on July 23, 2012, left the Sun with the speed of \(3050 \pm 260\) km s−1 and had the speed of \(2246 \pm 110\) km s−1 at 1 AU (Liu et al. 2014). It led to an extreme magnetic compression, the peak magnetic field exceeding 100 nT. This event also illustrates another important feature of the interplanetary propagation of CMEs. Sometimes two or more CMEs are expelled from the same region after each other. It appears that the first CME ploughs the road for the following CME, which can catch the leading CME and the two structures merge, sometimes called CME cannibalism. The combined, complicated structure can lead to very strong events, as was the case in this particular event.

Also ICMEs compressed between the heliospheric plasma sheet, typically a slow wind domain, and fast wind can lead to large magnetic fields and become very geoeffective, as was the case in the March 17, 2015, event (Kataoka et al. 2015). For further review of the large variety of the interactions of ICMEs with their surroundings and each other, including the interactions between ICME shocks, we refer to Manchester et al. (this issue).

For space weather in the magnetosphere the southward component \(B_{S}\) of the interplanetary magnetic field (IMF) is a key parameter. It is important to keep in mind that there are different causes for strong ICME-related \(B_{S}\) (e.g., Kilpua et al., this issue). The flux ropes with strong helical magnetic field form in the corona, whereas the enhanced sheath region \(B_{S}\) accumulates during the propagation. A major difficulty in understanding evolution of the ICMEs during the propagation from the Sun is that there are no detailed observations between the corona and 1 AU. Observing and tracing radio emissions associated with the propagating CME and its leading shock wave has been used to reconstruct some aspects of propagation characteristics (Reiner et al. 2007; Corona-Romero et al. 2015). The heliospheric imagers such as onboard STEREO help but, for example, the essential parameter, the strength of the magnetic field can be measured only very near the Sun and thereafter not until the ICME has propagated to 1 AU, unless there happens to be an interplanetary spacecraft with magnetic field instrument in the right place at the right time.

2.4 Shocks and Solar Energetic Particles

Solar energetic particle (SEP) events are traditionally divided into impulsive and gradual SEP events (e.g. Cane et al. 1986). The impulsive events are relatively short-lived, the energetic particles being observable for a few hours following the onset. These events are electron-rich and the 3He/4He ratio is of the order of one, sometimes even larger than 10. The impulsive events are typically associated with impulsive hard and soft X-ray flares but not necessarily with any CMEs. The gradual events are proton-rich, they are observable for days, and 3He/4He is much smaller than in the impulsive cases. The associated X-ray flares are typically soft. Almost all gradual events, some 96% of them, have been found to be associated with CMEs (Kahler et al. 1984).

Particles in impulsive events are thought to be accelerated close to the Sun by the rapid energy release in the impulsive phase of a solar flare and by the consequent strong wave activity. The high 3He content is a curious fact because only some 0.05% of all helium in the solar atmosphere is 3He. This indicates that the acceleration process must be highly selective to 3He, more precisely 3He++ whose gyro frequency is \(2/3\) of the proton gyro frequency. Thus interaction with Alfvén ion cyclotron waves at frequencies below the proton gyro frequency is a plausible candidate. The same acceleration mechanism that preferentially accelerates 3He appears to act, in a similar way, to enhance the fraction of partially ionized heavier atoms (Mason et al. 2016).

In the pre-CME times it was much harder to explain the gradual events, as practically the only solar activity they could be correlated with were flares. It is instructive to refer to the early studies as they shed an unbiased (if partly ignorant) light on the properties of the observed energetic particle populations, then termed Energetic Storm Particle (ESP) events (see, e.g. Bryant et al. 1962; Rao et al. 1967). It is true that large gradual events are associated with big flares, but today we know that the protons and other ions in gradual events can be accelerated to high energies at the shocks driven by fast ICMEs. Once the CMEs entered to the scene, the picture with a clear separation between impulsive and gradual events developed.

However, analogous to the problems with a bimodal picture of fast and slow solar wind, also this bimodality has problems with statistics, as discussed by Klein and Dalla (this issue). When correlating SEPs with flares, positive correlations are found. Flares and CMEs also correlate, in particular in the case of large flares. Even though the causal relationship between CMEs and flares is not straightforward, it could well be indirect with complex intermediary processes. Kahler (1982) introduced the concept of “big flare syndrome” to describe the danger of associating SEPs with flares based on the simultaneous appearance of big flares, CMEs and SEPs. The likely cause of the correlations is the dramatic reconfiguration of coronal structures, involving widespread magnetic reconnection and clearly releasing large amounts of stored magnetic energy. It is these underlying processes that are likely to trigger flares and launch CMEs—and ultimately causing the production of SEPs. A recent review of the topic has been provided by Reames (2015).

Another strong statistical bias toward attributing the gradual events to the ICME driven shocks arises because gradual events are orders of magnitude more intense than impulsive events. As pointed out by Trottet et al. (2015) both flare acceleration and ICME shock acceleration contribute to the production of protons in the energy range of tens of MeV and nearly-relativistic electron populations in large SEP events. Thus impulsive and gradual events cannot easily be distinguished, as the impulsive component can be hidden under the much more intense shock accelerated component of a gradual event. For practical space weather purposes, the strong events are, of course, critical because they are the most harmful to spacecraft and humans in space.

Further challenge with SEPs is their east-west variation. Most SEP events have been measured at one single point in space and the global picture has been composed putting together different events with different characteristics and using numerical modelling. The problem of how SEPs populate a broad range of solar longitudes dates back to the times before the association with ICMEs was suspected. Reinhard and Wibberenz (1974) studied a number of events using widely spaced spacecraft in the late 1960s and early 1970s. They attributed the longitudinal transport of SEPs to a three-step process: first, a rapid spreading of the accelerated population in the corona, followed by drifts in the highly structured high-altitude coronal arcades, and finally by a slower diffusion phase that extended the spatially broadening SEP populations further in the inner heliosphere.

Today the wide longitudinal extent of gradual events is, at least partially, understandable, as ICME-driven shocks are longitudinally wide and expand during the propagation. de Lucas et al. (2011) analyzed all ICME-driven shocks that had been observed during 1974–1986 by multiple spacecraft, including Helios. They found that there was a 50% chance to observe a shock at two locations separated by \(90^{\circ }\), after which the probability quickly dropped. Only in four cases the extent was in the range \(120^{\circ }\)–\(160^{\circ }\). However, the energetic particle enhancements can be longitudinally even wider than the shock, which could at least partially be attributed to cross-field diffusion in the interplanetary space.

The STEREO mission would have been ideal for further studies of the east-west extent of the shocks and SEPs. Unfortunately, the number of good events during the optimal separation of the two spacecraft was limited due to the long solar minimum 2008–2010. An interesting SEP event took place on November 11, 2011, and was observed by near-Earth spacecraft, both STEREO spacecraft and MESSENGER. Gómez-Herrero et al. (2015) found an almost circumsolar particle enhancement although the data indicated that all particles originated from a single active region on the Sun. Their conclusion was that the data were not consistent with a major role of interplanetary diffusion, as had been suggested by Dresing et al. (2012) to have been the case in an earlier event on January 17, 2010. Instead, in the November 2011 event the population appeared to have quickly spread already close to the Sun.

That the longitudinal extent of SEP events can have so different physical causes is a great challenge for modellers. While the extension of diffusion models to three spatial dimensions already is a formidable task, it may not be enough, as coronal drifts and other transport processes form a complex cocktail that may well vary from event to event.

2.5 Space Weather in the Magnetosphere

Most technological systems sensitive to space weather operate inside the magnetosphere, including those in the atmosphere or on the surface of the Earth. The global scale magnetospheric storms and substorms have been in the focus of STP research since the early days of in situ solar wind and magnetosphere observations (Akasofu and Chapman 1961; Akasofu 1964). It has become clear that the magnetosphere responds to the variations in the solar wind forcing in widely varying temporal and spatial scales. The internal magnetospheric processes and different magnetospheric domains are coupled to each other through complicated direct and feed-back mechanisms that often are non-linear. While there are good motivations to try to describe the magnetospheric activity in terms of different activity parameters, e.g., Dst, Kp, or AE (for definitions, see Mayaud 1980), they only give very coarse description of the magnetospheric dynamics. Space weather forecasts based on indices calculated from solar wind observations remain statistical and are not detailed enough if temporally or spatially localized forecasts are required. During the past 15 years the multi-satellite missions Cluster, THEMIS, and most recently MMS and Van Allen Probes (VAP) have opened new possibilities to deal with spatio-temporal structures and phenomena in the near-Earth solar wind, magnetosheath and magnetosphere. These observations together with improving physics-based numerical simulation tools carry a promise to qualitative improvements in our understanding of magnetospheric physics, but we are not yet there.

The most important drivers of intense magnetospheric activity are ICMEs with the associated shocks and sheaths, fast solar wind, and fast/slow stream interaction regions (SIR). As magnetic reconnection, introduced in the magnetospheric context by Dungey (1961), is a key element in the energy, momentum, and plasma transfer from the solar wind to the magnetosphere, the southward IMF (\(B_{S}\)) is a major contributor to the storm strength. However, it is not sufficient to consider the upstream solar magnetic field only. The magnetic field can have significant evolution in the magnetosheath and finally it is the shocked solar wind that interacts with the magnetopause. The sheath fluctuations are quite different behind the quasi-parallel and quasi-perpendicular sectors of the bow shock (for comprehensive reviews, see Bale et al. 2005; Burgess et al. 2005), and the kinetic effects must be taken into account, at least of ions. For numerical simulations, this calls for hybrid approach where electrons are treated as a fluid and ions are described by quasi-particles, as in the widely applied particle-in-cell simulations (see, e.g., Karimabadi et al. 2014), or in terms of distribution functions, as in the newly-developed Vlasov-hybrid method (von Alfthan et al. 2014).

The peak intensity of intense storms (defined here as \(\mathrm{Dst} < -100~\mbox{nT}\)) is observed equally often during ICME flux rope and sheath passages and only about 15% of intense storms take place during fast streams or SIRs (Huttunen and Koskinen 2004). The ICME flux ropes and sheaths tend to enhance the geospace response in different ways. Smoothly rotating flux ropes are more efficient drivers of large-scale convection and ring current, whereas the more fluctuating magnetic field in the sheath regions leads to stronger high-latitude effects (e.g., Huttunen et al. 2002; Pulkkinen et al. 2007). Also here the mixture of causes and effects, as well as involved feed-back mechanisms between different processes, make correlation studies non-straightforward.

The main, but not the only, contributor to the low-latitude magnetic deviations on ground, from which the Dst index is calculated, is the ring current. During intense storms the ring current dominates together with enhanced magnetopause current, the latter of which can be eliminated with rather good reliability (O’Brien and McPherron 2000), if the “pure” ring current effect is sought for. During moderate storms the contributions from the tail current become important, which are more difficult to handle (for a comprehensive reviews of the geospace current systems, see Ganushkina et al. 2015, and Ganushkina et al., this issue).

The positive correlation between fast solar wind and radiation belt electron fluxes has been known for a long time. High solar wind speed very quickly leads to enhancements of the lower-energy electron population, whereas the appearance of relativistic electrons can have a substantial delay. The wave–particle interactions needed to lift the energies to the relativistic range take their time (e.g., Horne et al. 2005).

Recently the Van Allen Probes (VAP) of NASA have shed much new light to the radiation belt dynamics (for a thorough review of VAP contributions, see Baker et al., this issue). A lucky coincidence was that a new ultrarelativistic electron belt was formed on September 2, 2012, soon after the VAP launch (Baker et al. 2013). It is becoming increasingly evident that “if you have seen one storm, you have seen one storm” (Geoff Reeves, private communication). In fact, energization by interplanetary shock-induced electric field impulses (e.g., Kanekal et al. 2014) as well as wave–particle interactions through both chorus (e.g., Thorne et al. 2013) and ULF (e.g., Mann et al. 2016) waves have been successfully applied to VAP observations of different storm events. As the characteristics of the various wave modes critically depend on the plasma background where they propagate, the coupling to the low-energy plasmas in the plasmasphere and ionosphere is of decisive importance. During strong magnetospheric activity the plasmapause is pushed closer to the Earth. On the other hand, the plasma outflow from the ionosphere is enhanced, which contributes to the seed particles for the ring current and radiation belt acceleration processes. This is still a poorly understood complex of feed-back processes, which may lead to quite different storm characteristics under otherwise similar the solar wind driving conditions. The VAP results strongly emphasize the importance to reach the heart of the electron belt instead of just skimming the low-altitude end of the radiation belt L-shells. This said, the great value of the long-term statistics from polar orbiting low-altitude satellites, such as SAMPEX, deserves to be acknowledged (e.g., Baker and Kanekal 2008).

It will take time to digest all new VAP information. From a space weather viewpoint an important challenge is to be able to associate the local magnetospheric processes with the solar wind drivers. There is a general tendency, when focusing on the details of magnetospheric processes, to oversimplify the parametrization of the solar and interplanetary drivers. Similarly, when assessing the effects of the interplanetary drivers, the complexity of the magnetospheric response is all too often oversimplified. For example, in studies of ICMEs and their geoefficiency it is tempting to lump the magnetospheric response into one single index, often Dst, or at most into a few of the indices. On the other hand, when trying to understand observations inside the magnetosphere, correlations are often sought only with some particular solar wind parameter, e.g., velocity, dynamic pressure, or magnetic field, or some combination of these, e.g., the epsilon parameter of Akasofu (1981), without taking into account the intrinsic structure and temporal evolution of the driver. An attempt to consider the details of both the driver and the response was presented by Kilpua et al. (2015). They investigated the cause–effect relationship by carefully considering which part of the shock and ICME was driving either the growth or decay of the electron belt during different storms. This was a promising step towards solving the long-standing problem, stated by Reeves et al. (2003), why some storms lead to electron enhancements, some others to depletions.

Finally, the role of substorms in this context is worthy of comment. The term “substorm” is sometimes misleading. Although one or more substorms almost always take place during magnetospheric storms, they are not building blocks of the storms but represent an independent mode of magnetospheric activity. Most substorms occur outside of storm periods and often without any clear trigger in the solar wind parameters. The substorm process intensifies the magnetosphere–ionosphere interaction, strengthens the ionosphere’s role as the magnetospheric plasma source and produce seed population for high-energy electrons through wave–particle interactions. The actual role of substorm process in the electron content of the radiation belts is yet to be understood (for further discussion, see Baker et al., this issue).

2.6 Space Weather Effects in the Upper Atmosphere and Ionosphere

The main space weather effects in the upper atmosphere and ionosphere are mediated through the electromagnetic magnetosphere–ionosphere coupling and precipitation of energetic charged particles to the upper atmosphere. While the statistical patterns of ionospheric current systems and magnetic field-aligned currents (FAC) have been established long time ago (Iijima and Potemra 1976), in the space weather context the dynamic changes and often highly localised perturbations are more critical. Again, packaging the ionospheric response in a small number of activity indices is far from sufficient.

In the high-latitude ionosphere the most visible space weather phenomena are the auroral manifestations of magnetospheric storms and substorms. During this process in particular the Region 1 FAC system, connecting the auroral ionosphere to the outer magnetospheric boundary, enhances as does the westward ionospheric electrojet, which is indexed with the electrojet index AL. While AL as well as the eastward electrojet index AU are routinely calculated as the lower (AL) and upper (AU) envelope of the horizontal magnetic field component recordings at a limited number of magnetometer stations below the statistical auroral oval (see Mayaud 1980), more detailed information can be obtained from individual magnetometers when they happen to be located under the most interesting ionospheric activity. Here the longitudinal chains, e.g., IMAGE in the Fennoscandian sector (http://space.fmi.fi/image), and CARISMA in Canada (http://www.carisma.ca) have turned out to be very valuable tools.

An interesting fact about the AL index is that it has the strongest coupling to the solar wind during relatively weak solar activity (see McPherron et al., this issue). In case of very strong activity the magnetosphere does not seem to be capable to take all of the energy the solar wind is offering, which also shows in the saturation of the cross polar cap potential (for a review of a number of proposed explanations for the saturation, see Borovsky et al. 2009). A fresh view on the saturation issue was recently given by Pulkkinen et al. (2016). They analyzed magnetosheath observations of the five-spacecraft THEMIS mission jointly with auroral electrojets calculated from IMAGE magnetometer data and argued that at least a part of the saturation takes place already in the magnetosheath before the solar wind directly interacts with the magnetopause.

While auroral displays caused by electron and proton precipitation during magnetospheric storms and substorms can be considered as the most pleasant face of space weather, the beautiful phenomenon has also its dark side. The enhanced and temporally variable particle precipitation causes perturbations in the ion content of the ionosphere disturbing radio wave communication and satellite navigation systems. Intense high-energy particle fluxes also increase the radiation dose to aircraft crews and passengers, forcing lowering of the cruise altitude to get more atmospheric shielding.

Finally, energetic particles penetrating into the middle atmosphere have effects on the local ion chemistry. The atmospheric effects of strong SEP events and energetic electron precipitation (EEP) are roughly comparable. However, sufficiently strong SEP events are rare, as only a small fraction of all CME-driven shocks are able to accelerate protons to high enough energies to have significant effects in the middle atmosphere. On the other hand, sufficiently energized EEP from the outer radiation belt is a common storm-time phenomenon, in particular during substorm activity, and cause observable changes in atmospheric molecular composition (Andersson et al. 2014). For further discussion of atmospheric space weather matters, see Sojka (this issue).

2.7 Space Weather Effects on Earth

In addition to the communications and positioning systems that can be perturbed by space weather-related ionospheric irregularities and direct noise from strong solar radio bursts, the list of ground-based space weather sensitive technological systems is long, including long conductive transmission lines, gas pipe lines, railway signal systems, prospecting and deep drilling, etc.

A fundamental phenomenon behind the most important ground-based effects is the electromagnetic induction leading to disturbance currents in conductor systems, called geomagnetically induced currents (GIC). The basic physics behind GIC is elementary electrodynamics. Fast changes in ionospheric currents cause fast variations in the magnetic field on ground. These induce an electric field that drives electric currents in the conductive earth. The currents seek their way through the best conductors in the vicinity.

From observed magnetic field variations and accurate conductivity models of both ground and the conductor system in question post-event analysis of GIC events is straightforward. Also nowcasting close enough to a good magnetometer is not a problem, as the calculations do not require large computing power (e.g., Pirjola and Viljanen 1998). While the role of ICMEs as ultimate drivers of large GICs is well-established (Huttunen et al. 2008), the path from the upstream solar wind down to the Earth is long and complicated and a real challenge lies in the forecasting. Rapid enough temporal changes in the ionospheric currents are related to substorms and often highly localized. They cannot be reliably predicted using current forecasting tools.

2.8 Space Climate—Long-term Variations

The Sun is a changing star and the changes determine the long-term space climate under which space weather variations take place. During the 400 years since the time of Galileo the solar cycle as seen in the appearance of the sunspots has shown great variability both in the cycle length and in the peak and bottom sunspot numbers (for a recent review, see Hathaway 2015). While the statistical and physics-based sunspot number models have become increasingly sophisticated, forecasting of the cycle is yet a big challenge. The peak of cycle 23 was not as strong as some of the most advanced models suggested, and the following long (2008–2010) and low valley before cycle 24 was not predicted at all. Also the physical properties of the minimum phase solar wind were intriguing. Based on Ulysses observations McComas et al. (2008) found that during the minimum before cycle 24 the fast solar wind was significantly less dense, cooler and carried less mass and momentum flux than during the previous minimum. Furthermore, the open magnetic flux was reduced and, as discussed by Schwadron et al. (this issue), this tendency has continued through cycle 24.

Cycle 24 is now approaching its end. It is becoming, in terms of sunspot number, the weakest cycle since cycle 14 (for regularly updated sunspot records see, e.g., http://sidc.oma.be/silso/). Is the state of the solar dynamo undergoing some kind of long-term change resembling, e.g., the Maunder minimum (e.g. Eddy 1976), is this a signature of the Gleissberg (1939) cycle, or is it just a question of “statistical fluctuation”, remains to be seen. As discussed by Schwadron et al. (this issue), the decreasing solar activity is enhancing the galactic cosmic ray fluxes to the inner heliosphere leading to enhanced levels of radiation doses with consequent hazard risks both in space and at airline altitudes. If this trend continues, it will shorten the time humans can be safely exposed to galactic cosmic rays. This is, evidently, a serious concern for those who are planning for manned missions to Mars.

The terrestrial space climate environment is also affected by the slowly changing geomagnetic field. Currently the dipole moment is decaying 5–6% per century and both the north and south magnetic poles have been moving toward the northwest. As discussed by Mandea (in this issue and references therein) the migration of the north magnetic pole has been accelerating until recently, but is now slowing down. The geomagnetic field is weakest over a large area, called South Atlantic Anomaly (SAA), containing the east cost of South America and the southern part of Atlantic Ocean. Whereas the nominal dipole field at the Earth’s surface is about 30 μT at the dipole equator and 60 μT at the poles, the minimum field of SAA is only 22 μT close to Asuncion in Paraguay. The SAA evolves naturally with the decaying and moving dipole moment. In the past the field minimum has been moving in the direction from the southern Africa towards South America. As the magnetic field is weakening above the SAA, more energetic particles can penetrate to lower altitudes with increasing risks to low-altitude satellites and aircraft. Thus the continuous updating of geomagnetic and ionospheric field models is also critical for space weather (for further discussion, see Mandea, this issue).

2.9 Extreme Space Weather

The most severe space weather events are often, but not always, associated with very complex solar activity. A historically important sequence of events, illustrating wide variety of space weather phenomena both on the Sun and in the near-Earth space, took place during the second half of May 1967 (for a recent review, see Knipp et al. 2016, and references therein). A particularly active region moved over the visible solar disc during 17–31 May. It produced several tens of significant flares and, once the activity intensified on 21 May, three major solar radio bursts (SRB) occurred on 21, 23 and 28 May. The burst on the 21st was associated with a large white-light flare and strong X-ray emissions. On 23 May the magnetic complexity enhanced and another white-light flare with solar proton emission took place. The X-ray flare activity and the SRB continued for several hours. The flare was followed with a gradual SEP event peaking at 1 AU on 25 May, suggesting the passage of an ICME-driven shock. The subsequent magnetospheric storm on 25–26 May was one of the strongest in the history of magnetic observations, the Dst index reaching \(-387~\mathrm{nT}\). This event took place before the existence of CMEs was discovered and there is no direct evidence that the magnetospheric storm would have been driven by an ICME. The geomagnetic indicators and the observation that the dayside magnetopause was inside geostationary orbit, however, point to an ICME. For further analysis of this event and its wide reaching consequences in geospace we refer to Knipp et al. (2016).

While magnetic and magnetospheric storms are driven by shocks, ICMEs and fast solar wind, during extreme solar activity also the electromagnetic connection from the Sun, in form of X-ray emissions and SRBs, can have serious consequences to communication and satellite navigation systems long before the plasma connection is established. The X-ray-enhanced ionization causes perturbations to signal propagation in the ionosphere, whereas strong radio bursts introduce noise directly at the receivers if the energy flux in the used frequency band is large enough.

The strongest SRB event thus far recorded at GPS frequencies took place in December 2006 during a sequence of X-class flares. The estimated flux of the burst on 6 December was about \(10^{6}\) SFU (solar flux unit) at 1.4 GHz, which was an order of magnitude more than in other large events. The noise caused long-duration (about 30 min) reduction of the carrier-to-noise ratio up to 25 dB (Cerruti et al. 2008; Carrano et al. 2009). Another burst exceeding \(10^{5}\) SFU took place on 13 December. Loss of lock with GPS satellites was experienced for around 2 hours following the event and for some receivers a complete loss of navigation solution experienced for 6–10 minutes. A curious fact is that these events took place close to solar minimum. The quiet Sun may sometimes be more dangerous than the active.

Evidently, strong space weather events have wide societal consequences already today, and an extreme event of the Carrington class or stronger may cause far reaching hazards. However, forecasting such events is difficult, if not impossible. Riley (2012) estimated the likelihood of extreme events statistically from past events assuming power law distribution of the peak Dst values. He found that an event with \(\mathrm{Dst} <-850~\mbox{nT}\) has a 12% likelihood to occur within the next decade. For an event exceeding \(\mathrm{Dst} <-1700~\mbox{nT}\) the likelihood dropped to 1.5%. Assuming log-normal or exponential distributions the likelihoods are a bit lower. Clearly, events of this size are rare but not unlikely.

As discussed by Riley (2012) the Dst exceeded –600  nT only once during the 20th century. That happened on 13 March 1989 when the peak Dst was about −640 nT. This was the storm causing the famous collapse of Hydro-Québec power distribution system. If the 23 July 2012 CME would have hit the Earth, the storm would most likely have been much stronger. The transit time to 1 AU was about the same as in the Carrington event and, as discussed in Sect. 2.3 and the peak IMF as observed by STEREO was larger 100 nT. According to existing models, Dst exceeding \(-1000~\mbox{nT}\) would have been expected, e.g., the Dst model by Temerin and Li (2006) predicts \(-1200~\mbox{nT}\). Of course, the Dst models have not been calibrated to so large events.

According to standard magnetopause models based on pressure balance, the July 2012 event would have pushed the dayside magnetopause almost down to the ionosphere. Also this prediction needs to be considered with some care, as we do not really know the strength of induced currents to counteract the push. As noted in Sect. 2.6, the polar cap potential has a tendency to saturate during extreme driving, which is an indication of counteraction by induction. However, it is quite likely that the event could have wiped a significant fraction of plasma and energetic particles away from the magnetosphere.

How serious can the consequences of the worst imaginable space weather event really be? The honest answer is that we do not know. Claims have been voiced that the outcome of a super event could be almost an armageddon of modern civilization. While such claims may give publicity to space weather efforts, scientists should stay calm and concentrate on serious and unbiased investigations of the extreme events and their effects on critical systems. One should not cry wolf too often (Aesop, the 6th century BCE). For further discussion of extreme events, see Riley et al. (this issue).

3 Space Weather Beyond Solar-Terrestrial Physics

Even if we leave the design and maintenance issues of technological space weather sensitive systems outside of our discussion, there are some interesting and challenging topics that are not directly solar-terrestrial physics but where improved STP understanding is needed.

3.1 Space Weather Elsewhere

The details of interaction between solar eruptions, solar wind and ICMEs with other planets depend on distinct plasma and magnetic field conditions at each individual planet. While the basic physics of these interactions is known, very little is known of the details. If we really had to make reliable space weather forecasts at other planets, as may some day be the case with Mars, we would be in much deeper trouble than at the Earth.

Mercury, the planet closest to the Sun, has the most direct influence by the solar wind. The planet’s magnetic field is so weak that it does not require a particularly strong event to open the dayside magnetosphere for a direct access for the solar wind down to the surface. As the planet has no atmosphere, and thus no ionosphere, there is no shielding against solar energetic particles, even less to Galactic cosmic rays. The MESSENGER mission has brought a wealth of new observational information of the dynamics of the Hermean magnetosphere. Due to the small scales of the system everything is happening much faster than at the Earth. In response to three ICMEs events discussed by Slavin et al. (2014), Mercury’s already dynamic magnetosphere was found to be significantly deformed, with the subsolar distance to the magnetopause reduced from 1.45 R☿ to as little as 1.03–1.12 R☿. Thick, low-\(\beta \) plasma depletion layers were observed, caused by the impact of the ICMEs in the inner magnetosheath adjacent to the subsolar magnetopause, leading to the magnetic pressure being nearly equal on both sides of the magnetopause. This vastly increased the magnetic reconnection rate, which already during normal conditions is much larger at Mercury than at the Earth (DiBraccio et al. 2013). The reconnection process led to a major erosion of the field at the magnetopause, with expected dynamic effects throughout the magnetosphere.

Venus is an opposite case. Venus does not have a magnetic field but its atmosphere is dense and hot. The direct interaction between the solar wind with the ionosphere leads to continuous outflow of plasma to space but not much is known how space weather events modify the escape. Remarkable examples are the so-called Hot Flow Anomalies in the Venusian foreshock region, which are also known to take place in the foreshocks of the Earth, Mercury and Mars (Collinson et al. 2012, and references therein). Hot Flow Anomalies are driven by solar wind discontinuities and in case of Venus they are bigger than the planet and lead to strongly enhanced loss of plasma from the ionosphere of the planet.

Mars is probably the most interesting planet for space weather considerations. Given its complex and fragmented environmental make-up (e.g., Dubinin et al. 2008) the impact of solar wind drivers is correspondingly complex. The planet has a tenuous atmosphere, an ionosphere and some localized magnetic structures. On Mars, the lower and upper atmospheres are strongly coupled and loss of gas from the upper atmosphere to space is an important process that shapes the planetary atmosphere and its dynamics. It has been shown (Jakosky et al. 2015) that, in addition to the expected magnetospheric effects (compression of the bow shock and magnetosheath, formation of aurora), the escape rate of ions is also significantly enhanced during the passage of an ICME.

There are and will be technological systems on the surface of Mars, and recently a very ambitious private plan to send humans to Mars was announced by Elon Musk at the 67th International Astronautical Conference. Before this can become reality, serious space weather hazards during the transit to Mars need to be assessed and taken into account. The very weak shield against both solar and galactic cosmic rays provided by the Martian magnetosphere and atmosphere means that, once on the surface, human explorers will need continuous and reliable space weather services to survive.

Disruptions of the distant plasma tails of comets by ICMEs have been repeatedly observed remotely (e.g. Kuchar et al. 2008). A fortuitous observation by ESA’s Rosetta mission to comet 67P/Churyumov–Gerasimenko (Edberg et al. 2016) gave a chance to study a widespread disruption in the near-environment of an active comet caused by an impacting ICME.

The giant gas planets have very strong magnetospheres and thick atmospheres. Consequently their plasma dynamics is much more internally controlled than at the Earth. As discussed in Sect. 2.3, fast ICMEs slow down toward the ambient solar wind speed while propagating through the solar system. This together with the shocks being dissipative weakens the ICME effects at outer planets. Space weather processes are still important as illustrated, e.g., by the Jovian auroras. An interesting group of celestial bodies, again with greatly variable characteristics, are the big moons of Jupiter and Saturn, which move both inside and outside of the planets’ magnetospheres. Thus far we have very little observational facts about space weather phenomena there.

The Sun is only one of the more than 100 billion stars in the Milky Way. Many of them are magnetically active, not too different from the Sun, and the biggest sunspots can be identified in the observed light-curves. There are no direct observations of stellar winds yet but there is no reason to doubt their existence. Stellar flares are easier to observe and many magnetically active stars are known to have continuous flare activity similar to our Sun. Present day observations are limited to X- and M-class flares. There are rare observations of 2–3 orders of magnitude larger flares than the largest observed in the Sun. If we again extend the distribution beyond observations, it is possible that such flares could happen in the Sun once in a few thousand years.

3.2 Anthropogenic Space Weather

Although space weather is ultimately driven by solar activity, humans can and have caused space weather events and modified the environment where space weather phenomena take place. The high-altitude nuclear weapon tests 1958–1962 introduced significant new trapped particle populations that remained observable for several years. In particular the 1.4-megaton Starfish Prime in July 1962 at an altitude of 400 km resulted in an artificial radiation belt that has been claimed to have destroyed three satellites and caused problems to at least three more soon after the detonation. In 1962 this was a large fraction of all satellites in orbit.

Another man-made effect is thought to be related to the question: “Why is there a slot region between outer and inner belt electrons?” Radio waves from ground-based VLF transmitters have been proposed to scatter the electrons into the atmospheric loss cone and clean up the region. The recent VAP data supports this interpretation indicating that the VLF waves really form an almost impenetrable barrier against the outer belt electrons at \(L = 2.8\) (Foster et al. 2016). For further discussion of anthropogenic space weather issues, see Gombosi et al. (this issue).

3.3 Space Weather in the Public Eye

Regardless of the risks that motivate most space weather activities, we must not forget the beautiful face of space weather. Time after time it is equally fascinating to be out in the silent arctic winter night watching a major auroral event and see the entire sky full of moving colourful displays. Although we know that the auroras are at 100-km altitude, the impression that they sweep the nearby hilltops is unavoidable. The feeling is simply unforgettable.

Because space weather is visible and beautiful, it has great educational potential for physics students and for wider public as well. It is remarkably easy to make a well-received public presentation of space weather. One just needs to collect a few breathtaking pictures and movies of solar eruptions at one end and of auroras at the other, and prepare to tell a few facts that have been discussed above. This works particularly well in those parts of the world where people have seen real auroras with their own eyes. And the number of these people is increasing also elsewhere due to the fairly intense auroral tourism, which is a growing but thus far underexploited form of business.

4 Concluding Remarks

Since the advent of in situ observations in the solar wind and the magnetosphere impressive progress in understanding the physics behind space weather phenomena has taken place. We know what the most relevant solar phenomena driving space weather are, and what their consequences in the near-Earth space can be, but not always are. In fact, our detailed knowledge and modelling capabilities are far from sufficient for reliable forecasting of hazardous space weather conditions in space or on ground. There is both room and need for further research, including new observations, advanced data analysis, and numerical modelling.

Observations of space weather in space, whether remotely of solar processes or in situ of charged particles and electromagnetic fields, are technologically and economically demanding. A particular concern is that the observations are often difficult to repeat, which is generally considered as one of the backbones of the scientific method. New observations and re-analysis of old data continuously modify our understanding of the scientific foundations of space weather. The recent VAP observations provide great illustrations of the need for fresh data with improved instrumentation.

As it is not always possible to directly probe the physical processes, at least not continuously, various proxies are needed, such as empirical estimates of energy input from the solar wind to the magnetosphere, based on known energy output mechanisms and theoretical ideas of the input processes. To obtain the proxies statistical analyses are called for. Too often the “statistics” in this field is based on way too few samples, in particular when dealing with rare space weather events, which in turn are the most important from the practical point of view. An issue are the complicated interdependencies of observable phenomena, which may or may not have causal connections, e.g., SPEs vs. CMEs, SPEs vs. solar flares, or CMEs vs. solar flares. More sophisticated statistical tools than simple linear correlation analyses are definitely called for.

In the field of numerical modelling and forecasting the space weather modellers are nowhere near their colleagues in the atmospheric weather business. There is much room for improved numerical modelling and simulations throughout the road from the Sun to the ionosphere. Some of the challenges are really serious, as physical processes often have their roots in the spatial and temporal scales of electron motion while the modellers are still struggling to include ion kinetic effects in the fluid models. Also the vast range of relevant physical parameters is a serious issue, in particular from the solar surface to the corona where much of space weather originates.

The challenges to improve practical space weather services cannot be met using more efficient computational methods and tools alone. Deep understanding of the underlying physics and innovative ideas to improve the understanding remain basic requirements of all progress in space weather activities. The decisive role of temporally and spatially sufficiently wide observational input for both theoretical and practical aspects of space weather must always be kept in mind.