DADA: data assimilation for the detection and attribution of weather and climate-related events

Abstract

We describe a new approach that allows for systematic causal attribution of weather and climate-related events, in near-real time. The method is designed so as to facilitate its implementation at meteorological centers by relying on data and methods that are routinely available when numerically forecasting the weather. We thus show that causal attribution can be obtained as a by-product of data assimilation procedures run on a daily basis to update numerical weather prediction (NWP) models with new atmospheric observations; hence, the proposed methodology can take advantage of the powerful computational and observational capacity of weather forecasting centers. We explain the theoretical rationale of this approach and sketch the most prominent features of a “data assimilation–based detection and attribution” (DADA) procedure. The proposal is illustrated in the context of the classical three-variable Lorenz model with additional forcing. The paper concludes by raising several theoretical and practical questions that need to be addressed to make the proposal operational within NWP centers.

Appendices

Appendix A Illustration of the computational benefit of the DADA approach

To illustrate the computational benefit, let Y be for instance a d-variate autoregressive process defined by Y _t+1 = A Y _t +w _t , where w _t is an i.i.d. noise having known PDF g(⋅) and where A has the usual properties that insure stationarity (Gardiner 2004). We then have:

$$\begin{array}{@{}rcl@{}} && f(\mathbf{y}) = \prod\limits_{t=1}^{T} g(\mathbf{y}_{t}-\mathbf{A}\mathbf{y}_{t-1})\pi(\mathbf{y}_{0})\,, \end{array} $$

(7a)

$$\begin{array}{@{}rcl@{}} && P(\phi(\mathbf{Y})\geq u) = {\int}_{\phi(\mathbf{y})\geq u} \prod\limits_{t=1}^{T} g(\mathbf{y}_{t}-\mathbf{A}\mathbf{y}_{t-1}) \pi(\mathbf{y}_{0}) \mathrm{d}y_{1,0} {\ldots} \mathrm{d}y_{d,0} {\ldots} \mathrm{d}y_{d,T}\,, \end{array} $$

(7b)

with π(⋅) the prior PDF on the initial state Y ₀. Equation 7a shows that f(y) can be easily computed using a closed-form expression, while P(ϕ(Y)≥u) in Eq. 7b is an integral on d×T+1 dimensions which must instead be evaluated by using, for instance, a computationally quite costly Monte-Carlo (MC) simulation.

Appendix B Data Assimilation

The state-estimation problem for the system given by Eq. 4a and 4b has an exact solution given by the following sequential Kalman filter (KF) equations:

$$\begin{array}{@{}rcl@{}} {\mathbf{x}^{a}_{t}} &=& {\mathbf{x}^{f}_{t}} +\mathbf K(\mathbf{y}_{t}-\mathbf H {\mathbf{x}^{f}_{t}}) \,, \end{array} $$

(8a)

$$\begin{array}{@{}rcl@{}} \mathbf{P^{a}_{t}} &=& (\mathbf I -\mathbf K\mathbf H) \mathbf{P^{f}_{t}} \,, \end{array} $$

(8b)

$$\begin{array}{@{}rcl@{}} \mathbf{x}^{f}_{t+1} &=& \mathbf M {\mathbf{x}^{a}_{t}} \,, \end{array} $$

(8c)

$$\begin{array}{@{}rcl@{}} \mathbf P^{f}_{t+1} &=& \mathbf M \mathbf{P^{a}_{t}}\mathbf M^{\prime} + \mathbf Q \,. \end{array} $$

(8d)

where $^{\prime }$ denotes the transpose operation. Here Eq. 8a and 8b are referred to as the analysis step and denoted by a superscript a, while the forecast step is given by Eq. 8c and 8d, and is denoted by a superscript f (Ide et al. 1997). The vector ${\mathbf {x}^{a}_{t}}$ and the matrix $\mathbf {P^{a}_{t}}$ are the mean and covariance of X _t conditional on (Y ₁,...,Y _t )=(y ₁,...,y _t ); $\mathbf K=\mathbf {P^{f}_{t}}\mathbf H^{\prime }(\mathbf H\mathbf {P^{f}_{t}}\mathbf H^{\prime }+\mathbf R)^{-1}$ is the so-called Kalman gain matrix; while Q and R are the covariances associated with v _t and w _t , respectively. Following (Wiener 1949), one distinguishes between filtering, in which ${\mathbf {x}^{a}_{t}}$ and $\mathbf {P^{a}_{t}}$ are conditioned only on the previous and current observations (y ₀,...,y _t ), and smoothing, in which they are conditioned on the entire sequence, 0≤t≤T. Furthermore, the sequential algorithm needs to be initialized at time t=0 with ${\mathbf {x}_{0}^{f}}$ and $\mathbf {P}_{0}^{f}$, which thus represent the a priori mean and covariance of X ₀, respectively, and have to be prescribed by the user.

Appendix C Derivation of the model evidence

In this appendix, we outline the derivation of model evidence within a general Bayesian framework, and we apply the latter to the narrower KF context to obtain Eq. 5. Consider two consecutive cycles of a DA run, the first with state vector x _t and observation vector y _t at instant t and the subsequent one with state vector x _t+1 and observation vector y _t+1 at instant t+1. We plan to find a tractable expression for the model evidence p(y _t ,y _t+1).

The model evidence provided by the full sequence of observations y=(y ₀,...,y _T ) will be inferred by recursion, using the results of this two-observation setting. In order to decouple the two cycles, one first has to spell out the Bayesian inference p(y _t ,y _t+1) = p(y _t )p(y _t+1|y _t ). We look for a tractable expression for p(y _t+1|y _t ) by further introducing the states x _t+1 and x _t as intermediate random variables:

$$\begin{array}{@{}rcl@{}} \begin{array}{lll} p(\mathbf{y}_{t+1} | \mathbf{y}_{t}) &=& {\int}_{\mathbf{x}_{t+1}} \! p(\mathbf{y}_{t+1} | \mathbf{y}_{t}, \mathbf{x}_{t+1}) p(\mathbf{x}_{t+1} | \mathbf{y}_{t})\,\mathrm{d}\mathbf{x}_{t+1} \\ & = & {\int}_{\mathbf{x}_{t+1}} \! p(\mathbf{y}_{t+1} | \mathbf{x}_{t+1}) \left\{{\int}_{\mathbf{x}_{t}} \! p(\mathbf{x}_{t+1} | \mathbf{x}_{t})\, p(\mathbf{x}_{t} | \mathbf{y}_{t})\,\mathrm{d}\mathbf{x}_{t}\right\} \mathrm{d}\mathbf{x}_{t+1}\,, \end{array} \end{array} $$

(9)

where p(y _t+1|x _t+1) is the likelihood of the observation vector y _t+1 conditional on the state vector x _t+1 and it is known from Eq. 4b. The conditional PDF p(x _t |y _t ) of x _t on y _t at instant t — which appears on the right-hand side of the above equation — is referred to as the analysis PDF in the DA literature, where it is denoted by a superscript a (Ide et al. 1997), and it constitutes the main DA output. The integral ${\int }_{\mathbf {x}_{t}} \! p(\mathbf {x}_{t+1} | \mathbf {x}_{t}) p(\mathbf {x}_{t} | \mathbf {y}_{t})\,\mathrm {d}\mathbf {x}_{t}=p(\mathbf {x}_{t+1} | \mathbf {y}_{t})$, in which p(x _t+1|x _t ) is known from the model dynamics given by Eq. 4a, propagates this analysis PDF further in time, to instant t+1. Hence, the result of this integration coincides with the forecast PDF, denoted by superscript f in the DA literature (Ide et al. 1997). It follows that this decomposition is tractable using a DA scheme that is able to estimate the conditional and forecast PDFs.

Next, let us apply the general Bayesian inference (9) to the case in which all the PDFs involved are Gaussian; this requires, in turn, that both the dynamics and observation models M and H be linear, and that the input statistics all be Gaussian. In this case, the Kalman filter allows for the exact computation of the PDFs mentioned in Eq. 9, which turn out to be Gaussian.

In the following, ${\mathcal {N}}(\overline {\mathbf {x}},\mathbf {P})$ designates the Gaussian PDF of mean $\overline {\mathbf {x}}$ and covariance matrix P. In this context, the analysis PDF at instant t is ${\mathcal {N}}({\mathbf {x}^{a}_{t}},{\mathbf {P}^{a}_{t}})$, where ${\mathbf {x}^{a}_{t}}$ and ${\mathbf {P}^{a}_{t}}$ are the analysis state and error covariance matrix at instant t. As a result of the linearity assumptions, the forecast PDF at instant t+1 is given by a Gaussian distribution ${\mathcal {N}}(\mathbf {x}^{f}_{t+1},\mathbf {P}^{f}_{t+1})$, where $\mathbf {x}^{f}_{t+1}$ and $\mathbf {P}^{f}_{t+1}$ are the forecast state and error covariance matrix at instant t+1. Further, the integration on x _t+1 in Eq. 9 can readily be performed under these circumstances, with the outcome that p(y _t+1|y _t ) is distributed as ${\mathcal {N}}(\mathbf {H}\,\mathbf {x}^{f}_{t+1},\mathbf {R}+\mathbf {H}\,\mathbf {P}^{f}_{t+1}\,\mathbf {H}^{\prime })$.

The desired model evidence f(y) can then be computed by recursion on successive time steps as:

$$\begin{array}{@{}rcl@{}} f(\mathbf{y}) = p(\mathbf{y}_{0})\,\prod\limits_{t=1}^{T} (2\pi)^{-\frac{d}{2}}|\boldsymbol{\Sigma}_{t}|^{-\frac{1}{2}} \exp \left\{ -\frac{1}{2}(\mathbf{y}_{t}-{\mathbf{H}\mathbf{x}^{f}_{t}})'\boldsymbol{\Sigma}_{t}^{-1}(\mathbf{y}_{t}-{\mathbf{H}\mathbf{x}^{f}_{t}}) \right\}\,; \end{array} $$

(10)

here p(y ₀) represents the prior PDF of the initial state, $\boldsymbol {\Sigma }_{t} = \mathbf {R}+{\mathbf {H}\mathbf {P}^{f}_{t}}\mathbf {H}^{\prime }$, and this expression coincides with Eq. 5 and can be evaluated with the help of any DA method that yields the forecast states and forecast error covariance matrices, such as the KF or the EnKF. Note that the traditional standard Kalman smoother would give the same result as the KF, since they share the same forecasts.

Finally, Eqs. 9 and 10 above show that the likelihood f(y) may be obtained as a by-product of the inference on the state vector x, which usually is the main purpose in numerical weather prediction. This idea may actually be highlighted in even greater generality by considering the equality:

$$ {f(\mathbf{y}) = \frac{p(\mathbf{y}\vert\mathbf{x}) p(\mathbf{x})}{p(\mathbf{x}\vert\mathbf{y})}\,.} $$

(11)

While Eq. 11 is a direct consequence of Bayes theorem, it also illustrates a point that is arguably not so intuitive. The likelihood f(y) is obtained here as the ratio of two quantities: a numerator p(y|x)p(x) that is a model premise inherently postulated by Eq. 4a and 4b, and a denominator p(x|y) that may be viewed as the end result of the primary inferxence on x. In other words, estimating f(y) requires only a straightforward division, provided x has been previously inferred.

Equation 11 thus expresses with great clarity and simplicity a fundamental idea buttressing our proposal, as it provides a general theoretical justification for the suggestion of deriving the likelihood from an inferential treatment that focuses on x. To put it succintly, this equation basically says, “He who can do more can do less.” In the context of DA, whose end purpose is to infer the state vector x out of an observation y — i.e., the more part — it is possible to obtain the likelihood as a by-product thereof — i.e., the less part — and thus almost for free.

Appendix D PDF of the state vector

We associate a label ω∈Ω with each realization of the random process v _t that drive the model given by Eq. 6. The PDF of the state vector x _t can be obtained as the numerical solution of the corresponding Fokker-Planck equation, and it is the mean over Ω of the sample measures obtained for each realization ω of the noise v _t and (Chekroun et al. 2011, and references therein). Each sample measure is supported on a random attractor that may have very fine structure and be time-dependent (Chekroun et al. 2011, Figs. 1–3 and supplementary material), but the PDF is supported smoothly, in the counterfactual world in which λ ₀=0, on a “thickened” version of the fairly well-known strange attractor of the original L63 model. The latter PDF represents its attractor in dynamic system’s terminology.