Error covariance tuning in variational data assimilation: application to an operating hydrological model

Abstract

Because the true state of complex physical systems is out of reach for real-world data assimilation problems, error covariances are uncertain and their specification remains very challenging. These error covariances are crucial ingredients for the proper use of data assimilation methods and for an effective quantification of the a posteriori errors of the state estimation. Therefore, the estimation of these covariances often involves at first a chosen specification of the matrices, followed by an adaptive tuning to correct their initial structure. In this paper, we propose a flexible combination of existing covariance tuning algorithms, including both online and offline procedures. These algorithms are applied in a specific order such that the required assumption of current tuning algorithms are fulfilled, at least partially, by the application of the ones at the previous steps. We use our procedure to tackle the problem of a multivariate and spatially-distributed hydrological model based on a precipitation-flow simulator with real industrial data. The efficiency of different algorithmic schemes is compared using real data with both quantitative and qualitative analysis. Numerical results show that these proposed algorithmic schemes improve significantly short-range flow forecast. Among the several tuning methods tested, recently developed CUTE and PUB algorithms are in the lead both in terms of history matching and forecast.

Appendix: Convergence of D05 iterative method

1.1 Justification of convergence in the ideal case

The convergence of D05 iterative method is proved by Bathmann (2018) in the ideal case, i.e., when the expectation in Eq. 14 is error-free and the current iterative matrix $\mathbf{R }_{n}$ stays always invertible.

Following the notation of Bathmann (2018), let

$$\begin{aligned} \mathbf{G }&= \mathbf{H }\mathbf{B }\mathbf{H }^T \end{aligned}$$

(36)

$$\begin{aligned} \mathbf{D }&= \mathbf{H }\mathbf{B }\mathbf{H }^T + \mathbf{R }_\text {E}, \end{aligned}$$

(37)

respectively denote the projection of the background matrix in the observation space and its sum with the exact observation matrix $\mathbf{R }_\text {E}$. Therefore, we have necessarily

$$\begin{aligned} \mathbf{D }-\mathbf{G }= \mathbf{R }_\text {E}. \end{aligned}$$

(38)

According to Bathmann (2018), the updating formulation of Eq. 14 is equivalent to:

$$\begin{aligned} \mathbf{R }_{n+1} = \mathbf{R }_{n} (\mathbf{G }+\mathbf{R }_{n})^{-1} \mathbf{D }. \end{aligned}$$

(39)

where n is the current iteration. It is obvious that the exact observation matrix $\mathbf{R }_\text {E}$ is a fixed point of Eq. 39. In fact, when $\mathbf{R }_\text {E}$ is SPD, the iterative process of Eq. 39 converges necessarily to the exact covariance $\mathbf{R }_\text {E}$. The interested readers are referred to Bathmann (2018) and Ménard (2016). We describe briefly their algebraic proof based on the two following lemmas.

Lemma 1

If $\mathbf{D }$ and $\mathbf{G }$ is SPD and $\mathbf{D }-\mathbf{G }$ is also SPD, then $\lambda _{max} (\mathbf{D }^{-1}\mathbf{G }) < 1$. Otherwise $\lambda _{max} (\mathbf{D }^{-1}\mathbf{G }) \ge 1$.

Lemma 2

For the matrix sequence ${\mathbf{R }_n}$ defined in Eq. 39, let $\mathbf{M }= \mathbf{D }^{-1} \mathbf{G }$, then

$$\begin{aligned} \mathbf{R }^{-1}_{n} = \mathbf{R }^{-1}_\text {E} + \mathbf{M }^n [\mathbf{R }^{-1}_{0} - \mathbf{R }^{-1}_\text {E}] \end{aligned}$$

(40)

The convergence could thus be derived as shown in Theorem 1.

Theorem 1

If the fixed point $\mathbf{R }_\text {E} = \mathbf{D }- \mathbf{G }$ is SPD then the D05 iterations converge to $\mathbf{R }_\text {E}$. Otherwise the iterations diverge to a singular matrix.

Proof by Bathmann: If $\mathbf{R }_\text {E}$ is SPD. By Lemma 2.

$$\begin{aligned} \mathbf{R }^{-1}_{n} = \mathbf{R }^{-1}_\text {E} + \mathbf{M }^n [\mathbf{R }^{-1}_{0} - \mathbf{R }^{-1}_\text {E}] \end{aligned}$$

(41)

where $\mathbf{M }= \mathbf{D }^{-1} \mathbf{G }$. As $\lambda _{max} (\mathbf{M }) < 1$, by Lemma 1., $\mathbf{M }^n \longrightarrow 0$, thus $\mathbf{R }^{-1}_{A,n} \longrightarrow \mathbf{R }^{-1}_{E}$ since they are both non-singular. If $\mathbf{R }_\text {E}$ is not SPD, then $\lambda _{max} (\mathbf{M }) \ge 1$, thus $\mathbf{M }^n$ diverges. We can deduce that $||\mathbf{R }^{-1}_{n}|| \longrightarrow \infty$ and therefore $\{ \mathbf{R }_n \}$ diverges to a singular matrix.

The case when $\mathbf{B }$ matrix is incorrectly specified is discussed in Ménard (2016), where it is proved that $\mathbf{R }_{n}$ will become rank deficient if the eigenvalues of $\mathbf{B }$ are overestimated.

1.2 Necessary regularization

As pointed out by Bathmann (2018), the Eq. 39 could not ensure the symmetricity of the updated matrix $\mathbf{R }_{n+1}$. An operation to enforce the symmetricity is necessary which leads the Eq. 39 to:

$$\begin{aligned} \mathbf{R }_{n+1} = \frac{1}{2}(\mathbf{R }_{n} + \mathbf{R }_{n}^T) \Big (\mathbf{G }+\frac{1}{2}(\mathbf{R }_{n} + \mathbf{R }_{n}^T) \Big )^{-1} \mathbf{D }. \end{aligned}$$

(42)

Being discussed in Bathmann (2018), the study of the convergence of Eq. 42 remains, for instance, an open question. It is mentioned in Bathmann (2018) and Ménard (2016) that an extra-regularization, e.g. via a hybrid method, is also needed to ensure that all the eigenvalues to be strictly positive.

1.3 Limitations of Desroziers method

1.3.1 Non-convergence of regularized matrix sequence

We found that unlike Eq. 39, the regularized sequence of Eq. 42 could have another fixed, different from the true observation covariance (i.e. $\mathbf{R }_\text {E} = \mathbf{D }-\mathbf{G }$). The proof is given by a counter-example:

$$\begin{aligned} \mathbf{G }= \begin{bmatrix} 1.5 &{} 1 \\ 1 &{} 4 \end{bmatrix}, \mathbf{D }= \begin{bmatrix} 3 &{} 2 \\ 2 &{} 3 \end{bmatrix}, \mathbf{R }= \begin{bmatrix} 1 &{} 1 \\ 2 &{} 1 \end{bmatrix}, \end{aligned}$$

(43)

which satisfies

$$\begin{aligned} \mathbf{R }= \frac{1}{2}(\mathbf{R }+ \mathbf{R }^T) \Big (\mathbf{G }+\frac{1}{2}(\mathbf{R }+ \mathbf{R }^T) \Big )^{-1} \mathbf{D }, \end{aligned}$$

(44)

and $\mathbf{R }$ is not SPD. Therefore, the proof of Bathmann (2018) is no longer valid for regularized matrix sequence as the equation has other fixed points other than $\mathbf{R }_\text {E}$.

1.3.2 Negative eigenvalues

The appearance of negative eigenvalues is known as an important challenge of D05 iterative methods (see Bathmann 2018; Ménard 2016). In this work, we found that the assumption of symmetric positiveness of $\mathbf{R }_{n}$ (for all n) should be added in the proof of Sect. 1 otherwise this proof could be completely misleading. In fact, when the $\mathbf{R }$ matrix possess negative eigenvalues, the term $\frac{1}{2}(\mathbf{y }-{\mathcal {H}}(\mathbf{x }))^T \mathbf{R }^{-1} (\mathbf{y }-{\mathcal {H}}(\mathbf{x }))$ which no longer represents a real norm, could have negative values, leading to a different expression of the analyzed state $\mathbf{x }_a$. As consequence, the Desroziers diagnosis formulation, which is established via a “BLUE” type resolution, is no longer valid. This effect is illustrated with a simple 2D example:

$$\begin{aligned} \mathbf{x }_b = \begin{bmatrix} 0 \\ 0 \end{bmatrix}, \mathbf{B }= \mathbf{I }_{2,2}, \mathbf{H }= \begin{bmatrix} 1 &{} 0 \\ 1 &{} 1 \end{bmatrix}, \mathbf{R }= \begin{bmatrix} 1 &{} 0 \\ 0 &{} -1 \end{bmatrix}, \mathbf{y }= \begin{bmatrix} 0 \\ 0 \end{bmatrix}. \end{aligned}$$

(45)

$$\begin{aligned} \text {let} \quad \mathbf{x }= \begin{bmatrix} \mathbf{x }_1 \\ \mathbf{x }_2 \end{bmatrix}, \quad \text {thus} \quad {\mathcal {J}}(x) = \mathbf{x }^2_1-2 \mathbf{x }_1 \mathbf{x }_2. \end{aligned}$$

(46)

It is obvious that the objective function ${\mathcal {J}}$ does not process a minimum in ${\mathbb {R}}/{\infty }$. However, the Desroziers diagnosis take into account the BLUE formulation, which writes as

$$\begin{aligned} \mathbf{x }_a = \mathbf{x }_b+\mathbf{K }(\mathbf{y }-\mathbf{H } \mathbf{x }_b) = \begin{bmatrix} 0 \\ 0 \end{bmatrix}. \end{aligned}$$

(47)

Therefore, because of this incoherence emerged by the non-symmetricity of $\mathbf{R }_n$, although the mathematical proof in Sect. 1 is without fault, the application of Desroziers iterative method may probably not lead to the true observation covariance even in the ideal case described in Bathmann (2018).