Adaptation of the optimal fingerprint method for climate change detection using a well-conditioned covariance matrix estimate

Abstract

The “optimal fingerprint” method, usually used for detection and attribution studies, requires to know, or, in practice, to estimate the covariance matrix of the internal climate variability. In this work, a new adaptation of the “optimal fingerprints” method is presented. The main goal is to allow the use of a covariance matrix estimate based on an observation dataset in which the number of years used for covariance estimation is close to the number of observed time series. Our adaptation is based on the use of a regularized estimate of the covariance matrix, that is well-conditioned, and asymptotically more precise, in the sense of the mean square error. This method is shown to be more powerful than the basic “guess pattern fingerprint”, and than the classical use of a pseudo-inverted truncation of the empirical covariance matrix. The construction of the detection test is achieved by using a bootstrap technique particularly well-suited to estimate the internal climate variability in real world observations. In order to validate the efficiency of the detection algorithm with climate data, the methodology presented here is first applied with pseudo-observations derived from transient regional climate change scenarios covering the 1960–2099 period. It is then used to perform a formal detection study of climate change over France, analyzing homogenized observed temperature series from 1900 to 2006. In this case, the estimation of the covariance matrix is only based on a part of the observation dataset. This new approach allows the confirmation and extension of previous results regarding the detection of an anthropogenic climate change signal over the country.

Appendices

Appendix 1

Ledoit estimate $\hat{C}_I$

The main concepts of Ledoit and Wolf (2004) to lead to the Ledoit regularized estimate ${\hat{C}}_I$ are here reviewed.

The first question addressed is the following: given the empirical estimate of a p × p covariance matrix, ${\hat{C}},$ deduced from a n × p sample X, can a linear combination of the form $\tilde{C} = \gamma \hat{C} + \rho I_p$ be a more precise estimate of C than $\hat{C}$ ? The answer is yes, in the sense of the mean square error:

$$ \hbox{E} \left(|\tilde{C}-C|_T^2 \right), $$

(21)

for the classical norm:

$$ |A|_T ^2=\frac{\hbox{Tr}(A^{\prime}A)}{{p}}. $$

(22)

When C is known, the optimal values γ _o and ρ _o of γ and ρ are given by Ledoit and Wolf (2004):

$$ \gamma_o=\frac{{\alpha^2}}{{\delta^2}}, $$

(23)

$$ \rho_o=\frac{{\beta^2 \nu}}{{\delta^2}}, $$

(24)

where

$$ \nu=\langle C,I_p \rangle_T=\frac{\hbox{Tr}(C)}{{p}}, $$

(25)

〈.,.〉 _T being the scalar product associated with the norm |.| _T , and:

$$ \alpha^2=| C - \nu I_p |_T^2, $$

(26)

$$ \beta^2=\hbox{E}\left(| \hat{C}-C |_T^2 \right), $$

(27)

$$ \delta^2=\hbox{E}\left(| \hat{C} - \nu I_p |_T^2 \right). $$

(28)

In the case when C is unknown, consistent estimators $\hat{\gamma}$ and $\hat{\rho}$ of those two optimal coefficients can be constructed, and they are shown to be convergent under general asymptotics hypothesis. Note that the “general asymptotics” framework, also called “large dimensional”, is larger than the classical asymptotics framework for statistics, and it especially covers the case n = p, and n,p →∞. More precisely, estimates of the coefficients ν,α,β and δ are first defined:

$$ \hat{\nu}=(\hat{C},I_p)=\frac{{\hbox{Tr}}(\hat{C})}{{p}}, $$

(29)

$$ \hat{\delta}^2=|\hat{C} - \hat{\nu} I_p |_T^2, $$

(30)

$$ \hat{\beta}^2=\hbox{min} \left(\hat{\delta}^2, \frac{{1}}{{n^2}} \sum_{i=1}^{n} | \psi_i \psi_i^{\prime} - \hat{C} |_T^2 \right), $$

(31)

$$ \hat{\alpha}^2=\hat{\delta}^2 - \hat{\beta}^2. $$

(32)

Then, the estimates of γ _o and ρ _o are deduced:

$$ \hat{\gamma}=\frac{{\hat{\alpha}^2}}{{\hat{\delta}^2}}, $$

(33)

$$ \hat{\rho}=\frac{{\hat{\beta}^2 \hat{\nu}}}{{\hat{\delta}^2}}. $$

(34)

These estimates lead to a new estimate of the covariance matrix, namely the Ledoit regularized estimate:

$$ \hat{C}_I=\hat{\gamma} \hat{C}+\hat{\rho} I_p. $$

(35)

Appendix 2

Bootstrap and validation implementation

4.1 Simulating ${{\mathcal{D}}}_{\hat{C}_I,g}$

The procedure implemented to simulate the distribution ${{\mathcal{D}}}_{\hat{C}_I,g}$ is presented here.

Step 0 From the observed n × p sample X, containing the observation vectors $(\psi _{i} )_{{1 \le i \le n}},$ the estimate $\hat{C}_I$ of C is computed.

Step 1 $\hat{C}_I$ is used to simulate a set of N samples $(X_{j}^{*} )_{{1 \le j \le N}},$ each one of size n × p, with a Monte Carlo technique: similarly to the matrix X, the lines of a X ^* _j matrix (equivalent to one of the ψ _i ) are independent random vectors, the distribution of which is a $N(0,\hat{C}_I).$ The X ^* _j are independent between them.

Step 2 From X ^* _j , an estimate $\hat{C}_j^*$ of $\hat{C}_I$ is computed, using the estimation method presented in Appendix 1.

Furthermore, an estimate $\hat{f}_j^* = {(\hat{C}_j^*)}^{-1} g$ of $\hat{f}_o$ is computed.

Step 3 For each j, the conditional distribution ${{\mathcal{D}}}_{\hat{C}_I,g}$ given X ^* _j is known:

$$ {\mathcal{D}}_{\hat{C}_I,g} |X_j^*=N \left(0,\frac{{\hat{f}_j^{*'} \hat{C}_I \hat{f}_j^*}}{{\hat{f}_j^{*'} \hat{C}_j^* \hat{f}_j^*}} \right), $$

(36)

so that the conditional probability density function associated can be deduced.

Now, the ${{\mathcal{D}}}_{\hat{C}_I,g}$ distribution being the mixture of the conditional distributions, its density is approximated by computing the mean of the N conditional densities simulated.

4.2 Validation

The question addressed here is: in what measure the ${{\mathcal{D}}}_{\hat{C}_I,g}$ -distribution can be used instead of the ${{\mathcal{D}}}_{C,g}$ one for computing the p-value of the test?

Using the distribution ${{\mathcal{D}}}_{\hat{C}_I,g},$ we can proposed a ROF test totally defined by the sample $(\psi _{i} )_{{1 \le i \le n}}.$ This test, denoted ${{\mathbb{T}}}_{\hat{C}_I^{-1} g}$ is characterized by its rejection region ${{\mathbb{W}}}_{\hat{C}_I^{-1}g}:$

$$ {{\mathbb{W}}}_{\hat{C}_I^{-1}g}=\left\{ (\psi_{i})_{1 \leq i \leq n+1}, \frac{{\langle \psi_{n+1}, \hat{C}_I^{-1} g \rangle}}{ {\sqrt{(\hat{C}_I^{-1} g)' \hat{C}_I (\hat{C}_I^{-1} g)}}} \geq {\mathcal{D}}_{\hat{C}_I,g}^{(\alpha)} \right\}, $$

(37)

where ${\mathcal{D}} _{\hat{C}_I,g}^{(\alpha)}$ denotes the (1−α) quantile of the distribution ${{\mathcal{D}}}_{\hat{C}_I,g},$ estimated from the $(\psi _{i} )_{{1 \le i \le n}}$ using the procedure presented in the previous paragraph. Note that we use here a specific notation ${{\mathbb{T}}}_{\hat{C}_I^{-1} g}$ , because this test couldn’t be defined by a rejection region of the form (6). So that ${{\mathbb{T}}}_{\hat{C}_I^{-1} g}$ isn’t part of the T _f family.

The validation proposed here focuses on the test level. To verify that the ${{\mathbb{T}}}_{\hat{C}_I^{-1} g}$ level is nominal, it is equivalent to verify that:

$$ \hbox{P}_{H_0} ({{\mathbb{W}}}_{\hat{C}_I^{-1}g})=\alpha. $$

(38)

The validation of this equality can be made similarly with the previous procedure.

Step 0 A “true” covariance matrix, C is fixed arbitrarily.

Step 1 C is used to simulate a set of N ₀ samples $(X_{k} )_{{1 \le k \le N_{0} }} ,$ similarly to the (X ^* _j ) of the previous paragraph: each (X _k ) is of dimension n × p, and its lines are independent random vectors, the distribution of which is a N(0,C). Note that X _k is equivalent to the unique set of available observations X. The regularized covariance estimate $\hat{C}_I^{(k)}$ is computed, for each k.

Step 2 Following the previously presented simulation procedure, the distribution ${{\mathcal{D}}}_{\hat{C}_I^{(k)},g}$ is generated. The candidate threshold ${{\mathcal{D}}}_{\hat{C}_I^{(k)},g}^{(\alpha)}$ is deduced.

Step 3 The conditional distribution ${{\mathcal{D}}}_{C,g}$ given X _k being known, the level and the power conditionally to X _k can be deduced. Finally, the unconditioned level and power are approximated, using a relatively large number of samples N ₀.

As mentioned in Sect. 3.3, this validation procedure has been successfully applied to different starting matrices C.

Appendix 3

Discussing the sensitivity to the choice of q

We first study the tests $T_{C_q^+g}$ . We denote respectively by λ ² _i , g _i and ψ _i , the eigenvalues of C, and the coordinates of g and ψ in the C eigenvectors basis. The detection variable associated with $T_{C_q^+g}$ can be written:

$$ d_{C_q^+ g} = \langle \psi,C_q^+ g \rangle=\sum_{i=1}^{q} \frac{{\psi_i g_i}}{{\lambda_i^2}}. $$

(39)

Note that $\hbox{Var} (\frac{{\psi_1}}{{\lambda_1}},\ldots,\frac{{\psi_p}}{{\lambda_p}})=I_p,$ that is to say that the $z_i=\frac{{\psi_i}}{{\lambda_i}}$ are random variables, independents and identically distributed, and, under the hypothesis H ₀, with N(0,1) distribution. So,

$$ d_{C_q^+ g}=\sum_{i=1}^{q} z_i \frac{{g_i}}{{\lambda_i}}, $$

(40)

is a random walk, with weights $\frac{{g_i}}{{\lambda_i}}$ .

This formula highlights the role of the family $(\frac{{g_i}}{ {\lambda_i}})$ , and shows that the results are generally “unstable” with respect to the choice of q, even when q is large. Indeed, some stability occurs only when the coefficients $(\frac{{g_i}}{{\lambda_i}})$ become “small”, as q goes to p. In such a case, the tests $T_{C_q^+g}$ tend to $T_{C^{-1} g}$ (and they are close to each other), so the choice of q doesn’t matter, as soon as q is large.

However, in the case of climate study, there isn’t any evidence that this condition on the $(\frac{{g_i}}{{\lambda_i}})$ family is satisfied. Consequently, under the hypothesis H ₀, the probability to reject H ₀ for at least one of the considered tests can become important (clearly greater than the individual level α of each test). This illustrate the potential danger of the method and of the choice of q.

A similar study can be made for the tests $T_{\hat{C}^+_q g}.$ Denoting respectively by $\hat{\lambda}_i^2,$ $\hat{g}_i$ and $\hat{\psi}_i$ the eigenvalues of $\hat{C},$ and the coordinates of g and ψ in the $\hat{C}$ eigenvectors basis,

$$ \hat{d}_{\hat{C}_q^+ g}=\langle \psi,\hat{C}_q^+ g \rangle=\sum_{i=1}^{q} \frac{{\hat{\psi}_i}}{{\hat{\lambda}_i}} \frac{{\hat{g}_i}}{{\hat{\lambda}_i}}. $$

(41)

Here, the main differences with the previous case is that, first, the random variables $\frac{{\hat{\psi}_i}}{{\hat{\lambda}_i}}$ are not exactly independent and identically distributed, and, second, the $\hat{\lambda}_i$ family decreases more rapidly than the λ _i one, due to estimation errors. Consequently to this last point, the sensitivity to the choice of q is increased.