The Area Skill Score Statistic for Evaluating Earthquake Predictability Experiments

Abstract

Rigorous predictability experimentation requires a statistical characterization of the performance metric used to evaluate the participating models. We explore the properties of the area skill score measure and consider issues related to experimental discretization. For the case of continuous alarm functions and continuous observations, we present exact analytical solutions that describe the distribution of the area skill score for unskilled predictors, and we also describe how a Gaussian distribution with known mean and variance can be used to approximate the area skill score distribution. We quantify the deviation of the exact distribution from the Gaussian approximation by specifying the kurtosis excess as a function of the number of observed target earthquakes. For numerical earthquake predictability experiments that involve discretization of the study region and observations, we explore simulation procedures for estimating the area skill score distribution, and we present efficient algorithms for various experimental scenarios. When more than one target earthquake occurs within a given space/time/magnitude bin, the probabilities of predicting individual events are not independent, and this requires special consideration. Having presented the statistical properties of the area skill score, we describe and illustrate a preliminary procedure for comparing earthquake prediction strategies based on alarm functions.

Appendices

Appendix 1: Expected Value of Molchan Trajectory Jump 〈τ _k 〉

We seek 〈τ _k 〉, the expectation of the kth Molchan trajectory jump of an unskilled alarm function, where expectation is defined as

$$ \left\langle X \right\rangle = \int\limits_{ - \infty }^{\infty } {xf\left( x \right)\,} dx. $$

(23)

Here f(x) is the probability density function; therefore, we need to find the probability density for τ _k .

We can find the probability density by taking the derivative of the cumulative density function. For τ ₁, this is the probability that the trajectory has experienced at least one jump prior to reaching τ. In other words, it is the probability of covering τ and obtaining 1, 2, 3,…, or N hits. This probability is given by summing binomial terms:

$$ D_{{\tau_{1} }} \left( \tau \right) = \sum\limits_{j = 1}^{N} {\left( {\begin{array}{*{20}c} N \\ j \\ \end{array} } \right)\tau^{j} \left( {1 - \tau } \right)^{N - j} } . $$

(24)

We can express (24) in the following closed form:

$$ D_{{\tau_{1} }} \left( \tau \right) = \left\{ {\begin{array}{*{20}c} {0,\;\tau < 0} \hfill \\ {1 - \left( {1 - \tau } \right)^{N} ,\;\tau \in \left[ {0,1} \right]} \hfill \\ {1,\;\tau > 1} \hfill \\ \end{array} } \right.. $$

(25)

By differentiating (25) with respect to τ, we obtain the probability density:

$$ p_{{\tau_{1} }} \left( \tau \right) = {\frac{{dD_{{\tau_{1} }} \left( \tau \right)}}{d\tau }} = \left\{ {\begin{array}{*{20}c} {N\left( {1 - \tau } \right)^{N - 1} ,\tau \in \left[ {0,1} \right]} \hfill \\ {\begin{array}{*{20}c} 0 & {\text{otherwise}} \\ \end{array} } \hfill \\ \end{array} } \right.. $$

(26)

Now we can substitute (26) into (23) to obtain the expectation, changing the limits of integration to isolate the region where the probability density is nonzero:

$$ \begin{aligned} \left\langle {\tau_{1} } \right\rangle & = \int\limits_{0}^{1} {\tau p_{{\tau_{1} }} \left( \tau \right)d\tau } = \int\limits_{0}^{1} {\tau N\left( {1 - \tau } \right)^{N - 1} d\tau } \\ & = \left. { - {\frac{{\left( {1 - \tau } \right)^{N} \left( {N\tau + 1} \right)}}{N + 1}}} \right|_{0}^{1} \\ & = {\frac{1}{N + 1}}. \\ \end{aligned} $$

(27)

This shows that the first hit is expected to be obtained by unskilled alarm functions when they cover ${\tfrac{1}{N + 1}}$ of the study region. Similarly, we can express the c.d.f., p.d.f., and expectation for the next jump:

$$ D_{{\tau_{2} }} \left( \tau \right) = 1 - \left( {1 - \tau } \right)^{N} - N\tau \left( {1 - \tau } \right)^{N - 1} , $$

(28)

$$ \begin{aligned} p_{{\tau_{2} }} \left( \tau \right) & = N\left( {1 - \tau } \right)^{N - 1} - \left[ { - N\tau \left( {N - 1} \right)\left( {1 - \tau } \right)^{N - 2} + N\left( {1 - \tau } \right)^{N - 1} } \right] \\ & = N\left( {N - 1} \right)\tau \left( {1 - \tau } \right)^{N - 2} , \\ \end{aligned} $$

(29)

$$ \begin{aligned} \left\langle {\tau_{2} } \right\rangle & = \int\limits_{0}^{1} {N\left( {N - 1} \right)\tau^{2} \left( {1 - \tau } \right)^{N - 2} d\tau } \\ & = \left. { - {\frac{{\left( {1 - \tau } \right)^{N - 1} \left( {\left( {N - 1} \right)\tau \left( {N\tau + 2} \right) + 2} \right)}}{N + 1}}} \right|_{0}^{1} \\ & = {\frac{2}{N + 1}}. \\ \end{aligned} $$

(30)

These equations show that, on average, unskilled alarm functions obtain two hits once they have covered ${\tfrac{2}{N + 1}}$ of the study region. Likewise for the following jump, we have

$$ D_{{\tau_{3} }} \left( \tau \right) = 1 - \left( {1 - \tau } \right)^{N} - N\tau \left( {1 - \tau } \right)^{N - 1} - \left( {\begin{array}{*{20}c} N \\ 2 \\ \end{array} } \right)\left( \tau \right)^{2} \left( {1 - \tau } \right)^{N - 2} , $$

(31)

$$ \begin{aligned} p_{{\tau_{3} }} \left( \tau \right) & = N\left( {1 - \tau } \right)^{N - 1} - \left[ { - N\left( {N - 1} \right)\tau \left( {1 - \tau } \right)^{N - 2} + N\left( {1 - \tau } \right)^{N - 1} } \right] \\ & \quad - \left[ { - \left( {\begin{array}{*{20}c} N \\ 2 \\ \end{array} } \right)\left( {N - 2} \right)\left( \tau \right)^{2} \left( {1 - \tau } \right)^{N - 3} + 2\left( {\begin{array}{*{20}c} N \\ 2 \\ \end{array} } \right)\tau \left( {1 - \tau } \right)^{N - 2} } \right] \\ & = N\left( {N - 1} \right)\tau \left( {1 - \tau } \right)^{N - 2} + \left( {\begin{array}{*{20}c} N \\ 2 \\ \end{array} } \right)\left( {N - 2} \right)\left( \tau \right)^{2} \left( {1 - \tau } \right)^{N - 3} - N\left( {N - 1} \right)\tau \left( {1 - \tau } \right)^{N - 2} \\ & = {\frac{{N\left( {N - 1} \right)\left( {N - 2} \right)}}{2}}\left( \tau \right)^{2} \left( {1 - \tau } \right)^{N - 3} , \\ \end{aligned} $$

(32)

$$ \begin{aligned} \left\langle {\tau_{3} } \right\rangle & = {\frac{{N\left( {N - 1} \right)\left( {N - 2} \right)}}{2}}\int\limits_{0}^{1} {\left( \tau \right)^{3} \left( {1 - \tau } \right)^{N - 3} d\tau } \\ & = \left. {{\frac{{N\left( {N - 1} \right)\left( {N - 2} \right)}}{2}}\left( {1 - \tau } \right)^{N} \left( { - {\frac{2N + 3}{{N\left( {N + 1} \right)}}} - {\frac{\tau }{N + 1}} - {\frac{3}{{\left( {N - 1} \right)\left( {\tau - 1} \right)}}} + {\frac{1}{{\left( {2 - N} \right)\left( {\tau - 1} \right)^{2} }}}} \right)} \right|_{0}^{1} \\ & = {\frac{{N\left( {N - 1} \right)\left( {N - 2} \right)}}{2}}\left( {{\frac{2N + 3}{{N\left( {N + 1} \right)}}} - {\frac{3}{{\left( {N - 1} \right)}}} - {\frac{1}{{\left( {2 - N} \right)}}}} \right) \Rightarrow \\ \end{aligned} $$

$$ \left\langle {\tau_{3} } \right\rangle = {\frac{3}{N + 1}}. $$

(33)

For an inductive proof, we assume that $\left\langle {\tau_{N - 1} } \right\rangle = {\frac{N - 1}{N + 1}}$ and compute 〈τ _N 〉. At this point, we can get a compact expression for the c.d.f. by returning to the original formulation in (24), such that

$$ \begin{aligned} D_{{\tau_{N} }} \left( \tau \right) & = \sum\limits_{j = N}^{N} {\left( {\begin{array}{*{20}c} N \\ j \\ \end{array} } \right)\tau^{j} \left( {1 - \tau } \right)^{N - j} } \\ & = \left( \tau \right)^{N} . \\ \end{aligned} $$

(34)

Then,

$$ p_{{\tau_{N} }} \left( \tau \right) = N\left( \tau \right)^{N - 1} , $$

(35)

$$ \begin{aligned} \left\langle {\tau_{N} } \right\rangle & = \int\limits_{0}^{1} {N\tau \left( \tau \right)^{N - 1} d\tau } \\ & = N\int\limits_{0}^{1} {\left( \tau \right)^{N} d\tau } \\ & = \left. {{\frac{{N\left( \tau \right)^{N + 1} }}{N + 1}}} \right|_{0}^{1} \\ \end{aligned} $$

$$ \left\langle {\tau_{N} } \right\rangle = {\frac{N}{N + 1}}. $$

(36)

This completes the proof and thus, for all jumps, the expected value of the jump is described:

$$ \left\langle {\tau_{k} } \right\rangle = {\frac{k}{N + 1}}. $$

(37)

Appendix 2: Number of Nonzero Sums of a Set’s Elements

In the case of a discretized reference model, the set of attainable values of τ is finite and its elements are the nonzero linear combinations of the reference model values with coefficients equal to zero or one. In other words, for a reference model specified in n bins, the set Τ _n of attainable τ values is comprised of the nonzero sums of the n reference model values. We denote the cardinality of this set |Τ _n |.

Theorem.

|Τ _n | = 2ⁿ − 1.

Proof.

In the case where n = 1, it is clear that there is only one nonzero sum: Τ ₁ = {1}, |Τ ₁| = 1. In the case where n = 2, we represent a reference model as the set {n ₁, n ₂}. In this case, the set of attainable τ values is {n ₁, n ₂, n ₁ + n ₂}; |Τ ₂| = 3. When n = 3, we represent a reference model as the set {n ₁, n ₂, n ₃} and the set of sums is {n ₁, n ₂, n ₃ , n ₁ + n ₂ , n ₁ + n ₃ , n ₂ + n ₃ , n ₁ + n ₂ + n ₃}; |Τ ₃| = 7. We assume that the relation holds for all values of n up to and including (x − 1) and we consider n = x. The set Τ _x will contain all elements of Τ_(x − 1) as well as each of these elements added to n _(x + 1); the only additional sum in Τ _x is n _(x + 1). Thus the cardinality |Τ _x | is twice the cardinality |Τ_(x − 1)| plus one additional element:

$$ \left| {{\rm T}_{x} } \right| = 2\left| {{\rm T}_{{\left( {x - 1} \right)}} } \right| + 1 = 2\left( {2^{{\left( {x - 1} \right)}} - 1} \right) + 1 = 2^{x} - 1. $$