End-to-End Statistical Model Checking for Parametric ODE Models

Abstract

We propose a simulation-based technique for the verification of structural parameters in Ordinary Differential Equations. This technique is an adaptation of Statistical Model Checking, often used to verify the validity of biological models, to the setting of Ordinary Differential Equations systems. The aim of our technique is to search the parameter space for the parameter values that induce solutions that best fit experimental data under variability, with any metrics of choice. To do so, we discretize the parameter space and use statistical model checking to grade each individual parameter value w.r.t experimental data. Contrary to other existing methods, we provide statistical guarantees regarding our results that take into account the unavoidable approximation errors introduced through the numerical resolution of the ODE system performed while simulating. In order to show the potential of our technique, we present its application to two case studies taken from the literature, one relative to the growth of a jellyfish population, and another concerning a prey-predator model.

Appendices

Appendices

In these appendices, we provide the complete proofs of Lemma 1 and Theorem 2.

A Proof of Lemma 1

First, we recall the definition of stability of an approximation method.

Definition 2 (Method stability)

We say that the approximation method determined by Eq. (7) is stable if there exists a constant $\mathcal {K}> 0$, called stability constant, such that, for any two sequences $(y_k)_{0 \le k \le J}$ and $(\widetilde{y}_k)_{0 \le k \le J}$ defined as $y_{k+1} = y_k + h \, \varPhi (\tau _k, y_k, {\boldsymbol{\lambda }}, h)$ and $\widetilde{y}_{k+1} = \widetilde{y}_k + h \,\varPhi (\tau _k, \widetilde{y}_k, {\boldsymbol{\lambda }}, h ) + \eta _k$ respectively, ($0 \le k < J$), with ${\boldsymbol{\lambda }}\in \textbf{W}$ and $\eta _k \in \mathbb {R}$, we have

$$\begin{aligned} \max _{0 \le k \le J} |y_{k} - \widetilde{y}_k| \le \mathcal {K}\big (|y_0 - \widetilde{y}_0| + \sum _{0 \le k \le J} |\eta _k| \big ). \end{aligned}$$

(22)

It is well-known that if $\varPhi$ is $\kappa$-Lipschitz w.r.t. y, i.e. $\forall t \in [0,T]$, $\forall y, \widetilde{y} \in \mathbb {R}$, $\forall {\boldsymbol{\lambda }}\in \textbf{W}$ and $\forall h \in \mathbb {R}$, $\left| \varPhi (t, y, {\boldsymbol{\lambda }}, h) - \varPhi (t, \widetilde{y}, {\boldsymbol{\lambda }}, h)\right| \le \kappa \left| y - y_2\right|$, then stability is ensured (see for instance [3] or [4]).

Now, we fix ${\boldsymbol{\lambda }}^* \in \textbf{W}$ and ${\boldsymbol{\lambda }}_1, {\boldsymbol{\lambda }}_2 \in \mathcal {B}_{{\boldsymbol{\lambda }}^*}$, and we consider the approximate solutions $y^{{\boldsymbol{\lambda }}_1},y^{{\boldsymbol{\lambda }}_2}$ to Eq. (1) relative to ${\boldsymbol{\lambda }}_1$ and ${\boldsymbol{\lambda }}_2$ and starting from $z_0$.

$$\begin{aligned} \begin{aligned} {\left\{ \begin{array}{ll} y_0^{{\boldsymbol{\lambda }}_1} &{}= z_0,\\ y_{k+1}^{{\boldsymbol{\lambda }}_1} &{}= y_k^{{\boldsymbol{\lambda }}_1} + h\, \varPhi (t_k, y_k^{{\boldsymbol{\lambda }}_1}, {\boldsymbol{\lambda }}_1, h),\\ \end{array}\right. } {\left\{ \begin{array}{ll} y_0^{{\boldsymbol{\lambda }}_2} &{}= z_0,\\ y_{k+1}^{{\boldsymbol{\lambda }}_2} &{}= \widetilde{y}_k + h\, \varPhi (t_k, y_k^{{\boldsymbol{\lambda }}_2}, {\boldsymbol{\lambda }}_2, h).\\ \end{array}\right. } \end{aligned} \end{aligned}$$

We recall that the exact solutions to Eq. (1) relative to ${\boldsymbol{\lambda }}_1$ and ${\boldsymbol{\lambda }}_2$ and starting from $z_0$ are denoted $z^{{\boldsymbol{\lambda }}_1}$ and $z^{{\boldsymbol{\lambda }}_2}$ respectively. For $i \in \lbrace 1, 2 \rbrace$ and $0 \le k \le J$, we introduce the consistency error on $y^{{\boldsymbol{\lambda }}_i}$ at step k:

$$\begin{aligned} \epsilon _{h, k}({\boldsymbol{\lambda }}_i) = |z^{{\boldsymbol{\lambda }}_i}(\tau _k) - y^{{\boldsymbol{\lambda }}_i}(\tau _k)|. \end{aligned}$$

(23)

The consistency errors satisfy $\epsilon _h({\boldsymbol{\lambda }}_i) = \max _{0 \le k \le J} \epsilon _{h, k}({\boldsymbol{\lambda }}_i)$, for $i \in \lbrace 1, 2 \rbrace$, where $\epsilon _h({\boldsymbol{\lambda }}_i)$ is the global approximation error (defined by Eq. (10)). The proof of Lemma 1 can be derived from the following theorem.

Theorem 3 (Stability with respect to consistency error)

Assume that the function $\varPhi$ defined in Eq. (7) is $\kappa _1$-Lipschitz w.r.t. ${\boldsymbol{\lambda }}$ and $\kappa _2$-Lipschitz continuous w.r.t. y. Then the approximation method is stable w.r.t. the consistency error, i.e. there exists $\mathcal {K}> 0$ such that

$$\begin{aligned} \forall {\boldsymbol{\lambda }}_1, {\boldsymbol{\lambda }}_2 \in \mathcal {B}_{{\boldsymbol{\lambda }}^*}, \max _{0 \le k \le J} |\epsilon _{h,k}({\boldsymbol{\lambda }}_1) - \epsilon _{h,k}({\boldsymbol{\lambda }}_2)| \le \mathcal {K}\Vert {\boldsymbol{\lambda }}_1 - {\boldsymbol{\lambda }}_2\Vert , \end{aligned}$$

(24)

where $\Vert \cdot \Vert$ is the Euclidean norm defined in Sect. 2.1.

Proof

(of Theorem 3). By assumption, $\varPhi$ is $\kappa _1$-Lipschitz continuous w.r.t. ${\boldsymbol{\lambda }}$:

$$\begin{aligned} \forall t, y, h \in \mathbb {R}, \forall {\boldsymbol{\lambda }}_1, {\boldsymbol{\lambda }}_2 \in \mathcal {B}_{{\boldsymbol{\lambda }}^*}, \left| \varPhi (t, y, {\boldsymbol{\lambda }}_1, h) - \varPhi (t, y, {\boldsymbol{\lambda }}_2, h)\right| \le \kappa _1 \left| {\boldsymbol{\lambda }}_1 - {\boldsymbol{\lambda }}_2 \right| . \end{aligned}$$

It follows that

$$\begin{aligned} \begin{aligned} |y_{k+1}^{{\boldsymbol{\lambda }}_1} - y_{k+1}^{{\boldsymbol{\lambda }}_2}|&\le |y_k^{{\boldsymbol{\lambda }}_1} - y_k^{{\boldsymbol{\lambda }}_2}| + h|\varPhi (t_k, y_k^{{\boldsymbol{\lambda }}_1}, {\boldsymbol{\lambda }}_1, h) - \varPhi (t_k, y_k^{{\boldsymbol{\lambda }}_2}, {\boldsymbol{\lambda }}_2, h)|\\&\le |y_k^{{\boldsymbol{\lambda }}_1} - y_k^{{\boldsymbol{\lambda }}_2}| + h|\varPhi (t_k, y_k^{{\boldsymbol{\lambda }}_1},{\boldsymbol{\lambda }}_1, h) - \varPhi (t_k, y_k^{{\boldsymbol{\lambda }}_1}, {\boldsymbol{\lambda }}_2, h)|\\&\quad + h|\varPhi (t_k, y_k^{{\boldsymbol{\lambda }}_1}, {\boldsymbol{\lambda }}_2, h) - \varPhi (t_k, y_k^{{\boldsymbol{\lambda }}_2},{\boldsymbol{\lambda }}_2, h)|\\&\le (1+h\kappa _2)|y_k^{{\boldsymbol{\lambda }}_1} - y_k^{{\boldsymbol{\lambda }}_2}| + h\kappa _1\Vert {\boldsymbol{\lambda }}_1 - {\boldsymbol{\lambda }}_2\Vert ,\\ \end{aligned} \end{aligned}$$

for $0 \le k \le J$. We write $|y_k^{{\boldsymbol{\lambda }}_1} - y_k^{{\boldsymbol{\lambda }}_2}| = \varDelta _{y,k}$ and $\Vert {\boldsymbol{\lambda }}_1 - {\boldsymbol{\lambda }}_2\Vert = \varDelta _{{\boldsymbol{\lambda }}}$, and we get

$$\begin{aligned} \varDelta _{y,k+1} \le (1+h\kappa _2)\varDelta _{y,k} + h\kappa _1 \varDelta _{{\boldsymbol{\lambda }}}. \end{aligned}$$

(25)

Applying the discrete Gronwall lemma (see for instance [7], VIII.2.3), we deduce

$$\begin{aligned} \max _{0 \le k \le J} \varDelta _{y,k} \le e^{\kappa _2T} \big (\varDelta _{y,0} + \sum _{0 \le j \le k-1} h\kappa _1\varDelta _{{\boldsymbol{\lambda }}} \big ) \end{aligned}$$

which leads to

$$\begin{aligned} \max _{0 \le k \le J} |y_k^{{\boldsymbol{\lambda }}_1} - y_k^{{\boldsymbol{\lambda }}_2}| \le e^{\kappa _2T} T\kappa _1 \Vert {\boldsymbol{\lambda }}_1 - {\boldsymbol{\lambda }}_2\Vert , \end{aligned}$$

since $y_0^{{\boldsymbol{\lambda }}_1} = y_0^{{\boldsymbol{\lambda }}_2} = z_0$ and $h\,J = T$.

Furthermore, it is proved in [4] that if $\varPhi$ is Lipschitz continuous w.r.t. ${\boldsymbol{\lambda }}$, then the exact solution $z_{{\boldsymbol{\lambda }}}$ is also Lipschitz continuous w.r.t. ${\boldsymbol{\lambda }}$ that is, there exists $\kappa _3 > 0$ such that

$$\begin{aligned} \forall {\boldsymbol{\lambda }}_1,{\boldsymbol{\lambda }}_2 \in \mathcal {B}_{{\boldsymbol{\lambda }}^*}, \forall t \in [0,T], |z^{{\boldsymbol{\lambda }}_1}(t) - z^{{\boldsymbol{\lambda }}_2}(t)| \le \kappa _3 \Vert {\boldsymbol{\lambda }}_1 - {\boldsymbol{\lambda }}_2\Vert . \end{aligned}$$

(26)

Finally, we have

$$\begin{aligned} \begin{aligned} |\epsilon _{h, k}({\boldsymbol{\lambda }}_1) - \epsilon _{h, k}({\boldsymbol{\lambda }}_2)|&\le |z^{{\boldsymbol{\lambda }}_1}(\tau _k) - y^{{\boldsymbol{\lambda }}_1}(\tau _k) - z^{{\boldsymbol{\lambda }}_2}(\tau _k) - y^{{\boldsymbol{\lambda }}_2}(\tau _k)|\\&\le |z^{{\boldsymbol{\lambda }}_1}(\tau _k) - z^{{\boldsymbol{\lambda }}_2}(\tau _k)| + |y^{{\boldsymbol{\lambda }}_1}(\tau _k) - y^{{\boldsymbol{\lambda }}_2}(\tau _k)|\\&\le \mathcal {K}\Vert {\boldsymbol{\lambda }}_1 - {\boldsymbol{\lambda }}_2\Vert , \end{aligned} \end{aligned}$$

with $\mathcal {K}= \kappa _3 + T\kappa _1 e^{\kappa _2T}$, which completes the proof of Theorem 3.    $\square$

It remains to show that Theorem 3 implies Lemma 1.

Proof

(of Lemma 1). Let $(h_i)_{i \ge 0}$ be a sequence of discretization steps such that $\lim _{i \rightarrow \infty } h_i = 0$. Since the approximation method given by (7) is assumed to be convergent, each function $\epsilon _{h_i}(\cdot )$ defined in Eq. (23) is pointwise convergent to 0.

Furthermore, we recall that $\varPhi$ is Lipschitz continuous w.r.t. ${\boldsymbol{\lambda }}\in \textbf{W}$. Hence, Theorem 3 implies that the functions $\big (\epsilon _{h_i}(\cdot )\big )_{i \ge 0}$ defined in Eq. (23) are also Lipschitz continuous, with uniform Lipschitz constant $\mathcal {K}$:

$$\begin{aligned} \left| \epsilon _{h_i}({\boldsymbol{\lambda }}_1) - \epsilon _{h_i}({\boldsymbol{\lambda }}_2)\right| \le \mathcal {K}\left\| {\boldsymbol{\lambda }}_1 - {\boldsymbol{\lambda }}_2\right\| , \quad \forall {\boldsymbol{\lambda }}_1, {\boldsymbol{\lambda }}_2 \in \mathcal {B}_{{\boldsymbol{\lambda }}^*}, \quad \forall i \in \mathbb {N}. \end{aligned}$$

Consequently, the functions $\big (\epsilon _{h_i}(\cdot )\big )_{i \ge 0}$ are uniformly equicontinuous. Hence, Arzelà-Ascoli Theorem [8] implies that the sequence $\big (\epsilon _{h_i}(\cdot )\big )_{i \ge 0}$ converges uniformly to 0 on $\mathcal {B}_{{\boldsymbol{\lambda }}^*}$, thus $\forall \varepsilon > 0$, $\exists i^* \in \mathbb {N}$, $\forall i \ge i^*$, $\forall {\boldsymbol{\lambda }}\in \mathcal {B}_{{\boldsymbol{\lambda }}^*}$, $\epsilon _{h_i}({\boldsymbol{\lambda }}) < \varepsilon$, and Lemma 1 is proved.    $\square$

Remark 2 (Computation of a sufficiently small integration step)

We emphasize that Lemma 1 can be supplemented by an explicit choice of a sufficiently small integration step h, provided the integration method comes with appropriate estimates of their global error. Notably, the accuracy of the Runge-Kutta 4 method, which we use for the numerical treatment of our case studies, has been thoroughly studied (see [16] for instance), and it is known that its inherent error can be bounded in terms of the successive derivatives of the function f involved in Eq. (1), up to order 4.

B Proof of Theorem 2

First Step. We begin the proof of Theorem 2 by showing how to compute an estimator $\hat{p}_1^\varepsilon$ of the probability $p_1^\varepsilon$ defined in (14).

Let $({\boldsymbol{\lambda }}_i)_\mathbb {N}$ be a sequence of values in the ball $\mathcal {B}_{{\boldsymbol{\lambda }}^*}$. We write $B_i$ the random variable corresponding to the test “$\varphi _1^\varepsilon ({\boldsymbol{\lambda }}_i)$ holds”: all the $B_i$ are i.i.d. variables and follow a Bernoulli’s law of parameter $p_1^\varepsilon$. We write $b_i$ the evaluation of $B_i$. We introduce the transfer function $g_1\,: \mathcal {B}_{{\boldsymbol{\lambda }}^*}\rightarrow \{0,1\}$ corresponding to the test regarding $\varphi _1^\varepsilon ({\boldsymbol{\lambda }}_i)$, defined by $g_1({\boldsymbol{\lambda }}_i) = 1$ if $\varphi _1^\varepsilon ({\boldsymbol{\lambda }}_i)$ holds, 0 otherwise. Next, we consider

$$\begin{aligned} G = \mathbb {E}(g_1(X)) = \int _{\mathcal {B}_{{\boldsymbol{\lambda }}^*}} g_1(x)f_X(x)\textrm{d}x, \end{aligned}$$

(27)

where $f_X$ is defined by a uniform distribution, that is, $f_X(x) = \frac{1}{\left| \mathcal {B}_{{\boldsymbol{\lambda }}^*}\right| }$, $x \in \mathcal {B}_{{\boldsymbol{\lambda }}^*}$. We produce a sample $(x_1, x_2, \dots , x_N)$ of the variable X in $\mathcal {B}_{{\boldsymbol{\lambda }}^*}$, and use it to compute the Monte-Carlo estimator G. By virtue of the Law of Large Numbers, the sample mean satisfies: $\overline{g}_N = \frac{1}{N} \sum _{i=1}^N g_1{x_i}$. The Central Limit Theorem states that the variable $Z = \frac{\overline{g}_N - G}{\sigma _{\overline{g}_N}}$ approximately follows a Standard Normal Distribution $\mathcal {N}(0,1)$; hence, for a risk $\theta$, we can bound the error $|\alpha _N|$ of swapping G with $\overline{g}_N$ by building confidence intervals:

$$\begin{aligned} \mathbb {P}\left( |\alpha _N|\le \chi _{1 - \frac{\theta }{2}} \frac{\sigma _{g_1}}{\sqrt{N}}\right) = 1 - \theta , \end{aligned}$$

(28)

where $\chi _{1 - \frac{\theta }{2}}$ is the quantile of the Standard Normal Distribution $\mathcal {N}(0,1)$ and $\sigma _{g_1}$ is the variance of $g_1$.

Since we are interested in finding $p_1^\varepsilon$ with a certain confidence, we can perform this process after setting the desired target error $\alpha$ and risk $\theta$, knowing how many simulations must be ran using Hoeffding’s inequality [12]:

$$\begin{aligned} \theta = \mathbb {P}(\overline{g}_{N} \notin [p_1^\varepsilon - \alpha , p_1^\varepsilon + \alpha ]) \le 2 \exp (-2 \alpha ^2{N}), \end{aligned}$$

or equivalently $N \ge \frac{\log (2/\theta )}{2\alpha ^2}$. Here, it is worth emphasizing that N can be chosen independently of $\varepsilon$.

Further, the variance of $\overline{g}_{N}$ can be expressed with the variance of $g_1(X)$:

$$\begin{aligned} \sigma _{g_1}^2 = \mathbb {E}\left( [g_1(X) - \mathbb {E}(g_1(X))]^2\right) = \int _{\mathcal {B}_{{\boldsymbol{\lambda }}^*}} (g_1(x))^2 f_X(x) \textrm{d}x - G^2. \end{aligned}$$

We consider i.i.d. samples, hence $\sigma _{g_1}^2$ can be estimated with the variance $S_{g_1}^2$:

$$\begin{aligned} \sigma _{g_1}^2 \simeq S_{g_1}^2 = \frac{1}{N}\sum _{i = 1}^{N} (g_1({\boldsymbol{\lambda }}_i)^2 - \overline{g}_{N}^2). \end{aligned}$$

It follows that $\sigma _{g_1}$ can be estimated with its empirical counterpart $\hat{\sigma }_{g_1} = \sqrt{S_{g_1}^2}$, which shows that the error displays a $1/\sqrt{N}$ convergence.

Finally, after estimating $\sigma _{g_1}$, we can find $\hat{p}_1^\varepsilon$ using the variance of Bernoulli’s law $\hat{\sigma }_{g_1}^2 = \hat{p}_1^\varepsilon \times (1 - \hat{p}_1^\varepsilon )$. We conclude that the probability that $\varphi _1^\varepsilon ({\boldsymbol{\lambda }})$ holds is estimated by $\hat{p}_1^\varepsilon = \frac{1}{2} \left( 1 \pm \sqrt{1 - 4 \hat{\sigma }_{g_1}^2}\right)$, with an error $\alpha$ and a risk $\theta$, provided we perform ${N} \ge \frac{\log (2/\theta )}{2\alpha ^2}$ simulations. It follows that

$$\begin{aligned} \mathbb {P}\big ({p}_1^\varepsilon \in [\hat{p}_1^\varepsilon - \alpha , \hat{p}_1^\varepsilon + \alpha ] \big ) \ge 1 - \theta . \end{aligned}$$

(29)

Similarly, we determine an estimator $\hat{p}_2^\varepsilon$ of $p_2^\varepsilon$ by running $N \ge \frac{\log (2/\theta )}{2\alpha ^2}$ additional simulations, and obtain a confidence interval satisfying

$$\begin{aligned} \mathbb {P}\big ({p}_2^\varepsilon \in [\hat{p}_2^\varepsilon - \alpha , \hat{p}_2^\varepsilon + \alpha ] \big ) \ge 1 - \theta . \end{aligned}$$

(30)

Second Step. Now, let us show how a confidence interval for the probability p can be derived from the confidence intervals given in (29), (30), involving the estimators $\hat{p}_1^\varepsilon$ and $\hat{p}_2^\varepsilon$ respectively. The independence of the samples used to determine the estimators $\hat{p}_1^\varepsilon$, $\hat{p}_2^\varepsilon$ guarantees that

$$\begin{aligned} \mathbb {P}(p \in [\hat{p}_1^\varepsilon - \alpha , \hat{p}_2^\varepsilon + \alpha ]) = \mathbb {P}\big (\lbrace p \ge \hat{p}_1^\varepsilon - \alpha \rbrace \big ) \times \mathbb {P}\big (\lbrace p \le \hat{p}_2^\varepsilon + \alpha \rbrace \big ). \end{aligned}$$

By virtue of (29), we have $\mathbb {P}(p_1^\varepsilon \ge \hat{p}_1^\varepsilon - \alpha ) \ge 1 - \theta$. Next, the estimate (15) implies $\mathbb {P}(p \ge \hat{p}_1^\varepsilon - \alpha ) \ge \mathbb {P}(p_1^\varepsilon \ge \hat{p}_1^\varepsilon - \alpha ) \ge 1 - \theta$. Similarly, we have $\mathbb {P}(p \le \hat{p}_2^\varepsilon + \alpha ) \ge 1 - \theta$, and finally $\mathbb {P}(p \in [\hat{p}_1^\varepsilon - \alpha , \hat{p}_2^\varepsilon + \alpha ]) \ge (1 - \theta )^2 = 1 - \xi$, since $\theta = 1 - \sqrt{1 - \xi }$.

Third Step. Finally, let us prove how Lemma 1 guarantees that proper values of h and $\varepsilon$ can be found, in order to control the distance between $\hat{p}_1$ and $\hat{p}_2$.

Indeed, the continuity of the probability measure $\mathbb {P}$ ensures that there exists $\varepsilon _0 > 0$ such that $\left| p_1^\varepsilon - p_2^\varepsilon \right| \le \alpha$, for $\varepsilon < \varepsilon _0$. Next, we write

$$\begin{aligned} \left| \hat{p}_1^\varepsilon - \hat{p}_2^\varepsilon \right| \le \left| \hat{p}_1^\varepsilon - p_1^\varepsilon \right| + \left| \hat{p}_2^\varepsilon - p_2^\varepsilon \right| + \left| p_1^\varepsilon - p_2^\varepsilon \right| , \end{aligned}$$

hence we have, for $\varepsilon < \varepsilon _0$:

$$\begin{aligned} \begin{aligned} \mathbb {P}\big (\left| \hat{p}_1^\varepsilon - \hat{p}_2^\varepsilon \right| \le 3\alpha \big )&\ge \mathbb {P}(\left| \hat{p}_1^\varepsilon - p_1^\varepsilon \right| \le \alpha ) \times \mathbb {P}(\left| \hat{p}_2^\varepsilon - p_2^\varepsilon \right| \le \alpha ) \times \mathbb {P}(\left| p_1^\varepsilon - p_2^\varepsilon \right| \le \alpha )\\&\ge (1-\theta )^2 \times 1 = 1 - \xi . \end{aligned} \end{aligned}$$

In parallel, Lemma 1 guarantees that for h sufficiently small, the global stability error can be uniformly bounded on $\mathcal {B}_{{\boldsymbol{\lambda }}^*}$ by $\varepsilon _0$. The proof is complete.    $\square$