Appendix A Proofs for Sections 2 and 3
Proof of Lemma 1
We first observe that
$$\begin{aligned} - 2 - \displaystyle \frac{nM}{n-1}< -\frac{3}{n} < 0. \end{aligned}$$
Then the calculus gives
$$\begin{aligned} p(0)= & {} \displaystyle \frac{9M(Mn+ 2n -2)}{2(n-1)^2}> 0, \\ p\left( -\frac{3}{n}\right)= & {} - \displaystyle \frac{9(M n^2 + 2(n-1)(n-3))}{2 n^3} < 0, \\ p\left( - 2 - \displaystyle \frac{nM}{n-1} \right)= & {} \frac{(Mn + 2n-2)(M(n-3) + 2n-2)}{2 (n-1)^2} > 0 \\ \lim _{\mu \rightarrow -\infty } p(\mu )= & {} - \infty . \end{aligned}$$
The intermediate value theorem applied to \(p(\mu )\) thus provides a proof of the result. \(\square \)
Proof of Proposition 2
In order to simplify computations, we make the following change of parameters:
$$\begin{aligned} M = 2 N, \; n = \frac{a + N}{a}. \end{aligned}$$
The new parameters a et N verify \(a > 0\) and \(N > 0\) and the matrix Q becomes
$$\begin{aligned} Q= \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0&{}3&{}0\\ a &{}-1+a-2\,N&{}-2\,a+2\,N&{}0&{}1\\ 0&{}6\,a&{}-6\,a&{}0&{}0\\ 0&{}0&{}0&{}0&{}0\\ 0&{}0&{}0&{}0&{}0 \end{array} \right] . \end{aligned}$$
The matrix Q has the double eigenvalue 0 and it is easy to check that the corresponding eigenspace is generated by the following non colinear vectors:
$$\begin{aligned} V_1 = [N,N-1,N-1,2 N, N -a -1]^T, \quad V_2 = [1,1,1,1,1]^T. \end{aligned}$$
We then consider the change of basis matrix
$$\begin{aligned} P_1 = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 1&{}0&{}0&{}N&{}1\\ 0&{}\quad 1&{}\quad 0&{}\quad N-1&{}\quad 1\\ 0&{}\quad 0&{}\quad 1&{}\quad N-1&{}\quad 1\\ 0&{}\quad 0&{}\quad 0&{}\quad 2\,N&{}\quad 1 \\ 0&{}\quad 0&{}\quad 0&{}\quad N-a-1&{}\quad 1\end{array} \right] , \end{aligned}$$
whose inverse is
$$\begin{aligned} P_1^{-1} = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 1&{}\quad 0&{}\quad 0&{}\quad -{\frac{a+1}{N+a+1}}&{}\quad -{\frac{N}{ N+a+1}}\\ 0&{}\quad 1&{}\quad 0&{}\quad -{\frac{a}{N+a+1}}&{}-\quad {\frac{N+1}{N+ a+1}}\\ 0&{}\quad 0&{}\quad 1&{}\quad -{\frac{a}{N+a+1}}\quad &{}-{\frac{N+1}{N+a+ 1}}\\ 0&{}\quad 0&{}\quad 0&{}\quad \frac{1}{N+a+1} &{}\quad - \frac{1}{N+a+1}\\ 0&{}\quad 0&{}\quad 0&{}\quad -{\frac{N-a-1}{N+a+1}}&{}\quad {\frac{2N}{N+a+1}}\end{array} \right] . \end{aligned}$$
We have thus obtained a partial diagonalization (by blocks) of the matrix Q:
$$\begin{aligned} Q_2 = P_1^{-1} Q P_1 = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0&{}0&{}0\\ a &{}-1+a-2\,N&{}-2\,a+2\,N&{}0&{}0\\ 0&{}6\,a&{}-6\,a&{}0&{}0 \\ 0&{}0&{}0&{}0&{}0\\ 0&{}0&{}0&{}0&{}0 \end{array} \right] , \end{aligned}$$
(24)
and we put
$$\begin{aligned} R := \left[ \begin{array}{c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0\\ a&{}-1+a- 2\,N&{}-2\,a+2\,N\\ 0&{}6\,a&{}-6\,a\end{array} \right] . \end{aligned}$$
The characteristic polynomial of R is the polynomial \(p(\mu )\), which has the following expressions using the new parameters:
$$\begin{aligned} p(\mu ) = \mu ^3 + (5N+5a+4) \mu ^2 + 3\left( 2 N^2 +4 aN + 3N + 2a^2 + 7a+1\right) \mu + 18 a (N + a +1), \end{aligned}$$
which has three strictly negative real roots. These roots are all distinct by Lemma 1.
If \(\mu \) is an eigenvalue of R, we then determine a corresponding eigenvector \(W(\mu )\) by solving the equation \(R W(\mu ) = \mu W(\mu )\).
The computations show that we can choose
$$\begin{aligned} W(\mu ) = \left[ \begin{array}{c} \mu ^2 + (2N + 5a+1) \mu + 6a(a+1) \\ a(\mu + 6a) \\ 6 a^2 \end{array} \right] . \end{aligned}$$
We then consider the \(3 \times 3\) passage matrix \(P_2\), whose column vectors are the \(W(\mu _i), \, i=1,2,3\), where the \(\mu _i\) are the three eigenvalues of R:
$$\begin{aligned} P_2 = \left[ W(\mu _1), \; W(\mu _2), \; W(\mu _3) \right] . \end{aligned}$$
Some easy computations on the rows show that the determinant of \(P_2\) is a Van der Monde determinant, and hence: \(\mathrm {det}(P_2) = (\mu _1 - \mu _2)(\mu _1-\mu _3)(\mu _2 - \mu _3) \ne 0\).
The computations of the inverse \(P_2^{-1}\) gives
$$\begin{aligned} P_2^{-1} = \left[ Z(\mu _1), Z(\mu _2), Z(\mu _3)\right] ^T, \end{aligned}$$
where
$$\begin{aligned} Z(\mu ) = \frac{1}{p'(\mu )} \; \left[ \begin{array}{c} 1 \\ \displaystyle \frac{\mu + 3N+3}{a} \\ -\displaystyle \frac{\mu ^2 + (3N+a+4)\mu + 3(N+a+1)}{a \mu } \end{array}\right] , \end{aligned}$$
with
$$\begin{aligned} p'(\mu ) = 3 \mu ^2 + 2(5 N + 5 a +4)\mu + 3\left( 2 N^2 +4 aN + 3N + 2a^2 + 7a+1\right) . \end{aligned}$$
We further obtain
$$\begin{aligned} e^{t R}= & {} P_2 \; \left[ \begin{array}{c@{\quad }c@{\quad }c} e^{\mu _1 t} &{} 0 &{} 0\\ 0 &{} e^{\mu _2 t} &{}0 \\ 0 &{} 0 &{} e^{\mu _3 t} \end{array} \right] \; P_2^{-1}, \\= & {} \sum _{j=1}^3 e^{\mu _j} \; W(\mu _j) Z(\mu _j)^T. \end{aligned}$$
We then introduce the matrices \(A(\mu )\) and B defined by
$$\begin{aligned} A(\mu ) := P_1 \; \bar{W}(\mu ) \bar{Z}(\mu ) ^T \; P_1^{-1}, \end{aligned}$$
where \(\bar{W}_j(\mu )\) (resp. \(\bar{Z}(\mu )\)) is a vector of length 5 obtained by adding two null coordinates to \(W(\mu )\) (resp. \(Z(\mu )\)), and
$$\begin{aligned} B = P_1 \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 1 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 0 &{}\quad 1 \end{array} \right] \; P_1^{-1}. \end{aligned}$$
Emphasizing the rank one property of matrix \(A(\mu )\), let us define the vectors
$$\begin{aligned} E_1 = \left[ \begin{array}{c} \displaystyle \frac{3 M}{v} \\ 1 \\ \displaystyle \frac{3 M}{u} \\ 0 \\ 0 \end{array} \right] , \quad E_2 = \left[ \begin{array}{c} M u \\ (n-1) u v \\ (n-1)(n-2) M v \\ \displaystyle \frac{3 M u}{\mu } \\ \displaystyle \frac{(n-1) uv}{\mu } \end{array} \right] , \end{aligned}$$
where \(\mu \) is an eigenvalue of R, and u, v are functions of \(\mu \) defined by
$$\begin{aligned} u = (n-1)\mu + 3 M,\quad v = 2 \mu + 3(M+2). \end{aligned}$$
Note that, since
$$\begin{aligned} p\left( \displaystyle -\frac{3M}{n-1}\right)= & {} - \displaystyle \frac{9}{2} \cdot \frac{(n-2) [2(n-1) + (n-3)M] M^2}{(n-1)^3} <0,\\ p\left( \displaystyle -\frac{3(M+2)}{2}\right)= & {} \displaystyle \frac{9}{8} \cdot \frac{ [2(n-1) + (n-3)M] M^2}{(n-1)^2} >0, \end{aligned}$$
\(\displaystyle -\frac{3M}{n-1}\) and \(\displaystyle -\frac{3(M+2)}{2}\) cannot be eigenvalues and thus \(u \ne 0\) et \(v \ne 0\).
With \(\delta (M,n,\mu ) = 2(n-1)^2 p'(\mu )\), we have
$$\begin{aligned} A(\mu ) = \displaystyle \frac{1}{\delta (M,n,\mu )} \; E_1 E_2^T, \end{aligned}$$
(25)
which gives the expression of \(A(\mu )\) in Proposition 2.
The stated result easily follows. \(\square \)
Proof of Proposition 4
Using Eq. (15) and Proposition 2, we deduce that
$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = -3 \ \frac{A(\mu _1)(i,4)+A(\mu _1)(i,5)}{3A(\mu _1)(i,1)+A(\mu _1)(i,2)}, \quad i=1,2,3. \end{aligned}$$
Note that the matrix \(E_1 E_2^T\) introduced in the proof of Proposition 2 can also be factorized in a different manner. If we define the \(3 \times 3\) matrix C, the \(5 \times 3\) matrix \(D_1\) and the \(3 \times 5\) matrix \(D_2\) by
$$\begin{aligned} C:= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c} \displaystyle \frac{u}{v} &{} u &{} 1 \\ u &{} uv &{} v \\ 1 &{} v &{} \displaystyle \frac{v}{u} \end{array} \right] ,\quad D_1 := \left[ \begin{array}{c@{\quad }c@{\quad }c} 3 M &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 3M \\ 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 \end{array} \right] ,\\ D_2:= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} M &{} 0 &{} 0 &{} \displaystyle \frac{3M}{\mu } &{} 0 \\ 0 &{} n-1 &{} 0 &{} 0 &{} \displaystyle \frac{n-1}{\mu } \\ 0 &{} 0 &{} (n-1)(n-2) &{} 0 &{} 0 \end{array} \right] , \end{aligned}$$
we may check that, for any value of \(\mu \), we have \(E_1 E_2^T = D_1 C D_2\).
The vectors \(V_1\) and \(V_2\), defined by
$$\begin{aligned} V_1 := \left[ \begin{array}{c} 3 \\ 0 \\ 0 \\ vp \\ 0 \end{array} \right] , \;\; V_2 := \left[ \begin{array}{c} 0 \\ 1 \\ 0 \\ 0 \\ -\mu \end{array} \right] , \end{aligned}$$
verify \(D_2 V_j = 0\) for \(j=1,2\), and hence, using Eq. (25), \(A(\mu )V_j = 0\), for \(j=1,2\).
Therefore, for \(i=1,2,3\),
$$\begin{aligned} 3A(\mu )(i,1) - \mu A(\mu )(i,4) = 0, \; A(\mu )(i,2) - \mu A(\mu )(i,5) = 0, \end{aligned}$$
and the result follows. \(\square \)
Proof of Lemma 2
Note that \(-3\alpha \) and \(-3\beta \), with \(\alpha , \beta \) defined in Eq. (10), are the roots of the polynomial \(q_1(X)= q(X/3)\), where
$$\begin{aligned} q(X) = X^2 + \left( 1 + \frac{nM}{n-1}\right) X + \frac{M}{n-1}, \end{aligned}$$
and on the open subset \(D = \left\{ (n,M):\; n> 2, \; M>0 \right\} \) of \(\mathbb {R}^2\) the polynomials p(X) and \(q_1(X)\) have a common root if and only their resultant \(R(n,M) = \text{ Res }(p(X),q_1(X),X)\) with respect to X is null [see e.g. Lang (2002, Chapter IV–8)].
Because \(R(n,M) = - \frac{M^3}{9(n-1)^2} < 0\) on D, and because the roots \(-3 \alpha \) and \(\mu _3\) are continuous functions of (n, M) on D, the inequality \(-3 \alpha < \mu _3\) for one value of \((n,M) \in D\) implies the inequality everywhere on D. The same is true for the inequality \(\mu _1 < -3 \beta \).
This achieves the proof of the lemma. \(\square \)
Proof of Proposition 5
We will only give the proof of (i), which corresponds to the case of three genes sampled from the same deme. The proofs ot the analogous results (ii) and (iii), corresponding to the two other sample schemes, are similar.
When \(t \rightarrow 0\), using the well-known relation \(P_t = I_5 + tQ + o(t)\), where \(I_5\) denotes the identity matrix, we have the following Taylor expansions:
$$\begin{aligned} P_t(1,1)&= 1 + \left( -\frac{3M}{2}-3\right) t + o(t),\\ P_t(1,2)&= \frac{3M}{2} t + o(t). \end{aligned}$$
In particular, we have
$$\begin{aligned} \frac{1}{3P_t(1,1)+P_t(1,2)} = \frac{1}{1-(M+3)t +o(t)} = 1+(M+3)t+o(t). \end{aligned}$$
Using Eqs. (7)–(11), and a Taylor expansion of order 2 of the exponential in the neighborhood of 0, we easily obtain
$$\begin{aligned} F_{2,s}(u)&= u - \frac{M+1}{2}u^2 + o\left( u^2\right) ,\\ F_{2,d}(u)&= \frac{M}{2(n-1)}u^2 + o\left( u^2\right) . \end{aligned}$$
When substituting the above expressions into Eq. (11), straightforward calculations give
$$\begin{aligned} \mathbb {P}_1\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = u - \frac{M}{2}ut - \frac{M+1}{2}u^2 + o\left( u^2 + t^2\right) . \end{aligned}$$
Further, using Eq. (18), let us denote
$$\begin{aligned} h(u,t):=\mathbb {P}\left( T_2^{(3),\lambda _1} \le u | T_3^{(3),\lambda _1} = t\right) =1 - \left( \frac{1 - P_{t+u}(1,4)-P_{t+u}(1,5)}{1 - P_{t}(1,4)-P_{t}(1,5)}\right) ^{1/3}. \end{aligned}$$
We will write a Taylor expansion of order 2 of h(u, t) in the neighborhood of (0, 0). We have \(h(0,0)=0\) and direct computations give
$$\begin{aligned} \frac{\partial h}{\partial u}(u,t)&= \frac{1}{3} \times \left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) ^{-1/3} \times \left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) ^{-2/3}\\&\quad \times \left( P_{t+u}Q(1,4)+P_{t+u}Q(1,5) \right) ,\\ \frac{\partial h}{\partial t}(u,t)&= \frac{\left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) ^{-2/3}}{3\left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) ^{4/3}} \\&\quad \times \{ \left( P_{t+u}Q(1,4)+P_{t+u}Q(1,5) \right) \left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) \\&\quad - \left( P_{t}Q(1,4)+P_{t}Q(1,5) \right) \left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) \}. \end{aligned}$$
Using the fact that \(P_0 = I_5,\) and \(Q(1,4)=3, Q(1,5)=0\), we obtain
$$\begin{aligned} \frac{\partial h}{\partial u}(0,0)=1, \frac{\partial h}{\partial t}(0,0)=0. \end{aligned}$$
We further compute the second partial derivatives of h in (0, 0). With \(Q^2\) being the square matrix of the rate matrix Q, and using the fact that \(Q^2(1,4) = -9\left( \frac{M}{2}+1\right) \) and \(Q^2(1,5) = \frac{3M}{2},\) we easily derive
$$\begin{aligned} \frac{\partial ^2 h}{\partial u \partial t}(0,0)&= \frac{1}{3} \left\{ \left( Q(1,4)+Q(1,5)\right) ^2 + Q^2(1,4)+Q^2(1,5) \right\} = -M,\\ \frac{\partial ^2 h}{\partial u^2}(0,0)&= \frac{1}{3} \left\{ \frac{2}{3}\left( Q(1,4)+Q(1,5)\right) ^2 + Q^2(1,4)+Q^2(1,5) \right\} = -(M+1),\\ \frac{\partial ^2 h}{\partial t^2}(0,0)&= 0, \end{aligned}$$
The Taylor expansion of order 2 of h(u, t) near (0, 0) finally gives
$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _1} \le u | T_3^{(3),\lambda _1} = t\right) = u - Mut - \frac{M+1}{2}u^2 + o\left( u^2 + t^2\right) , \end{aligned}$$
which finishes the proof. \(\square \)
Proof of Proposition 6
Because \(0< \beta < \alpha \), using Eqs. (7) and (8), we get
$$\begin{aligned} F_{2,s}(u) = 1 - \frac{a}{\alpha } e^{-\alpha u} - \frac{1-a}{\beta } e^{-\beta u} = 1 - \frac{1-a}{\beta } e^{-\beta u} + o\left( e^{-\beta u}\right) \end{aligned}$$
and
$$\begin{aligned} F_{2,d}(u) = 1 - \frac{c}{\alpha } e^{-\alpha u} + \frac{c}{\beta } e^{-\beta u} = 1 + \frac{c}{\beta } e^{-\beta u} + o\left( e^{-\beta u}\right) . \end{aligned}$$
Therefore, using Eq. (11), we obtain
$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = 1 - K_{1,i}(n,M,t) e^{-\beta u} + o\left( e^{-\beta u}\right) , \end{aligned}$$
where \(K_{1,i}(n,M,t)\), given by (20), is strictly positive because \(0< a < 1\) and \(c < 0\).
Using Proposition 2 we get
$$\begin{aligned} \sum _{j=1}^3 P_{t+u}(i,j) = \left( A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1(t+u)} + o\left( e^{\mu _1(t+u)}\right) . \end{aligned}$$
Therefore, using Eq. (18) and relation \(1 - P_t(i,4) - P_t(i,5) = \sum _{j=1}^3 P_t(i,j)\), we obtain
$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right) = 1 - K_{2,i}(n,M,t)\; e^{\frac{\mu _1}{3} u} + o\left( e^{\frac{\mu _1}{3} u}\right) , \end{aligned}$$
where
$$\begin{aligned} K_{2,i}(n,M,t) :=\left( \frac{A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)}{P_t(i,1) + P_t(i,2) + P_t(i,3)} \right) ^{\frac{1}{3}} \; e^{\frac{\mu _1}{3} t}. \end{aligned}$$
Because \(\frac{\mu _1}{3} < - \beta \) from Lemma 2, we have \( \displaystyle e^{\frac{\mu _1}{3} u} = o(e^{-\beta u}), \) which achieves the proof of (19). \(\square \)
Proof of Proposition 7
We will first prove a useful lemma. \(\square \)
Lemma 3
Let us define \(\phi (\mu )\) by
$$\begin{aligned} \phi (\mu ) := \displaystyle \frac{3 A(\mu )(1,1)}{3 A(\mu )(1,1) + A(\mu )(1,2)}, \end{aligned}$$
where the matrix \(A(\mu )\) is given in Proposition 2.
Then,
-
(i)
\(0< \phi (\mu _i)< - \frac{\mu _i}{3} < \alpha , \; i=1,2,3.\)
-
(ii)
Defining h(n) by
$$\begin{aligned} h(n) = \frac{(n-1)\left( 25 - 9n + 5 \sqrt{9n^2 - 18n + 25}\right) }{12 n^2}, \end{aligned}$$
we have the following results:
-
(a)
If \(M > h(n)\), then \(\phi (\mu _1)< \beta< \phi (\mu _2)< \phi (\mu _3) < \alpha .\)
-
(b)
If \(M = h(n)\), then \(\phi (\mu _1)< \beta = \phi (\mu _2)< \phi (\mu _3) < \alpha .\)
-
(c)
If \(M < h(n)\), then \(\phi (\mu _1)< \phi (\mu _2)< \beta< \phi (\mu _3) < \alpha .\)
Proof
From Proposition 2 and using \(p(\mu _i)=0\), we get
$$\begin{aligned} \phi (\mu _i)= & {} \displaystyle \frac{1}{3} \frac{(n-1)\mu _i^2 + (nM + n-1)\mu _i + 6M}{Mn + 2(n-1)} \\= & {} \displaystyle \frac{3(n-1)}{nM + 2(n-1)} \; q\left( \frac{\mu _i}{3}\right) - \displaystyle \frac{M(\mu _i n + 3)}{nM + 2(n-1)} - \displaystyle \frac{\mu _i}{3}. \end{aligned}$$
From Lemma 1, it follows that \(\mu n + 3 >0\). Because \( -\alpha< \displaystyle \frac{\mu _i}{3} < -\beta \) from Lemma 2, we get \(q\left( \frac{\mu _i}{3}\right) < 0\), that proves \(\phi (\mu _i) < - \frac{\mu _i}{3}\).
Now, using again that \(p(\mu _i)=0\), we obtain the following expression for \(\phi (\mu _i)\)
$$\begin{aligned} \phi (\mu _i) = \frac{3 M}{2(n-1)\mu _i + 3nM + 6(n-1)}. \end{aligned}$$
(26)
Introducing the new variable \(\nu = \displaystyle \frac{3 M}{2(n-1)\mu + 3nM + 6(n-1)}\), and thus \(\mu = \displaystyle - \frac{3}{2} \frac{(nM + 2n-2))\nu - M}{(n-1) \nu }\), we get that \(\phi (\mu _i)\), for \(i=1,2,3\) are the three real roots of the polynomial
$$\begin{aligned} r(\nu )= & {} 4(n-1)\left( nM + 2n-2\right) \nu ^3 - \left[ n^2 M^2 + 2(n-1)(3n+4) M + 8(n-1)^2 \right] \nu ^2 \\&+ 2 M (2 nM + 5n-5) \nu - 3M^2. \end{aligned}$$
The coefficient signs of r(X) show that r(X) has no negative roots, implying that \(\phi (\mu _i) >0, \; i=1,2,3\), which achieves the proof of (i).
The set of (n, M) such that r(X) and \(q(-X)\) have a common root is obtained by computing their resultant with respect to X, denoted by R(n, M):
$$\begin{aligned} R(n,M) = \text{ Res }(r(X),q(-X),X) = M^3 \left[ 6n^2 M^2 + (n-1)(9n-25) M - 6(n-1)^2 \right] . \end{aligned}$$
In the domain \(D = \{(n,M),\; n>2, \; M>0 \}\) the curve \(6n^2 M^2 + (n-1)(9n-25) M - 6(n-1)^2 = 0\) is identical to the graph of the function \(M = h(n), \; n > 2\).
The set \(D {\setminus } \{(n,h(n)): \; n > 2 \}\) has two connex open components in which the relative position of \(\beta , \alpha \) and \(\phi (\mu _i), \; i=1,2,3\) are the same.
In the component \(D_1 := \{(n,M):\; n>2,\; M > h(n)\}\), one may choose \(n=3, \; M=\frac{4}{3}\) for which
$$\begin{aligned} \beta= & {} \displaystyle \frac{3}{2} - \displaystyle \frac{\sqrt{57}}{6} \approx 0.241694, \\ \alpha= & {} \displaystyle \frac{3}{2} + \displaystyle \frac{\sqrt{57}}{6} \approx 2.758305,\\ \phi (\mu _1)= & {} \displaystyle \frac{3 - \sqrt{5}}{4} \approx 0.190983, \\ \phi (\mu _2)= & {} \displaystyle \frac{1}{3}, \\ \phi (\mu _3)= & {} \displaystyle \frac{3 + \sqrt{5}}{4} \approx 1.3090167, \end{aligned}$$
that proves \((ii-a)\).
In the component \(D_2 := \{(n,M):\; n>2,\; M < h(n)\}\), one may choose \(n=3, \; M= \frac{1}{2} \) for which
$$\begin{aligned} \beta= & {} \displaystyle \frac{7 - \sqrt{33}}{8} \approx 0.156929, \\ \alpha= & {} \displaystyle \frac{7 + \sqrt{33}}{8} \approx 1.593070, \\ \phi (\mu _1)\approx & {} 0.104089, \\ \phi (\mu _2)\approx & {} 0.146359, \\ \phi (\mu _3)\approx & {} 1.118868, \end{aligned}$$
that proves \((ii-c)\).
The curve arc \(\{(n,h(n)): n > 2 \}\) may be parametrized by
$$\begin{aligned} n = \displaystyle \frac{3u^2 + 8u-3}{3(u^2-1)}, \quad M = \displaystyle \frac{8u}{(u+3)^2},\quad u \in ]1,3[, \end{aligned}$$
and we obtain
$$\begin{aligned} \beta= & {} \displaystyle \frac{u-1}{u+3}, \\ \alpha= & {} \displaystyle \frac{3(u+1)}{u+3}, \\ \phi (\mu _1)= & {} \displaystyle \frac{39u + 51 -3 \sqrt{-71u^2 + 442u +529}}{40(u+3)}, \\ \phi (\mu _2)= & {} \displaystyle \frac{u-1}{u+3}, \\ \phi (\mu _3)= & {} \displaystyle \frac{39u + 51 +3 \sqrt{-71u^2 + 442u +529}}{40(u+3)}, \end{aligned}$$
and \((ii-b)\) is proved. \(\square \)
Lemma 3 is now used to prove Proposition 7. First note that we have
$$\begin{aligned} \phi (\mu ) = \displaystyle \frac{3 A(\mu )(i,1)}{3 A(\mu )(i,1) + A(\mu )(i,2)}, \end{aligned}$$
for every \(i=1,2,3\).
When \(t \rightarrow + \infty \), \(P_t(i,j) = A(\mu _1)(i,j) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) ,\; j=1,2\) and thus
$$\begin{aligned}&\mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) \\&\quad = \frac{3 A(\mu _1)(i,1)}{3 A(\mu _1)(i,1) + A(\mu _1)(i,2)} F_{2,s}(u) + \frac{A(\mu _1)(i,2)}{3 A(\mu _1)(i,1) + A(\mu _1)(i,2)} F_{2,d}(u) + o(1) \\&\quad = \phi (\mu _1) F_{2,s}(u) + (1 - \phi (\mu _1)) F_{2,d}(u) + o(1). \end{aligned}$$
Using the definitions of \(F_{2,s}(u)\) and \(F_{2,d}(u)\) in Eqs. (7) and (8), we get
$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = 1 - c_1 e^{-\beta u} - c_2 e^{-\alpha u} + o(1), \end{aligned}$$
where \(c_1\) and \(c_2\) are given by (23).
On another hand, we have \(P_t(i,j) = A(\mu _1)(i,j) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) ,\; j=1,2,3\), we obtain
$$\begin{aligned} \sum _{j=1}^3 P_{t+u}(i,j)= & {} \left( (A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1 (t+u)} + o\left( e^{\mu _1 t}\right) ,\\ \sum _{j=1}^3 P_t(i,j)= & {} \left( (A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) . \end{aligned}$$
Using the fact that \(1 - P_t(i,4) - P_t(i,5) = \sum _{j=1}^3 P_t(i,j)\), this implies
$$\begin{aligned} \mathbb {P}(T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t) = 1 - e^{ \frac{\mu _1}{3} u} + o(1), \end{aligned}$$
and thus (21) holds.
It remains to show that \(K_3(n,M,u) >0\).
Using \(c_1 + c_2 = 1\), we may write
$$\begin{aligned} K_3(n,M,u) = \frac{\beta - \phi (\mu _1)}{\beta -\alpha } \left( e^{- \alpha u} - e^{\frac{\mu _1}{3} u}\right) + \frac{\phi (\mu _1)-\alpha }{\beta -\alpha } \left( e^{ \frac{\mu _1}{3} u} - e^{-\beta u}\right) . \end{aligned}$$
From Lemma 2, \(-\alpha< \frac{\mu _1}{3} < -\beta \) and thus \(e^{-\alpha u}< e^{\frac{\mu _1}{3} u} < e^{-\beta u}\). On the other hand, from Lemma 3 we have \(\phi (\mu _1)< \beta < \alpha \). Therefore \(K_3(n,M,u) >0\). \(\square \)
Appendix B The case of \(n=2\) islands
If we now consider the ancestral lineage process for a sample of three genes in the case of a symmetrical n-island model with \(n=2\) two islands, we only have the following four possible configurations:
-
1.
the three lineages are in the same island,
-
2.
two lineages are in the same island and the third one is in the other island,
-
3.
there are only two ancestral lineages left and they are in the same island,
-
4.
there are only two ancestral lineages left and they are in different islands.
The corresponding transition rate matrix is:
$$\begin{aligned} Q = \begin{pmatrix} -\frac{3M}{2}-3 &{}\quad \frac{3M}{2} &{}\quad 3 &{}\quad 0\\ \frac{M}{2} &{}\quad -\frac{M}{2}-1 &{}\quad 0 &{}\quad 1 \\ 0 &{}\quad 0&{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \end{pmatrix}, \end{aligned}$$
The characteristic polynomial of Q is \(\chi _Q(\mu ) = -\mu ^2 p(\mu )\), with
$$\begin{aligned} p(\mu ) = \mu ^2 + 2(M+2) \mu + 3(M+1). \end{aligned}$$
The matrix Q has the double eigenvalue 0 and the corresponding eigenspace of dimension 2 can be generated by the vectors \(\displaystyle \left[ \frac{M}{2},\frac{M}{2}-1, M, -1\right] ^T\) and \(\displaystyle [1,1,1,1]^T\). The two other eigenvalues are \(\mu _1 = - M - 2 + \sqrt{M^2+M+1}\) and \(\mu _2 = - M - 2 - \sqrt{M^2+M+1}\).
An eigenvector for \(\mu _1\) (resp. \(\mu _2\)) is \([3M, 2 \mu _1+3M+6, 0, 0]^T\) (resp. \([3M, 2 \mu _2+3M+6, 0, 0]^T\)), and we may consider the following change of basis matrix P given by
$$\begin{aligned} P = \begin{pmatrix} 3M &{}\quad 3M &{}\quad \frac{M}{2} &{}\quad 1 \\ 2 \mu _1+3M+6 &{}\quad 2 \mu _2+3M+6 &{}\quad \frac{M}{2}-1 &{}\quad 1 \\ 0 &{}\quad 0 &{}\quad M &{}\quad 1 \\ 0 &{}\quad 0 &{}\quad -1 &{}\quad 1 \end{pmatrix}. \end{aligned}$$
From
$$\begin{aligned} P^{-1} = \begin{pmatrix} \displaystyle \frac{2 \mu _1+M+2}{12M(\mu _1 + M+2)} &{} \displaystyle \frac{1}{4(\mu _1+M+2)} &{} \displaystyle \frac{2\mu _1+M+2}{4 M \mu _1(\mu _1 + M +2)} &{} \displaystyle \frac{1}{4\mu _1(\mu _1+M+2)} \\ \displaystyle \frac{2 \mu _2+M+2}{12M(\mu _2 + M+2)} &{} \displaystyle \frac{1}{4(\mu _2+M+2)} &{} \displaystyle \frac{2\mu _2+M+2}{4 M \mu _2(\mu _2 + M +2)} &{} \displaystyle \frac{1}{4\mu _2(\mu _2+M+2)} \\ 0 &{} 0 &{}\displaystyle \frac{1}{M+1} &{} - \displaystyle \frac{1}{M+1} \\ 0 &{} 0 &{}\displaystyle \frac{1}{M+1} &{}\displaystyle \frac{M}{M+1} \end{pmatrix}, \end{aligned}$$
and the equality
$$\begin{aligned} e^{t Q} = P \; \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} e^{\mu _1 t} &{} 0 &{} 0 &{} 0 \\ 0 &{} e^{\mu _2} &{} 0 &{} 0 \\ 0 &{} 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 0 &{} 1 \end{array} \right] \; P^{-1}, \end{aligned}$$
a proof of the following proposition is obtained.
Proposition 8
The transition kernel \(P_t = e^{t Q}\) is given by
$$\begin{aligned} P_t = e^{\mu _1 t} A(\mu _1) + e^{\mu _2 t} A(\mu _2) + B, \end{aligned}$$
where
$$\begin{aligned} A(\mu )= & {} \displaystyle \frac{1}{\delta (M,\mu )} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 2 \mu +M+2 &{} 3 M &{} \displaystyle \frac{3(2\mu +M+2)}{\mu } &{} \displaystyle \frac{3M}{\mu } \\ M &{} 2\mu +3M+6 &{} \displaystyle \frac{3M}{\mu } &{} \displaystyle \frac{2\mu +3M+6}{\mu } \\ 0 &{} 0 &{}0 &{} 0 \\ 0 &{} 0 &{}0 &{} 0 \end{array} \right] ,\\ B= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 0 &{}\quad b &{}\quad 1 - b \\ 0 &{} \quad 0 &{}\quad 1-b &{} \quad b \\ 0 &{}\quad 0 &{}\quad 1 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 1 \end{array}\right] , \end{aligned}$$
with \(\delta (M,\mu ) = 4(\mu + M+2)\) and \(b= \displaystyle \frac{M+2}{2(M+1)}\).
The IICRs \(\lambda _i(\cdot )/3\), for the initial sample configurations \(i=1\) and \(i=2\), correspond to the size-change functions
$$\begin{aligned} \lambda _i(t) = \frac{3(1 - P_t(i,3) - P_t(i,4))}{3P_t(i,1) + P_t(i,2)}, \end{aligned}$$
and the following proposition is verified.
Proposition 9
When \(t \rightarrow \infty \), \(\lambda _i(\cdot ),\; i=1,2\), have the following limit
$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = - \frac{3}{\mu _1}. \end{aligned}$$
In Fig. 7 we plot the two functions \(\lambda _i(\cdot ),\; i=1,2\) for \(n=2\) demes and \(M=1\) (left), respectively \(M=0.1\) (right). The dashed line indicates the common asymptotic value \(- \frac{3}{\mu _1}\). We can also show, using Proposition 8, that \(\displaystyle \lim _{M \rightarrow \infty } \lambda _i(t) = 2\) for every \(t > 0\) and \(i=1,2\). This corresponds to the dotted line in the left panel of Fig. 7.
Proof
Using Proposition 8 we get
$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = - \frac{3(A(\mu _1)(i,3) + A(\mu _1)(i,4))}{3A(\mu _1)(i,1) + A(\mu _1)(i,2)}. \end{aligned}$$
Then the relations \(3 A(\mu _1)(i,1) = \mu _1 A(\mu _1)(i,3)\) and \(A(\mu _1)(i,2) = \mu _1 A(\mu _1)(i,4)\) allow to prove the result. \(\square \)
The conditional cumulative distribution functions \(\mathbb {P}\left( T_2^{(3),\lambda _i} \le \cdot | T_3^{(3),\lambda _i} = t\right) \) and \(\mathbb {P}_i(T_2^{(3),2,M} \le \cdot | T_3^{(3),2,M} = t) \) are given, for every \(t > 0\), by formulas analogous to (11) and (18):
$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),2,M} \le u | T_3^{(3),2,M} = t\right)&= \frac{F_{2,s}(u)\frac{d}{dt} P_t(i,3)+F_{2,d}(u)\frac{d}{dt} P_t(i,4)}{f_{T_3^{(3),2,M},i}(t)} \nonumber \\&= \frac{3F_{2,s}(u)P_t(i,1)+F_{2,d}(u)P_t(i,2)}{3P_t(i,1)+P_t(i,2)}, \end{aligned}$$
(27)
$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right)&= 1- \exp \{-(\varLambda _i(t+u) - \varLambda _i(t))\}\nonumber \\&= 1 - \left( \frac{1 - P_{t+u}(i,3)-P_{t+u}(i,4)}{1 - P_{t}(i,3)-P_{t}(i,4)}\right) ^{1/3}. \end{aligned}$$
(28)
In order to compare, for \(u, t > 0\), the conditional cumulative distribution functions given in Eqs. (27) and (28), let us introduce the functions
$$\begin{aligned} g_i(u,t) := \mathbb {P}_i\left( T_2^{(3),2,M} \le u | T_3^{(3),2,M} = t \right) -\mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right) , \; i=1,2. \end{aligned}$$
Proposition 10
The functions \(g_i(u,t),\; i=1,2\) have the following asymptotic behaviour:
-
1.
For (u, t) in the neighborhood of (0, 0), we have
$$\begin{aligned} g_1(u,t)= & {} \displaystyle \frac{M}{2} u t + o\left( u^2+t^2\right) , \\ g_2(u,t)= & {} \displaystyle -\frac{u}{3} + \frac{6 M + 1}{18} u^2 + \frac{7 M}{6} u t + o\left( u^2+t^2\right) . \end{aligned}$$
-
2.
For fixed \(t > 0\), when \(u \rightarrow +\infty \),
$$\begin{aligned} g_i(u,t) = - K_{1,i} (M,t) e^{-\beta u} + o(e^{-\beta u}),\; i=1,2, \end{aligned}$$
where \(K_{1,i}(M,t) > 0\) is given by
$$\begin{aligned} K_{1,i}(M,t) = \displaystyle \frac{3 P_t(i,1)}{3P_t(i,1) + P_t(i,2)} \frac{1-a}{\beta } - \displaystyle \frac{P_t(i,2)}{3P_t(i,1) + P_t(i,2)} \frac{c}{\beta }, \end{aligned}$$
where constants \(\beta , a, c\) are defined in Eqs. (9) and (10) with \(n=2\), i.e.
$$\begin{aligned}&\beta = M + \frac{1}{2}\left( 1 - \left( 4M^2+1\right) ^{1/2}\right) ,\quad a = \frac{1}{2} \left( 1 + \left( 4M^2+1\right) ^{-1/2}\right) ,\\&\quad c = -M \left( 4M^2+1\right) ^{-1/2}. \end{aligned}$$
-
3.
For fixed \(u \ge 0\), we have
$$\begin{aligned} \displaystyle \lim _{t \rightarrow +\infty } g_i(u,t) = -K_3(M,u),\quad i=1,2, \end{aligned}$$
where \(K_3(M,u) > 0\) is given by
$$\begin{aligned} K_3(M,u) = c_1 e^{-\beta u} + c_2 e^{-\alpha u} - e^{\frac{\mu _1}{3} u}, \end{aligned}$$
with
$$\begin{aligned} \alpha= & {} M + \frac{1}{2}\left( 1 + \left( 4M^2+1\right) ^{1/2}\right) ,\quad \beta = M + \frac{1}{2}\left( 1 - \left( 4M^2+1\right) ^{1/2}\right) ,\; \\ \phi \left( \mu _1\right)= & {} \displaystyle \frac{3M}{2\left( \mu _1 + 3M + 3\right) },\quad c_1 = \frac{\phi \left( \mu _1\right) - \alpha }{\beta -\alpha },\quad c_2 = \frac{\beta - \phi \left( \mu _1\right) }{\beta -\alpha }. \end{aligned}$$
The proof is similar to the one given in the case \(n > 2\).
Most of the calculations of “Appendices A and B” section were made and/or verified using the computer algebra system Maple: programs and tracks of their execution are available from the authors.