Skip to main content

Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes

Abstract

The increasing amount of genomic data currently available is expanding the horizons of population genetics inference. A wide range of methods have been published allowing to detect and date major changes in population size during the history of species. At the same time, there has been an increasing recognition that population structure can generate genetic data similar to those generated under models of population size change. Recently, Mazet et al. (Heredity 116(4):362–371, 2016) introduced the idea that, for any model of population structure, it is always possible to find a panmictic model with a particular function of population size-change having an identical distribution of \(T_{2}\) (the time of the first coalescence for a sample of size two). This implies that there is an identifiability problem between a panmictic and a structured model when we base our analysis only on \(T_2\). In this paper, based on an analytical study of the rate matrix of the ancestral lineage process, we obtain new theoretical results about the joint distribution of the coalescence times \((T_3,T_2)\) for a sample of three haploid genes in a n-island model with constant size. Even if, for any \(k \ge 2\), it is always possible to find a size-change scenario for a panmictic population such that the marginal distribution of \(T_k\) is exactly the same as in a n-island model with constant population size, we show that the joint distribution of the coalescence times \((T_3,T_2)\) for a sample of three genes contains enough information to distinguish between a panmictic population and a n-island model of constant size.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  • Beaumont MA (1999) Detecting population expansion and decline using microsatellites. Genetics 153(4):2013–2029

    Google Scholar 

  • Beaumont MA (2004) Recent developments in genetic data analysis: what can they tell us about human demographic history? Heredity 92(5):365–379

    Article  Google Scholar 

  • Bhaskar A, Song YS (2014) Descartes’ rule of signs and the identifiability of population demographic models from genomic variation data. Ann Stat 42(6):2469–2493

    MathSciNet  Article  MATH  Google Scholar 

  • Chikhi L, Sousa VC, Luisi P, Goossens B, Beaumont MA (2010) The confounding effects of population structure, genetic diversity and the sampling scheme on the detection and quantification of population size changes. Genetics 186(3):983–995

    Article  Google Scholar 

  • Chikhi L, Rodríguez W, Grusea S, Santos P, Boitard S, Mazet O (2018) The IICR (inverse instantaneous coalescence rate) as a summary of genomic diversity: insights into demographic inference and model choice. Heredity 120:13–24

    Article  Google Scholar 

  • Goossens B, Chikhi L, Ancrenaz M, Lackman-Ancrenaz I, Andau P, Bruford MW et al (2006) Genetic signature of anthropogenic population collapse in orang-utans. PLoS Biol 4(2):285

    Article  Google Scholar 

  • Griffiths RC, Tavaré S (1994) Sampling theory for neutral alleles in a varying environment. Philos Trans R Soc B Biol Sci 344(1310):403–410

    Article  Google Scholar 

  • Griffiths RC, Tavaré S (1998) The age of a mutation in a general coalescent tree. Stoch Models 14(1–2):273–295

    MathSciNet  Article  MATH  Google Scholar 

  • Heller R, Chikhi L, Siegismund HR (2013) The confounding effect of population structure on Bayesian skyline plot inferences of demographic history. PLoS ONE 8(5):e62992

    Article  Google Scholar 

  • Herbots HM (1994) Stochastic models in population genetics: genealogy and genetic differentiation in structured populations. PhD thesis, University of London. https://qmro.qmul.ac.uk/xmlui/handle/123456789/1482?show=full

  • Hudson RR (1983) Properties of a neutral allele model with intragenic recombination. Theor Popul Biol 23(2):183–201

    Article  MATH  Google Scholar 

  • Hudson RR (2002) Generating samples under a Wright–Fisher neutral model of genetic variation. Bioinformatics 18(2):337–338

    MathSciNet  Article  Google Scholar 

  • Hudson RR et al (1990) Gene genealogies and the coalescent process. Oxf Surv Evol Biol 7(1):44

    Google Scholar 

  • Kim J, Mossel E, Rácz MZ, Ross N (2015) Can one hear the shape of a population history? Theor Popul Biol 100:26–38 ISSN 0040-5809

    Article  MATH  Google Scholar 

  • Kingman JFC (1982) The coalescent. Stoch Process Appl 13(3):235–248

    MathSciNet  Article  MATH  Google Scholar 

  • Lang S (2002) Algebra, rev. 3 edn. Springer, Berlin

    Book  MATH  Google Scholar 

  • Lapierre M, Lambert A, Achaz G (2017) Accuracy of demographic inferences from the site frequency spectrum: the case of the Yoruba population. Genetics 206(1):439–449

    Article  Google Scholar 

  • Mazet O, Rodríguez W, Chikhi L (2015) Demographic inference using genetic data from a single individual: separating population size variation from population structure. Theor Popul Biol 104:46–58

    Article  MATH  Google Scholar 

  • Mazet O, Rodríguez W, Grusea S, Boitard S, Chikhi L (2016) On the importance of being structured: instantaneous coalescence rates and human evolution-lessons for ancestral population size inference? Heredity 116(4):362–371

    Article  Google Scholar 

  • Myers S, Fefferman C, Patterson N (2008) Can one learn history from the allelic spectrum? Theor Popul Biol 73(3):342–348

    Article  MATH  Google Scholar 

  • Nei M, Takahata N (1993) Effective population size, genetic diversity, and coalescence time in subdivided populations. J Mol Evol 37(3):240–244

    Article  Google Scholar 

  • Nickalls RWD (1993) A new approach to solving the cubic: Cardan’s solution revealed. Math Gazette 77:354–359. http://www.nickalls.org/dick/papers/maths/cubic1993.pdf

  • Nielsen R, Wakeley J (2001) Distinguishing migration from isolation: a Markov Chain Monte Carlo approach. Genetics 158(2):885–896

    Google Scholar 

  • Norris JR (1998) Markov Chains. Cambridge series in statistical and probabilistic mathematics. Cambridge University Press, Cambridge

    Google Scholar 

  • Notohara M (1990) The coalescent and the genealogical process in geographically structured population. J Math Biol 29(1):59–75

    MathSciNet  Article  MATH  Google Scholar 

  • Peter BM, Wegmann D, Excoffier L (2010) Distinguishing between population bottleneck and population subdivision by a Bayesian model choice procedure. Mol Ecol 19(21):4648–4660

    Article  Google Scholar 

  • Quéméré E, Amelot X, Pierson J, Crouau-Roy B, Chikhi L (2012) Genetic data suggest a natural prehuman origin of open habitats in northern Madagascar and question the deforestation narrative in this region. Proc Natl Acad Sci USA 109(32):13028–13033

    Article  Google Scholar 

  • Rodríguez W (2016) Estimation de l’histoire démographique des populations à partir de génomes entièrement séquencés. PhD thesis, University Paul Sabatier, Toulouse, June 2016

  • Rogers AR, Harpending H (1992) Population growth makes waves in the distribution of pairwise genetic differences. Mol Biol Evol 9(3):552–569

    Google Scholar 

  • Schiffels S, Durbin R (2014) Inferring human population size and separation history from multiple genome sequences. Nat Genet 46(8):919–925

    Article  Google Scholar 

  • Slatkin M (1991) Inbreeding coefficients and coalescence times. Genet Res 58(02):167–175

    Article  Google Scholar 

  • Storz JF, Beaumont MA (2002) Testing for genetic evidence of population expansion and contraction: an empirical analysis of microsatellite DNA variation using a hierarchical Bayesian model. Evolution 56(1):154–166

    Article  Google Scholar 

  • Tajima F (1983) Evolutionary relationship of DNA sequences in finite populations. Genetics 105:437–460

    Google Scholar 

  • Terhorst J, Song YS (2015) Fundamental limits on the accuracy of demographic inference based on the sample frequency spectrum. Proc Natl Acad Sci USA 112(25):7677–7682

    Article  Google Scholar 

  • Vallander SS (1973) Calculation of the Wasserstein distance between probability distributions on the line. Theory Probab Appl 18:784–786

    Article  MATH  Google Scholar 

  • Wakeley J (1999) Nonequilibrium migration in human history. Genetics 153(4):1863–1871

    Google Scholar 

  • Weissman DB, Hallatschek O (2017) Minimal-assumption inference from population-genomic data. eLife 6:e24836

    Article  Google Scholar 

  • Wilkinson-Herbots HM (1998) Genealogy and subpopulation differentiation under various models of population structure. J Math Biol 37(6):535–585

    MathSciNet  Article  MATH  Google Scholar 

  • Wright S (1931) Evolution in mendelian populations. Genetics 16(2):97

    Google Scholar 

Download references

Acknowledgements

The authors wish to thank Josué M. Corujo Rodríguez for very interesting discussions in preparing this article. The authors also thank the anonymous reviewers for their reading and for valuable suggestions. This research was funded through the 2015–2016 BiodivERsA COFUND call for research proposals, with the national funders ANR (ANR-16-EBI3-0014), FCT (Biodiversa/0003/2015) and PT-DLR (01LC1617A), under the INFRAGECO (Inference, Fragmentation, Genomics, and Conservation) Project (https://infrageco-biodiversa.org/). The research was also supported by the LABEX entitled TULIP (ANR-10-LABX-41), as well as the Pôle de Recherche et d’Enseignement Suprieur (PRES) and the Région Midi-Pyrénées, France. We finally thank the LIA BEEG-B (Laboratoire International Associé—Bioinformatics, Ecology, Evolution, Genomics and Behaviour) (CNRS) and the PESSOA program for facilitating travel and collaboration between EDB, IMT and INSA in Toulouse and the IGC, in Portugal.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Simona Grusea.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Appendices

Appendix A Proofs for Sections 2 and 3

Proof of Lemma 1

We first observe that

$$\begin{aligned} - 2 - \displaystyle \frac{nM}{n-1}< -\frac{3}{n} < 0. \end{aligned}$$

Then the calculus gives

$$\begin{aligned} p(0)= & {} \displaystyle \frac{9M(Mn+ 2n -2)}{2(n-1)^2}> 0, \\ p\left( -\frac{3}{n}\right)= & {} - \displaystyle \frac{9(M n^2 + 2(n-1)(n-3))}{2 n^3} < 0, \\ p\left( - 2 - \displaystyle \frac{nM}{n-1} \right)= & {} \frac{(Mn + 2n-2)(M(n-3) + 2n-2)}{2 (n-1)^2} > 0 \\ \lim _{\mu \rightarrow -\infty } p(\mu )= & {} - \infty . \end{aligned}$$

The intermediate value theorem applied to \(p(\mu )\) thus provides a proof of the result. \(\square \)

Proof of Proposition 2

In order to simplify computations, we make the following change of parameters:

$$\begin{aligned} M = 2 N, \; n = \frac{a + N}{a}. \end{aligned}$$

The new parameters a et N verify \(a > 0\) and \(N > 0\) and the matrix Q becomes

$$\begin{aligned} Q= \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0&{}3&{}0\\ a &{}-1+a-2\,N&{}-2\,a+2\,N&{}0&{}1\\ 0&{}6\,a&{}-6\,a&{}0&{}0\\ 0&{}0&{}0&{}0&{}0\\ 0&{}0&{}0&{}0&{}0 \end{array} \right] . \end{aligned}$$

The matrix Q has the double eigenvalue 0 and it is easy to check that the corresponding eigenspace is generated by the following non colinear vectors:

$$\begin{aligned} V_1 = [N,N-1,N-1,2 N, N -a -1]^T, \quad V_2 = [1,1,1,1,1]^T. \end{aligned}$$

We then consider the change of basis matrix

$$\begin{aligned} P_1 = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 1&{}0&{}0&{}N&{}1\\ 0&{}\quad 1&{}\quad 0&{}\quad N-1&{}\quad 1\\ 0&{}\quad 0&{}\quad 1&{}\quad N-1&{}\quad 1\\ 0&{}\quad 0&{}\quad 0&{}\quad 2\,N&{}\quad 1 \\ 0&{}\quad 0&{}\quad 0&{}\quad N-a-1&{}\quad 1\end{array} \right] , \end{aligned}$$

whose inverse is

$$\begin{aligned} P_1^{-1} = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 1&{}\quad 0&{}\quad 0&{}\quad -{\frac{a+1}{N+a+1}}&{}\quad -{\frac{N}{ N+a+1}}\\ 0&{}\quad 1&{}\quad 0&{}\quad -{\frac{a}{N+a+1}}&{}-\quad {\frac{N+1}{N+ a+1}}\\ 0&{}\quad 0&{}\quad 1&{}\quad -{\frac{a}{N+a+1}}\quad &{}-{\frac{N+1}{N+a+ 1}}\\ 0&{}\quad 0&{}\quad 0&{}\quad \frac{1}{N+a+1} &{}\quad - \frac{1}{N+a+1}\\ 0&{}\quad 0&{}\quad 0&{}\quad -{\frac{N-a-1}{N+a+1}}&{}\quad {\frac{2N}{N+a+1}}\end{array} \right] . \end{aligned}$$

We have thus obtained a partial diagonalization (by blocks) of the matrix Q:

$$\begin{aligned} Q_2 = P_1^{-1} Q P_1 = \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0&{}0&{}0\\ a &{}-1+a-2\,N&{}-2\,a+2\,N&{}0&{}0\\ 0&{}6\,a&{}-6\,a&{}0&{}0 \\ 0&{}0&{}0&{}0&{}0\\ 0&{}0&{}0&{}0&{}0 \end{array} \right] , \end{aligned}$$
(24)

and we put

$$\begin{aligned} R := \left[ \begin{array}{c@{\quad }c@{\quad }c} -3-3\,N&{}3\,N&{}0\\ a&{}-1+a- 2\,N&{}-2\,a+2\,N\\ 0&{}6\,a&{}-6\,a\end{array} \right] . \end{aligned}$$

The characteristic polynomial of R is the polynomial \(p(\mu )\), which has the following expressions using the new parameters:

$$\begin{aligned} p(\mu ) = \mu ^3 + (5N+5a+4) \mu ^2 + 3\left( 2 N^2 +4 aN + 3N + 2a^2 + 7a+1\right) \mu + 18 a (N + a +1), \end{aligned}$$

which has three strictly negative real roots. These roots are all distinct by Lemma 1.

If \(\mu \) is an eigenvalue of R, we then determine a corresponding eigenvector \(W(\mu )\) by solving the equation \(R W(\mu ) = \mu W(\mu )\).

The computations show that we can choose

$$\begin{aligned} W(\mu ) = \left[ \begin{array}{c} \mu ^2 + (2N + 5a+1) \mu + 6a(a+1) \\ a(\mu + 6a) \\ 6 a^2 \end{array} \right] . \end{aligned}$$

We then consider the \(3 \times 3\) passage matrix \(P_2\), whose column vectors are the \(W(\mu _i), \, i=1,2,3\), where the \(\mu _i\) are the three eigenvalues of R:

$$\begin{aligned} P_2 = \left[ W(\mu _1), \; W(\mu _2), \; W(\mu _3) \right] . \end{aligned}$$

Some easy computations on the rows show that the determinant of \(P_2\) is a Van der Monde determinant, and hence: \(\mathrm {det}(P_2) = (\mu _1 - \mu _2)(\mu _1-\mu _3)(\mu _2 - \mu _3) \ne 0\).

The computations of the inverse \(P_2^{-1}\) gives

$$\begin{aligned} P_2^{-1} = \left[ Z(\mu _1), Z(\mu _2), Z(\mu _3)\right] ^T, \end{aligned}$$

where

$$\begin{aligned} Z(\mu ) = \frac{1}{p'(\mu )} \; \left[ \begin{array}{c} 1 \\ \displaystyle \frac{\mu + 3N+3}{a} \\ -\displaystyle \frac{\mu ^2 + (3N+a+4)\mu + 3(N+a+1)}{a \mu } \end{array}\right] , \end{aligned}$$

with

$$\begin{aligned} p'(\mu ) = 3 \mu ^2 + 2(5 N + 5 a +4)\mu + 3\left( 2 N^2 +4 aN + 3N + 2a^2 + 7a+1\right) . \end{aligned}$$

We further obtain

$$\begin{aligned} e^{t R}= & {} P_2 \; \left[ \begin{array}{c@{\quad }c@{\quad }c} e^{\mu _1 t} &{} 0 &{} 0\\ 0 &{} e^{\mu _2 t} &{}0 \\ 0 &{} 0 &{} e^{\mu _3 t} \end{array} \right] \; P_2^{-1}, \\= & {} \sum _{j=1}^3 e^{\mu _j} \; W(\mu _j) Z(\mu _j)^T. \end{aligned}$$

We then introduce the matrices \(A(\mu )\) and B defined by

$$\begin{aligned} A(\mu ) := P_1 \; \bar{W}(\mu ) \bar{Z}(\mu ) ^T \; P_1^{-1}, \end{aligned}$$

where \(\bar{W}_j(\mu )\) (resp. \(\bar{Z}(\mu )\)) is a vector of length 5 obtained by adding two null coordinates to \(W(\mu )\) (resp. \(Z(\mu )\)), and

$$\begin{aligned} B = P_1 \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 1 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 0 &{}\quad 1 \end{array} \right] \; P_1^{-1}. \end{aligned}$$

Emphasizing the rank one property of matrix \(A(\mu )\), let us define the vectors

$$\begin{aligned} E_1 = \left[ \begin{array}{c} \displaystyle \frac{3 M}{v} \\ 1 \\ \displaystyle \frac{3 M}{u} \\ 0 \\ 0 \end{array} \right] , \quad E_2 = \left[ \begin{array}{c} M u \\ (n-1) u v \\ (n-1)(n-2) M v \\ \displaystyle \frac{3 M u}{\mu } \\ \displaystyle \frac{(n-1) uv}{\mu } \end{array} \right] , \end{aligned}$$

where \(\mu \) is an eigenvalue of R, and uv are functions of \(\mu \) defined by

$$\begin{aligned} u = (n-1)\mu + 3 M,\quad v = 2 \mu + 3(M+2). \end{aligned}$$

Note that, since

$$\begin{aligned} p\left( \displaystyle -\frac{3M}{n-1}\right)= & {} - \displaystyle \frac{9}{2} \cdot \frac{(n-2) [2(n-1) + (n-3)M] M^2}{(n-1)^3} <0,\\ p\left( \displaystyle -\frac{3(M+2)}{2}\right)= & {} \displaystyle \frac{9}{8} \cdot \frac{ [2(n-1) + (n-3)M] M^2}{(n-1)^2} >0, \end{aligned}$$

\(\displaystyle -\frac{3M}{n-1}\) and \(\displaystyle -\frac{3(M+2)}{2}\) cannot be eigenvalues and thus \(u \ne 0\) et \(v \ne 0\).

With \(\delta (M,n,\mu ) = 2(n-1)^2 p'(\mu )\), we have

$$\begin{aligned} A(\mu ) = \displaystyle \frac{1}{\delta (M,n,\mu )} \; E_1 E_2^T, \end{aligned}$$
(25)

which gives the expression of \(A(\mu )\) in Proposition 2.

The stated result easily follows. \(\square \)

Proof of Proposition 4

Using Eq. (15) and Proposition 2, we deduce that

$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = -3 \ \frac{A(\mu _1)(i,4)+A(\mu _1)(i,5)}{3A(\mu _1)(i,1)+A(\mu _1)(i,2)}, \quad i=1,2,3. \end{aligned}$$

Note that the matrix \(E_1 E_2^T\) introduced in the proof of Proposition 2 can also be factorized in a different manner. If we define the \(3 \times 3\) matrix C, the \(5 \times 3\) matrix \(D_1\) and the \(3 \times 5\) matrix \(D_2\) by

$$\begin{aligned} C:= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c} \displaystyle \frac{u}{v} &{} u &{} 1 \\ u &{} uv &{} v \\ 1 &{} v &{} \displaystyle \frac{v}{u} \end{array} \right] ,\quad D_1 := \left[ \begin{array}{c@{\quad }c@{\quad }c} 3 M &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 3M \\ 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 \end{array} \right] ,\\ D_2:= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} M &{} 0 &{} 0 &{} \displaystyle \frac{3M}{\mu } &{} 0 \\ 0 &{} n-1 &{} 0 &{} 0 &{} \displaystyle \frac{n-1}{\mu } \\ 0 &{} 0 &{} (n-1)(n-2) &{} 0 &{} 0 \end{array} \right] , \end{aligned}$$

we may check that, for any value of \(\mu \), we have \(E_1 E_2^T = D_1 C D_2\).

The vectors \(V_1\) and \(V_2\), defined by

$$\begin{aligned} V_1 := \left[ \begin{array}{c} 3 \\ 0 \\ 0 \\ vp \\ 0 \end{array} \right] , \;\; V_2 := \left[ \begin{array}{c} 0 \\ 1 \\ 0 \\ 0 \\ -\mu \end{array} \right] , \end{aligned}$$

verify \(D_2 V_j = 0\) for \(j=1,2\), and hence, using Eq. (25), \(A(\mu )V_j = 0\), for \(j=1,2\).

Therefore, for \(i=1,2,3\),

$$\begin{aligned} 3A(\mu )(i,1) - \mu A(\mu )(i,4) = 0, \; A(\mu )(i,2) - \mu A(\mu )(i,5) = 0, \end{aligned}$$

and the result follows. \(\square \)

Proof of Lemma 2

Note that \(-3\alpha \) and \(-3\beta \), with \(\alpha , \beta \) defined in Eq. (10), are the roots of the polynomial \(q_1(X)= q(X/3)\), where

$$\begin{aligned} q(X) = X^2 + \left( 1 + \frac{nM}{n-1}\right) X + \frac{M}{n-1}, \end{aligned}$$

and on the open subset \(D = \left\{ (n,M):\; n> 2, \; M>0 \right\} \) of \(\mathbb {R}^2\) the polynomials p(X) and \(q_1(X)\) have a common root if and only their resultant \(R(n,M) = \text{ Res }(p(X),q_1(X),X)\) with respect to X is null [see e.g. Lang (2002, Chapter IV–8)].

Because \(R(n,M) = - \frac{M^3}{9(n-1)^2} < 0\) on D, and because the roots \(-3 \alpha \) and \(\mu _3\) are continuous functions of (nM) on D, the inequality \(-3 \alpha < \mu _3\) for one value of \((n,M) \in D\) implies the inequality everywhere on D. The same is true for the inequality \(\mu _1 < -3 \beta \).

This achieves the proof of the lemma. \(\square \)

Proof of Proposition 5

We will only give the proof of (i), which corresponds to the case of three genes sampled from the same deme. The proofs ot the analogous results (ii) and (iii), corresponding to the two other sample schemes, are similar.

When \(t \rightarrow 0\), using the well-known relation \(P_t = I_5 + tQ + o(t)\), where \(I_5\) denotes the identity matrix, we have the following Taylor expansions:

$$\begin{aligned} P_t(1,1)&= 1 + \left( -\frac{3M}{2}-3\right) t + o(t),\\ P_t(1,2)&= \frac{3M}{2} t + o(t). \end{aligned}$$

In particular, we have

$$\begin{aligned} \frac{1}{3P_t(1,1)+P_t(1,2)} = \frac{1}{1-(M+3)t +o(t)} = 1+(M+3)t+o(t). \end{aligned}$$

Using Eqs. (7)–(11), and a Taylor expansion of order 2 of the exponential in the neighborhood of 0, we easily obtain

$$\begin{aligned} F_{2,s}(u)&= u - \frac{M+1}{2}u^2 + o\left( u^2\right) ,\\ F_{2,d}(u)&= \frac{M}{2(n-1)}u^2 + o\left( u^2\right) . \end{aligned}$$

When substituting the above expressions into Eq. (11), straightforward calculations give

$$\begin{aligned} \mathbb {P}_1\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = u - \frac{M}{2}ut - \frac{M+1}{2}u^2 + o\left( u^2 + t^2\right) . \end{aligned}$$

Further, using Eq. (18), let us denote

$$\begin{aligned} h(u,t):=\mathbb {P}\left( T_2^{(3),\lambda _1} \le u | T_3^{(3),\lambda _1} = t\right) =1 - \left( \frac{1 - P_{t+u}(1,4)-P_{t+u}(1,5)}{1 - P_{t}(1,4)-P_{t}(1,5)}\right) ^{1/3}. \end{aligned}$$

We will write a Taylor expansion of order 2 of h(ut) in the neighborhood of (0, 0). We have \(h(0,0)=0\) and direct computations give

$$\begin{aligned} \frac{\partial h}{\partial u}(u,t)&= \frac{1}{3} \times \left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) ^{-1/3} \times \left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) ^{-2/3}\\&\quad \times \left( P_{t+u}Q(1,4)+P_{t+u}Q(1,5) \right) ,\\ \frac{\partial h}{\partial t}(u,t)&= \frac{\left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) ^{-2/3}}{3\left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) ^{4/3}} \\&\quad \times \{ \left( P_{t+u}Q(1,4)+P_{t+u}Q(1,5) \right) \left( 1 - P_{t}(1,4)-P_{t}(1,5) \right) \\&\quad - \left( P_{t}Q(1,4)+P_{t}Q(1,5) \right) \left( 1 - P_{t+u}(1,4)-P_{t+u}(1,5) \right) \}. \end{aligned}$$

Using the fact that \(P_0 = I_5,\) and \(Q(1,4)=3, Q(1,5)=0\), we obtain

$$\begin{aligned} \frac{\partial h}{\partial u}(0,0)=1, \frac{\partial h}{\partial t}(0,0)=0. \end{aligned}$$

We further compute the second partial derivatives of h in (0, 0). With \(Q^2\) being the square matrix of the rate matrix Q, and using the fact that \(Q^2(1,4) = -9\left( \frac{M}{2}+1\right) \) and \(Q^2(1,5) = \frac{3M}{2},\) we easily derive

$$\begin{aligned} \frac{\partial ^2 h}{\partial u \partial t}(0,0)&= \frac{1}{3} \left\{ \left( Q(1,4)+Q(1,5)\right) ^2 + Q^2(1,4)+Q^2(1,5) \right\} = -M,\\ \frac{\partial ^2 h}{\partial u^2}(0,0)&= \frac{1}{3} \left\{ \frac{2}{3}\left( Q(1,4)+Q(1,5)\right) ^2 + Q^2(1,4)+Q^2(1,5) \right\} = -(M+1),\\ \frac{\partial ^2 h}{\partial t^2}(0,0)&= 0, \end{aligned}$$

The Taylor expansion of order 2 of h(ut) near (0, 0) finally gives

$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _1} \le u | T_3^{(3),\lambda _1} = t\right) = u - Mut - \frac{M+1}{2}u^2 + o\left( u^2 + t^2\right) , \end{aligned}$$

which finishes the proof. \(\square \)

Proof of Proposition 6

Because \(0< \beta < \alpha \), using Eqs. (7) and (8), we get

$$\begin{aligned} F_{2,s}(u) = 1 - \frac{a}{\alpha } e^{-\alpha u} - \frac{1-a}{\beta } e^{-\beta u} = 1 - \frac{1-a}{\beta } e^{-\beta u} + o\left( e^{-\beta u}\right) \end{aligned}$$

and

$$\begin{aligned} F_{2,d}(u) = 1 - \frac{c}{\alpha } e^{-\alpha u} + \frac{c}{\beta } e^{-\beta u} = 1 + \frac{c}{\beta } e^{-\beta u} + o\left( e^{-\beta u}\right) . \end{aligned}$$

Therefore, using Eq. (11), we obtain

$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = 1 - K_{1,i}(n,M,t) e^{-\beta u} + o\left( e^{-\beta u}\right) , \end{aligned}$$

where \(K_{1,i}(n,M,t)\), given by (20), is strictly positive because \(0< a < 1\) and \(c < 0\).

Using Proposition 2 we get

$$\begin{aligned} \sum _{j=1}^3 P_{t+u}(i,j) = \left( A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1(t+u)} + o\left( e^{\mu _1(t+u)}\right) . \end{aligned}$$

Therefore, using Eq. (18) and relation \(1 - P_t(i,4) - P_t(i,5) = \sum _{j=1}^3 P_t(i,j)\), we obtain

$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right) = 1 - K_{2,i}(n,M,t)\; e^{\frac{\mu _1}{3} u} + o\left( e^{\frac{\mu _1}{3} u}\right) , \end{aligned}$$

where

$$\begin{aligned} K_{2,i}(n,M,t) :=\left( \frac{A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)}{P_t(i,1) + P_t(i,2) + P_t(i,3)} \right) ^{\frac{1}{3}} \; e^{\frac{\mu _1}{3} t}. \end{aligned}$$

Because \(\frac{\mu _1}{3} < - \beta \) from Lemma 2, we have \( \displaystyle e^{\frac{\mu _1}{3} u} = o(e^{-\beta u}), \) which achieves the proof of (19). \(\square \)

Proof of Proposition 7

We will first prove a useful lemma. \(\square \)

Lemma 3

Let us define \(\phi (\mu )\) by

$$\begin{aligned} \phi (\mu ) := \displaystyle \frac{3 A(\mu )(1,1)}{3 A(\mu )(1,1) + A(\mu )(1,2)}, \end{aligned}$$

where the matrix \(A(\mu )\) is given in Proposition 2.

Then,

  1. (i)

    \(0< \phi (\mu _i)< - \frac{\mu _i}{3} < \alpha , \; i=1,2,3.\)

  2. (ii)

    Defining h(n) by

    $$\begin{aligned} h(n) = \frac{(n-1)\left( 25 - 9n + 5 \sqrt{9n^2 - 18n + 25}\right) }{12 n^2}, \end{aligned}$$

    we have the following results:

    1. (a)

      If \(M > h(n)\), then \(\phi (\mu _1)< \beta< \phi (\mu _2)< \phi (\mu _3) < \alpha .\)

    2. (b)

      If \(M = h(n)\), then \(\phi (\mu _1)< \beta = \phi (\mu _2)< \phi (\mu _3) < \alpha .\)

    3. (c)

      If \(M < h(n)\), then \(\phi (\mu _1)< \phi (\mu _2)< \beta< \phi (\mu _3) < \alpha .\)

Proof

From Proposition 2 and using \(p(\mu _i)=0\), we get

$$\begin{aligned} \phi (\mu _i)= & {} \displaystyle \frac{1}{3} \frac{(n-1)\mu _i^2 + (nM + n-1)\mu _i + 6M}{Mn + 2(n-1)} \\= & {} \displaystyle \frac{3(n-1)}{nM + 2(n-1)} \; q\left( \frac{\mu _i}{3}\right) - \displaystyle \frac{M(\mu _i n + 3)}{nM + 2(n-1)} - \displaystyle \frac{\mu _i}{3}. \end{aligned}$$

From Lemma 1, it follows that \(\mu n + 3 >0\). Because \( -\alpha< \displaystyle \frac{\mu _i}{3} < -\beta \) from Lemma 2, we get \(q\left( \frac{\mu _i}{3}\right) < 0\), that proves \(\phi (\mu _i) < - \frac{\mu _i}{3}\).

Now, using again that \(p(\mu _i)=0\), we obtain the following expression for \(\phi (\mu _i)\)

$$\begin{aligned} \phi (\mu _i) = \frac{3 M}{2(n-1)\mu _i + 3nM + 6(n-1)}. \end{aligned}$$
(26)

Introducing the new variable \(\nu = \displaystyle \frac{3 M}{2(n-1)\mu + 3nM + 6(n-1)}\), and thus \(\mu = \displaystyle - \frac{3}{2} \frac{(nM + 2n-2))\nu - M}{(n-1) \nu }\), we get that \(\phi (\mu _i)\), for \(i=1,2,3\) are the three real roots of the polynomial

$$\begin{aligned} r(\nu )= & {} 4(n-1)\left( nM + 2n-2\right) \nu ^3 - \left[ n^2 M^2 + 2(n-1)(3n+4) M + 8(n-1)^2 \right] \nu ^2 \\&+ 2 M (2 nM + 5n-5) \nu - 3M^2. \end{aligned}$$

The coefficient signs of r(X) show that r(X) has no negative roots, implying that \(\phi (\mu _i) >0, \; i=1,2,3\), which achieves the proof of (i).

The set of (nM) such that r(X) and \(q(-X)\) have a common root is obtained by computing their resultant with respect to X, denoted by R(nM):

$$\begin{aligned} R(n,M) = \text{ Res }(r(X),q(-X),X) = M^3 \left[ 6n^2 M^2 + (n-1)(9n-25) M - 6(n-1)^2 \right] . \end{aligned}$$

In the domain \(D = \{(n,M),\; n>2, \; M>0 \}\) the curve \(6n^2 M^2 + (n-1)(9n-25) M - 6(n-1)^2 = 0\) is identical to the graph of the function \(M = h(n), \; n > 2\).

The set \(D {\setminus } \{(n,h(n)): \; n > 2 \}\) has two connex open components in which the relative position of \(\beta , \alpha \) and \(\phi (\mu _i), \; i=1,2,3\) are the same.

In the component \(D_1 := \{(n,M):\; n>2,\; M > h(n)\}\), one may choose \(n=3, \; M=\frac{4}{3}\) for which

$$\begin{aligned} \beta= & {} \displaystyle \frac{3}{2} - \displaystyle \frac{\sqrt{57}}{6} \approx 0.241694, \\ \alpha= & {} \displaystyle \frac{3}{2} + \displaystyle \frac{\sqrt{57}}{6} \approx 2.758305,\\ \phi (\mu _1)= & {} \displaystyle \frac{3 - \sqrt{5}}{4} \approx 0.190983, \\ \phi (\mu _2)= & {} \displaystyle \frac{1}{3}, \\ \phi (\mu _3)= & {} \displaystyle \frac{3 + \sqrt{5}}{4} \approx 1.3090167, \end{aligned}$$

that proves \((ii-a)\).

In the component \(D_2 := \{(n,M):\; n>2,\; M < h(n)\}\), one may choose \(n=3, \; M= \frac{1}{2} \) for which

$$\begin{aligned} \beta= & {} \displaystyle \frac{7 - \sqrt{33}}{8} \approx 0.156929, \\ \alpha= & {} \displaystyle \frac{7 + \sqrt{33}}{8} \approx 1.593070, \\ \phi (\mu _1)\approx & {} 0.104089, \\ \phi (\mu _2)\approx & {} 0.146359, \\ \phi (\mu _3)\approx & {} 1.118868, \end{aligned}$$

that proves \((ii-c)\).

The curve arc \(\{(n,h(n)): n > 2 \}\) may be parametrized by

$$\begin{aligned} n = \displaystyle \frac{3u^2 + 8u-3}{3(u^2-1)}, \quad M = \displaystyle \frac{8u}{(u+3)^2},\quad u \in ]1,3[, \end{aligned}$$

and we obtain

$$\begin{aligned} \beta= & {} \displaystyle \frac{u-1}{u+3}, \\ \alpha= & {} \displaystyle \frac{3(u+1)}{u+3}, \\ \phi (\mu _1)= & {} \displaystyle \frac{39u + 51 -3 \sqrt{-71u^2 + 442u +529}}{40(u+3)}, \\ \phi (\mu _2)= & {} \displaystyle \frac{u-1}{u+3}, \\ \phi (\mu _3)= & {} \displaystyle \frac{39u + 51 +3 \sqrt{-71u^2 + 442u +529}}{40(u+3)}, \end{aligned}$$

and \((ii-b)\) is proved. \(\square \)

Lemma 3 is now used to prove Proposition 7. First note that we have

$$\begin{aligned} \phi (\mu ) = \displaystyle \frac{3 A(\mu )(i,1)}{3 A(\mu )(i,1) + A(\mu )(i,2)}, \end{aligned}$$

for every \(i=1,2,3\).

When \(t \rightarrow + \infty \), \(P_t(i,j) = A(\mu _1)(i,j) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) ,\; j=1,2\) and thus

$$\begin{aligned}&\mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) \\&\quad = \frac{3 A(\mu _1)(i,1)}{3 A(\mu _1)(i,1) + A(\mu _1)(i,2)} F_{2,s}(u) + \frac{A(\mu _1)(i,2)}{3 A(\mu _1)(i,1) + A(\mu _1)(i,2)} F_{2,d}(u) + o(1) \\&\quad = \phi (\mu _1) F_{2,s}(u) + (1 - \phi (\mu _1)) F_{2,d}(u) + o(1). \end{aligned}$$

Using the definitions of \(F_{2,s}(u)\) and \(F_{2,d}(u)\) in Eqs. (7) and (8), we get

$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),n,M} \le u | T_3^{(3),n,M} = t\right) = 1 - c_1 e^{-\beta u} - c_2 e^{-\alpha u} + o(1), \end{aligned}$$

where \(c_1\) and \(c_2\) are given by (23).

On another hand, we have \(P_t(i,j) = A(\mu _1)(i,j) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) ,\; j=1,2,3\), we obtain

$$\begin{aligned} \sum _{j=1}^3 P_{t+u}(i,j)= & {} \left( (A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1 (t+u)} + o\left( e^{\mu _1 t}\right) ,\\ \sum _{j=1}^3 P_t(i,j)= & {} \left( (A(\mu _1)(i,1) + A(\mu _1)(i,2) + A(\mu _1)(i,3)\right) e^{\mu _1 t} + o\left( e^{\mu _1 t}\right) . \end{aligned}$$

Using the fact that \(1 - P_t(i,4) - P_t(i,5) = \sum _{j=1}^3 P_t(i,j)\), this implies

$$\begin{aligned} \mathbb {P}(T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t) = 1 - e^{ \frac{\mu _1}{3} u} + o(1), \end{aligned}$$

and thus (21) holds.

It remains to show that \(K_3(n,M,u) >0\).

Using \(c_1 + c_2 = 1\), we may write

$$\begin{aligned} K_3(n,M,u) = \frac{\beta - \phi (\mu _1)}{\beta -\alpha } \left( e^{- \alpha u} - e^{\frac{\mu _1}{3} u}\right) + \frac{\phi (\mu _1)-\alpha }{\beta -\alpha } \left( e^{ \frac{\mu _1}{3} u} - e^{-\beta u}\right) . \end{aligned}$$

From Lemma 2, \(-\alpha< \frac{\mu _1}{3} < -\beta \) and thus \(e^{-\alpha u}< e^{\frac{\mu _1}{3} u} < e^{-\beta u}\). On the other hand, from Lemma 3 we have \(\phi (\mu _1)< \beta < \alpha \). Therefore \(K_3(n,M,u) >0\). \(\square \)

Appendix B The case of \(n=2\) islands

If we now consider the ancestral lineage process for a sample of three genes in the case of a symmetrical n-island model with \(n=2\) two islands, we only have the following four possible configurations:

  1. 1.

    the three lineages are in the same island,

  2. 2.

    two lineages are in the same island and the third one is in the other island,

  3. 3.

    there are only two ancestral lineages left and they are in the same island,

  4. 4.

    there are only two ancestral lineages left and they are in different islands.

The corresponding transition rate matrix is:

$$\begin{aligned} Q = \begin{pmatrix} -\frac{3M}{2}-3 &{}\quad \frac{3M}{2} &{}\quad 3 &{}\quad 0\\ \frac{M}{2} &{}\quad -\frac{M}{2}-1 &{}\quad 0 &{}\quad 1 \\ 0 &{}\quad 0&{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 0 &{}\quad 0 &{}\quad 0 \end{pmatrix}, \end{aligned}$$

The characteristic polynomial of Q is \(\chi _Q(\mu ) = -\mu ^2 p(\mu )\), with

$$\begin{aligned} p(\mu ) = \mu ^2 + 2(M+2) \mu + 3(M+1). \end{aligned}$$

The matrix Q has the double eigenvalue 0 and the corresponding eigenspace of dimension 2 can be generated by the vectors \(\displaystyle \left[ \frac{M}{2},\frac{M}{2}-1, M, -1\right] ^T\) and \(\displaystyle [1,1,1,1]^T\). The two other eigenvalues are \(\mu _1 = - M - 2 + \sqrt{M^2+M+1}\) and \(\mu _2 = - M - 2 - \sqrt{M^2+M+1}\).

An eigenvector for \(\mu _1\) (resp. \(\mu _2\)) is \([3M, 2 \mu _1+3M+6, 0, 0]^T\) (resp. \([3M, 2 \mu _2+3M+6, 0, 0]^T\)), and we may consider the following change of basis matrix P given by

$$\begin{aligned} P = \begin{pmatrix} 3M &{}\quad 3M &{}\quad \frac{M}{2} &{}\quad 1 \\ 2 \mu _1+3M+6 &{}\quad 2 \mu _2+3M+6 &{}\quad \frac{M}{2}-1 &{}\quad 1 \\ 0 &{}\quad 0 &{}\quad M &{}\quad 1 \\ 0 &{}\quad 0 &{}\quad -1 &{}\quad 1 \end{pmatrix}. \end{aligned}$$

From

$$\begin{aligned} P^{-1} = \begin{pmatrix} \displaystyle \frac{2 \mu _1+M+2}{12M(\mu _1 + M+2)} &{} \displaystyle \frac{1}{4(\mu _1+M+2)} &{} \displaystyle \frac{2\mu _1+M+2}{4 M \mu _1(\mu _1 + M +2)} &{} \displaystyle \frac{1}{4\mu _1(\mu _1+M+2)} \\ \displaystyle \frac{2 \mu _2+M+2}{12M(\mu _2 + M+2)} &{} \displaystyle \frac{1}{4(\mu _2+M+2)} &{} \displaystyle \frac{2\mu _2+M+2}{4 M \mu _2(\mu _2 + M +2)} &{} \displaystyle \frac{1}{4\mu _2(\mu _2+M+2)} \\ 0 &{} 0 &{}\displaystyle \frac{1}{M+1} &{} - \displaystyle \frac{1}{M+1} \\ 0 &{} 0 &{}\displaystyle \frac{1}{M+1} &{}\displaystyle \frac{M}{M+1} \end{pmatrix}, \end{aligned}$$

and the equality

$$\begin{aligned} e^{t Q} = P \; \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} e^{\mu _1 t} &{} 0 &{} 0 &{} 0 \\ 0 &{} e^{\mu _2} &{} 0 &{} 0 \\ 0 &{} 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} 0 &{} 1 \end{array} \right] \; P^{-1}, \end{aligned}$$

a proof of the following proposition is obtained.

Proposition 8

The transition kernel \(P_t = e^{t Q}\) is given by

$$\begin{aligned} P_t = e^{\mu _1 t} A(\mu _1) + e^{\mu _2 t} A(\mu _2) + B, \end{aligned}$$

where

$$\begin{aligned} A(\mu )= & {} \displaystyle \frac{1}{\delta (M,\mu )} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c} 2 \mu +M+2 &{} 3 M &{} \displaystyle \frac{3(2\mu +M+2)}{\mu } &{} \displaystyle \frac{3M}{\mu } \\ M &{} 2\mu +3M+6 &{} \displaystyle \frac{3M}{\mu } &{} \displaystyle \frac{2\mu +3M+6}{\mu } \\ 0 &{} 0 &{}0 &{} 0 \\ 0 &{} 0 &{}0 &{} 0 \end{array} \right] ,\\ B= & {} \left[ \begin{array}{c@{\quad }c@{\quad }c@{\quad }c@{\quad }c} 0 &{}\quad 0 &{}\quad b &{}\quad 1 - b \\ 0 &{} \quad 0 &{}\quad 1-b &{} \quad b \\ 0 &{}\quad 0 &{}\quad 1 &{}\quad 0 \\ 0 &{}\quad 0 &{} \quad 0 &{}\quad 1 \end{array}\right] , \end{aligned}$$

with \(\delta (M,\mu ) = 4(\mu + M+2)\) and \(b= \displaystyle \frac{M+2}{2(M+1)}\).

The IICRs \(\lambda _i(\cdot )/3\), for the initial sample configurations \(i=1\) and \(i=2\), correspond to the size-change functions

$$\begin{aligned} \lambda _i(t) = \frac{3(1 - P_t(i,3) - P_t(i,4))}{3P_t(i,1) + P_t(i,2)}, \end{aligned}$$

and the following proposition is verified.

Proposition 9

When \(t \rightarrow \infty \), \(\lambda _i(\cdot ),\; i=1,2\), have the following limit

$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = - \frac{3}{\mu _1}. \end{aligned}$$

In Fig. 7 we plot the two functions \(\lambda _i(\cdot ),\; i=1,2\) for \(n=2\) demes and \(M=1\) (left), respectively \(M=0.1\) (right). The dashed line indicates the common asymptotic value \(- \frac{3}{\mu _1}\). We can also show, using Proposition 8, that \(\displaystyle \lim _{M \rightarrow \infty } \lambda _i(t) = 2\) for every \(t > 0\) and \(i=1,2\). This corresponds to the dotted line in the left panel of Fig. 7.

Proof

Using Proposition 8 we get

$$\begin{aligned} \lim _{t \rightarrow \infty } \lambda _i(t) = - \frac{3(A(\mu _1)(i,3) + A(\mu _1)(i,4))}{3A(\mu _1)(i,1) + A(\mu _1)(i,2)}. \end{aligned}$$

Then the relations \(3 A(\mu _1)(i,1) = \mu _1 A(\mu _1)(i,3)\) and \(A(\mu _1)(i,2) = \mu _1 A(\mu _1)(i,4)\) allow to prove the result. \(\square \)

Fig. 7
figure 7

Plot of the functions \(\lambda _i(\cdot ),\; i=1,2\) for \(n=2\) and \(M=1\) (left), respectively \(M=0.1\) (right). The dashed line corresponds to the asymptotic value \(- 3/\mu _1\). The dotted line in the left panel corresponds to the limit value \(n=2\) when \(M \rightarrow \infty \)

The conditional cumulative distribution functions \(\mathbb {P}\left( T_2^{(3),\lambda _i} \le \cdot | T_3^{(3),\lambda _i} = t\right) \) and \(\mathbb {P}_i(T_2^{(3),2,M} \le \cdot | T_3^{(3),2,M} = t) \) are given, for every \(t > 0\), by formulas analogous to (11) and (18):

$$\begin{aligned} \mathbb {P}_i\left( T_2^{(3),2,M} \le u | T_3^{(3),2,M} = t\right)&= \frac{F_{2,s}(u)\frac{d}{dt} P_t(i,3)+F_{2,d}(u)\frac{d}{dt} P_t(i,4)}{f_{T_3^{(3),2,M},i}(t)} \nonumber \\&= \frac{3F_{2,s}(u)P_t(i,1)+F_{2,d}(u)P_t(i,2)}{3P_t(i,1)+P_t(i,2)}, \end{aligned}$$
(27)
$$\begin{aligned} \mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right)&= 1- \exp \{-(\varLambda _i(t+u) - \varLambda _i(t))\}\nonumber \\&= 1 - \left( \frac{1 - P_{t+u}(i,3)-P_{t+u}(i,4)}{1 - P_{t}(i,3)-P_{t}(i,4)}\right) ^{1/3}. \end{aligned}$$
(28)

In order to compare, for \(u, t > 0\), the conditional cumulative distribution functions given in Eqs. (27) and (28), let us introduce the functions

$$\begin{aligned} g_i(u,t) := \mathbb {P}_i\left( T_2^{(3),2,M} \le u | T_3^{(3),2,M} = t \right) -\mathbb {P}\left( T_2^{(3),\lambda _i} \le u | T_3^{(3),\lambda _i} = t\right) , \; i=1,2. \end{aligned}$$

Proposition 10

The functions \(g_i(u,t),\; i=1,2\) have the following asymptotic behaviour:

  1. 1.

    For (ut) in the neighborhood of (0, 0), we have

    $$\begin{aligned} g_1(u,t)= & {} \displaystyle \frac{M}{2} u t + o\left( u^2+t^2\right) , \\ g_2(u,t)= & {} \displaystyle -\frac{u}{3} + \frac{6 M + 1}{18} u^2 + \frac{7 M}{6} u t + o\left( u^2+t^2\right) . \end{aligned}$$
  2. 2.

    For fixed \(t > 0\), when \(u \rightarrow +\infty \),

    $$\begin{aligned} g_i(u,t) = - K_{1,i} (M,t) e^{-\beta u} + o(e^{-\beta u}),\; i=1,2, \end{aligned}$$

    where \(K_{1,i}(M,t) > 0\) is given by

    $$\begin{aligned} K_{1,i}(M,t) = \displaystyle \frac{3 P_t(i,1)}{3P_t(i,1) + P_t(i,2)} \frac{1-a}{\beta } - \displaystyle \frac{P_t(i,2)}{3P_t(i,1) + P_t(i,2)} \frac{c}{\beta }, \end{aligned}$$

    where constants \(\beta , a, c\) are defined in Eqs. (9) and (10) with \(n=2\), i.e.

    $$\begin{aligned}&\beta = M + \frac{1}{2}\left( 1 - \left( 4M^2+1\right) ^{1/2}\right) ,\quad a = \frac{1}{2} \left( 1 + \left( 4M^2+1\right) ^{-1/2}\right) ,\\&\quad c = -M \left( 4M^2+1\right) ^{-1/2}. \end{aligned}$$
  3. 3.

    For fixed \(u \ge 0\), we have

    $$\begin{aligned} \displaystyle \lim _{t \rightarrow +\infty } g_i(u,t) = -K_3(M,u),\quad i=1,2, \end{aligned}$$

    where \(K_3(M,u) > 0\) is given by

    $$\begin{aligned} K_3(M,u) = c_1 e^{-\beta u} + c_2 e^{-\alpha u} - e^{\frac{\mu _1}{3} u}, \end{aligned}$$

    with

    $$\begin{aligned} \alpha= & {} M + \frac{1}{2}\left( 1 + \left( 4M^2+1\right) ^{1/2}\right) ,\quad \beta = M + \frac{1}{2}\left( 1 - \left( 4M^2+1\right) ^{1/2}\right) ,\; \\ \phi \left( \mu _1\right)= & {} \displaystyle \frac{3M}{2\left( \mu _1 + 3M + 3\right) },\quad c_1 = \frac{\phi \left( \mu _1\right) - \alpha }{\beta -\alpha },\quad c_2 = \frac{\beta - \phi \left( \mu _1\right) }{\beta -\alpha }. \end{aligned}$$

The proof is similar to the one given in the case \(n > 2\).

Most of the calculations of “Appendices A and B” section were made and/or verified using the computer algebra system Maple: programs and tracks of their execution are available from the authors.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Grusea, S., Rodríguez, W., Pinchon, D. et al. Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes. J. Math. Biol. 78, 189–224 (2019). https://doi.org/10.1007/s00285-018-1272-4

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00285-018-1272-4

Keywords

  • Inverse instantaneous coalescence rate (IICR)
  • Population structure
  • Population size change
  • Demographic history
  • Rate matrix
  • Structured coalescent

Mathematics Subject Classification

  • 92D15
  • 60J27
  • 60J35
  • 60E05
  • 15A18