The consequences of gene flow for local adaptation and differentiation: a two-locus two-deme model

Abstract

We consider a population subdivided into two demes connected by migration in which selection acts in opposite direction. We explore the effects of recombination and migration on the maintenance of multilocus polymorphism, on local adaptation, and on differentiation by employing a deterministic model with genic selection on two linked diallelic loci (i.e., no dominance or epistasis). For the following cases, we characterize explicitly the possible equilibrium configurations: weak, strong, highly asymmetric, and super-symmetric migration, no or weak recombination, and independent or strongly recombining loci. For independent loci (linkage equilibrium) and for completely linked loci, we derive the possible bifurcation patterns as functions of the total migration rate, assuming all other parameters are fixed but arbitrary. For these and other cases, we determine analytically the maximum migration rate below which a stable fully polymorphic equilibrium exists. In this case, differentiation and local adaptation are maintained. Their degree is quantified by a new multilocus version of $F_\mathrm{ST}$ and by the migration load, respectively. In addition, we investigate the invasion conditions of locally beneficial mutants and show that linkage to a locus that is already in migration-selection balance facilitates invasion. Hence, loci of much smaller effect can invade than predicted by one-locus theory if linkage is sufficiently tight. We study how this minimum amount of linkage admitting invasion depends on the migration pattern. This suggests the emergence of clusters of locally beneficial mutations, which may form ‘genomic islands of divergence’. Finally, the influence of linkage and two-way migration on the effective migration rate at a linked neutral locus is explored. Numerical work complements our analytical results.

1 Introduction

Migration in a geographically structured population may have opposing effects on the genetic composition of that population and, hence, on its evolutionary potential. On the one hand, gene flow caused by migration may be so strong that it not only limits but hinders local adaptation by swamping the whole population with a genotype that has high fitness in only one or a few demes. On the other hand, if migration is sufficiently weak, gene flow may replenish local populations with genetic variation and contribute to future adaptation. In this case, locally adapted genotypes may coexist in the population and maintain high levels of genetic variation as well as differentiation between subpopulations. For reviews of the corresponding, well developed one-locus theory, see Karlin (1982), Lenormand (2002), and Nagylaki and Lou (2008).

If selection acts on more than one locus, additional questions arise immediately. For instance, what are the consequences of the genetic architecture, such as linkage between loci, relative magnitude of locus effects, or epistasis, on the degree of local adaptation and of differentiation achieved for a given amount of gene flow? What are the consequences for genetic variation at linked neutral sites? What genetic architectures can be expected to evolve under various forms of spatially heterogeneous selection?

For selection acting on multiple loci, the available theory is much less well developed than for a single locus. One of the main reasons is that the interaction of migration and selection, even if the latter is nonepistatic, leads to linkage disequilibrium (LD) between loci (Li and Nei 1974; Christiansen and Feldman 1975; Slatkin 1975; Barton 1983). LD causes substantial, often insurmountable, complications in the analysis of multilocus models. Therefore, many multilocus studies are primarily numerical and focus on quite specific situations or problems. For instance, Spichtig and Kawecki (2004) investigated numerically the influence of the number of loci and of epistasis on the degree of polymorphism if selection acts antagonistically in two demes. Yeaman and Whitlock (2011) showed that concentrated genetic architecture, i.e., clusters of linked, locally beneficial alleles, evolve if stabilizing selection acts on a trait such that the fitness optima in two demes differ.

Linkage disequilibrium is also essential for the evolution of recombination. The evolution of recombination in heterogeneous environments has been studied by a number of authors (e.g., Charlesworth and Charlesworth 1979; Pylkov et al. 1998; Lenormand and Otto 2000), and the results depend strongly on the kind of variability of selection across environments, the magnitude of migration, and the sign and strength of epistasis.

Recent years have seen some advances in developing general theory for multilocus migration-selection models. The focus of this work was on the properties of the evolutionary dynamics and the conditions for the maintenance of multilocus polymorphism in limiting or special cases, such as weak or strong migration (Bürger 2009a; Bürger 2009b), or in the Levene model (Nagylaki 2009; Bürger 2009c; Bürger 2010; Barton 2010; Chasnov 2012). This progress was facilitated by the fact that in each case, LD is weak or absent.

Using a continent-island-model framework, Bürger and Akerman (2011) and Bank et al. (2012) analyzed the effects of gene flow on local adaptation, differentiation, the emergence of Dobzhansky-Muller incompatibilities, and the maintenance of polymorphism at two linked diallelic loci. They obtained analytical characterizations of the possible equilibrium configurations and bifurcation patterns for wide ranges of parameter combinations. In these models, typically high LD is maintained. In particular, explicit formulas were derived for the maximum migration rate below which a fully polymorphic equilibrium can be maintained, as well as for the minimum migration rate above which the island is swamped by the continental haplotype.

Here, we explore the robustness of some of these results by admitting arbitrary (forward and backward) migration between two demes. This generalization leads to substantial mathematical complications, but also to new biological insight. Because our focus is on the consequences of gene flow for local adaptation and differentiation, we assume divergent selection among the demes, i.e., alleles $A_1$ and $B_1$ are favored in deme 1, and $A_2$ and $B_2$ are favored in deme 2. The loci may recombine at an arbitrary rate. By ignoring epistasis and dominance, we assume genic selection. Mutation and random drift are neglected. Because we assume evolution in continuous time, our model also describes selection on haploids.

The model is set up in Sect. 2. In Sect. 3, we derive the equilibrium and stability structure for several important special cases. These include weak, strong, highly asymmetric, and super-symmetric migration, no or weak recombination, independent or strongly recombining loci, and absence of genotype-environment interaction. In Sect. 4, we study the dependence of the equilibrium and stability patterns on the total migration rate while keeping the ratio of migration rates, the recombination rate, and the selection coefficients constant (but arbitrary). In particular, we derive the possible bifurcation patterns for the cases of independent loci (linkage equilibrium) and for completely linked loci. With the help of perturbation theory, we obtain the equilibrium and stability configurations for weak or strong migration, highly asymmetric migration, and weak or strong recombination. For these cases, we determine the maximum migration rate below which a stable, fully polymorphic equilibrium is maintained, and the minimum migration rate above which the population is monomorphic. Numerical work complements our analytical results.

The next four sections are devoted to applications of the theory developed in Sects. 3 and 4. In Sects. 5 and 6, we use the migration load and a new, genuine multilocus, fixation index ($F_\mathrm{ST}$), respectively, to quantify the dependence of local adaptation and of differentiation on various parameters, especially, on the migration and the recombination rate. In Sect. 7, we investigate the invasion conditions for a mutant of small effect ($A_1$) that is beneficial in one deme but disadvantageous in the other deme. We assume that the mutant is linked to a polymorphic locus which is in selection-migration balance. We show that linkage between the loci facilitates invasion. Therefore, in such a scenario, clusters of locally adapted alleles are expected to emerge (cf. Yeaman and Whitlock 2011; Bürger and Akerman 2011). In Sect. 8, we study the strength of barriers to gene flow at neutral sites linked to the selected loci by deriving an explicit approximation for the effective migration rate at a linked neutral site. Our results are summarized and discussed in Sect. 9. Several purely technical proofs are relegated to the Appendix.

2 The model

We consider a sexually reproducing population of monoecious, diploid individuals that is subdivided into two demes connected by genotype-independent migration. Within each deme, there is random mating. We assume that two diallelic loci are under genic selection, i.e., there is no dominance or epistasis, and different alleles are favored in different demes. We assume soft selection, i.e., population regulation occurs within each deme. We ignore random genetic drift and mutation and employ a deterministic continuous-time model to describe evolution. A continuous-time model is obtained from the corresponding discrete-time model in the limit of weak evolutionary forces (here, selection, recombination, and migration).

We denote the rate at which individuals in deme 1 (deme 2) are replaced by immigrants from the other deme by $m_1 \ge 0$ ($m_2 \ge 0$). Then $m=m_1+m_2$ is the total migration rate. The recombination rate between the two loci is designated by $\rho \ge 0$.

Alleles at locus $\mathsf{A }$ are denoted by $A_1$ and $A_2$, at locus $\mathsf{B }$ by $B_1$ and $B_2$. We posit that $A_1$ and $B_1$ are favored in deme 1, whereas $A_2$ and $B_2$ are favored in deme 2. In deme $k$ ($k=1,2$), we assign the Malthusian parameters $\tfrac{1}{2}\alpha _k$ and $-\tfrac{1}{2}\alpha _k$ to $A_1$ and $A_2$, and $\tfrac{1}{2}\beta _k$ and $-\tfrac{1}{2}\beta _k$ to $B_1$ and $B_2$. Because we assume absence of dominance and of epistasis, the resulting fitness matrix for the genotypes reads

(2.1)

By relabeling alleles, we can assume without loss of generality $\alpha _1>0>\alpha _2$ and $\beta _1>0>\beta _2$. Hence, $A_1B_1$ and $A_2B_2$ may be called the locally adapted haplotypes in deme 1 and deme 2, respectively. By relabeling loci, we can assume $\beta _1\ge \alpha _1$. We define

$$\begin{aligned} \theta =\alpha _1\beta _2-\alpha _2\beta _1. \end{aligned}$$

(2.2)

By exchanging demes, i.e., by the transformation $\tilde{\alpha }_k=-\alpha _{k^*}$ and $\tilde{\beta }_k=-\beta _{k^*}$ (where $k^*$ denotes the deme $\ne k$), or by exchanging loci, i.e., by the transformation $\tilde{\alpha }_k=\beta _k$ and $\tilde{\beta }_k=\alpha _k$, we can further assume $\theta \ge 0$ without loss of generality, cf. Appendix A.1.

The fitness matrix (2.1) is also obtained if the two loci contribute additively to a quantitative trait that is under linear directional selection in each deme (Bürger 2009c). Then $\theta =0$ if the genotypic values are deme independent, i.e., if there is no genotype-environment interaction on the trait level.

Because in the case $\theta =0$ degenerate features can occur, it will be treated separately (Sects. 3.9 and 3.10). Therefore, unless stated otherwise, we always impose the following assumptions on our parameters:

$$\begin{aligned} \alpha _1>0>\alpha _2\quad \text{ and } \quad \beta _1>0>\beta _2, \end{aligned}$$

(2.3a)

and

$$\begin{aligned} \beta _1>\alpha _1, \end{aligned}$$

(2.3b)

and

$$\begin{aligned} \theta >0. \end{aligned}$$

(2.3c)

From (2.3a) and (2.3c), we infer

$$\begin{aligned} \beta _2<\alpha _2 \Rightarrow \alpha _1<\beta _1. \end{aligned}$$

(2.4)

Therefore, locus $\mathsf{A }$ is under weaker selection than locus $\mathsf{B }$ in both demes, i.e., $|\alpha _k|\le |\beta _k|$ for $k=1,2$, if and only if $\beta _2<\alpha _2$ holds.

The population can be described by the gamete frequencies in each of the demes. We denote the frequencies of the four possible gametes $A_1B_1$, $A_1B_2$, $A_2B_1$, and $A_2B_2$ in deme $k$ by $x_{1,k}$, $x_{2,k}$, $x_{3,k}$, and $x_{4,k}$. Then the state space is $S_4 \times S_4$, where $S_4=\{(x_1,x_2,x_3,x_4):\, x_i \ge 0\, \text{ and }\, \sum _{i=1}^4x_i=1\}$ is the simplex.

The following differential equations for the evolution of gamete frequencies in deme $k$ can be derived straightforwardly:

$$\begin{aligned} \dot{x}_{i,k} = \frac{d}{dt} x_{i,k}= x_{i,k}(w_{i,k}-\bar{w}_k) - \eta _i \rho D_k + m_k(x_{i,k^*}-x_{i,k}). \end{aligned}$$

(2.5)

Here the marginal fitness $w_{i,k}$ of gamete $i$ and the mean fitness $\bar{w}_k$ in deme $k$ are calculated from (2.1), $\eta _1=\eta _4=-\eta _2=-\eta _3=1$, and $D_{k}=x_{1,k}x_{4,k}-x_{2,k}x_{3,k}$ is the linkage-disequilibrium (LD) measure. We note that $D_k>0$ corresponds to an excess of the locally adapted haplotypes in deme $k$. The equations (2.5) also describe the dynamics of a haploid population if in deme $k$ we assign the fitnesses $\alpha _k$, $-\alpha _k$, $\beta _k$, $-\beta _k$ to the alleles $A_1$, $A_2$, $B_1$, $B_2$, respectively.

Instead of gamete frequencies it is often more convenient to work with allele frequencies and the LD measures $D_k$. We write $p_k=x_{1,k}+x_{2,k}$ and $q_k=x_{1,k}+x_{3,k}$ for the frequencies of $A_1$ and $B_1$ in deme $k$. Then the gamete frequencies $x_{i,k}$ are calculated from the $p_k$, $q_k$, and $D_k$ by

$$\begin{aligned} x_{1,k}&= p_k q_k+D_k, \quad \qquad \, \quad x_{2,k} =p_k(1-q_k)-D_k,\end{aligned}$$

(2.6a)

$$\begin{aligned} x_{3,k}&= (1-p_k)q_k-D_k,\quad x_{4,k} =(1-p_k)(1-q_k)+D_k\,. \end{aligned}$$

(2.6b)

The constraints $x_{i,k} \ge 0$ and $\sum _{i=1}^4 x_{i,k}=1$ for $i=1,2,3,4,$ and $k=1,2$ transform into $0\le p_k, q_k \le 1$ and $-\min \left\{ p_k q_k,(1-p_k)(1-q_k)\right\} \le D_k \le \min \left\{ p_k(1-q_k),(1-p_k)q_k\right\}$. It follows that $p_k$, $q_k$, and $D_k$ evolve according to

$$\begin{aligned} \dot{p}_k&= \alpha _k p_k(1-p_k)+\beta _k D_k + m_k(p_{k^*}-p_k),\end{aligned}$$

(2.7a)

$$\begin{aligned} \dot{q}_k&= \beta _k q_k(1-q_k)+\alpha _k D_k + m_k(q_{k^*}-q_k),\end{aligned}$$

(2.7b)

$$\begin{aligned} \dot{D}_k&= \left[ \alpha _k(1-2p_k)+\beta _k(1-2q_k)-\rho \right] D_k \nonumber \\&\quad + m_k \left[ (D_{k^*}-D_k)+(p_{k^*}-p_k)(q_{k^*}-q_k)\right] . \end{aligned}$$

(2.7c)

We emphasize that, because we are treating a continuous-time model, the parameters $\rho$, $m_k$, $\alpha _k$, and $\beta _k$ are rates (of recombination, migration, growth), whence they can be arbitrarily large. Their magnitude is determined by the time scale. By rescaling time, for instance to units of $\rho$ or $m$, the number of independent parameters could be reduced by one without changing the equilibrium properties.

3 Equilibria and their stability

We distinguish three types of equilibria: (i) monomorphic equilibria (ME), (ii) single-locus polymorphisms (SLPs), and (iii) full (two-locus) polymorphisms (FPs). The first two types are boundary equilibria, whereas FPs are internal equilibria (except when $\rho =0$). The stability properties of the ME and the coordinates and conditions for admissibility of the SLPs can be derived explicitly. However, the stability conditions for the SLPs and the conditions for existence or stability of FPs could be derived only for a number of limiting cases. These include strong recombination, weak or no recombination, weak, strong, or highly asymmetric migration.

3.1 Existence of boundary equilibria

The four ME, corresponding to fixation of one of the gametes, exist always. Their coordinates are as follows:

$$\begin{aligned}&\mathsf M _1 \; (A_1B_1 \text{ fixed }): \quad \hat{p}_k=1,\; \hat{q}_k=1,\; \hat{D}_k=0 \; \text{ for }\, k=1,2, \\&\mathsf M _2 \; (A_1B_2 \text{ fixed }): \quad \hat{p}_k=1,\; \hat{q}_k=0,\; \hat{D}_k=0 \; \text{ for }\, k=1,2, \\&\mathsf M _3 \; (A_2B_1 \text{ fixed }): \quad \hat{p}_k=0,\; \hat{q}_k=1,\; \hat{D}_k=0 \; \text{ for }\, k=1,2, \\&\mathsf M _4 \; (A_2B_2 \text{ fixed }): \quad \hat{p}_k=0,\; \hat{q}_k=0,\; \hat{D}_k=0 \; \text{ for }\, k=1,2, \end{aligned}$$

where a $\hat{}$ signifies an equilibrium. There are up to four SLPs, one in each marginal one-locus system. We denote the SLPs where $B_1$ or $B_2$ is fixed by $\mathsf P _\mathsf{A ,1}$ or $\mathsf P _\mathsf{A ,2}$, respectively, and the SLPs where $A_1$ or $A_2$ is fixed by $\mathsf P _\mathsf{B ,1}$ or $\mathsf P _\mathsf{B ,2}$. Their coordinates and the conditions for their admissibility can be calculated explicitly (Eyland 1971). We define

$$\begin{aligned} \sigma _k = \frac{m_k}{\alpha _k} \quad \text{ and }\quad \tau _k = \frac{m_k}{\beta _k}\,. \end{aligned}$$

(3.1)

By (2.3a), we have

$$\begin{aligned} \sigma _1>0>\sigma _2 \quad \text{ and }\quad \tau _1>0>\tau _2\,. \end{aligned}$$

(3.2)

In addition, it is easy to show that the assumptions (2.3) imply:

$$\begin{aligned}&\sigma _1+\sigma _2\ge 0 \; \Rightarrow \; \tau _1+\tau _2<\sigma _1+\sigma _2, \end{aligned}$$

(3.3a)

$$\begin{aligned}&\sigma _1+\sigma _2<0 \; \Rightarrow \; \tau _1+\tau _2<0. \end{aligned}$$

(3.3b)

If locus $\mathsf{B }$ is fixed (for $B_1$ or $B_2$), the equilibrium allele frequencies at locus $\mathsf{A }$ are

$$\begin{aligned} \hat{p}_1^\mathsf A = \frac{1}{2}\left( 1-2\sigma _1 + \sqrt{1-4\sigma _1\sigma _2}\right) \,, \quad \hat{p}_2^\mathsf A = \frac{1}{2}\left( 1-2\sigma _2 - \sqrt{1-4\sigma _1\sigma _2}\right) \,.\qquad \end{aligned}$$

(3.4)

If locus $\mathsf{A }$ is fixed, the equilibrium allele frequencies at locus $\mathsf{B }$ are given by

$$\begin{aligned} \hat{q}_1^\mathsf B = \frac{1}{2}\left( 1-2\tau _1 + \sqrt{1-4\tau _1\tau _2}\right) \,, \quad \hat{q}_2^\mathsf B = \frac{1}{2}\left( 1-2\tau _2 - \sqrt{1-4\tau _1\tau _2}\right) \,.\qquad \end{aligned}$$

(3.5)

Thus, the four SLPs have the following coordinates:

$$\begin{aligned}&\mathsf P _\mathsf{A ,1}: \quad \hat{p}_1=\hat{p}_1^\mathsf A ,\; \hat{p}_2=\hat{p}_2^\mathsf A ,\; \hat{q}_1=\hat{q}_2=1,\; \hat{D}_1=\hat{D}_2=0, \end{aligned}$$

(3.6a)

$$\begin{aligned}&\mathsf P _\mathsf{A ,2}: \quad \hat{p}_1=\hat{p}_1^\mathsf A ,\; \hat{p}_2=\hat{p}_2^\mathsf A ,\; \hat{q}_1=\hat{q}_2=0,\; \hat{D}_1=\hat{D}_2=0, \end{aligned}$$

(3.6b)

$$\begin{aligned}&\mathsf P _\mathsf{B ,1}: \quad \hat{p}_1=\hat{p}_2=1,\; \hat{q}_1=\hat{q}_1^\mathsf B ,\; \hat{q}_2=\hat{q}_2^\mathsf B ,\; \hat{D}_1=\hat{D}_2=0, \end{aligned}$$

(3.6c)

$$\begin{aligned}&\mathsf P _\mathsf{B ,2}: \quad \hat{p}_1=\hat{p}_2=0,\; \hat{q}_1=\hat{q}_1^\mathsf B ,\; \hat{q}_2=\hat{q}_2^\mathsf B ,\; \hat{D}_1=\hat{D}_2=0, \end{aligned}$$

(3.6d)

The equilibria $\mathsf P _\mathsf{A ,1}$ and $\mathsf P _\mathsf{A ,2}$ are admissible if and only if

$$\begin{aligned} |\sigma _1+\sigma _2| < 1, \end{aligned}$$

(3.7)

and the equilibria $\mathsf P _\mathsf{B ,1}$ and $\mathsf P _\mathsf{B ,2}$ are admissible if and only if

$$\begin{aligned} |\tau _1+\tau _2| < 1. \end{aligned}$$

(3.8)

The SLPs leave the state space through one of their ‘neighboring’ ME if $|\sigma _1+\sigma _2|$ or $|\tau _1+\tau _2|$ increases above 1. In particular, we find

$$\begin{aligned} \sigma _1+\sigma _2 \downarrow -1&\iff \mathsf P _\mathsf{A ,1} \rightarrow \mathsf M _1 \text{ and } \, \mathsf P _\mathsf{A ,2} \rightarrow \mathsf M _2,\end{aligned}$$

(3.9a)

$$\begin{aligned} \sigma _1+\sigma _2 \uparrow 1&\iff \mathsf P _\mathsf{A ,1} \rightarrow \mathsf M _3 \text{ and } \, \mathsf P _\mathsf{A ,2} \rightarrow \mathsf M _4, \end{aligned}$$

(3.9b)

$$\begin{aligned} \tau _1+\tau _2 \downarrow -1&\iff \mathsf P _\mathsf{B ,1} \rightarrow \mathsf M _1 \text{ and } \,\mathsf P _\mathsf{B ,2} \rightarrow \mathsf M _3,\end{aligned}$$

(3.9c)

$$\begin{aligned} \tau _1+\tau _2 \uparrow 1&\iff \mathsf P _\mathsf{B ,1} \rightarrow \mathsf M _2 \text{ and } \, \mathsf P _\mathsf{B ,2} \rightarrow \mathsf M _4. \end{aligned}$$

(3.9d)

Throughout, we use $\downarrow$ to indicated convergence from above and $\uparrow$ to indicate convergence from below. Figure 1 illustrates the location of the possible equilibria.

Fig. 1

Location of equilibria. In terms of gamete frequencies, the state space is $S_4\times S_4$, where each $S_4$ corresponds to one deme. This figure shows (schematically) the location in $S_4$ of all boundary equilibria and of the stable internal equilibrium $\mathsf F$. $\mathsf F$ converges to $\mathsf{F _\infty }$ if $\rho \rightarrow \infty$ and to $\mathsf{F _0}$ if $\rho \rightarrow 0$. The LE manifold is indicated by hatching

The SLPs are asymptotically stable within their marginal one-locus system if and only if they are admissible. Then they are also globally asymptotically stable within their marginal system (Eyland 1971). (We use globally stable in the sense that at least all trajectories from the interior of the designated set converge to the equilibrium.) The reader may notice that (3.7) and (3.8) are precisely the conditions for maintaining a protected polymorphism at locus $\mathsf{A }$ and $\mathsf{B }$, respectively.

3.2 Stability of monomorphic equilibria

At each monomorphic equilibrium, the characteristic polynomial factors into three quadratic polynomials. Two of them determine stability with respect to the marginal one-locus systems, whereas the third determines stability with respect to the interior of the state space. The stability properties of the monomorphic equilibria are as follows. The proof is given in Appendix A.2.

Proposition 3.1

$\mathsf M _1$ is asymptotically stable if

$$\begin{aligned} \sigma _1+\sigma _2<-1\quad \text{ and } \quad \tau _1+\tau _2<-1 \end{aligned}$$

(3.10)

and one of the following conditions hold:

$$\begin{aligned} \rho \ge \min \{-\alpha _2, -\beta _2 \} \end{aligned}$$

(3.11a)

or

$$\begin{aligned} \rho < \min \{-\alpha _2, -\beta _2 \} \text{ and } \, m_2 > -\frac{(\alpha _1+\beta _1+\rho +m_1)(\alpha _2+\beta _2+\rho )}{\alpha _1+\beta _1+\rho }.\qquad \quad \end{aligned}$$

(3.11b)

$\mathsf M _2$ is always unstable.

$\mathsf M _3$ is asymptotically stable if

$$\begin{aligned} \sigma _1+\sigma _2>1, \text{ and } , \tau _1+\tau _2<-1. \end{aligned}$$

(3.12)

$\mathsf M _4$ is asymptotically stable if

$$\begin{aligned} \sigma _1+\sigma _2>1, \text{ and } , \tau _1+\tau _2>1 \end{aligned}$$

(3.13)

and one of the following conditions hold:

$$\begin{aligned} \rho \ge \alpha _1 \end{aligned}$$

(3.14a)

or

$$\begin{aligned} \rho < \alpha _1 \text{ and } \, m_2 > \frac{(\alpha _1+\beta _1-\rho -m_1)(\alpha _2+\beta _2-\rho )}{\alpha _1+\beta _1-\rho }. \end{aligned}$$

(3.14b)

If one of the inequalities in (3.10), (3.12), or (3.13), or one of the inequalities for $m_2$ in (3.11b) or (3.14b) is reversed, the respective equilibrium is unstable.

If we assumed $\theta <0$, then $\mathsf M _3$ would always be unstable and $\mathsf M _2$ would be stable if $\sigma _1+\sigma _2<-1$ and $\tau _1+\tau _2>1$.

The above result shows that each of $\mathsf M _1$, $\mathsf M _3$, or $\mathsf M _4$ can be stable, but never simultaneously. For sufficiently loose linkage, the stability of a ME is determined solely by its stability within the two marginal one-locus systems in which it occurs. Stability of $\mathsf M _3$ is independent of the recombination rate. For given migration rates, the equilibria $\mathsf M _1$ and $\mathsf M _4$ may be stable for high recombination rates but unstable for low ones. For a low total migration rate ($m_1+m_2$), no ME is stable. For a sufficiently high total migration rate, there is a globally asymptotically stable ME (Sect. 4.5).

3.3 Stability of single-locus polymorphisms

As already mentioned, a single-locus polymorphism is globally attracting within its marginal one-locus system whenever it is admissible. Although the coordinates of the SLPs are given explicitly, the conditions for stability within the full, six-dimensional system on $S_4\times S_4$ are uninformative because the four eigenvalues that determine stability transversal to the marginal one-locus system are solutions of a complicated quartic equation.

In the following we treat several limiting cases in which the conditions for stability of the SLPs and for existence and stability of FPs can be obtained explicitly.

3.4 Weak migration

The equilibrium and stability structure for weak migration can be deduced from the model with no migration by perturbation theory. In the absence of migration ($m_1=m_2=0$), the two subpopulations evolve independently. Because selection is nonepistatic and there is no dominance, in each deme the fittest haplotype becomes eventually fixed. In fact, mean fitness is nondecreasing (Ewens 1969). Our assumptions about fitness, i.e., (2.1) and (2.3a), imply that in deme 1 the equilibrium with $\hat{p}_1=\hat{q}_1=1$ and $\hat{D}_1=0$ is globally attracting, and in deme 2 the equilibrium with $\hat{p}_2=\hat{q}_2=0$ and $\hat{D}_2=0$ is globally attracting. Therefore, in the combined system, i.e., on $S_4\times S_4$, but still with $m_1=m_2=0$, the (unique) globally attracting equilibrium is given by

$$\begin{aligned} \hat{p}_1=\hat{q}_1=1, \;\; \hat{p}_2=\hat{q}_2=0, \;\; \hat{D}_1=\hat{D}_2=0\,. \end{aligned}$$

(3.15)

All other equilibria are on the boundary and unstable.

Because, generically, all equilibria in the system without migration are hyperbolic and it is a gradient system (Shahshahani 1979; Bürger 2000, p. 42), Theorem 5.4 in Bürger (2009a) applies and shows that the perturbation $\mathsf F$ of the equilibrium (3.15) is globally asymptotically stable for sufficiently small migration rates $m_1$ and $m_2$. Boundary equilibria remain unstable for sufficiently small migration rates. It is straightforward to calculate the coordinates of the perturbed equilibrium to leading order in $m_1$ and $m_2$. They are given by

$$\begin{aligned} \hat{p}_1 \!&= \! 1\!-\!\frac{m_1}{\alpha _1} \frac{\alpha _1\!+\!\rho }{\alpha _1\!+\!\beta _1\!+\!\rho } \,, \quad \hat{q}_1\! =\! 1\!-\!\frac{m_1}{\beta _1}\frac{\beta _1\!+\!\rho }{\alpha _1\!+\!\beta _1+\rho } \,, \quad \hat{D}_1 \!=\! \frac{m_1}{\alpha _1+\beta _1+\rho } \,,\qquad \qquad \end{aligned}$$

(3.16a)

$$\begin{aligned} \hat{p}_2 \!&= \! \frac{m_2}{-\alpha _2}\frac{\rho -\alpha _2}{\rho -\alpha _2-\beta _2} \,, \; \hat{q}_2 \!=\! \frac{m_2}{-\beta _2}\frac{\rho -\beta _2}{\rho -\alpha _2-\beta _2} \,, \; \hat{D}_2 \!=\! \frac{m_2}{\rho -\alpha _2-\beta _2} \,.\qquad \qquad \end{aligned}$$

(3.16b)

Therefore, we conclude

Proposition 3.2

For sufficiently weak migration, there is a unique, globally attracting, fully polymorphic equilibrium $\mathsf F$. To leading order in $m_1$ and $m_2$, its coordinates are given by (3.16).

Proposition 3.2 remains valid if the assumptions (2.3b) and (2.3c) are dropped. Apart from the obvious fact that migration reduces differences between subpopulations, the above approximations show that the lower the recombination rate, the smaller is this reduction. Thus, for given (small) migration rates, differentiation between subpopulations is always enhanced by reduced recombination. Linkage disequilibria within subpopulations are always positive.

3.5 Linkage equilibrium

If recombination is so strong relative to selection and migration that linkage equilibrium (LE) can be assumed, i.e., if $\frac{1}{\rho }\max _{k=1,2}\{|\alpha _k|,|\beta _k|,m_k\}\rightarrow 0$, the dynamics (2.7) simplifies to

$$\begin{aligned} \dot{p}_1&= \alpha _1 p_1(1-p_1) + m_1 (p_2-p_1) \,, \end{aligned}$$

(3.17a)

$$\begin{aligned} \dot{p}_2&= \alpha _2 p_2(1-p_2) + m_2 (p_1-p_2) \,, \end{aligned}$$

(3.17b)

$$\begin{aligned} \dot{q}_1&= \beta _1 q_1(1-q_1) + m_1 (q_2-q_1) \,,\end{aligned}$$

(3.17c)

$$\begin{aligned} \dot{q}_2&= \beta _2 q_2(1-q_2) + m_2 (q_1-q_2)\,, \end{aligned}$$

(3.17d)

which is defined on $[0,1]^4$.

In (3.17), the differential equations for the two loci are decoupled, i.e., (3.17a) and (3.17b) as well as (3.17c) and (3.17d) form closed systems. Thus, the dynamics of the full system is a Cartesian product of the two one-locus dynamics. Therefore, in addition to the ME and to the SLPs determined above, the following internal equilibrium, denoted by $\mathsf{F _\infty }$, may exist

$$\begin{aligned} \hat{p}_1^{\infty } = \hat{p}_1^\mathsf A , \quad \hat{p}_2^{\infty } = \hat{p}_2^\mathsf A , \quad \hat{q}_1^{\infty } = \hat{q}_1^\mathsf B , \quad \hat{q}_2^{\infty } = \hat{q}_2^\mathsf B , \end{aligned}$$

(3.18)

where the $\hat{p}_k^\mathsf{A }$ and $\hat{q}_k^\mathsf B$ are given by (3.4) and (3.5), respectively. No other internal equilibrium can exist. This equilibrium is admissible if and only if (3.7) and (3.8) are satisfied, i.e., if and only if all four SLPs are admissible.

Because in the one-locus model the FP is globally asymptotically stable (hence, it attracts all trajectories from the interior of the state space) whenever it is admissible (Eyland 1971; Hadeler and Glas 1983, Theorem 2; Nagylaki and Lou 2008, Section 4.3.2), and because the full dynamics is the Cartesian product of the one-locus dynamics, the fully polymorphic equilibrium $\mathsf{F _\infty }$ is globally asymptotically stable whenever it is admissible. Similarly, we conclude that a boundary equilibrium is globally asymptotically stable whenever it is asymptotically stable in the full system. These results in combination with those in Sects. 3.1 and 3.2 yield the following proposition.

Proposition 3.3

Assume (3.17). Then a globally asymptotically stable equilibrium exists always. This equilibrium is internal, hence equals $\mathsf{F _\infty }$ (3.18), if and only if (3.7) and (3.8) hold. It is a SLP if one of (3.7) or (3.8) is violated, and a ME if both (3.7) and (3.8) are violated.

If, by variation of parameters, the internal equilibrium leaves (or enters) the state space, generically, it does so through one of the SLPs. The precise conditions are:

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{A ,1}}&\;\iff \; \tau _1+\tau _2 \downarrow -1 \text{ and } \, |\sigma _1+\sigma _2|<1, \end{aligned}$$

(3.19a)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{A ,2}}&\;\;\text{ does } \text{ not } \text{ occur }, \end{aligned}$$

(3.19b)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{B ,1}}&\;\iff \; \sigma _1+\sigma _2 \downarrow -1 \text{ and } \, |\tau _1+\tau _2|<1, \end{aligned}$$

(3.19c)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{B ,2}}&\;\iff \; \sigma _1+\sigma _2 \uparrow 1 \text{ and } \, |\tau _1+\tau _2|<1. \end{aligned}$$

(3.19d)

When, upon leaving the state space, $\mathsf{F _\infty }$ collides with a boundary equilibrium (SLP or ME), the respective boundary equilibrium becomes globally asymptotically stable.

We note that $\mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{A ,2}}$ does not occur because it requires $\tau _1+\tau _2 \uparrow 1$ and $|\sigma _1+\sigma _2|<1$, which is impossible by (3.3). We leave the simple determination of the conditions for bifurcations of $\mathsf{F _\infty }$ with one of the ME to the interested reader.

Proposition 3.3 can be extended straightforwardly to an arbitrary number of loci because the dynamics at each locus is independent of that at the other loci. This decoupling of loci occurs because there is no epistasis.

3.6 Strong recombination: quasi-linkage equilibrium

If recombination is strong, a regular perturbation analysis of the internal equilibrium $\mathsf{F _\infty }$ of (3.17) can be performed. The allele frequencies and linkage disequilibria can be calculated to order $1/\rho$. Formally, we set

$$\begin{aligned} m_k = \mu _k/\rho \quad (k=1,2), \end{aligned}$$

(3.20)

keep $\sigma _k$ and $\tau _k$ constant, and let $\rho \rightarrow \infty$. Then, we obtain

$$\begin{aligned} \hat{p}_1&= \hat{p}_1^{\infty } + \frac{\sigma _1}{\rho } \frac{\sigma _2(\beta _1-\beta _2)+\beta _1\sqrt{1-4\sigma _1\sigma _2}}{\sqrt{1-4\sigma _1\sigma _2}} \left( \hat{q}_1^{\infty }-\hat{q}_2^{\infty } \right) + O(\rho ^{-2})\,,\qquad \quad \end{aligned}$$

(3.21a)

$$\begin{aligned} \hat{q}_1&= \hat{q}_1^{\infty } +\frac{\tau _1}{\rho } \frac{\tau _2(\alpha _1-\alpha _2)+\alpha _1\sqrt{1-4\tau _1\tau _2}}{\sqrt{1-4\tau _1\tau _2}} \left( \hat{p}_1^{\infty }-\hat{p}_2^{\infty } \right) + O(\rho ^{-2})\,,\qquad \quad \end{aligned}$$

(3.21b)

$$\begin{aligned} \hat{D}_1&= \frac{m_1}{\rho }\left( \hat{p}_1^{\infty }-\hat{p}_2^{\infty }\right) \left( \hat{q}_1^{\infty }-\hat{q}_2^{\infty } \right) + O(\rho ^{-2})\,, \end{aligned}$$

(3.21c)

and analogous formulas hold for the second deme. Because LD is of order $1\!/\!\rho$, this approximation may be called the quasi-linkage equilibrium approximation of the fully polymorphic equilibrium (Kimura 1965; Turelli and Barton 1990; Nagylaki et al. 1999). We note that LD is positive in both demes and increases with increasing differentiation between the demes, increasing migration, or decreasing recombination.

Proposition 5.1 in Bürger (2009a) shows that in every small neighborhood of an equilibrium of the model with LE (3.17), there is one equilibrium of the perturbed system, and it has the same stability properties as the unperturbed equilibrium. Because of the simple structure of (3.17), a stronger result can be obtained. In an isolated one-locus system on $[0,1]^2$ (e.g., (3.17a) and (3.17b)), every trajectory from the interior converges to the unique asymptotically stable equilibrium (Sect. 3.5), and the chain-recurrent points (Conley 1978) are the equilibria. Therefore, the same holds for the LE dynamics (3.17), and the regular global perturbation result of Nagylaki et al. (1999) (the proof of their Theorem 2.3) applies for large $\rho$. Hence the dynamical behavior with strong recombination is qualitatively the same as that under LE. We conclude that for sufficiently strong recombination every asymptotically stable equilibrium is globally asymptotically stable.

3.7 No recombination

Let recombination be absent, i.e., $\rho =0$. Then, effectively, we have a one-locus model in which the four alleles correspond to the four gametes $A_1B_1$, $A_1B_2$, $A_2B_1$, $A_2B_2$. In deme $k$, they have the selection coefficients $\tfrac{1}{2}(\alpha _k+\beta _k)$, $\tfrac{1}{2}(\alpha _k-\beta _k)$, $\tfrac{1}{2}(-\alpha _k+\beta _k)$, $-\tfrac{1}{2}(\alpha _k+\beta _k)$, respectively. According to Theorem 2.4 of Nagylaki and Lou (2001), generically, no more than two gametes can be present at an equilibrium. We will prove a stronger result and characterize all possible equilibria and their local stability.

Because $\rho =0$, there may be a polymorphic equilibrium at which only the gametes $A_1B_1$ and $A_2B_2$ are present. We call it $\mathsf{F _0}$ and set

$$\begin{aligned} \kappa _k = \frac{m_k}{\alpha _k+\beta _k} \quad (k=1,2)\,. \end{aligned}$$

(3.22)

Then one-locus theory (Sect. 3.1) informs us that $\mathsf{F _0}$ is admissible if and only if

$$\begin{aligned} |\kappa _1+\kappa _2| < 1. \end{aligned}$$

(3.23)

Its coordinates are given by

$$\begin{aligned} \hat{p}_1^0&= \hat{q}_1^0 = \frac{1}{2}\left( 1-2\kappa _1 + \sqrt{1-4\kappa _1\kappa _2}\right) \,, \end{aligned}$$

(3.24a)

$$\begin{aligned} \hat{p}_2^0&= \hat{q}_2^0 = \frac{1}{2}\left( 1-2\kappa _2 - \sqrt{1-4\kappa _1\kappa _2}\right) \,, \end{aligned}$$

(3.24b)

$$\begin{aligned} \hat{D}_k^0&= \hat{p}_k(1-\hat{p}_k) \quad (k=1,2)\,, \end{aligned}$$

(3.24c)

where $\hat{p}_k^0=\hat{q}_k^0=\hat{x}_{1,k}^0$, $\hat{x}_{2,k}^0=\hat{x}_{3,k}^0=0$, and $\hat{x}_{4,k}^0=1-\hat{x}_{1,k}^0$ ($k=1,2$). Within the subsystem in which only the gametes $A_1B_1$ and $A_2B_2$ are present, $\mathsf{F _0}$ is asymptotically stable whenever it is admissible. One-locus theory implies that

$$\begin{aligned} \kappa _1+\kappa _2\downarrow -1&\iff \mathsf{F _0}\rightarrow \mathsf M _1,\end{aligned}$$

(3.25a)

$$\begin{aligned} \kappa _1+\kappa _2\uparrow 1&\iff \mathsf{F _0}\rightarrow \mathsf M _4. \end{aligned}$$

(3.25b)

A simple application of Corollary 3.9 of Nagylaki and Lou (2007) shows that the gamete $A_1B_2$ will always be lost (to apply their result, recall assumptions (2.3) and use $\gamma _{22}=\gamma _{23}=0$, $\frac{\alpha _1}{\alpha _1+\beta _1}<\gamma _{21}<\frac{\alpha _2}{\alpha _2+\beta _2}$, $\gamma _{24}=1-\gamma _{21}$). This strengthens the result in Sect. 3.2 that $\mathsf M _2$ is always unstable. Thus, we are left with the analysis of the tri-gametic system consisting of $A_1B_1$, $A_2B_1$, and $A_2B_2$. (If $\theta <0$, then gamete $A_2B_1$ is lost.)

In Appendix A.3 it is proved that $\mathsf{F _0}$ is the only equilibrium at which both loci are polymorphic except when

$$\begin{aligned} m_1m_2 = {\tilde{m}}\end{aligned}$$

(3.26)

holds, where

$$\begin{aligned} {\tilde{m}}= -\alpha _1\alpha _2\beta _1\beta _2(\alpha _1+\beta _1)(\alpha _2+\beta _2)/\theta ^2. \end{aligned}$$

(3.27)

If (3.26) holds, then there is a line of internal equilibria connecting $\mathsf{F _0}$ with $\mathsf P _\mathsf{A ,1}$ or $\mathsf P _\mathsf{B ,2}$ (or $\mathsf M _3$); see Appendix A.3.

We find that $\mathsf{F _0}$ is asymptotically stable if

$$\begin{aligned} m_1m_2 < {\tilde{m}}, \end{aligned}$$

(3.28)

and unstable if the inequality is reversed (Appendix A.4). For sufficiently small migration rates, Proposition 3.2 implies that $\mathsf{F _0}=\mathsf F$ and $\mathsf{F _0}$ is globally asymptotically stable. If the inequality in (3.28) is reversed, $\mathsf{F _0}$ may or may not be admissible.

Of course, if $\mathsf{F _0}$ is asymptotically stable, then the equilibria $\mathsf M _1$ and $\mathsf M _4$ are unstable; cf. (3.25). The following argument shows that $\mathsf M _3$ cannot be simultaneously stable with $\mathsf{F _0}$. We rewrite (3.28) as

$$\begin{aligned} \kappa _1\kappa _2 > -\frac{\alpha _1\alpha _2\beta _1\beta _2}{\theta ^2} = -\frac{\sigma _1\sigma _2\tau _1\tau _2}{(\sigma _1\tau _2-\sigma _2\tau _1)^2}\,. \end{aligned}$$

(3.29)

Because

$$\begin{aligned} \kappa _k^{-1} = \sigma _k^{-1} + \tau _k^{-1}\,, \end{aligned}$$

(3.30)

(3.29) becomes

$$\begin{aligned} (\sigma _1\tau _2-\sigma _2\tau _1)^2 + (\sigma _1+\tau _1)(\sigma _2+\tau _2) < 0\,. \end{aligned}$$

(3.31)

Since $\mathsf M _3$ is asymptotically stable if (3.12) holds and because, as is easy to show, (3.12) and (3.31) are incompatible, the assertion follows. It can also be shown from (3.12) and (3.31) that $\mathsf M _3$ cannot become stable when $\mathsf{F _0}$ loses its stability except in the degenerate case when $\sigma _1+\sigma _2=1$ and $\tau _1+\tau _2=-1$.

In our tri-gametic system, $\mathsf P _\mathsf{A ,1}$ and $\mathsf P _\mathsf{B ,2}$ are the only possible SLPs. They may exist simultaneously with $\mathsf{F _0}$ if (3.28) holds, i.e., if $\mathsf{F _0}$ is stable, but not otherwise (Appendix A.5). If (3.28) holds, both are unstable (if admissible). $\mathsf P _\mathsf{A ,1}$ or $\mathsf P _\mathsf{B ,2}$ have an eigenvalue 0 if and only if (3.26) holds or if they leave or enter the state space through a ME. In Appendix A.5 it is shown that $\mathsf P _\mathsf{A ,1}$ is asymptotically stable if and only if

$$\begin{aligned} \tau _1+\tau _2<-1 \quad \text{ and } \quad |\sigma _1+\sigma _2|<1 \quad \text{ and }\quad m_1m_2 > {\tilde{m}}, \end{aligned}$$

(3.32)

and $\mathsf P _\mathsf{B ,2}$ is asymptotically stable if and only if

$$\begin{aligned} 1<\sigma _1+\sigma _2 \; \text{ and } \, |\tau _1+\tau _2|<1 \quad \text{ and }\quad m_1m_2 > {\tilde{m}}. \end{aligned}$$

(3.33)

Hence, if $m_1m_2$ increases above ${\tilde{m}}$, the SLP that is admissible becomes asymptotically stable. Upon collision of the stable SLP with one of the adjacent ME, the corresponding ME becomes stable and remains so for all higher migration rates. We summarize these findings as follows:

Proposition 3.4

Except in the degenerate case when (3.26) holds, only equilibria with at most two gametes present exist. If (3.23) is satisfied, the equilibrium $\mathsf{F _0}$ given by (3.24) is admissible. If, in addition, (3.28) is fulfilled, then $\mathsf{F _0}$ is asymptotically stable. For sufficiently small migration rates, it is globally asymptotically stable. If $\mathsf{F _0}$ is unstable or not admissible, then one of the ME $(\mathsf M _1$, $\mathsf M _3$, $\mathsf M _4)$ or one of the SLPs $(\mathsf P _\mathsf{A ,1}$, $\mathsf P _\mathsf{B ,2})$ is asymptotically stable. If (3.26) holds, then there is a line of equilibria with three gametes present.

The proposition shows that, except for the nongeneric case when (3.26) holds, there is always precisely one stable equilibrium point. Numerical results support the conjecture that the stable equilibrium is globally asymptotically stable. Bifurcation patterns as functions of $m$ are derived in Sect. 4.8.

In addition to $\mathsf{F _0}$, there exists a second FP on the edge connecting $\mathsf M _2$ and $\mathsf M _3$. Although its coordinates can be calculated easily, it is not of interest here as it is unstable for every choice of selection and migration parameters. This unstable equilibrium leaves the state space under small perturbations, i.e., if $\rho >0$.

3.8 Highly asymmetric migration

All special cases treated above suggest that there always exists a globally asymptotically stable equilibrium. This, however, is generally not true as was demonstrated by the analysis of the two-locus continent-island (CI) model in Bürger and Akerman (2011). There, all possible bifurcation patterns were derived and it was shown that the fully polymorphic equilibrium can be simultaneously stable with a boundary equilibrium. For highly asymmetric migration rates, the equilibrium and stability structure can be obtained by a perturbation analysis of this CI model.

Therefore, we first summarize the most relevant features of the analysis in Bürger and Akerman (2011). Because in that analysis the haplotype $A_2B_2$ is fixed on the continent (here, deme 2) and there is no back migration ($m_2=0$), it is sufficient to treat the dynamics on the island (here, deme 1) where immigration of $A_2B_2$ occurs at rate $m_1$. Thus, the state space is $S_4$.

It was shown that up to two internal (fully polymorphic) equilibria, denoted by $\mathsf E _+$ and $\mathsf E _-$, may exist. Only one ($\mathsf E _+$) can be stable. Two SLPs, $\mathsf E _\mathsf{A }$ and $\mathsf E _\mathsf B$, may exist. At $\mathsf E _\mathsf{A }$, locus $\mathsf{A }$ is polymorphic and allele $B_2$ is fixed; at $\mathsf E _\mathsf B$, locus $\mathsf{B }$ is polymorphic and allele $A_2$ is fixed. $\mathsf E _\mathsf{A }$ ($\mathsf E _\mathsf B$) is admissible if and only if $m_1<\alpha _1$ ($m_1<\beta _1$). $\mathsf E _\mathsf{A }$ is always unstable. Finally, there always exists the monomorphic equilibrium $\mathsf E _\mathsf C$ at which the haplotype $A_2B_2$ is fixed on the island. The equilibrium coordinates of all equilibria were obtained explicitly. In addition, it was proved (see also Bank et al. 2012, Supporting Information, Theorem S.4) that precisely the following two types of bifurcation patterns can occur:

Type 1 There exists a critical migration rate $m^\bullet >0$ such that:

• If $0<m_1<m^\bullet$, a unique internal equilibrium, $\mathsf E _+$, exists. It is asymptotically stable and, presumably, globally asymptotically stable.

• At $m_1=m^\bullet$, $\mathsf E _+$ leaves the state space through a boundary equilibrium ($\mathsf E _\mathsf B$ or $\mathsf E _\mathsf C$) by an exchange-of-stability bifurcation.

• If $m_1>m^\bullet$, a boundary equilibrium ($\mathsf E _\mathsf B$ or $\mathsf E _\mathsf C$) is asymptotically stable and, presumably, globally stable.

Type 2 There exist critical migration rates $m^\circ$ and $m^\bullet$ satisfying $m^\bullet >m^\circ >0$ such that:

• If $0<m_1<m^\circ$, there is a unique internal equilibrium $(\mathsf E _+)$. It is asymptotically stable and, presumably, globally stable.

• At $m_1=m^\circ$, an unstable equilibrium $(\mathsf E _-)$ enters the state space by an exchange-of-stability bifurcation with a boundary equilibrium ($\mathsf E _\mathsf B$ or $\mathsf E _\mathsf C$).

• If $m^\circ <m_1<m^\bullet$, there are two internal equilibria, one asymptotically stable $(\mathsf E _+)$, the other unstable $(\mathsf E _-)$, and one of the boundary equilibria ($\mathsf E _\mathsf B$ or $\mathsf E _\mathsf C$) is asymptotically stable.

• At $m_1=m^\bullet$, the two internal equilibria merge and annihilate each other by a saddle-node bifurcation.

• If $m_1>m^\bullet$, a boundary equilibrium ($\mathsf E _\mathsf B$ or $\mathsf E _\mathsf C$) is asymptotically stable and, presumably, globally stable.

For sufficiently large migration rates ($m>m^{\bullet \bullet }\ge m^\bullet$), $\mathsf E _\mathsf C$ is globally asymptotically stable in both cases. Bifurcation patterns of Type 2 occur only if the recombination rate is intermediate, i.e., if $\rho$ is about as large as $\alpha _1$.

By imbedding the CI model into the two-deme dynamics, (2.5) or (2.7), perturbation theory can be applied to obtain analogous results for highly asymmetric migration, i.e., for sufficiently small $m_2/m_1$ (Karlin and McGregor 1972). This is so because all equilibria in the CI model are hyperbolic except when collisions between equilibria occur (Bürger and Akerman 2011). Since the coordinates of the internal equilibria $\mathsf E _+$ and $\mathsf E _-$ were derived, the perturbed equilibrium frequencies can be obtained. Because they are too complicated to be informative, we do not present them. The perturbation of $\mathsf E _+$, denoted by $\mathsf F$, is asymptotically stable. As $\mathsf E _-$ is internal, it cannot be lost by a small perturbation. Also the boundary equilibria and their stability properties are preserved under small perturbations. In particular, $\mathsf E _\mathsf C$ gives rise to $\mathsf M _4$, and the SLPs $\mathsf E _\mathsf{A }$ and $\mathsf E _\mathsf B$ give rise to $\mathsf P _\mathsf{A ,2}$ and $\mathsf P _\mathsf{B ,2}$, respectively,

If recombination is intermediate, (at least) under highly asymmetric two-way migration, one stable and one unstable FP can coexist. In this case the stable FP, $\mathsf F$, is simultaneously stable with either $\mathsf M _4$ or $\mathsf P _\mathsf{B ,2}$. Although there is precisely one (perturbed) equilibrium in a small neighborhood of every equilibrium of the CI model, we can not exclude that other internal equilibria or limit sets are generated by perturbation.

3.9 The case $\theta =0$

The analyses in the previous sections are based on the assumptions (2.3), in particular, on $\theta >0$. However, many of the results obtained above remain valid if $\theta =0$. Here, we point out the necessary adjustments.

Without loss of generality, we can assume

$$\begin{aligned} |\alpha _k|\le |\beta _k|\quad \text{ for } k=1,2 \end{aligned}$$

(3.34)

in addition to $\theta =0$ and (2.3a). Then we observe that

$$\begin{aligned} \sigma _1\!+\!\sigma _2\!=\!\frac{\beta _2}{\alpha _2}(\tau _1\!+\!\tau _2)\!\ge \! \tau _1\!+\!\tau _2 \quad \text{ and } \quad \kappa _1\!+\!\kappa _2\!=\!\frac{\beta _2}{\alpha _2\!+\!\beta _2}(\tau _1\!+\!\tau _2)<\tau _1\!+\!\tau _2.\qquad \quad \end{aligned}$$

(3.35)

Therefore, either

$$\begin{aligned}&0<\kappa _1+\kappa _2<\tau _1+\tau _2\le \sigma _1+\sigma _2 \end{aligned}$$

(3.36a)

or

$$\begin{aligned} 0>\kappa _1+\kappa _2>\tau _1+\tau _2\ge \sigma _1+\sigma _2 \end{aligned}$$

(3.36b)

or

$$\begin{aligned} \kappa _1+\kappa _2 = \tau _1+\tau _2 = \sigma _1+\sigma _2 = 0 \end{aligned}$$

(3.36c)

applies, where equality in (3.36a) and (3.36b) holds if $\alpha _k=\beta _k$ ($k=1,2$). In addition,

$$\begin{aligned} \kappa _1+\kappa _2=0 \;\iff \; \tau _1+\tau _2=0 \;\iff \; \sigma _1+\sigma _2=0. \end{aligned}$$

(3.37)

With these preliminaries, we can treat the changes required in the above propositions if $\theta =0$.

From (3.36) we infer that in Proposition 3.1 not only $\mathsf M _2$ but also $\mathsf M _3$ is always unstable. In addition, if $0>\tau _1+\tau _2\ge \sigma _1+\sigma _2$, then $\mathsf M _1$ is asymptotically stable for sufficiently strong migration, whereas $\mathsf M _4$ is stable for sufficiently strong migration if $0<\tau _1+\tau _2\le \sigma _1+\sigma _2$ holds.

As already noted, Proposition 3.2 remains valid independently of the value of $\theta$.

In Proposition 3.3, the only SLPs through which the internal equilibrium $\mathsf{F _\infty }$ can leave the state space are $\mathsf{P _\mathsf{B ,1}}$ and $\mathsf{P _\mathsf{B ,2}}$; see (3.19c) and (3.19d). The reason is that, except when $\sigma _1+\sigma _2=0$ (and (3.37) applies), $\mathsf{P _\mathsf{A ,1}}$ and $\mathsf{P _\mathsf{A ,2}}$ are only admissible if $\mathsf{P _\mathsf{B ,1}}$ and $\mathsf{P _\mathsf{B ,2}}$ are. Thus, the locus under weaker selection always becomes monomorphic at lower rates of gene flow than the locus under stronger selection.

If $\rho =0$ (Proposition 3.4), $\mathsf{F _0}$ is asymptotically stable whenever it is admissible because $\tilde{m}\rightarrow \infty$ as $\theta \rightarrow 0$; see (3.28). In addition, (3.36) implies that $\mathsf{F _0}$ persists stronger gene flow than the SLPs, which are always unstable; see (3.32) and (3.33).

In the highly symmetric case of (3.36c), SLPs cannot be lost. Thus, $\mathsf{F _\infty }$ is always admissible and globally stable, cf. Proposition 3.3. If $\rho =0$, (3.37) implies that $\mathsf{F _0}$ exists always (and is stable). In the next section we show that in this highly symmetric case the FP is always admissible for arbitrary recombination rates.

3.10 The super-symmetric case

In many, especially ecological, applications highly symmetric migration-selection models are studied. Frequently made assumptions are that the migration rates between the demes are identical ($m_1=m_2$), selection in deme 2 mirrors that in deme 1 ($\alpha _k=-\alpha _{k^*}$), and the loci are equivalent ($\alpha _k=\beta _k$). Thus, $\theta =0$ and (3.36c) holds, which we assume now.

Conditions (3.7) and (3.8) imply that all four SLPs are admissible. Hence, all monomorphisms are unstable. In addition, it can be proved that all SLPs are unstable (Appendix A.6). If migration is weak, a globally asymptotically stable, fully polymorphic equilibrium ($\mathsf F$) exists (Proposition 3.2).

Because every boundary equilibrium is hyperbolic for every parameter choice, the index theorem of Hofbauer (1990) can be applied. Since none of the boundary equilibria is saturated, it follows that an internal equilibrium with index 1 exists. For small migration rates, this is $\mathsf F$ because it is unique. Since the boundary equilibria are always hyperbolic, no internal equilibrium can leave the state space through the boundary. However, we cannot exclude that the internal equilibrium undergoes a pitchfork or a Hopf bifurcation. Numerical results support the conjecture that the internal equilibrium is unique and globally attracting, independently of the strength of migration. This is a very special feature of this super-symmetric case; cf. Proposition 4.3.

3.11 General case

Because a satisfactory analysis for general parameter choices seems out of reach, we performed extensive numerical work to determine the possible equilibrium structures. In no case did we find more complicated equilibrium structures than indicated above, i.e., apparently there are never more than two internal equilibria. If there is one internal equilibrium, it appears to be globally asymptotically stable. If there are two internal equilibria, then one is unstable and the other is simultaneously stable with one boundary equilibrium (as in the CI model). Apparently, two internal equilibria occur only for sufficiently asymmetric migration rates and only if the recombination rate is of similar magnitude as the selection coefficients.

A glance at the dynamical equations (2.7) reveals that an internal equilibrium can be in LE only if $p_1=p_2$ or $q_1=q_2$. From (3.18), (3.4) and (3.5), we find that this can occur only if $|\sigma _1+\sigma _2|=1$ or $|\tau _1+\tau _2|=1$, i.e., for a boundary equilibrium. Thus, internal equilibria always exhibit LD.

For low migration rates as well as for high recombination rates, there is a unique, fully polymorphic equilibrium which is globally asymptotically stable and exhibits positive LD (Sects. 3.4 or 3.6). We denote the (presumably unique) asymptotically stable, fully polymorphic equilibrium by $\mathsf F$. If migration is weak, or recombination is weak, or recombination is strong, we have proved that $\mathsf F$ is unique. Useful approximations are available for weak migration or strong recombination; see (3.16) or (3.21). Finally, for sufficiently high migration rates one of the monomorphic equilibria is globally asymptotically stable.

4 Bifurcation patterns and maintenance of polymorphism

Here we study how genetic variation and polymorphism depend on the strength and pattern of migration. In particular, we are interested in determining how the maximum migration rate that permits genetic polymorphism depends on the other parameters. For this end, we explore properties of our model, such as the possible bifurcation patterns, as functions of the total migration rate $m$. We do this by assuming that $\alpha _1$, $\alpha _2$, $\beta _1$, $\beta _2$, $\rho$, and the migration ratio

$$\begin{aligned} \phi =\frac{m_1}{m}, \end{aligned}$$

(4.1)

where $m>0$ and $0\le \phi \le 1$, are constant. The values $\phi =0$ and $\phi =1$ correspond to one-way migration, as in the CI model. If $\phi =\tfrac{1}{2}$, migration between the demes is symmetric, an assumption made in many studies of migration-selection models. Fixing $\phi$ and treating $m$ as the only migration parameter corresponds to the migration scheme introduced by Deakin (1966).

4.1 Important quantities

We define several important quantities that will be needed to describe our results and summarize the relevant relations between them. Let

$$\begin{aligned} \phi ^\mathsf{A }&= \frac{\alpha _1}{\alpha _1-\alpha _2}, \end{aligned}$$

(4.2a)

$$\begin{aligned} \phi ^\mathsf{B }&= \frac{\beta _1}{\beta _1-\beta _2}, \end{aligned}$$

(4.2b)

$$\begin{aligned} \phi ^{\mathsf{F _0}}&= \frac{\alpha _1+\beta _1}{\alpha _1+\beta _1-(\alpha _2+\beta _2)}, \end{aligned}$$

(4.2c)

$$\begin{aligned} \tilde{\phi }^\mathsf{AB }&= \frac{\alpha _1\beta _1(\alpha _2-\beta _2)}{\alpha _1\beta _1(\alpha _2-\beta _2)-\alpha _2\beta _2(\alpha _1-\beta _1)}, \end{aligned}$$

(4.2d)

$$\begin{aligned} \phi ^\mathsf{AB }&= \frac{\alpha _1\beta _1(\alpha _2+\beta _2)}{\alpha _1\beta _1(\alpha _2+\beta _2)-\alpha _2\beta _2(\alpha _1+\beta _1)}, \end{aligned}$$

(4.2e)

$$\begin{aligned} \phi ^\mathsf{M _1}&= \frac{\alpha _1(\beta _2+\rho )(\alpha _1+\beta _1+\rho )}{\alpha _1(\beta _2+\rho )(\alpha _1+\beta _1+\rho )-\alpha _2(\beta _1+\rho )(\alpha _2+\beta _2+\rho )}, \end{aligned}$$

(4.2f)

$$\begin{aligned} \tilde{\phi }^\mathsf{M _1}&= \frac{\beta _1(\alpha _2+\rho )(\alpha _1+\beta _1+\rho )}{\beta _1(\alpha _2+\rho )(\alpha _1+\beta _1+\rho )-\beta _2(\alpha _1+\rho )(\alpha _2+\beta _2+\rho )}, \end{aligned}$$

(4.2g)

$$\begin{aligned} \phi ^\mathsf{M _4}&= \frac{\beta _1(\alpha _2-\rho )(\alpha _1+\beta _1-\rho )}{\beta _1(\alpha _2-\rho )(\alpha _1+\beta _1-\rho )-\beta _2(\alpha _1-\rho )(\alpha _2+\beta _2-\rho )}, \end{aligned}$$

(4.2h)

$$\begin{aligned} \tilde{\phi }^\mathsf{M _4}&= \frac{\alpha _1(\beta _2-\rho )(\alpha _1+\beta _1-\rho )}{\alpha _1(\beta _2-\rho )(\alpha _1+\beta _1-\rho )-\alpha _2(\beta _1-\rho )(\alpha _2+\beta _2-\rho )}, \end{aligned}$$

(4.2i)

$$\begin{aligned} \phi ^\mathsf{A \mathsf{F _0}}&= \frac{\alpha _1(\alpha _1+\beta _1)(2\alpha _2+\beta _2)}{\alpha _1(\alpha _1+\beta _1)(2\alpha _2+\beta _2)-\alpha _2(\alpha _2+\beta _2)(2\alpha _1+\beta _1)}, \end{aligned}$$

(4.2j)

$$\begin{aligned} \phi ^\mathsf{B \mathsf{F _0}}&= \frac{\beta _1(\alpha _1+\beta _1)(\alpha _2+2\beta _2)}{\beta _1(\alpha _1+\beta _1)(\alpha _2+2\beta _2)-\beta _2(\alpha _2+\beta _2)(\alpha _1+2\beta _1)}, \end{aligned}$$

(4.2k)

and

$$\begin{aligned} m^\mathsf{A }&= \frac{\alpha _1\alpha _2}{\alpha _1-\phi (\alpha _1-\alpha _2)}, \end{aligned}$$

(4.3a)

$$\begin{aligned} m^\mathsf{B }&= \frac{\beta _1\beta _2}{\beta _1-\phi (\beta _1-\beta _2)}, \end{aligned}$$

(4.3b)

$$\begin{aligned} m^{\mathsf{F _0}}&= \frac{(\alpha _1+\beta _1)(\alpha _2+\beta _2)}{\alpha _1+\beta _1-\phi (\alpha _1+\beta _1-\alpha _2-\beta _2)},\end{aligned}$$

(4.3c)

$$\begin{aligned} m^\mathsf{M _1}&= \frac{-(\alpha _1+\beta _1+\rho )(\alpha _2+\beta _2+\rho )}{\alpha _1+\beta _1+\rho -\phi (\alpha _1+\beta _1-\alpha _2-\beta _2)}, \end{aligned}$$

(4.3d)

$$\begin{aligned} m^\mathsf{M _4}&= \frac{(\alpha _1+\beta _1-\rho )(\alpha _2+\beta _2-\rho )}{\alpha _1+\beta _1-\rho -\phi (\alpha _1+\beta _1-\alpha _2-\beta _2)}, \end{aligned}$$

(4.3e)

$$\begin{aligned} m^{*}&= \frac{1}{\theta }\sqrt{\frac{-\alpha _1\alpha _2\beta _1\beta _2(\alpha _1+\beta _1)(\alpha _2+\beta _2)}{\phi (1-\phi )}}. \end{aligned}$$

(4.3f)

We set $m^\mathsf{A }=\infty$, $m^\mathsf{B }=\infty$, and $m^{\mathsf{F _0}}=\infty$ if $\phi =\phi ^\mathsf{A }$, $\phi =\phi ^\mathsf{B }$, and $\phi =\phi ^{\mathsf{F _0}}$, respectively. Similarly, we set $m^{*}=\infty$ if $\theta =0$, $\phi =0$, or $\phi =1$.

The quantities $m^\mathsf{A }$, $m^\mathsf{B }$, and $m^{\mathsf{F _0}}$ yield the bounds for the intervals of total migration rates $m$ in which the SLPs at $\mathsf{A }$, $\mathsf{B }$, and the polymorphic equilibrium $\mathsf{F _0}$, respectively, are admissible:

$$\begin{aligned} -1 < \sigma _1+\sigma _2 < 1&\iff -1 < \frac{m}{m^\mathsf{A }} < 1 ,\end{aligned}$$

(4.4a)

$$\begin{aligned} -1 < \tau _1+\tau _2 < 1&\iff -1 < \frac{m}{m^\mathsf{B }} < 1 ,\end{aligned}$$

(4.4b)

$$\begin{aligned} -1 < \kappa _1+\kappa _2 <1&\iff -1 < \frac{m}{m^{\mathsf{F _0}}} < 1 . \end{aligned}$$

(4.4c)

Here, the left and the right inequalities correspond, and we have

$$\begin{aligned} m^\mathsf{A }>0&\iff \phi >\phi ^\mathsf{A },\end{aligned}$$

(4.5a)

$$\begin{aligned} m^\mathsf{B }>0&\iff \phi >\phi ^\mathsf{B },\end{aligned}$$

(4.5b)

$$\begin{aligned} m^{\mathsf{F _0}}>0&\iff \phi >\phi ^{\mathsf{F _0}}. \end{aligned}$$

(4.5c)

From (2.3), we obtain

$$\begin{aligned}&\qquad \qquad m^\mathsf{A }\ne 0, \quad m^\mathsf{B }\ne 0, \quad m^{\mathsf{F _0}}\ne 0, \quad m^{*}>0,\end{aligned}$$

(4.6)

$$\begin{aligned}&\alpha _2\le m^\mathsf{A }\le \alpha _1, \quad \beta _2\le m^\mathsf{B }\le \beta _1, \quad \alpha _2+\beta _2\le m^{\mathsf{F _0}}\le \alpha _1+\beta _1. \end{aligned}$$

(4.7)

The quantities $m^\mathsf{M _1}$ and $m^\mathsf{M _4}$ occur in the stability conditions of the monomorphic equilibria $\mathsf M _1$ and $\mathsf M _4$ (Proposition 4.1), and $m^{*}$ determines the range of stability of $\mathsf{F _0}$; see (4.56). They satisfy

$$\begin{aligned} \!-\!(\alpha _1\!+\!\beta _1\!+\!\rho )\le m^\mathsf{M _1}\le -(\alpha _2+\beta _2+\rho ), \quad \alpha _2+\beta _2-\rho \le m^\mathsf{M _4}\le \alpha _1+\beta _1-\rho .\nonumber \\ \end{aligned}$$

(4.8)

We note that $m^\mathsf{A }$, $m^\mathsf{B }$, $m^{\mathsf{F _0}}$, and $m^\mathsf{M _4}$ assume their minima if $\phi =0$ and their maxima if $\phi =1$, whereas $m^\mathsf{M _1}$ assumes its minimum or maximum at $\phi =1$ or $\phi =0$, respectively. $m^{*}$ is a convex function of $\phi$, and symmetric around its minimum $\phi =1/2$.

The definitions of (several of) the quantities $\phi ^\mathsf X$ are motivated by the following relations:

$$\begin{aligned} m^\mathsf{A }= m^\mathsf{B }&\iff \phi =\tilde{\phi }^\mathsf{AB }\quad \text{ and } \, m^\mathsf{A }<0, \end{aligned}$$

(4.9a)

$$\begin{aligned} m^\mathsf{A }= -m^\mathsf{B }&\iff \phi =\phi ^\mathsf{AB }\quad \text{ and } \, m^\mathsf{A }>0, \end{aligned}$$

(4.9b)

$$\begin{aligned} m^\mathsf{M _1}= -m^\mathsf{A }&\iff \phi =\phi ^\mathsf{M _1}\quad \text{ and } \, m^\mathsf{A }<0, \end{aligned}$$

(4.9c)

$$\begin{aligned} m^\mathsf{M _1}= -m^\mathsf{B }&\iff \phi =\tilde{\phi }^\mathsf{M _1}\quad \text{ and } \, m^\mathsf{B }<0, \end{aligned}$$

(4.9d)

$$\begin{aligned} m^\mathsf{M _4}= m^\mathsf{A }&\iff \phi =\tilde{\phi }^\mathsf{M _4}\quad \text{ and } \, m^\mathsf{A }>0, \end{aligned}$$

(4.9e)

$$\begin{aligned} m^\mathsf{M _4}= m^\mathsf{B }&\iff \phi =\phi ^\mathsf{M _4}\quad \text{ and } \, m^\mathsf{B }>0, \end{aligned}$$

(4.9f)

$$\begin{aligned} m^{\mathsf{F _0}}= -m^\mathsf{B }&\iff \phi =\phi ^\mathsf{B \mathsf{F _0}}\quad \text{ and } \, m^\mathsf{B }<0, \end{aligned}$$

(4.9g)

$$\begin{aligned} -m^{\mathsf{F _0}}= m^\mathsf{A }&\iff \phi =\phi ^\mathsf{A \mathsf{F _0}}\quad \text{ and } \, m^\mathsf{A }>0, \end{aligned}$$

(4.9h)

where we have

$$\begin{aligned} m^{\mathsf{F _0}}=m^\mathsf{M _4}= -m^\mathsf{M _1}\;\iff \; \rho =0. \end{aligned}$$

(4.10)

The following relations apply to $m^{*}$:

$$\begin{aligned} m^\mathsf{A }= -m^\mathsf{B }= m^{*}&\iff \phi =\phi ^\mathsf{AB }, \end{aligned}$$

(4.11a)

$$\begin{aligned} m^\mathsf{A }= m^{\mathsf{F _0}}= m^{*}&\iff \phi =\phi ^\mathsf{M _4}, \end{aligned}$$

(4.11b)

$$\begin{aligned} -m^\mathsf{B }= -m^{\mathsf{F _0}}=m^{*}&\iff \phi =\phi ^\mathsf{M _1}, \end{aligned}$$

(4.11c)

where we derived (4.11b) and (4.11c) from (4.9c) and (4.9f) using (4.10).

In the following, we summarize the most important inequalities between the quantities $\phi ^\mathsf X$:

$$\begin{aligned}&0<\phi ^\mathsf{A }< \phi ^\mathsf{A \mathsf{F _0}}< \phi ^\mathsf{AB }< \phi ^\mathsf{B \mathsf{F _0}}< \phi ^\mathsf{B }<1,\end{aligned}$$

(4.12)

$$\begin{aligned}&\qquad \beta _2 < \alpha _2 \; \iff \; 0 < \tilde{\phi }^\mathsf{AB }< \phi ^\mathsf{A },\end{aligned}$$

(4.13)

$$\begin{aligned}&\qquad \qquad \phi ^\mathsf{A \mathsf{F _0}}< \phi ^{\mathsf{F _0}}< \phi ^\mathsf{B \mathsf{F _0}}. \end{aligned}$$

(4.14)

They can be derived straightforwardly from their definitions and our general assumption (2.3). Finally, if $\rho =0$ and $\tilde{\theta }=\alpha _1\alpha _2-\beta _1\beta _2$, the following relations hold:

$$\begin{aligned} 0<\phi ^\mathsf{M _1}<\phi ^\mathsf{A }<\phi ^\mathsf{A \mathsf{F _0}}<\phi ^{\mathsf{F _0}}\le \phi ^\mathsf{AB }<\phi ^\mathsf{B \mathsf{F _0}}<\phi ^\mathsf{B }<\phi ^\mathsf{M _4}<1&\iff \tilde{\theta }\le 0,\nonumber \\\end{aligned}$$

(4.15a)

$$\begin{aligned} 0<\phi ^\mathsf{M _1}<\phi ^\mathsf{A }<\phi ^\mathsf{A \mathsf{F _0}}<\phi ^\mathsf{AB }< \phi ^{\mathsf{F _0}}<\phi ^\mathsf{B \mathsf{F _0}}<\phi ^\mathsf{B }<\phi ^\mathsf{M _4}<1&\iff \tilde{\theta }> 0, \nonumber \\ \end{aligned}$$

(4.15b)

and

$$\begin{aligned} \tilde{\phi }^\mathsf{AB }<\phi ^\mathsf{M _1}\quad \text{ if } \;\beta _2<\alpha _2. \end{aligned}$$

(4.15c)

Additional relations that are needed only in the proofs may be found in Appendix A.7.

4.2 Admissibility of SLPs

We begin by expressing the conditions for admissibility of the SLPs in terms of the total migration rate $m$ and the migration ratio $\phi$. Since, by (3.7), (3.8), and (4.4), every SLP is admissible if $m$ is sufficiently small and leaves the state space at a uniquely defined critical migration rate, it is sufficient to determine this critical rate and the monomorphism through which it leaves the state space. Using (4.3a), (4.3b), (4.5), and (4.4), we infer from (3.9) that

$$\begin{aligned} \phi <\phi ^\mathsf{A } \text{ and } \, m\uparrow -m^\mathsf{A }&\iff \mathsf P _\mathsf{A ,1} \rightarrow \mathsf M _1 \text{ and } \, \mathsf P _\mathsf{A ,2} \rightarrow \mathsf M _2,\end{aligned}$$

(4.16a)

$$\begin{aligned} \phi >\phi ^\mathsf{A } \text{ and } \, m\uparrow m^\mathsf{A }&\iff \mathsf P _\mathsf{A ,1} \rightarrow \mathsf M _3 \text{ and } \, \mathsf P _\mathsf{A ,2} \rightarrow \mathsf M _4, \end{aligned}$$

(4.16b)

$$\begin{aligned} \phi <\phi ^\mathsf{B } \text{ and } \, m\uparrow -m^\mathsf{B }&\iff \mathsf P _\mathsf{B ,1} \rightarrow \mathsf M _1 \text{ and } \, \mathsf P _\mathsf{B ,2} \rightarrow \mathsf M _3,\end{aligned}$$

(4.16c)

$$\begin{aligned} \phi >\phi ^\mathsf{B } \text{ and } \, m\uparrow m^\mathsf{B }&\iff \mathsf P _\mathsf{B ,1} \rightarrow \mathsf M _2 \text{ and } \, \mathsf P _\mathsf{B ,2} \rightarrow \mathsf M _4. \end{aligned}$$

(4.16d)

In particular, no SLP is admissible if

$$\begin{aligned} m>\max \{|m^\mathsf{A }|,|m^\mathsf{B }|\}. \end{aligned}$$

(4.17)

We observe that locus $\mathsf{A }$ is polymorphic and locus $\mathsf{B }$ is monomorphic if and only if

$$\begin{aligned} |m^\mathsf{B }|<m<|m^\mathsf{A }|. \end{aligned}$$

(4.18)

If $\beta _2<\alpha _2$, we infer from (10.18c) and (10.18d) that (4.18) holds if and only if

$$\begin{aligned} \tilde{\phi }^\mathsf{AB }< \phi < \phi ^\mathsf{AB }. \end{aligned}$$

(4.19)

Therefore, (2.4) implies that if locus $\mathsf{A }$ is under weaker selection than locus $\mathsf{B }$ in both demes ($|\beta _k|>|\alpha _k|$), then there is a range of values $\phi$ and $m$ such that locus $\mathsf{A }$ is polymorphic whereas $\mathsf{B }$ is monomorphic. This is in contrast to the CI model or highly asymmetric migration rates or $\theta =0$, where it is always the locus under weaker selection that first loses its polymorphism while $m$ increases. This is a pure one-locus result and a consequence of the classical condition for a protected polymorphism, e.g., (3.7). With two-way migration, a locus with alleles of small and similar (absolute) effects in the demes ($\alpha _1\approx -\alpha _2$) may be maintained polymorphic for higher migration rates than a locus with alleles of large and very different (absolute) effects.

4.3 Stability of monomorphic equilibria

Here, we reformulate the stability conditions of the ME derived in Sect. 3.2 in terms of $m$ and $\phi$.

Proposition 4.1

$\mathsf M _1$ is asymptotically stable if

$$\begin{aligned} \phi <\phi ^\mathsf{A } \text{ and } \, m > \max \{-m^\mathsf{A },-m^\mathsf{B }, m^\mathsf{M _1}\}. \end{aligned}$$

(4.20)

$\mathsf M _2$ is always unstable.

$\mathsf M _3$ is asymptotically stable if

$$\begin{aligned} \phi ^\mathsf{A }<\phi <\phi ^\mathsf{B } \text{ and } \, m>\max \{m^\mathsf{A },-m^\mathsf{B }\}. \end{aligned}$$

(4.21)

$\mathsf M _4$ is asymptotically stable if

$$\begin{aligned} \phi >\phi ^\mathsf{B } \text{ and } \, m > \max \{m^\mathsf{A },m^\mathsf{B },m^\mathsf{M _4}\}. \end{aligned}$$

(4.22)

If in these conditions one inequality is reversed, the corresponding equilibrium is unstable.

Proof

We prove only that the statement about $\mathsf M _1$ is equivalent to that in Proposition 3.1. The others follow analogously or are immediate.

From Proposition 3.1 and (4.3a), (4.3b), (4.3d), and (4.4), we infer immediately that $\mathsf M _1$ is asymptotically stable if and only if

$$\begin{aligned} 1<\frac{m}{-m^\mathsf{A }} \text{ and } \, 1<\frac{m}{-m^\mathsf{B }} \end{aligned}$$

(4.23)

and

$$\begin{aligned} m > m^\mathsf{M _1}. \end{aligned}$$

(4.24)

The possible inequalities between $-m^\mathsf{A }$, $-m^\mathsf{B }$, and $m^\mathsf{M _1}$ are given in (10.32) and (10.33). By (4.5a), (4.5b), and (4.12), it follows that (4.23) is feasible if and only if $\phi <\phi ^\mathsf{A }$. Thus if $\phi \ge \phi ^\mathsf{A }$, $\mathsf M _1$ is unstable. Therefore, (4.23) and (4.24) are equivalent to (4.20). $\square$

Remark 4.2

• (i) We have $\max \{-m^\mathsf{A },-m^\mathsf{B },m^\mathsf{M _1}\}=m^\mathsf{M _1}$ in (4.20) if and only if

  $$\begin{aligned}&\phi <\min \{\phi ^\mathsf{M _1},\tilde{\phi }^\mathsf{M _1}\} \text{ and } \, \rho < -\alpha _2 \text{ and } \, \beta _2<\alpha _2, \text{ or } \end{aligned}$$

  (4.25a)
  $$\begin{aligned}&\phi <\phi ^\mathsf{M _1} \text{ and } \,\rho < -\beta _2 \text{ and } \, \beta _2\ge \alpha _2. \end{aligned}$$

  (4.25b)

• (ii) We have $\max \{m^\mathsf{A },m^\mathsf{B },m^\mathsf{M _4}\}=m^\mathsf{M _4}$ in (4.22) if and only if

  $$\begin{aligned} \phi >\phi ^\mathsf{M _4} \text{ and } \, \rho <\alpha _1. \end{aligned}$$

  (4.26)

• (iii) An internal equilibrium in LD can leave or enter the state space through $\mathsf M _1$ or $\mathsf M _4$ only if $m=m^\mathsf{M _1}$ or $m=m^\mathsf{M _4}$, respectively. If (4.25) or (4.26) holds, then $\mathsf M _1$ or $\mathsf M _4$, respectively, become asymptotically stable by the bifurcation.

Proof of Remark 4.2

If $\beta _2\ge \alpha _2$, statement (i) is an immediate consequence of (10.32a) and (10.32b) because $\phi <\phi ^\mathsf{A }$ implies $0<\min \{-m^\mathsf{A },-m^\mathsf{B }\}$. If $\beta _2<\alpha _2$, then (10.32a), (10.32b), and (10.32c) show that $\max \{-m^\mathsf{A },-m^\mathsf{B }, m^\mathsf{M _1}\}=m^\mathsf{M _1}$ if (a) $\rho <\rho ^\mathsf{M _1}$ (10.23) and $\tilde{\phi }^\mathsf{AB }<\phi <\phi ^\mathsf{M _1}$ or (b) $\rho <\rho ^\mathsf{M _1}$ and $\phi \le \tilde{\phi }^\mathsf{AB }$ or (c) $\rho ^\mathsf{M _1}<\rho <-\alpha _2$ and $\phi <\tilde{\phi }^\mathsf{M _1}$, where $\rho ^\mathsf{M _1}<-\alpha _2$ by (10.26a). Invoking (10.31), we can combine conditions (a), (b), and (c) to obtain (4.25a).

Statement (ii) follows directly from (10.18f) and (10.35a).

Statement (iii) follows by observing that only internal equilibria in LD will depend on $\rho$, the factor $t_3$ (10.4) in the characteristic polynomial at $\mathsf M _1$ is the only one that depends on $\rho$, and $t_3$ gives rise to an eigenvalue zero if and only if $m=m^\mathsf{M _1}$. An analogous argument holds for $\mathsf M _4$. $\square$

The asymmetry between (4.25) and (4.26) results from the fact that $\alpha _1<\beta _1$ is assumed, whereas $\beta _2<\alpha _2$ or $\beta _2\ge \alpha _2$ is possible. The reader may recall the comments made below Proposition 3.1. In addition, we note that if the fitness parameters and $\rho$ and $\phi$ are fixed, a stable ME remains stable if $m$ is increased. This is not necessarily so if $m$ and $\phi$ are varied simultaneously. For related phenomena in the one-locus case, see Karlin (1982) and Nagylaki (2012). In Sect. 4.5, we will prove global convergence to one of the asymptotically stable ME if $m$ is sufficiently large.

4.4 Weak migration

We recall from Proposition 3.2 that for sufficiently weak migration, there is a fully polymorphic equilibrium, it is globally asymptotically stable, and exhibits positive LD in both demes.

4.5 Strong migration

Proposition 4.3

For sufficiently large $m$, one of the monomorphic equilibria $\mathsf M _1$, $\mathsf M _3$, or $\mathsf M _4$ is globally attracting. This equilibrium is $\mathsf M _1$, $\mathsf M _3$, or $\mathsf M _4$ if $\phi <\phi ^\mathsf{A }$, $\phi ^\mathsf{A }<\phi <\phi ^\mathsf{B }$, or $\phi ^\mathsf{B }<\phi$, respectively.

Proof

The proof is based on the perturbation results about the strong-migration limit in Section 4.2 of Bürger (2009a). The strong-migration limit is obtained if $\max _{k=1,2}\{|\alpha _k|,|\beta _k|,\rho \}/m\rightarrow 0$. In this limit, the demes become homogeneous and the system of differential equations (2.7) converges to a system, where in each deme

$$\begin{aligned} \dot{p}&= \alpha p(1-p)+\beta D,\end{aligned}$$

(4.27a)

$$\begin{aligned} \dot{q}&= \beta q(1-q)+\alpha D,\end{aligned}$$

(4.27b)

$$\begin{aligned} \dot{D}&= \left[ \alpha (1-2p)+\beta (1-2q)-\rho \right] D \end{aligned}$$

(4.27c)

holds with $p_1=p_2=p$, $q_1=q_2=q$, $D_1=D_2=D$. Here,

$$\begin{aligned} \alpha = (1-\phi )\alpha _1+\phi \alpha _2 \quad \text{ and } \, \beta = (1-\phi )\beta _1+\phi \beta _2 \end{aligned}$$

(4.28)

are the spatially averaged selection coefficients and averaging is performed with respect to the Perron-Frobenius eigenvector $(1-\phi ,\phi )$ of the migration matrix (see Section 4.2 in Bürger 2009a for a much more general treatment starting with a multilocus model in discrete time). Therefore, Proposition 4.10 in Bürger (2009a) applies and, provided $m$ is sufficiently large, all trajectories of (2.7) converge to a manifold on which the allele frequencies and the linkage disequilibria in both demes are nearly identical. In addition, in the neighborhood of each hyperbolic equilibrium of (4.27) there is exactly one equilibrium of (2.7), and it has the same stability.

In the present case, the conclusion of Proposition 4.10 in Bürger (2009a) can be considerably strengthened. Because the system (4.27) describes evolution in an ordinary two-locus model under genic selection, the ME representing the gamete of highest fitness is globally asymptotically stable. In fact, (4.27) is also a generalized gradient system for which Lemma 2.2 of Nagylaki et al. (1999) holds. Therefore, the analog of statement (c) in Theorem 4.3 of Bürger (2009a) applies and yields global convergence to the unique stable equilibrium.

Finally, it is an easy exercise to show that, in the strong-migration limit, i.e., with fitnesses averaged according to (4.28), gamete $A_1B_1$, $A_2B_1$, or $A_2B_2$ has highest fitness if $\phi <\phi ^\mathsf{A }$, $\phi ^\mathsf{A }<\phi <\phi ^\mathsf{B }$, or $\phi >\phi ^\mathsf{B }$, respectively. Since there is no dominance, the corresponding ME is the unique stable equilibrium. $\square$

4.6 Linkage equilibrium

We shall establish all possible equilibrium configurations and their dependence on the parameters under LE. In Fig. 2, the equilibrium configurations are displayed as schematic bifurcation diagrams with the total migration rate $m$ as the bifurcation parameter. In Theorem 4.4, we assign to each diagram its pertinent parameter combinations.

Fig. 2

Bifurcation diagrams for LE. Diagrams (a)–(c) display the equilibrium configurations listed in Theorem 4.4. Each line indicates one admissible equilibrium as a function of the total migration rate $m$. Only equilibria are shown that can be stable or are involved in a bifurcation with an equilibrium that can be stable. Lines are drawn such that intersections occur if and only if the corresponding equilibria collide. Solid lines represent asymptotically stable equilibria, dashed lines unstable equilibria. The meaning of the superscripts $\mathsf{X }$ and $\mathsf{Y }$ is given in (L1)–(L5)

In order to have only one bifurcation diagram covering cases that can be obtained from each other by simple symmetry considerations but are structurally equivalent otherwise, we use the sub- and superscripts $\mathsf{X }$ and $\mathsf{Y }$ in the labels of Fig. 2. For an efficient presentation of the results, we define

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf{P _\mathsf{B ,1}},\; \mathsf P _\mathsf Y =\mathsf{P _\mathsf{A ,1}},\; \mathsf M _\mathsf i =\mathsf M _1,\; m^\mathsf X =-m^\mathsf{B },\; m^\mathsf Y =-m^\mathsf{A }, \end{aligned}$$

(L1)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf{P _\mathsf{A ,1}},\; \mathsf P _\mathsf Y =\mathsf{P _\mathsf{B ,1}},\; \mathsf M _\mathsf i =\mathsf M _1,\; m^\mathsf X =-m^\mathsf{A },\; m^\mathsf Y =-m^\mathsf{B }, \end{aligned}$$

(L2)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf{P _\mathsf{A ,1}},\; \mathsf P _\mathsf Y =\mathsf{P _\mathsf{B ,2}},\; \mathsf M _\mathsf i =\mathsf M _3,\; m^\mathsf X =m^\mathsf{A },\; m^\mathsf Y =-m^\mathsf{B }, \end{aligned}$$

(L3)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf{P _\mathsf{B ,2}},\; \mathsf P _\mathsf Y =\mathsf{P _\mathsf{A ,1}},\; \mathsf M _\mathsf i =\mathsf M _3,\; m^\mathsf X =-m^\mathsf{B },\; m^\mathsf Y =m^\mathsf{A }, \end{aligned}$$

(L4)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf{P _\mathsf{B ,2}},\; \mathsf P _\mathsf Y =\mathsf{P _\mathsf{A ,2}},\; \mathsf M _\mathsf i =\mathsf M _4,\; m^\mathsf X =m^\mathsf{B },\; m^\mathsf Y =m^\mathsf{A }. \end{aligned}$$

(L5)

Theorem 4.4

Assume LE, i.e., (3.17). Figure 2 shows all possible bifurcation diagrams that involve bifurcations with equilibria that can be stable for some $m$ given the other parameters.

A. Diagram (a) in Fig. 2 occurs generically. It occurs if and only if one of the following cases applies:

$$\begin{aligned} \phi <\tilde{\phi }^\mathsf{AB }\quad \text{ and } \quad \beta _2<\alpha _2 \quad \text{ and } \quad (\text{ L }1) \end{aligned}$$

(4.29)

or

$$\begin{aligned} \tilde{\phi }^\mathsf{AB }<\phi <\phi ^\mathsf{A }\quad \text{ and } \quad \beta _2<\alpha _2 \quad \text{ and } \quad \, (\text{ L }2) \end{aligned}$$

(4.30)

or

$$\begin{aligned} \phi <\phi ^\mathsf{A }\quad \text{ and } \quad \alpha _2\le \beta _2 \quad \text{ and } \quad (\text{ L }2) \end{aligned}$$

(4.31)

or

$$\begin{aligned} \phi ^\mathsf{A }<\phi <\phi ^\mathsf{AB }\quad \text{ and } \quad (\text{ L }3) \end{aligned}$$

(4.32)

or

$$\begin{aligned} \phi ^\mathsf{AB }<\phi <\phi ^\mathsf{B }\quad \text{ and } \quad (\text{ L }4) \end{aligned}$$

(4.33)

or

$$\begin{aligned} \phi ^\mathsf{B }<\phi \quad \text{ and } \quad (\text{ L }5). \end{aligned}$$

(4.34)

B. The following two diagrams occur only if the parameters satisfy particular relations.

Diagram (b) in Fig. 2 applies if one of the following two cases holds:

$$\begin{aligned} \phi =\tilde{\phi }^\mathsf{AB }\quad \text{ and } \quad \beta _2<\alpha _2 \quad \text{ and } \quad (\text{ L }1) \end{aligned}$$

(4.35)

or

$$\begin{aligned} \phi =\phi ^\mathsf{AB }\quad \text{ and } \quad (\text{ L }4). \end{aligned}$$

(4.36)

Diagram (c) in Fig. 2 applies if one of the following two cases holds:

$$\begin{aligned} \phi =\phi ^\mathsf{A }\quad \text{ and } \quad \mathsf P _\mathsf X =\mathsf{P _\mathsf{A ,1}}\quad \text{ and } \quad m^\mathsf Y =-m^\mathsf{B }\end{aligned}$$

(4.37)

or

$$\begin{aligned} \phi =\phi ^\mathsf{B }\quad \text{ and } \quad \mathsf P _\mathsf X =\mathsf{P _\mathsf{B ,2}}\quad \text{ and } \quad m^\mathsf Y =m^\mathsf{A }. \end{aligned}$$

(4.38)

C. Figure 3 shows the order in which the bifurcation diagrams of Fig. 2 arise if $\phi$ is increased from 0 to 1.

Fig. 3

Order in which the bifurcation diagrams of Fig. 2 occur as $\phi$ increases from 0 to 1

Proof

We prove parts A and B simultaneously, essentially by rewriting the conditions in Proposition 3.3 on admissibility and stability of the equilibria in terms of $m$, $m^\mathsf{A }$, and $m^\mathsf{B }$ (4.3).

From (3.19) and (4.4), we infer easily:

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{A ,1}}&\;\iff \; m \uparrow -m^\mathsf{B } \text{ and } \, 0<-m^\mathsf{B }<|m^\mathsf{A }|, \end{aligned}$$

(4.39a)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{A ,2}}&\;\;\text{ does } \text{ not } \text{ occur }, \end{aligned}$$

(4.39b)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{B ,1}}&\;\iff \; m \uparrow -m^\mathsf{A } \text{ and } \, 0<-m^\mathsf{A }<-m^\mathsf{B }, \end{aligned}$$

(4.39c)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{B ,2}}&\;\iff \; m \uparrow m^\mathsf{A } \text{ and } \, 0<m^\mathsf{A }<|m^\mathsf{B }|, \end{aligned}$$

(4.39d)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf M _1&\;\iff \; m \uparrow -m^\mathsf{A }=-m^\mathsf{B }\; \text{ and } 0<-m^\mathsf{A }=-m^\mathsf{B }, \end{aligned}$$

(4.39e)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf M _3&\;\iff \; m \uparrow m^\mathsf{A }=-m^\mathsf{B }\; \text{ and } 0<m^\mathsf{A }=-m^\mathsf{B }, \end{aligned}$$

(4.39f)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf M _2&\text{ or } \mathsf{F _\infty }\rightarrow \mathsf M _4 \text{ do } \text{ not } \text{ occur. } \end{aligned}$$

(4.39g)

Invoking the relations (10.18), we can rewrite conditions (4.39a), (4.39c)–(4.39f) in the form

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{A ,1}}\!&\iff \! m \uparrow \!-\!m^\mathsf{B } \text{ and } \text{ either } \nonumber \\&\quad \phi <\phi ^\mathsf{AB } \text{ if } \alpha _2\!\le \!\beta _2 \text{ or } \tilde{\phi }^\mathsf{AB }<\phi <\phi ^\mathsf{AB } \text{ if } \beta _2\!<\!\alpha _2, \nonumber \\ \end{aligned}$$

(4.40a)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{B ,1}}\!&\iff \! m \uparrow -m^\mathsf{A } \text{ and } \beta _2<\alpha _2 \text{ and } \phi <\tilde{\phi }^\mathsf{AB }, \end{aligned}$$

(4.40b)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf{P _\mathsf{B ,2}}\!&\iff \! m \uparrow m^\mathsf{A } \text{ and } \phi >\phi ^\mathsf{AB }, \end{aligned}$$

(4.40c)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf M _1 \!&\iff \! m \uparrow -m^\mathsf{A }\!=\!-m^\mathsf{B } \text{ and } \beta _2<\alpha _2 \text{ and } \phi =\tilde{\phi }^\mathsf{AB }, \end{aligned}$$

(4.40d)

$$\begin{aligned} \mathsf{F _\infty }\rightarrow \mathsf M _3 \!&\iff \! m \uparrow m^\mathsf{A }\!=\!-m^\mathsf{B } \text{ and } \phi =\phi ^\mathsf{AB }. \end{aligned}$$

(4.40e)

We conclude immediately that (4.40b) applies in case (4.29) (Part A), (4.40d) in case (4.35) (Part B), and (4.40e) in case (4.36) (Part B). From (4.12) and (4.13) we conclude that (4.40a) applies in the following cases: (4.30)–(4.32) (Part A), or (4.37) (Part B). Analogously we conclude that (4.40c) applies in the following cases: (4.33), (4.34) (Part A), or (4.38) (Part B).

From Proposition 3.3 and (4.16) we obtain:

$$\begin{aligned} \mathsf{P _\mathsf{A ,1}} \text{ is } \text{ globally } \text{ asymptotically } \text{ stable }&\iff -m^\mathsf{B }<m<|m^\mathsf{A }|,\end{aligned}$$

(4.41a)

$$\begin{aligned} \mathsf{P _\mathsf{B ,1}} \text{ is } \text{ globally } \text{ asymptotically } \text{ stable }&\iff -m^\mathsf{A }<m<-m^\mathsf{B },\end{aligned}$$

(4.41b)

$$\begin{aligned} \mathsf{P _\mathsf{B ,2}} \text{ is } \text{ globally } \text{ asymptotically } \text{ stable }&\iff m^\mathsf{A }<m<|m^\mathsf{B }|. \end{aligned}$$

(4.41c)

As $m \rightarrow \max \{|m^\mathsf{A }|,|m^\mathsf{B }|\}$, the stable SLP leaves the state space according to (4.16), which gives precisely the cases corresponding to diagrams (a) and (c). If $\phi =\phi ^\mathsf{A }$ ($m^\mathsf{A }=\infty$), $\mathsf{P _\mathsf{A ,1}}$ is always admissible, cf. (4.37). If $\phi =\phi ^\mathsf{B }$ ($m^\mathsf{B }=\infty$), $\mathsf{P _\mathsf{B ,2}}$ is always admissible, cf. (4.38).

A ME is globally asymptotically stable and only if

$$\begin{aligned} m \ge \max \{|m^\mathsf{A }|,|m^\mathsf{B }|\}. \end{aligned}$$

(4.42)

By Proposition 4.1 and Remark 4.2 this equilibrium is $\mathsf M _1$ if $\phi <\phi ^\mathsf{A }$ (cases (4.29)-(4.31), (4.35)), or $\mathsf M _3$ if $\phi ^\mathsf{A }<\phi <\phi ^\mathsf{B }$ (cases (4.32), (4.33), (4.36)), or $\mathsf M _4$ if $\phi ^\mathsf{B }<\phi$ (4.34). $\square$

The bifurcations of equilibria that cannot be stable can be derived easily from Sects. 4.2 and 4.3 and the above theorem by noting that these are boundary equilibria and corresponding pairs of SLPs are admissible for the same parameters; see (3.7) and (3.8). Inclusion of these bifurcations would require the introduction of subcases.

Corollary 4.5

Under the assumption of LE, the maximum migration rate, below which a stable two-locus polymorphism exists, is given by

$$\begin{aligned} m_{\max }^\infty = \min \{|m^\mathsf{A }|,|m^\mathsf{B }|\}. \end{aligned}$$

(4.43)

The corollary is a simple consequence of Proposition 3.3 and (4.40).

4.7 Strong recombination: quasi-linkage equilibrium

We recall from Sect. 3.6 that for sufficiently strong recombination, global convergence to the unique stable equilibrium occurs. From the coordinates (3.21) of the perturbed internal equilibrium, which is in quasi-linkage equilibrium, approximations could be derived for the critical migration rates at which the internal equilibrium collides with a boundary equilibrium and leaves the state space. It is not difficult to check with Mathematica that for large $\rho$, $\mathsf F$ collides with $\mathsf{P _\mathsf{B ,2}}$ if $m=m_{\max }^\rho (\mathsf{P _\mathsf{B ,2}})+O(\rho ^{-2})$, where

$$\begin{aligned} m_{\max }^\rho (\mathsf{P _\mathsf{B ,2}})&= m^\mathsf{A }- \frac{(m^\mathsf{A })^3}{\rho }\left[ \frac{\beta _1}{\alpha _1}\phi -\frac{\beta _2}{\alpha _2}(1-\phi ) \right] \nonumber \\&\times \left[ \frac{\phi }{\beta _1}-\frac{1-\phi }{\beta _2} - \sqrt{(m^\mathsf{A })^{-2}-\frac{4\phi (1-\phi )}{\beta _1\beta _2}} \right] . \end{aligned}$$

(4.44)

We note that $m_{\max }^\rho (\mathsf{P _\mathsf{B ,2}})>0$ if and only if $\phi >\phi ^\mathsf{AB }$, as is expected from (4.40c). Closer examination of (4.44) reveals that both $m_{\max }^\rho (\mathsf{P _\mathsf{B ,2}}) > m^\mathsf{A }$ and $m_{\max }^\rho (\mathsf{P _\mathsf{B ,2}}) < m^\mathsf{A }$ may hold.

Thus, the fully polymorphic equilibrium may be maintained for higher or lower migration rates than in the case of LE. This does not conform with the intuitive expectation that for reduced recombination $m_{\max }^\rho (\mathsf{P _\mathsf{B ,2}}) > m_{\max }^\infty$ should hold because the locally adapted haplotypes ($A_kB_k$ in deme $k$) are less frequently broken apart. However, numerical evaluation of (4.44) shows that $m_{\max }^\rho (\mathsf{P _\mathsf{B ,2}}) < m_{\max }^\infty$ occurs only for about 3% of the admissible parameter combinations and if it holds, $m_{\max }^\rho (\mathsf{P _\mathsf{B ,2}})$ is only very slightly less than $m_{\max }^\infty$ (results not shown). If $\rho$ is about as large as the largest selection coefficient or smaller, $m_{\max }$ increases with decreasing $\rho$. Expressions analogous to (4.44) can be obtained for collisions of $\mathsf F$ with the other equilibria.

4.8 No recombination

Our aim is to establish all possible equilibrium configurations and their dependence on the parameters if recombination is absent. In Fig. 4, the equilibrium configurations are displayed as schematic bifurcation diagrams with the total migration rate $m$ as the bifurcation parameter. In Theorem 4.6, we assign to each diagram its pertinent parameter combinations.

Fig. 4

Bifurcation diagrams for $\rho =0$. Diagrams (a)–(j) represent all equilibrium and stability configurations listed in Theorem 4.6. Each diagram displays the possible equilibria as a function of the total migration rate $m$. Each line indicates one admissible equilibrium, drawn if and only if it is admissible. Only equilibria are shown that can be stable or are involved in a bifurcation with an equilibrium that can be stable. Lines are drawn such that intersections occur if and only if the corresponding equilibria collide. Solid lines represent asymptotically stable equilibria, dashed lines unstable equilibria

In order to have only one bifurcation diagram covering cases that can be obtained from each other by simple symmetry considerations but are structurally equivalent otherwise, we use the sub- and superscripts $\mathsf{X }$ and $\mathsf{Y }$ in the labels of Fig. 4. For an efficient presentation of the results, we define

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf P _\mathsf{A ,1},\; \mathsf P _\mathsf Y = \mathsf P _\mathsf{B ,2},\; \mathsf M _\mathsf i = \mathsf M _1,\; m^\mathsf X = |m^\mathsf{A }|,\; m^\mathsf Y = |m^\mathsf{B }|, \end{aligned}$$

(R1)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf P _\mathsf{B ,2},\; \mathsf P _\mathsf Y = \mathsf P _\mathsf{A ,1},\; \mathsf M _\mathsf i = \mathsf M _4,\; m^\mathsf X = |m^\mathsf{B }|,\; m^\mathsf Y = |m^\mathsf{A }|, \end{aligned}$$

(R2)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf P _\mathsf{A ,1},\; \mathsf P _\mathsf Y = \mathsf P _\mathsf{B ,2},\; \mathsf M _\mathsf{i }=\mathsf M _4, \; m^\mathsf X = |m^\mathsf{A }|,\; m^\mathsf Y = |m^\mathsf{B }|, \end{aligned}$$

(R3)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf P _\mathsf{B ,2},\; \mathsf P _\mathsf Y = \mathsf P _\mathsf{A ,1},\;\mathsf M _\mathsf i = \mathsf M _1,\; m^\mathsf X = |m^\mathsf{B }|,\; m^\mathsf Y = |m^\mathsf{A }|, \end{aligned}$$

(R4)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf P _\mathsf{A ,1},\; \mathsf M _\mathsf i = \mathsf M _1,\; m^\mathsf X = |m^\mathsf{A }|, \end{aligned}$$

(R1′)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf P _\mathsf{B ,2},\; \mathsf M _\mathsf i = \mathsf M _4,\; m^\mathsf X = |m^\mathsf{B }|, \end{aligned}$$

(R2′)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf P _\mathsf{A ,1},\; \mathsf P _\mathsf Y = \mathsf P _\mathsf{B ,2},\; m^\mathsf X = |m^\mathsf{A }|,\; m^\mathsf Y = |m^\mathsf{B }|, \end{aligned}$$

(R3′)

$$\begin{aligned} \mathsf P _\mathsf X =\mathsf P _\mathsf{B ,2},\; \mathsf P _\mathsf Y = \mathsf P _\mathsf{A ,1},\; m^\mathsf X = |m^\mathsf{B }|,\; m^\mathsf Y = |m^\mathsf{A }|. \end{aligned}$$

(R4′)

Theorem 4.6

Let $\rho =0$. Figure 4 shows all possible bifurcation diagrams that involve bifurcations with equilibria that can be stable for some $m$ given the other parameters.

A. The following diagrams occur for an open set of parameters:

• 1. Diagram (a) in Fig. 4 applies if one of the following two cases holds:

  $$\begin{aligned} 0\le \phi < \phi ^\mathsf{M _1}\quad \text{ and } \, (\text{ R }1^{\prime }) \end{aligned}$$

  (4.45a)

  or

  $$\begin{aligned} \phi ^\mathsf{M _4}< \phi \le 1 \quad \text{ and } \, (\text{ R }2^{\prime }). \end{aligned}$$

  (4.45b)

• 2. Diagram (c) in Fig. 4 applies if one of the following two cases holds:

  $$\begin{aligned} \phi ^\mathsf{M _1}<\phi <\phi ^\mathsf{A }\quad \text{ and } \, (\text{ R }1^{\prime }) \end{aligned}$$

  (4.46a)

  or

  $$\begin{aligned} \phi ^\mathsf{B }<\phi <\phi ^\mathsf{M _4}\quad \text{ and } \, (\text{ R }2^{\prime }). \end{aligned}$$

  (4.46b)

• 3. Diagram (e) in Fig. 4 applies if one of the following two cases holds:

  $$\begin{aligned} \phi ^\mathsf{A }<\phi <\phi ^\mathsf{A \mathsf{F _0}}\quad \text{ and } \, (\text{ R }4) \end{aligned}$$

  (4.47a)

  or

  $$\begin{aligned} \phi ^\mathsf{B \mathsf{F _0}}<\phi <\phi ^\mathsf{B }\quad \text{ and } \, (\text{ R }3). \end{aligned}$$

  (4.47b)

• 4. Diagram (g) in Fig. 4 applies if one of the following four cases holds:

  $$\begin{aligned} \phi ^\mathsf{A \mathsf{F _0}}<\phi <\min \{\phi ^{\mathsf{F _0}},\phi ^\mathsf{AB }\} \quad \text{ and } \, (\text{ R }4) \end{aligned}$$

  (4.48a)

  or

  $$\begin{aligned} \phi ^{\mathsf{F _0}}<\phi <\phi ^\mathsf{AB }\quad \text{ and } \, (\text{ R }2) \end{aligned}$$

  (4.48b)

  or

  $$\begin{aligned} \phi ^\mathsf{AB }<\phi <\phi ^{\mathsf{F _0}}\quad \text{ and } \, (\text{ R }1) \end{aligned}$$

  (4.48c)

  or

  $$\begin{aligned} \max \{\phi ^{\mathsf{F _0}},\phi ^\mathsf{AB }\}<\phi <\phi ^\mathsf{B \mathsf{F _0}}\quad \text{ and } \, (\text{ R }3). \end{aligned}$$

  (4.48d)

  B. The following diagrams are degenerate, i.e., occur only if the parameters satisfy particular relations.

• 5. Diagram (b) in Fig. 4 applies if one of the following two cases holds:

  $$\begin{aligned} \phi =\phi ^\mathsf{M _1}\quad \text{ and } \, (\text{ R }1^{\prime }) \end{aligned}$$

  (4.49a)

  or

  $$\begin{aligned} \phi =\phi ^\mathsf{M _4}\quad \text{ and } \, (\text{ R }2^{\prime }). \end{aligned}$$

  (4.49b)

• 6. Diagram (d) in Fig. 4 applies if one of the following two cases holds:

  $$\begin{aligned} \phi =\phi ^\mathsf{A }\quad \text{ and } \, (\text{ R }1^{\prime }) \end{aligned}$$

  (4.50a)

  or

  $$\begin{aligned} \phi =\phi ^\mathsf{B }\quad \text{ and } \, (\text{ R }2^{\prime }). \end{aligned}$$

  (4.50b)

• 7. Diagram (f) in Fig. 4 applies if one of the following two cases holds:

  $$\begin{aligned} \phi =\phi ^\mathsf{A \mathsf{F _0}}\quad \text{ and } \, (\text{ R }2) \end{aligned}$$

  (4.51a)

  or

  $$\begin{aligned} \phi =\phi ^\mathsf{B \mathsf{F _0}}\quad \text{ and } \, (\text{ R }1). \end{aligned}$$

  (4.51b)

• 8. Diagram (h) in Fig. 4 applies if

  $$\begin{aligned} \phi =\phi ^{\mathsf{F _0}}=\phi ^\mathsf{AB }. \end{aligned}$$

  (4.52)

• 9. Diagram (i) in Fig. 4 applies if one of the following two cases holds:

  $$\begin{aligned} \phi =\phi ^{\mathsf{F _0}}> \phi ^\mathsf{AB }\quad \text{ and } \, (\text{ R }3^{\prime }). \end{aligned}$$

  (4.53a)

  or

  $$\begin{aligned} \phi =\phi ^{\mathsf{F _0}}< \phi ^\mathsf{AB }\quad \text{ and } \, (\text{ R }4^{\prime }) \end{aligned}$$

  (4.53b)

• 10. Diagram (j) in Fig. 4 applies if one of the following two cases holds:

  $$\begin{aligned} \phi =\phi ^\mathsf{AB }< \phi ^{\mathsf{F _0}}\quad \text{ and } \, \mathsf M _\textit{i}=\mathsf M _1 \end{aligned}$$

  (4.54a)

  or

  $$\begin{aligned} \phi =\phi ^\mathsf{AB }> \phi ^{\mathsf{F _0}}\quad \text{ and } \, \mathsf M _\textit{i}=\mathsf M _4. \end{aligned}$$

  (4.54b)

C. Figure 5 shows the order in which the bifurcation diagrams of Fig. 4 arise if $\phi$ is increased from 0 to 1.

Fig. 5

Order in which the bifurcation diagrams of Fig. 4 occur as $\phi$ increases from 0 to 1, where $\tilde{\theta }=\alpha _1\alpha _2-\beta _1\beta _2$

Proof

We prove parts A and B simultaneously and derive the statements about admissibility and stability of the equilibria by rewriting the conditions in Sect. 3.7 in terms of $m$, $m^\mathsf{A }$, $m^\mathsf{B }$, $m^{\mathsf{F _0}}$, and $m^{*}$ (4.3). These critical migration rates satisfy the relations given in (4.9), (4.11), (10.19) and (10.37).

We start by treating the bifurcations and stability of $\mathsf{F _0}$. Using (4.5c) and (4.4c), we infer from (3.25):

$$\begin{aligned} \mathsf{F _0}\rightarrow \mathsf M _1&\iff m\uparrow -m^{\mathsf{F _0}}\; \text{ and } \, \phi <\phi ^{\mathsf{F _0}}, \end{aligned}$$

(4.55a)

$$\begin{aligned} \mathsf{F _0}\rightarrow \mathsf M _4&\iff m\uparrow m^{\mathsf{F _0}}\; \text{ and } \, \phi >\phi ^{\mathsf{F _0}}. \end{aligned}$$

(4.55b)

From (4.15) we conclude that (4.55a) applies precisely in the following cases: (4.45a), (4.46a), (4.47a), (4.48a), (4.48c) (Part A), or (4.49a), (4.50a), (4.51a), (4.54a) (Part B). Similarly, (4.55b) applies in precisely the following cases: (4.45b), (4.46b), (4.47b), (4.48b), (4.48d) (Part A) or (4.49b), (4.50b), (4.51b), (4.54b) (Part B). $\mathsf{F _0}$ is admissible for every $m>0$ if and only if $\phi =\phi ^{\mathsf{F _0}}$, which corresponds to the remaining three cases (4.52) and (4.53b), (4.53a).

Condition (3.26), which determines when $\mathsf{F _0}$ changes stability, is equivalent to $m=m^{*}$. Therefore, Proposition 3.4 and the definitions of $m^{\mathsf{F _0}}$ and $m^{*}$ imply that $\mathsf{F _0}$ is asymptotically stable if and only if either

$$\begin{aligned} 0 < m < |m^{\mathsf{F _0}}| \le m^{*} \end{aligned}$$

(4.56a)

or

$$\begin{aligned} 0 < m < m^{*}< |m^{\mathsf{F _0}}| \end{aligned}$$

(4.56b)

holds, where

$$\begin{aligned} 0<|m^{\mathsf{F _0}}|\le m^{*}&\iff \phi \le \phi ^\mathsf{M _1}\; \text{ or } \; \phi \ge \phi ^\mathsf{M _4}, \end{aligned}$$

(4.57a)

$$\begin{aligned} 0<m^{*}<|m^{\mathsf{F _0}}|&\iff \phi ^\mathsf{M _1}<\phi <\phi ^\mathsf{M _4}. \end{aligned}$$

(4.57b)

If (4.56a) applies, according to (4.55), $\mathsf{F _0}$ leaves the state space at $m=-m^{\mathsf{F _0}}$ or $m=m^{\mathsf{F _0}}$ and exchanges stability with the respective monomorphism. By (4.57a), this occurs in the cases (4.45) or (4.49) of the theorem.

If (4.56b) applies, $\mathsf{F _0}$ loses stability at $m=m^{*}$ and, generically, either $\mathsf P _\mathsf{A ,1}$ or $\mathsf P _\mathsf{B ,2}$ is asymptotically stable if $m>m^{*}$ (see below). $\mathsf{F _0}$ remains admissible up to $m=|m^{\mathsf{F _0}}|$, when it collides with $\mathsf M _1$ or $\mathsf M _4$. By (4.15) and (4.57b), this occurs in the cases (4.46)–(4.48), (4.50), (4.51), (4.53), and (4.54).

Finally, if $\phi =\phi ^\mathsf{AB }$ (cases (4.52) and (4.54) in the theorem), $\mathsf M _3$ becomes stable. This follows from the statement below (3.31) together with (4.11a).

Next, we treat the bifurcations of the SLPs. The SLPs are admissible in intervals of the form $0<m<|m^\mathsf{A }|$ or $0<m<|m^\mathsf{B }|$ and leave the state space upon collision with a ME (Sect. 4.2). From (3.32) we conclude by simple calculations that $\mathsf P _\mathsf{A ,1}$ is asymptotically stable if and only if

$$\begin{aligned} m^{*}<m<|m^\mathsf{A }| \quad \text{ and } \, \phi ^\mathsf{M _1}<\phi <\phi ^\mathsf{AB }, \end{aligned}$$

(4.58)

as is the case in (4.46a), (4.47a), (4.48a) (if $\min \{\phi ^{\mathsf{F _0}}, \phi ^\mathsf{AB }\}=\phi ^\mathsf{AB }$), and (4.48b), as well as in (4.50a), (4.51a), and (4.53b).

From (3.33), we conclude that $\mathsf P _\mathsf{B ,2}$ is asymptotically stable if and only if

$$\begin{aligned} m^{*}<m<|m^\mathsf{B }| \quad \text{ and } \, \phi ^\mathsf{AB }<\phi <\phi ^\mathsf{M _4}, \end{aligned}$$

(4.59)

as is the case in (4.46b), (4.47b), (4.48c), and (4.48d) (if $\max \{\phi ^{\mathsf{F _0}}, \phi ^\mathsf{AB }\}=\phi ^\mathsf{AB }$), as well as in (4.50b), (4.51b), and (4.53a).

It remains to study the stability of the ME. For $\rho =0$, we infer from Sect. 4.1 and Proposition 4.1:

$$\begin{aligned} \mathsf M _1 \; \text{ is } \text{ asymptotically } \text{ stable } \!&\iff \! {\left\{ \begin{array}{ll} m\!>\!-m^{\mathsf{F _0}}\; \text{ and } \, \phi \!<\!\phi ^\mathsf{M _1}, \text{ or }\\ m\!>\!-m^\mathsf{A }\; \text{ and } \, \phi ^\mathsf{M _1}\le \phi \!<\!\phi ^\mathsf{A }, \end{array}\right. } \quad \end{aligned}$$

(4.60a)

$$\begin{aligned} \mathsf M _3 \; \text{ is } \text{ asymptotically } \text{ stable } \!&\iff \! m\!>\!\max \{|m^\mathsf{A }|,|m^\mathsf{B }|\} \; \text{ and } \,\phi ^\mathsf{A }\!<\!\phi \!<\!\phi ^\mathsf{B }, \qquad \quad \quad \end{aligned}$$

(4.60b)

$$\begin{aligned} \mathsf M _4 \; \text{ is } \text{ asymptotically } \text{ stable } \!&\iff \! {\left\{ \begin{array}{ll} m\!>\!m^\mathsf{B }\; \text{ and } \, \phi ^\mathsf{B }\!<\!\phi \le \phi ^\mathsf{M _1}, \text{ or }\\ m\!>\!m^{\mathsf{F _0}}\; \text{ and } \, \phi ^\mathsf{M _4}\!<\!\phi .\quad \quad \end{array}\right. } \end{aligned}$$

(4.60c)

In conjunction with the above results on $\mathsf{F _0}$ and the SLPs, this shows that, except in the degenerate cases (4.49), (4.50), (4.52), and (4.54), a ME becomes stable through a transcritical bifurcation with either $\mathsf{F _0}$, $\mathsf P _\mathsf{A ,1}$, or $\mathsf P _\mathsf{B ,2}$. In particular, $\mathsf M _1$ becomes asymptotically stable for large $m$ if (4.45a), (4.46a), or (4.49a) applies, $\mathsf M _3$ becomes asymptotically stable if one of (4.47), (4.48), (4.51), (4.52), (4.53), or (4.54) applies, and $\mathsf M _4$ becomes asymptotically stable if (4.45b), (4.46b), or (4.49b) applies. If $\phi =\phi ^\mathsf{A }$ or $\phi =\phi ^\mathsf{B }$ (4.50), then $\mathsf P _\mathsf{A ,1}$ or $\mathsf P _\mathsf{B ,2}$, respectively, is admissible and asymptotically stable for every $m$, and every ME is unstable. This finishes the proof of parts A and B.

Part C of Theorem 4.6 follows immediately from parts A and B by applying the relations in (4.15). $\square$

This theorem demonstrates that, for given selection parameters, the equilibrium structure, hence also the evolutionary dynamics, depends strongly on the degree $\phi$ of asymmetry of the migration rates. However, it is also important to note (and maybe counter intuitive) that for symmetric migration ($\phi =1/2$) any of the ten possible bifurcation diagrams may apply, simply by choosing the selection parameters accordingly.

The bifurcations of equilibria that cannot be stable can be derived easily from Sects. 4.2, 4.3, 3.7, and the above theorem by noting that these are boundary equilibria and corresponding pairs of SLPs are admissible for the same parameters. Inclusion of these bifurcations would require the introduction of subcases. In particular, $\mathsf{P _\mathsf{A ,2}}$, $\mathsf{P _\mathsf{B ,1}}$, and $\mathsf M _2$ are always unstable because gamete $A_1B_2$ is eventually lost. We observe from (10.20) and (10.38) that at most one pair of SLPs can be admissible if $\mathsf{F _0}$ is either unstable or not admissible. If this is the case, then one of these SLPs is asymptotically stable (Fig. 4).

Corollary 4.7

If $\rho =0$, the maximum migration rate, below which a stable two-locus polymorphism exists, is given by

$$\begin{aligned} m_{\max }^0 = \min \{|m^{\mathsf{F _0}}|,m^{*}\}. \end{aligned}$$

(4.61)

The corollary follows from the arguments surrounding (4.56) and (4.57).

4.9 Weak recombination

If $m\ne m^{*}$, a regular perturbation analysis of $\mathsf{F _0}$ yields the coordinates of a fully polymorphic (internal) equilibrium to leading order in $\rho$. This equilibrium, $\mathsf F$, is asymptotically stable (Karlin and McGregor 1972). We denote the first-order approximation of $\mathsf F$ by $\mathsf{F _\rho }$. Therefore, we have $\mathsf F =\mathsf{F _\rho }+o(\rho )$ and $\mathsf{F _\rho }=\mathsf{F _0}+O(\rho )$ as $\rho \rightarrow 0$. Because the coordinates of $\mathsf{F _\rho }$ are much too complicated to be informative, we refrain from presenting them.

For sufficiently small $\rho$, the following properties of $\mathsf{F _\rho }$ (hence, of $\mathsf F$) can be inferred from Proposition 4.1, Remark 4.2, and Theorem 4.6, Part A.1:

$$\begin{aligned} \mathsf{F _\rho }\rightarrow \mathsf M _1&\iff m\uparrow m^\mathsf{M _1}\quad \text{ and } \, \phi \le \phi ^\mathsf{M _1}, \end{aligned}$$

(4.62a)

$$\begin{aligned} \mathsf{F _\rho }\rightarrow \mathsf M _4&\iff m\uparrow m^\mathsf{M _4}\quad \text{ and } \,\phi ^\mathsf{M _4}\le \phi . \end{aligned}$$

(4.62b)

The above perturbation analysis can not be used to investigate the properties of the internal equilibrium $\mathsf F$ for given small positive $\rho$ when $m$ is varied in the proximity of $m^{*}$. Therefore, we performed numerical calculations to study the fate of $\mathsf F$ when $\rho$ is small and fixed, and $m$ increases. It suggests the following:

$$\begin{aligned} \mathsf F \rightarrow \mathsf P _\mathsf{A ,1}&\iff m\uparrow m^{*}_\mathsf A \quad \text{ and } \, \phi ^\mathsf{M _1}<\phi <\phi ^\mathsf{AB }, \end{aligned}$$

(4.63a)

$$\begin{aligned} \mathsf F \rightarrow \mathsf M _3&\iff m\uparrow m^{*}=m^\mathsf{A }=-m^\mathsf{B }\quad \text{ and } \, \phi =\phi ^\mathsf{AB }, \end{aligned}$$

(4.63b)

$$\begin{aligned} \mathsf F \rightarrow \mathsf P _\mathsf{B ,2}&\iff m\uparrow m^{*}_\mathsf B \quad \text{ and } \, \phi ^\mathsf{AB }<\phi <\phi ^\mathsf{M _4}, \end{aligned}$$

(4.63c)

where $m^{*}_\mathsf{A }$ and $m^{*}_\mathsf B$ are close to $m^{*}$. Thus, if $\rho$ is small, $\mathsf F$ stays close to $\mathsf{F _0}$ as $m$ increases from 0 until a value close to $m^{*}$ is reached. Then, within a very short interval of $m$, $\mathsf F$ moves ‘quickly’ along the manifold given by (10.8) and (10.11) to one of the boundary equilibria ($\mathsf{P _\mathsf{A ,1}}$, $\mathsf{P _\mathsf{B ,2}}$, or $\mathsf M _3$) on the ‘opposite’ side of the state space, where it exchanges stability upon collision with the respective equilibrium (at $m^{*}_\mathsf{A }$, $m^{*}_\mathsf B$, or $m^{*}$). $\mathsf F$ appears to be asymptotically stable whenever it is admissible.

If one of the cases in (4.62) applies, then $\mathsf{F _0}$ can be maintained for higher migration rates than $\mathsf F$ because $m^\mathsf{M _1}$ and $m^\mathsf{M _4}$ are decreasing functions in $\rho$. Numerical investigations support the conjecture that $\mathsf{F _0}$ can be maintained for higher migration rates than $\mathsf F$ whenever recombination is weak but positive. Thus, when recombination is weak, decreasing $\rho$ increases the maximum migration rate below which a stable, fully polymorphic equilibrium can be maintained.

4.10 Highly asymmetric migration

As already discussed in Sect. 3.8, by introducing weak back migration (i.e., $\phi$ close to 0 or 1) to the CI model, every equilibrium in the CI model gives rise to a unique equilibrium in a small neighborhood. This (perturbed) equilibrium has the same stability as the unperturbed. For weak or strong recombination, we can strengthen this conclusion. Because the CI model with $\rho =0$ is a generalized gradient system (Bürger and Akerman 2011, Section 3.4.4) and the LE dynamics (3.17) has a globally asymptotically stable equilibrium (Theorem 4.4), the proof of Theorem 2.3 of Nagylaki et al. (1999) applies and shows that in both cases the global dynamics remains qualitatively unchanged under small perturbations. In particular, no new equilibria or limit sets are generated by a small perturbation.

Therefore, if $\rho$ is sufficiently small and $\phi$ is sufficiently close to 0 or 1, we infer from Section 3.8 and Theorem 2 in Bürger and Akerman (2011) that the following bifurcation pattern applies (where $i=1$ or 4):

• If $0<m<m^\mathsf{M _i}$, a unique internal equilibrium, $\mathsf F$, exists. It is globally asymptotically stable.

• At $m=m^\mathsf{M _i}$, $\mathsf F$ leaves the state space through the ME $\mathsf M _i$ by an exchange-of-stability bifurcation.

• If $m>m^\mathsf{M _i}$, $\mathsf M _i$ is globally asymptotically stable.

This pattern is displayed in diagram (a) of Fig. 4, where $\mathsf{F _0}$ needs to be substituted by $\mathsf F$. We conjecture that it applies whenever $\rho$ is sufficiently small and either $\phi <\phi ^\mathsf{M _1}$ or $\phi >\phi ^\mathsf{M _4}$ holds. The bounds $\phi ^\mathsf{M _1}$ and $\phi ^\mathsf{M _4}$ follow from Remark 4.2 because $\tilde{\phi }^\mathsf{M _1}$ is not needed if $\rho$ is sufficiently small; see (10.32b). However, the upper bounds for $\rho$ given in Remark 4.2 are, in general, too large to guarantee the above bifurcation pattern. This is known from the CI model in which the monomorphic equilibrium ($\mathsf M _i$) may be simultaneously stable with the internal equilibrium $\mathsf F$ because an unstable internal equilibrium enters the state space at $m=m^\mathsf{M _i}$ through $\mathsf M _i$. If $\phi =1$, this may occur if $\tfrac{1}{3}(\alpha _1+\beta _1) < \rho < 3\alpha _1-\beta _1$, cf. (4.65b). For $\phi \ne 0$ or $\phi \ne 1$, we have not been able to determine the upper bound for $\rho$ below which $\mathsf F$ indeed leaves the state space through $\mathsf M _i$.

Now we treat large $\rho$. Proposition 4.1 and Remark 4.2 show that if $\rho >\max \{-\alpha _2,-\beta _2\}$, then $\mathsf M _1$ is asymptotically stable if and only if $\phi <\phi ^\mathsf{A }$ and $m>\max \{-m^\mathsf{A },-m^\mathsf{B }\}$, and if $\rho >\alpha _1$, then $\mathsf M _4$ is asymptotically stable if and only if $\phi >\phi ^\mathsf{B }$ and $m>\max \{m^\mathsf{A },m^\mathsf{B }\}$.

If, in addition to $\rho$ being sufficiently large, $\phi$ is small or large, then Theorem 4.4 implies that the internal equilibrium ($\mathsf F$) leaves the state space through $\mathsf{P _\mathsf{A ,1}}$, $\mathsf{P _\mathsf{B ,1}}$, or $\mathsf{P _\mathsf{B ,2}}$. The respective conditions are small perturbations of those given in (4.40a), (4.40b), of (4.40c), respectively. Combining theses conditions with those for the stability of the ME and observing (4.12) and (4.13), we conclude that the following bifurcation pattern applies if one of the conditions (a) $\alpha _2\le \beta _2$ and $\phi <\phi ^\mathsf{A }$, or (b) $\beta _2<\alpha _2$ and $\phi <\tilde{\phi }^\mathsf{AB }$, or (c) $\phi >\phi ^\mathsf{B }$ holds approximately:

• If $0<m<m^\bullet$, a unique internal equilibrium, $\mathsf F$, exists. It is asymptotically stable.

• At $m=m^\bullet$, $\mathsf F$ leaves the state space through a SLP by an exchange-of-stability bifurcation.

• If $m^\bullet <m<m^{\bullet \bullet }$, this SLP is asymptotically stable.

• If $m\ge m^{\bullet \bullet }$, then a ME is asymptotically stable.

If (a) holds, then $m^\bullet \approx -m^\mathsf{B }$ and the SLP and the ME are $\mathsf{P _\mathsf{A ,1}}$ and $\mathsf M _1$, respectively; if (b) holds, then $m^\bullet \approx -m^\mathsf{A }$ and the SLP and the ME are $\mathsf{P _\mathsf{B ,1}}$ and $\mathsf M _1$, respectively; if (c) holds, then $m^\bullet \approx m^\mathsf{A }$ and the SLP and the ME are $\mathsf{P _\mathsf{B ,2}}$ and $\mathsf M _4$, respectively. Finally, $m^{\bullet \bullet }=\max \{-m^\mathsf{A },-m^\mathsf{B }\}$ in (a) and (b), and $m^{\bullet \bullet }=\max \{m^\mathsf{A },m^\mathsf{B }\}$ in (c).

4.11 Maintenance of polymorphism

As already noted in Sect. 3.11, for general parameters the equilibrium configurations could not be determined analytically. To explore the potential of spatially heterogeneous selection in maintaining genetic variation in the presence of gene flow, we investigate the maximum total migration rate, $m_{\max }$, that admits a stable, fully polymorphic equilibrium. We have already shown that $m_{\max }=m_{\max }^\infty$ holds in the LE approximation (Corollary 4.5), and $m_{\max }=m_{\max }^0$ holds if $\rho =0$ (Corollary 4.7). From (10.20) and (10.38) we conclude that

$$\begin{aligned} m_{\max }^\infty \le m_{\max }^0, \end{aligned}$$

(4.64)

where, as is not difficult to show, equality holds if and only if $\phi =\phi ^\mathsf{AB }$.

For the CI model with $\phi =1$, Proposition 1 in Bürger and Akerman (2011) yields

In this case, the fully polymorphic equilibrium is globally asymptotically stable if (4.65a) or (4.65c) apply, but only locally stable if (4.65b) holds and $m$ is close to $m_{\max }$. A formula analogous to (4.65), but with $-\alpha _2$ and $-\beta _2$ instead of $\alpha _1$ and $\beta _1$, holds if $\phi =0$.

In general, we have no explicit formula for $m_{\max }$. However, extensive numerical work, as well as (4.65) and the considerations in Sect. 4.9 suggest that

$$\begin{aligned} m_{\max }\le m_{\max }^0 \end{aligned}$$

(4.66)

holds always. This is illustrated by Fig. 6, which displays the dependence of $m_{\max }$ on the migration ratio $\phi$ (Fig. 6a, c) and on the recombination rate $\rho$ (Fig. 6b, d) for two selection regimes. In Fig. 6a, b, locus $\mathsf{B }$ is under stronger selection in both demes. In Fig. 6c, d, each locus is under stronger selection in one deme.

Fig. 6

The maximum amount of gene flow, $m_{\max }$, admitting an asymptotically stable two-locus polymorphism as a function of $\phi$ or $\rho$. In panels a and b, locus $\mathsf{B }$ is under stronger selection than locus $\mathsf{A }$ in both demes ($\alpha _2=-2\alpha _1=-1$, $\beta _1=-\beta _2=2$, $\theta =1$). In c and d, different loci are under stronger selection in the two demes ($\alpha _1=-\beta _2=0.4$, $\beta _1=-\alpha _2=2$, $\theta =3.84$). Panels a and c show $m_{\max }$ as a function of $\phi$ for complete linkage ($m_{\max }^0$, (4.61)) and under linkage equilibrium ($m_{\max }^\infty$, (4.43)). Panels b and d display $m_{\max }$ for the indicated values of $\phi$ as a function of $\rho$. Here, $m_{\max }$ is obtained by determining numerically the critical migration rate when the stable internal equilibrium hits the boundary. This is done by computing when the leading eigenvalue at the boundary equilibrium is zero and by calculating the coordinates of the fully polymorphic equilibrium in a small neighborhood. In a and b, we have $\tilde{\phi }^\mathsf{AB }=\tfrac{1}{4}$ (indicated by the kink in the dashed line in a), $\phi ^\mathsf{M _1}=\tfrac{5}{17}$, $\phi ^\mathsf{A }=\tfrac{1}{3}$, $\phi ^\mathsf{AB }=\tfrac{3}{8}$, $\phi ^\mathsf{B }=\tfrac{1}{2}$, $\phi ^\mathsf{M _4}=\tfrac{5}{8}$. In c and d, we have $\phi ^\mathsf{M _1}=\tfrac{1}{26}$, $\phi ^\mathsf{A }=\tfrac{1}{6}$, $\phi ^\mathsf{AB }=\tfrac{1}{2}$, $\phi ^\mathsf{B }=\tfrac{5}{6}$, $\phi ^\mathsf{M _4}=\tfrac{25}{26}$

In Fig. 6a, c, $m_{\max }^\infty$ and $m_{\max }^0$ are shown as functions of $\phi$. The inequality (4.64) is a conspicuous feature in both cases. Also the shapes of $m_{\max }^\infty$ and $m_{\max }^0$ are conspicuous. The following properties are easy to prove: $m_{\max }^\infty$ is not differentiable at $\phi =\phi ^\mathsf{AB }$ and $\phi =\tilde{\phi }^\mathsf{AB }$, and $m_{\max }^0$ is not differentiable at $\phi =\phi ^\mathsf{M _1}$ and $\phi =\phi ^\mathsf{M _4}$. $m_{\max }^\infty$ and $m_{\max }^0$ are piecewise convex functions in $\phi$. If $\phi <\phi ^\mathsf{M _1}$, $m_{\max }^0$ increases in $\phi$; if $\phi ^\mathsf{M _4}<\phi$, $m_{\max }^0$ decreases in $\phi$; if $\phi ^\mathsf{M _1}<\phi <\phi ^\mathsf{M _4}$, $m_{\max }^0$ assumes its minimum at $\tfrac{1}{2}$ provided $\phi ^\mathsf{M _1}\le \tfrac{1}{2}\le \phi ^\mathsf{M _4}$. Therefore, $m_{\max }^0$ attains its maximum at $\phi ^\mathsf{M _1}$ or $\phi ^\mathsf{M _4}$. $m_{\max }^\infty$ increases if $\phi <\phi ^\mathsf{AB }$ and decreases if $\phi >\phi ^\mathsf{AB }$. It assumes its maximum at $\phi ^\mathsf{AB }$.

Notably, $m_{\max }^\infty =m_{\max }^0$ holds if $\phi =\phi ^\mathsf{AB }$. Numerical work suggests that indeed $m_{\max }=m_{\max }^\infty =m_{\max }^0$ holds independently of $\rho$ if $\phi =\phi ^\mathsf{AB }$. If $\theta =0$, then $\phi ^\mathsf{A }=\phi ^\mathsf{B }=\phi ^\mathsf{AB }=\phi ^{\mathsf{F _0}}$ and $m_{\max }^\infty =m_{\max }^0$ if $\phi =\phi ^\mathsf{AB }$. The latter condition is equivalent to (3.36c). Therefore, the analysis in Sect. 3.10 applies and shows that an internal equilibrium, which presumably is globally asymptotically stable, exists always, i.e., $m_{\max }=m^{\infty }_{\max }=m^0_{\max }=\infty$.

Figure 6b, d illustrates the effect of recombination on $m_{\max }$ for different values of $\phi$. In all cases investigated, $m_{\max }$ decreased monotonically with increasing $\rho$. These findings support the conjecture that (4.66) is always valid. Therefore, $m_{\max }^0-m_{\max }^\infty$ serves as a useful estimate for the sensitivity of $m_{\max }$ to variation in $\rho$. We can prove that $m_{\max }^0-m_{\max }^\infty$ is maximized at $\phi =\phi ^\mathsf{M _1}$, or at $\phi =\phi ^\mathsf{M _4}$, or at $\phi =0$ if $\beta _2<\alpha _2$.

Although we proved that $m_{\max }<m_{\max }^\infty$ can occur (Sect. 4.7), all numerical examples showed that $m_{\max }$ is only very slightly smaller than $m_{\max }^\infty$ in this case (results not shown). Therefore, our results suggest that the cases of LE (infinitely strong recombination) and of no recombination ‘essentially’ bracket the range of parameters for which both loci can be maintained polymorphic.

As Fig. 6a, c shows, the range of values $\phi$ for which the equilibrium structure can be expected to be similar to the CI model, i.e., $\phi <\min \{\phi ^\mathsf{M _1},\tilde{\phi }^\mathsf{AB }\}$ or $\phi >\phi ^\mathsf{M _4}$ (Sect. 4.10), can vary considerably.

Finally, we infer from Proposition 4.1 that none of the ME is stable if $m<\max \{|m^\mathsf{A }|,|m^\mathsf{B }|\}$. Hence, in this case at least one locus is maintained polymorphic. By contrast, we have shown in Sect. 4.2 that no SLP is admissible if $m>\max \{|m^\mathsf{A }|,|m^\mathsf{B }|\}$. However, as demonstrated by our results for $\rho =0$, an internal equilibrium may be asymptotically stable if $\max \{|m^\mathsf{A }|,|m^\mathsf{B }|\}<m<m_{\max }^0$. These results suggest that no genetic variability can be maintained if

$$\begin{aligned} m > \max \{|m^\mathsf{A }|,|m^\mathsf{B }|,m_{\max }^0\}. \end{aligned}$$

(4.67)

This bound is best possible if $\rho =0$. For sufficiently large $\rho$, the corresponding bound is $\max \{|m^\mathsf{A }|,|m^\mathsf{B }|\}$.

5 Migration load and local adaptation

Here, we briefly investigate some properties of the migration load of the subpopulations and of the total population. We use these migration loads as simple measures for local adaptation (but see Blanquart et al. 2012). Mean fitness in deme $k$ is given by $\bar{w}_k=\alpha _k(2p_k-1)+\beta _k(2q_k-1)$, with its maximum at $\alpha _k+\beta _k$. Therefore, the migration loads in demes 1 and 2, defined as the deviation of $\bar{w}_k$ from its maximum, are given by

$$\begin{aligned} L_1 = 2(\alpha _1(1-p_1)+\beta _1(1-q_1)) \quad \text{ and }\quad L_2 = 2(-\alpha _2p_2-\beta _2q_2). \end{aligned}$$

(5.1)

Assuming that the subpopulations are of equal size, we define the load of the total population by $L=\tfrac{1}{2}(L_1+L_2)$.

If migration is weak, we can calculate the migration load in each deme at the fully polymorphic equilibrium $\mathsf F$ (Proposition 3.2) to leading order in $m_1$ and $m_2$. For deme 1, we obtain

$$\begin{aligned} L_1 \approx 2m_1\frac{\alpha _1+\beta _1+2\rho }{\alpha _1+\beta _1+\rho }, \end{aligned}$$

(5.2)

and an analogous formula holds for deme 2. Obviously, the migration load increases with increasing migration rates $m_1$ or $m_2$, hence with $m$, in each of the demes and in the total population. Simple calculations show that each of the loads also increases with increasing recombination rate $\rho$ if migration is weak.

In general, however, the load in each deme does not always increase with increasing $m$. The reason is that for sufficiently strong migration, generically, first one locus, then one of the haplotypes becomes fixed (Proposition 4.3). If this is either $A_1B_1$ or $A_2B_2$, then the load in the corresponding deme will vanish for high migration rates, whereas that in the other deme will be very high. In such a case, the load of the total population may also decrease with increasing $m$. This occurs for large migration rates (not far below $m_{\max }$) and it can occur for completely linked loci as well as for loci in LE. In the CI model, the load always increases with the migration rate (Bürger and Akerman 2011)

Finally, although $L$ is increasing in $\rho$ if migration is weak, this is not necessarily so if migration is strong. By using a grid of parameter combinations, we showed numerically that in about 0.34% of more than $10^6$ combinations of $\alpha _1,\alpha _2,\beta _1,\beta _2,m$, and $\phi$, the total load $L$ at the equilibrium $\mathsf{F _\infty }$ is lower than that at $\mathsf{F _0}$ (results not shown). Again, this occurs for high migration rates, not far below the value $m_{\max }^\infty$ at which $\mathsf{F _\infty }$ leaves the state space. Then a population maintained fully polymorphic by tight linkage may have a higher total load than a population in which fixation of a locus or a haplotype is facilitated by high recombination. In all such cases, selection in one deme was (considerably) stronger than in the other, and in more than 70$\%$ of the cases, a specialist haplotype became fixed at very high migration rates. In summary, under a wide range of conditions in this model, reduced recombination is favored, but there are instances where increased recombination is favored (cf. Pylkov et al. 1998; Lenormand and Otto 2000).

6 $F_\mathrm{ST}$ and differentiation

The most commonly used measure for quantifying differentiation in spatially structured populations is $F_\mathrm{ST}$. For diallelic loci, $F_\mathrm{ST}$ can be defined as $F_\mathrm{ST}=\frac{\text{ Var }(p)}{\bar{p}(1-\bar{p})}$, where $\text{ Var }(p)$ is the variance of the allele frequencies in the total population and $\bar{p}$ is the allele frequency averaged over the demes. Estimators of multilocus $F_\mathrm{ST}$ are usually defined as weighted averages of one-locus $F_\mathrm{ST}$ estimators (e.g., Weir and Cockerham 1984; Leviyang and Hamilton 2011). Here, we extend Nagylaki’s (1998) approach and define a genuine multilocus version of $F_\mathrm{ST}$ that measures the covariance of the frequencies of (multilocus) haplotypes. We restrict attention to the diallelic two-locus case, but the extension to multiple multiallelic loci is evident. A general multilocus theory of fixation indices will be developed elsewhere.

Let $c_k$ denote the proportion of the population in deme $k$, so that $\sum _k c_k=1$. Then the frequency of haplotype $i$ in the entire population is $\bar{x}_i=\sum _k c_k x_{i,k}$. Because our subpopulations are randomly mating, the frequency of genotype $ij$ in the entire population is given by $\overline{x_ix_j}=\sum _k c_k x_{i,k}x_{j,k}$. Following eqs. (6a) and (6b) in Nagylaki (1998), we define $F_{\mathrm{ST},ij}$ as a standardized measure of the covariance of the frequencies of haplotypes $i$ and $j$:

$$\begin{aligned} \overline{x_i^2}&= \bar{x}_i^2 + F_{\mathrm{ST},ii}\bar{x}_i(1-\bar{x}_i), \end{aligned}$$

(6.1a)

$$\begin{aligned} \overline{x_ix_j}&= (1-F_{\mathrm{ST},ij})\bar{x}_i \bar{x}_j. \end{aligned}$$

(6.1b)

The multilocus, or haplotype, heterozygosity in the entire population can be defined as

$$\begin{aligned} \bar{h}_S = \sum _{i,j:i\ne j}\overline{x_ix_j} = \sum _i(\bar{x}_i-\overline{x_i^2}), \end{aligned}$$

(6.2)

where $\sum _i$ runs over all haplotypes. If the entire population were panmictic, its multilocus heterozygosity would be

$$\begin{aligned} h_T = \sum _i \bar{x}_i(1-\bar{x}_i). \end{aligned}$$

(6.3)

Thus, $1-h_T$ is the probability that two gametes chosen at random from the entire population are the same haplotype.

Following eq. (32) in Nagylaki (1998), we define $F_\mathrm{ST}$ by

$$\begin{aligned} F_\mathrm{ST}= \frac{1}{h_T}\sum _i \bar{x}_i(1-\bar{x}_i) F_{\mathrm{ST},ii}. \end{aligned}$$

(6.4)

Then $F_\mathrm{ST}$ can be written as

$$\begin{aligned} F_\mathrm{ST}= 1-\frac{\bar{h}_S}{h_T} = \frac{\sum _i \text{ Var }(x_i)}{\sum _i \bar{x}_i(1-\bar{x}_i)}, \end{aligned}$$

(6.5)

in direct generalization of the classical formula given above.

We focus on the dependence of the equilibrium value of $F_\mathrm{ST}$ on the migration parameters $m$ and $\phi$ and on the recombination rate $\rho$. Because we obtained the coordinates of the stable, fully polymorphic equilibrium equilibrium $\mathsf F$ explicitly only in special or limiting cases, explicit formulas for $F_\mathrm{ST}$ can be derived only in these cases. For instance, if migration is weak, we obtain from (3.16) that, to leading order in $m$,

$$\begin{aligned} F_\mathrm{ST}= 1 - m\left[ \frac{\phi }{c_2}\, \frac{\alpha _1\beta _1+(\alpha _1+\beta _1)\rho }{\alpha _1\beta _1(\alpha _1+\beta _1+\rho )} - \frac{1-\phi }{c_1}\frac{\alpha _2\beta _2-(\alpha _2+\beta _2)\rho }{\alpha _2\beta _2(\alpha _2+\beta _2-\rho )}\right] . \end{aligned}$$

(6.6)

Here, $F_\mathrm{ST}$ increases with decreasing $\rho$, and decreases with increasing $m$. Thus, stronger linkage leads to increased differentiation if migration is weak.

Figure 7 illustrates for two selection scenarios how $F_\mathrm{ST}$, evaluated at the stable, fully polymorphic equilibrium $\mathsf F$, depends on the total migration rate $m$ and the recombination rate $\rho$. In diagrams (a) and (c) of Fig. 7, it is assumed that locus $\mathsf{B }$ is under stronger selection than locus $\mathsf{A }$ in both demes. It shows that $F_\mathrm{ST}$ usually declines with increasing migration rate. However, there are a few instances, where $F_\mathrm{ST}$ increases if $m$ is slightly below the migration rate at which the fully polymorphic equilibrium loses admissibility. In diagrams (a) and (c) of Fig. 7, differentiation between the populations experiences the fastest decline for weak migration (relative to the selection parameters), whereas this is not necessarily so in diagrams (b) and (d). There, $F_\mathrm{ST}$ may experience its strongest decrease if migration is strong.

Fig. 7

$F_\mathrm{ST}$ as a function of the total migration rate $m$. In panels a and c, locus $\mathsf{B }$ is under stronger selection in both demes ($\alpha _1=\tfrac{1}{2}$, $\alpha _2=-1$, $\beta _1=-\beta _2=2$, $\theta =1$). In panels b and d, locus $\mathsf{A }$ is under stronger selection than $\mathsf{B }$ in deme 2, and locus $\mathsf{B }$ is under stronger selection than $\mathsf{A }$ in deme 1 ($\alpha _1=-\beta _2=0.4$, $\beta _1=-\alpha _2=2$, $\theta =3.84$). Note that in all cases, $F_\mathrm{ST}$ is also monotone decreasing in $\rho$. For $\rho =0$ and $\rho =\infty$ (LE), the lines are from numerical evaluation of (6.5) by substitution of the coordinates of $\mathsf{F _0}$ (3.24) or $\mathsf{F _\infty }$ (3.18). For other values of $\rho$, the numerically determined coordinates of the internal equilibrium are used

Figure 7 also shows that at large migration rates, $F_\mathrm{ST}$ may increase if the recombination rate increases, i.e., $F_\mathrm{ST}$ is not minimized under linkage equilibrium. However, this occurs only for large recombination rates, i.e., larger than the largest selection coefficient. This is compatible with the finding in Sect. 4.11 that at high recombination rates, $m_{\max }$ may (slightly) increase in $\rho$, and the finding in Sect. 5 that the load $L$ may decrease with increasing $m$. We note that this ‘aberrant’ behavior of $m_{\max }$, $L$, and $F_\mathrm{ST}$ does not necessarily occur for the same parameter combinations. Among more than $10^6$ parameter combinations of $\alpha _1,\alpha _2,\beta _1,\beta _2,m$, and $\phi$, we found no instance where $F_\mathrm{ST}$ evaluated at the equilibrium $\mathsf{F _\infty }$ was higher than that at $\mathsf{F _0}$ (results not shown). Importantly, if recombination is weak or migration is weak then $F_\mathrm{ST}$ apparently always increases with tighter linkage.

Comparison of our multilocus $F_\mathrm{ST}$ with averages of single-locus $F_\mathrm{ST}$ values showed that the multilocus $F_\mathrm{ST}$ declines somwehat faster at small migration rates than the averaged single-locus $F_\mathrm{ST}$. For large parameter regions, the qualitative behavior of these measures of differentiation is the same. Differences occur only for a subset of selection coefficients at high migration rates and high recombination rates. Finally, we mention that our multilocus $F_\mathrm{ST}$ is a sensitive measure of differentiation only if the effective number of haplotypes is low. This parallels the well known fact that the classical $F_\mathrm{ST}$ is a sensitive measure of differentiation only if the effective number of alleles is low (e.g., Nagylaki 1998, 2011). Thus, our multilocus $F_\mathrm{ST}$ may be most useful if applied to short sequences of DNA. A thorough and more general study is in preparation.

7 Invasion of a locally beneficial mutant

Differentiation between subpopulations can be increased by the invasion of mutants that establish a stable polymorphism at their locus. Therefore, we consider a locus ($\mathsf{A }$) at which a new mutant $A_1$ arises that is advantageous relative to the wild type $A_2$ in deme 1, but disadvantageous in deme 2. In terms of our model, we assume $\alpha _1>0>\alpha _2$. If locus $\mathsf{A }$ is isolated, this mutant can invade and become established in a stable polymorphism if and only if $|\sigma _1+\sigma _2|<1$; cf. (3.7) and (3.9). Using $m$ and $\phi$, this condition can be rewritten as

$$\begin{aligned} m < |m^\mathsf{A }|, \end{aligned}$$

(7.1)

see (4.3a) and (4.4a), or

$$\begin{aligned} \frac{m+\alpha _2}{m}\,\phi ^\mathsf{A }< \phi < \frac{m-\alpha _2}{m}\,\phi ^\mathsf{A }= \phi _\mathrm{inv}. \end{aligned}$$

(7.2)

We restrict attention to the case $\phi >\phi ^\mathsf{A }$ (4.2a) when the influx of the deleterious allele $A_2$ into deme 1 is sufficiently strong such that $\mathsf A _2$ is protected. (The case $\phi <\phi ^\mathsf{A }$ is symmetric and more suitable to study invasion of $\mathsf A _2$ under influx into deme 2 of $\mathsf A _1$ which is deleterious there.) Then the mutant $A_1$ can invade if any of the following equivalent conditions hold:

$$\begin{aligned}&\qquad \qquad \qquad m < m^\mathsf{A },\end{aligned}$$

(7.3a)

$$\begin{aligned}&\alpha _1 > \frac{\alpha _2m\phi }{\alpha _2-m(1-\phi )} = \frac{m_1}{1-m_2/\alpha _2}, \end{aligned}$$

(7.3b)

or

$$\begin{aligned} \phi ^\mathsf{A }< \phi < \phi _\mathrm{inv}, \end{aligned}$$

(7.3c)

where $\phi _\mathrm{inv}>1$ if and only if $m<\alpha _1$. Thus, $A_1$ can always invade if $m<\alpha _1$. For the CI model ($\phi =1$), each of the conditions in (7.3) simplifies to the well known invasion condition $m<\alpha _1$ (Haldane 1930). The conditions (7.3) show that invasion is facilitated whenever back migration is increased, either by keeping $m_1$ constant and increasing $m_2$, or by fixing $m$ and decreasing $\phi$.

For the CI model it was proved that invasion of a locally beneficial mutant is always facilitated by increased linkage to a locus in migration-selection balance (Bürger and Akerman 2011). In fact, mutants of arbitrarily small effect can invade provided they are sufficiently tightly linked to this polymorphic locus which may be considered as the background in which the new mutant appears.

Here, we investigate whether this is also the case with two-way migration. Thus, we assume that locus $\mathsf{B }$ is in migration-selection balance (which requires that analogs of (7.3) are satisfied for $\beta _1$ and $\beta _2$) and a locally beneficial mutant $A_1$ arises at the linked locus $\mathsf{A }$. Hence, the model in Sect. 2 applies and we assume (2.3).

Because we are mainly interested in the invasion properties of mutants of small effect, we assume that locus $\mathsf{B }$ is under stronger selection than $\mathsf{A }$, i.e., $|\alpha _k|<|\beta _k|$ in deme $k=1,2$. Before the mutant $A_1$ arises, the population is at the equilibrium $\mathsf{P _\mathsf{B ,2}}$ (where $|\tau _1+\tau _2|<1$ must hold for admissibility; see Sect. 3). $A_1$ can invade if $\mathsf{P _\mathsf{B ,2}}$ is unstable. Since the eigenvalues determining external stability are zeros of a complicated quartic equation, the stability of $\mathsf{P _\mathsf{B ,2}}$ cannot be determined analytically. We expect that the new stable equilibrium that will be reached is the fully polymorphic equilibrium $\mathsf F$. For the CI model, this was be proved in (Bürger and Akerman 2011). For the case of LE, it follows from Theorem 4.4.

Figure 8 displays typical results about the invasion of the mutant $A_1$. In Fig. 8a, the maximum recombination rate admitting invasion, denoted by $\rho _\text{ max }$, is shown as a function of $\phi$. In the shaded region, $A_1$ can invade. If $\phi \le \phi _\mathrm{inv}=0.55$, (7.3c) implies that $\mathsf A _1$ can always invade. If $\phi >\phi _\mathrm{inv}$, there exists $\rho _\text{ max }<\infty$, such that $A_1$ can invade only if $\rho <\rho _\text{ max }$, i.e., if $\mathsf A _1$ is sufficiently tightly linked to locus $\mathsf{B }$. In Fig. 8b, the minimum selection coefficient $\alpha _1$ necessary for invasion of $A_1$ is shown as a function of $\rho /m$ for various values of $\phi$. These values are obtained by computing when the leading eigenvalue that determines external stability of $\mathsf{P _\mathsf{B ,2}}$ equals zero.

Fig. 8

Invasion properties of locally beneficial alleles. In a, the maximum recombination rate between loci $\mathsf{A }$ and $\mathsf{B }$, below which invasion of $\mathsf A _1$ can occur, is displayed as a function of $\phi$. The parameters $\alpha _1=-\alpha _2=0.1$, $\beta _1=-2\beta _2=2$, and $m=1$ are fixed. Therefore, $\phi ^\mathsf{A }=\tfrac{1}{2}$ and $\phi _\mathrm{inv}=0.55$. In b, the minimum selective advantage $\alpha _1$ required for invasion of $\mathsf A _1$ is shown as a function of $\rho$ for different values of $\phi$. The parameters $\alpha _2=-0.1$, $\beta _1=-2\beta _2=2$, and $m=1$ are fixed

We conclude that, as in the CI model, mutants of arbitrarily small effect can invade provided they are sufficiently tightly linked to a locus that is already maintained in migration-selection balance. In addition, as shown by both panels in Fig. 8, increasingly symmetric migration facilitates the invasion and establishment of locally beneficial alleles.

8 The effective migration rate at a linked neutral site

Linkage to loci under selection may impede or enhance gene flow at a neutral marker locus. In the first case, linkage may act as a barrier to gene flow. This was shown by the work of Petry (1983), Bengtsson (1985), Barton and Bengtsson (1986), and Charlesworth et al. (1997), who developed and studied the concept of the effective migration rate as a measure of the ‘effective’ gene flow at a neutral site. More recently, the effective migration rate was studied for CI models with selection on a single locus in a class-structured population (Kobayashi et al. 2008) or with selection on two linked loci (Bürger and Akerman 2011). Fusco and Uyenoyama (2011) investigated the consequences of a selectively maintained polymorphism on the rate of introgression at a linked neutral site under symmetric migration between two demes.

Here, we derive an explicit expression for the effective migration rate at a neutral locus ($\mathsf N$) that is located between the two selected loci, $\mathsf{A }$ and $\mathsf{B }$. Recombination between locus $\mathsf{A }$ ($\mathsf{B }$) and the neutral locus occurs with rate $\rho _\mathsf{AN }$ ($\rho _\mathsf{NB }$) such that $\rho =\rho _\mathsf{AN }+\rho _\mathsf{NB }$. Thus, only one crossover event occurs in a sufficiently small time interval. We assume that $\rho _\mathsf{AN }$ and $\rho _\mathsf{NB }$ are positive, i.e., the neutral locus is not completely linked to a selected site. We consider two variants at the neutral locus, $N_1$ and $N_2$, each with arbitrary, positive initial frequency in at least one deme. The frequency of $N_1$ in deme $k(=1,2)$ is denoted by $n_k$. We model evolution at the three loci by a system of $7\times 2$ ordinary differential equations for the allele frequencies and linkage disequilibria ($p_1$, $p_2$, $q_1$, $q_2$, $D^\mathsf{AB }_1$, $D^\mathsf{AB }_2$, $n_1$, $n_2$, $D^\mathsf{AN }_1$, $D^\mathsf{AN }_2$, $D^\mathsf{NB }_1$, $D^\mathsf{NB }_2$, $D^\mathsf{ANB }_1$, $D^\mathsf{ANB }_2$). We refrain from presenting the equations for the allele frequencies at the neutral locus and the associated linkage disequilibria because they are a straightforward extension of those in Section 4.6 of Bürger and Akerman (2011).

Obviously, the equilibrium allele frequencies at the neutral locus are the same in each deme and given by the initial allele frequencies averaged over the two demes:

$$\begin{aligned} \hat{n}_1=\hat{n}_2=\hat{n}=\frac{m_2 n_{1}(0)}{m}+\frac{m_1 n_{2}(0)}{m}. \end{aligned}$$

(8.1)

The equilibrium frequencies at the two selected loci are independent of the neutral locus and, thus, the same as in the two-locus model treated above. The linkage disequilibria involving the neutral locus ($D^\mathsf{AN }_k$, $D^\mathsf{NB }_k$, and $D^\mathsf{ANB }_k$) are zero at equilibrium. By (8.1), there is a one-dimensional manifold of equilibria resulting from the absence of selection at the neutral locus.

We assume that parameters are such that the fully polymorphic equilibrium $\mathsf F$ is admissible and globally asymptotically stable. Using the above order for the allele frequencies and linkage equilibria, the Jacobian at the equilibrium $\mathsf F$ has block structure,

$$\begin{aligned} J= \begin{pmatrix} J_S &{}\quad 0 \\ 0 &{}\quad J_N \end{pmatrix}, \end{aligned}$$

(8.2)

where $J_S$ is the Jacobian describing convergence of $(p_1,p_2,q_1,q_2,D^\mathsf{AB }_1,D^\mathsf{AB }_2)$ to $\mathsf F$, and $J_N$ is the Jacobian describing convergence of $(n_1,n_2,D^\mathsf{AN }_1, D^\mathsf{AN }_2, D^\mathsf{NB }_1, D^\mathsf{NB }_2, D^\mathsf{ANB }_1, D^\mathsf{ANB }_2)$ to $(\hat{n},\hat{n},0,0,0,0,0,0)$.

Because zero is the leading eigenvalue of $J_N$, the rate of convergence to equilibrium at the neutral locus is determined by the second largest eigenvalue of $J_N$, which we denote by $\lambda _N$. We define the effective (total) migration rate by $m_\mathsf{eff }=-\lambda _N$ (Bengtsson 1985; Kobayashi et al. 2008; Bürger and Akerman 2011). It can be checked that under weak migration, i.e., to leading order in $m_1$ and $m_2$, one obtains

$$\begin{aligned} m_\mathsf{eff }= -\lambda _N = m_1\frac{\rho _\mathsf{AN }\rho _\mathsf{NB }}{(\rho _\mathsf{AN }+\alpha _1)(\rho _\mathsf{NB }+\beta _1)} + m_2\frac{\rho _\mathsf{AN }\rho _\mathsf{NB }}{(\rho _\mathsf{AN }-\alpha _2)(\rho _\mathsf{NB }-\beta _2)} \end{aligned}$$

(8.3)

(a Mathematica notebook is available on request). If the neutral site is linked only to one selected locus (e.g., because $\beta _1=\beta _2=0$), then

$$\begin{aligned} m_\mathsf{eff }= m_1\frac{\rho _\mathsf{AN }}{\rho _\mathsf{AN }+\alpha _1} + m_2\frac{\rho _\mathsf{AN }}{\rho _\mathsf{AN }-\alpha _2} \end{aligned}$$

(8.4)

is obtained. Thus, two linked selected loci act as a much stronger barrier to gene flow than a single selected locus, especially if the recombination rate between the two loci is not much larger than the selective coefficients. In Fig. 9, the approximation (8.3) of the effective migration rate $m_\mathsf{eff }$ is displayed as a function of $m$ for various parameter combinations and compared with the exact value obtained by numerical evaluation of $\lambda _N$.

Fig. 9

The effective migration rate $m_\mathsf{eff }$ as a function of $m$ for $\alpha _1=-\alpha _2=0.1$, $\beta _1=-\beta _2=0.2$, $\phi =\tfrac{1}{2}$, and $\rho =0.2$. Recall that $\rho =\rho _\mathsf{AN }+\rho _\mathsf{NB }$. Lines show the weak-migration approximation of $m_\mathsf{eff }$ (8.3). Symbols give the exact numerical value of $m_\mathsf{eff }=-\lambda _N$

We note that $m_\mathsf{eff }$ is (approximately) the sum of the two effective one-way migration rates (Bürger and Akerman 2011) and closely related to Kobayashi and Telschow’s (2011) effective recombination rate. Our result complements their explicit example on two-locus incompatibilities. We refer to their paper for the discussion of the relation of this concept of an effective migration rate to that of Bengtsson (1985) and for applications in the context of speciation theory.

9 Discussion

The purpose of this investigation was to improve our understanding of how genetic architecture, in particular recombination and locus effects, as well as the pattern and amount of migration determine polymorphism, local adaptation, and differentiation in a subdivided population inhabiting a heterogeneous environment. For simplicity, we restricted attention to two linked, diallelic loci and to migration between two demes. The study of diversifying selection in just two demes may also shape our intuition about clinal variation if the two subpopulations are from different ends of the cline. If alleles are beneficial in only one environment and detrimental in the other, local adaptation of subpopulations and differentiation between them can be obtained only if a (multilocus) polymorphism is maintained. Therefore, most of our mathematical results focus on existence and stability of polymorphic equilibria and on the dependence of the equilibrium configurations on the model parameters (migration rates, selection coefficients, recombination rate).

The model is introduced in Sect. 2. Sections 3 and 4 are devoted to the derivation of the possible equilibrium configurations and bifurcation patterns. They contain our main mathematical results. Explicit analytical results about existence and stability of equilibria were obtained for several limiting or special cases and are complemented by numerical work.

The conditions for admissibility of all single-locus polymorphisms (SLPs) are given in Sect. 3.1, those for asymptotic stability of the monomorphic equilibria (ME) in Proposition 3.1 in Sect. 3.2. The stability of SLPs could not generally be determined (Sect. 3.3). Weak migration is treated by perturbation methods in Sect. 3.4. For sufficiently weak migration, there exists a globally attracting fully polymorphic equilibrium, $\mathsf F$ (Proposition 3.2). Its approximate coordinates are given by (3.16).

The complete equilibrium and stability structure could be derived under the assumption of linkage equilibrium (Sect. 3.5). The unique, fully polymorphic equilibrium $\mathsf F =\mathsf{F _\infty }$ is admissible and globally attracting if and only if all four SLPs are admissible. Otherwise, one boundary equilibrium (SLP or ME) is globally asymptotically stable (Proposition 3.3). These results extend straightforwardly to an arbitrary number of diallelic loci. Based on these results, nonlinear perturbation theory establishes the existence of a globally stable, fully polymorphic equilibrium in a perturbed parameter range if recombination is sufficiently strong (Sect. 3.6). This equilibrium is in quasi-linkage equilibrium and given by (3.21).

Also for completely linked loci all equilibria and their local stability properties could be derived (Sect. 3.7). In this case, the fully polymorphic equilibrium $\mathsf{F _0}$ (3.24) may lose stability while it is admissible (3.28). At this threshold a boundary equilibrium becomes stable by a ‘jump bifurcation’ (Proposition 3.4). In general, however, more complicated equilibrium patterns than determined by Propositions 3.3 and 3.4 can occur, in particular, multiple stable equilibria.

In Sect. 3.8, we apply perturbation theory to infer the equilibrium properties under highly asymmetric migration from those derived for the continent-island model in Bürger and Akerman (2011) and Bank et al. (2012). There, a stable ($\mathsf F$) and an unstable fully polymorphic equilibrium may exist if recombination is intermediate, and $\mathsf F$ is simultaneously stable with a boundary equilibrium. In general (Sect. 3.11), we cannot exclude the existence of more than two internal equilibria or complicated dynamical behavior. Numerical searches produced no such instances. What can be shown easily is that, if $\rho <\infty$, any fully polymorphic equilibrium exhibits LD. In all cases, where an internal equilibrium was calculated (numerically or analytically), it exhibited positive LD.

In the super-symmetric case, in which selection in deme 2 mirrors that in deme 1 and migration is symmetric, an assumption made in several applications, a fully polymorphic equilibrium exists always and, presumably, is stable (Sect. 3.10). This is a highly degenerate situation because if $\theta \ne 0$, only a monomorphic equilibrium can be stable for sufficiently large migration rates (Proposition 4.3). If $\theta =0$ (Sect. 3.9), then a fully polymorphic equilibrium can exist for arbitrarily large migration rates if $\phi =\phi ^\mathsf{AB }$ (see also Sect. 4.11).

Whereas in Sect. 3 the focus was on the efficient presentation of the existence and stability results of equilibria, in Sect. 4 these results are used to derive the possible bifurcation patterns with the total migration rate $m$ as the bifurcation parameter. All possible bifurcation patterns could be derived under the assumption of LE (Theorem 4.4, Figs. 2, 3), and under the assumption of complete linkage (Theorem 4.6, Figs. 4, 5). The latter case is considerably more complex. Interestingly, in each case, every bifurcation pattern can occur for every ratio $\phi =m_1/m$ of migration rates by choosing the selection coefficients appropriately. Hence, the assumption of symmetric migration does not yield simpler equilibrium configurations than general migration if arbitrary selection coefficients are admitted.

In each of these cases (LE or $\rho =0$), we determined the maximum migration rate $m_{\max }$ admitting an asymptotically stable, fully polymorphic equilibrium (Corollaries 4.5 and 4.7). The maximum migration rate $m_{\max }^0$ for $\rho =0$ always exceeds or equals that ($m_{\max }^\infty$) for LE, i.e., $m_{\max }^\infty \le m_{\max }^0$. Although for strong recombination, $m_{\max }$ can be very slightly smaller than $m_{\max }^\infty$ (Sect. 4.7), in the vast majority of investigated cases, $m_{\max }$ is bracketed by $m_{\max }^\infty$ and $m_{\max }^0$ (Fig. 6, Sect. 4.11).

Proposition 4.3 demonstrates that a ME is globally attracting if migration is sufficiently strong (except in the degenerate case noted above). If we interpret the equilibria $\mathsf M _2$ and $\mathsf M _3$ as fixation of a generalist ($A_1B_2$ and $A_2B_1$ are haplotypes of intermediate fitness), and $\mathsf M _1$ and $\mathsf M _4$ as fixation of a specialist ($A_1B_1$ and $A_2B_2$ are the locally adapted haplotypes), then depending on the sign of $\theta$ one of the generalists becomes fixed for high $m$ if $\phi$ is intermediate (i.e., $\phi ^\mathsf{A }<\phi <\phi ^\mathsf{B }$ if $\theta >0$, $\phi ^\mathsf{B }<\phi <\phi ^\mathsf{A }$ if $\theta <0$; but note that, depending on the selection coefficients, both $\phi ^\mathsf{A }$ and $\phi ^\mathsf{B }$ can be arbitrarily close to 0 or 1.). The critical value $m$ as well as $\phi ^\mathsf{A }$ and $\phi ^\mathsf{B }$ are independent of $\rho$. Otherwise, one of the specialists becomes fixed for large $m$.

The fact that a generalist becomes fixed for strong migration is a distinct feature of (balanced) two-way migration: in the CI model or if migration is sufficiently asymmetric ($\phi <\phi ^\mathsf{A }$ or $\phi >\phi ^\mathsf{B }$ if $\theta >0$), one of the specialist haplotypes swamps the populations and becomes fixed. Another difference between highly asymmetric and more symmetric migration patterns is that in the first case, it is always the locus under weaker selection that first loses its polymorphism while $m$ increases, whereas this not necessarily so in the latter case (see Sect. 4.2 and Theorem 4.6, cases A3 and A4).

In summary, we determined quantitatively when the following three evolutionarily stable states discussed by Kawecki and Ebert (2004) occur: (i) existence of a single specialist optimally adapted to one deme and poorly to the other, (ii) existence of a single generalist type which has higher average fitness in the whole population than than any of the specialists, and (iii) existence of a set of specialists each adapted to its deme, i.e., coexistence in a polymorphism. Local adaptation and differentiation occur only in case (iii).

In Sect. 5, we used the migration load in each deme to quantify the degree of local adaptation. In Sect. 6 we introduced a new multilocus version of $F_\mathrm{ST}$ to measure differentiation. If migration is weak, then local adaptation and differentiation decrease with increasing migration rate and increase with increasing linkage between the loci (Fig. 7). In particular, for given (small) migration rate, local adaptation and differentiation are maximized if the fitness effects are concentrated on a single locus (corresponding to $\rho =0$ in our model). However, as discussed in Sect. 5, for high migration rates, the migration load of the total population can decrease with increasing recombination or migration rate. Similarly, at high recombination and migration rates, $F_\mathrm{ST}$ can increase with increasing migration or recombination rate. Thus, for given, relatively high migration rate, $F_\mathrm{ST}$ may be minimized at intermediate recombination rates. Apparently, it is always maximized in the absence of recombination.

In Sect. 7, we investigated the conditions for invasion of locally beneficial mutants. At an isolated locus, such a mutant can invade and become established in a migration-selection equilibrium if and only if its advantage exceeds a threshold that increases with the immigration rate of the wild type; see (7.3b). If, however, this mutant occurs at a locus that is linked to a locus that is already in migration-selection balance, then its invasion is facilitated, i.e., its local selective advantage can be smaller (Fig. 8b). Equivalently, for given selection coefficients and total migration rate, the minimum recombination rate needed for invasion increases if $\phi$, or the influx of the (deleterious) wild type relative to the efflux of the new mutant, increases (Fig. 8a). For the extreme case of one-way migration from a ‘continental’ population to an ‘island’ population that is adapting to a new environment, Bürger and Akerman (2011) proved that invasion of a locally beneficial mutant is always facilitated by increased linkage to a locus in migration-selection balance.

Thus, our results complement the numerical finding by Yeaman and Whitlock (2011) for a multilocus quantitative-genetic model that clusters of locally adaptive mutations, or concentrated genetic architectures, build up in spatially structured populations with opposing selection pressures in two demes. Because tighter linkage is required for invasion under increasingly asymmetric migration rates, more concentrated architectures and a greater advantage for recombination-reducing mechanisms (such as chromosome inversions) should be expected for highly asymmetric migration. In finite populations, invasion of new mutants occurs only with a certain probability, and genetic drift may erase polymorphism. Numerical work, supported by analytical methods, has already shed some light on the dependence of the probability of establishment of new, locally adaptive mutations on the recombination rate and other factors (Yeaman and Otto 2011; Feder et al. 2012). Analytical work on the role of genetic drift and finite population size on these issues is in progress.

Our results also show that, in the absence of epistasis and under the present form of balancing selection, reduced recombination between selected loci is favored, except when migration rates are sufficiently symmetric and high (Sect. 5). Selection inducing certain forms of epistasis may favor high recombination in structured populations more easily (Pylkov et al. 1998; Lenormand and Otto 2000; Bank et al. 2012). Therefore, general predictions about the emergence of clusters of locally adaptive mutations in regions of reduced recombination, or of genomic islands of speciation (Wu and Ting 2004) or of differentiation (Feder et al. 2012), can not be made in the absence of detailed information about epistasis and the spatial pattern of selection and migration. At least in the absence of epistasis, the most favorable situation for the emergence of such clusters should occur in populations that are adapting to a new environment, still receiving maladaptive gene flow but sending out only very few or no migrants (corresponding to a continent-island model).

In Sect. 8, we derived the approximation (8.3) for the effective migration rate at a linked neutral locus that is located between the selected loci. This approximation is simply the sum of the two effective migration rates under one-way migration (Bürger and Akerman 2011). Because in the present model, polymorphism at the selected loci is maintained by balancing selection, the effective migration rate may be greatly reduced compared with the actual migration rate (see Fig. 9). Thus, strong barriers against gene flow may build up at such neutral sites and enhance (neutral) differentiation (see Charlesworth and Charlesworth 2010, Chap. 8.3). Future work will have to study the actual amount and pattern of neutral diversity at such sites in finite populations.

Appendix A

1.1 Sufficiency of the assumptions (2.3)

By relabeling alleles, we can assume without loss of generality (2.3a). Generically, one of the following six parameter sets applies:

$$\begin{aligned} \theta&> 0, \;\;\alpha _1 < \beta _1, \; \text{ and } \, \alpha _2 \ge \beta _2, \end{aligned}$$

(10.1a)

$$\begin{aligned} \theta&> 0, \;\;\alpha _1 < \beta _1, \; \text{ and } \, \alpha _2 < \beta _2, \end{aligned}$$

(10.1b)

$$\begin{aligned} \theta&> 0, \;\;\alpha _1 \ge \beta _1, \; \text{ and } \, \alpha _2 < \beta _2, \end{aligned}$$

(10.1c)

$$\begin{aligned} \theta&< 0, \;\;\alpha _1 > \beta _1, \; \text{ and } \, \alpha _2 \le \beta _2, \end{aligned}$$

(10.1d)

$$\begin{aligned} \theta&< 0, \;\;\alpha _1 > \beta _1, \; \text{ and } \, \alpha _2 > \beta _2, \end{aligned}$$

(10.1e)

$$\begin{aligned} \theta&< 0, \;\;\alpha _1 \le \beta _1, \; \text{ and } \, \alpha _2 > \beta _2. \end{aligned}$$

(10.1f)

In addition, there are the following three parameter sets:

$$\begin{aligned} \theta&= 0, \;\;\alpha _1 < \beta _1 \; \text{ and } \, \alpha _2 > \beta _2,\end{aligned}$$

(10.1g)

$$\begin{aligned} \theta&= 0, \;\;\alpha _1 = \beta _1 \; \text{ and } \, \alpha _2 = \beta _2,\end{aligned}$$

(10.1h)

$$\begin{aligned} \theta&= 0, \;\;\alpha _1 > \beta _1 \; \text{ and } \, \alpha _2 < \beta _2. \end{aligned}$$

(10.1i)

The sets (10.1a)–(10.1i) yield the complete parameter space of the selection coefficients.

We show that the parameter sets (10.1c)–(10.1f) can be derived from (10.1a) and (10.1b) by simple transformations. Let $f$ denote the exchange of loci, i.e., $f(\alpha _k)=\beta _k$ and $f(\beta _k)=\alpha _k$, and $g$ the exchange of demes, i.e., $g(\alpha _k)=-\alpha _{k^*}$ and $g(\beta _k)=-\beta _{k^*}$. We observe that $\text{ sign } (f(\theta ))=\text{ sign }(g(\theta ))=-\text{ sign }(\theta )$ and

$$\begin{aligned} (10.1a)&\stackrel{f}{\rightarrow }&(10.1d) \stackrel{g}{\rightarrow } (10.1c) \stackrel{f}{\rightarrow } (10.1f),\end{aligned}$$

(10.2a)

$$\begin{aligned} (10.1b)&\stackrel{g}{\rightarrow }&(10.1e) \end{aligned}$$

(10.2b)

hold. Therefore, (2.3) is sufficient to describe the (generic) parameter region where $\theta \ne 0$. Since

$$\begin{aligned} (10.1g) \stackrel{f}{\rightarrow } (10.1i), \end{aligned}$$

(10.3)

(3.34) is sufficient to describe the degenerate cases $\theta =0$.

1.2 Proof of Proposition 3.1

At each monomorphic equilibrium, the characteristic polynomial factors into three quadratic polynomials, $P(\lambda ) = t_1(\lambda )t_2(\lambda )t_3(\lambda )$. Two of them, $t_1(\lambda )$ and $t_2(\lambda )$, determine stability with respect to the adjacent marginal one-locus systems. The corresponding conditions are already known from one-locus theory. The third, $t_3(\lambda )$, determines stability with respect to the interior of the state space.

In the following, we derive the stability conditions (3.10) and (3.11) for $\mathsf M _1$. Those for $\mathsf M _4$ can be deduced analogously or by symmetry considerations by taking into account that (2.3b) implies $\min \{\alpha _1,\beta _1\}=\alpha _1$. The stability analysis of $\mathsf M _2$ and $\mathsf M _3$ is much simpler and left to the reader.

For $\mathsf M _1$, it is straightforward to show that

$$\begin{aligned} t_1(\lambda )&= \lambda ^2 + [\alpha _1(1+\sigma _1)+\alpha _2(1+\sigma _2)]\lambda + \alpha _1\alpha _2(1+\sigma _1+\sigma _2), \end{aligned}$$

(10.4a)

$$\begin{aligned} t_2(\lambda )&= \lambda ^2 + [\beta _1(1+\tau _1)+\beta _2(1+\tau _2)]\lambda + \beta _1\beta _2(1+\tau _1+\tau _2), \end{aligned}$$

(10.4b)

$$\begin{aligned} t_3(\lambda )&= \lambda ^2+(\alpha _1+\alpha _2+\beta _1+\beta _2+2\rho +m_1+m_2)\lambda \nonumber \\&\quad + (\alpha _1+\beta _1+m_1+\rho )(\alpha _2+\beta _2+m_2+\rho )-m_1m_2. \end{aligned}$$

(10.4c)

Because $t_1^{\prime \prime }(\lambda )>0$ for every $\lambda$, $t_1^{\prime }(0)=\alpha _1(1+\sigma _1)+\alpha _2(1+\sigma _2)>0$ if $\sigma _2<-1$, $t_1(0)>0$ if and only if $\sigma _1+\sigma _2<-1$, and $\min _\lambda \{t_1(\lambda )\}<0$, we conclude that the two eigenvalues emanating from $t_1$ are negative if and only if

$$\begin{aligned} \sigma _1+\sigma _2 < -1. \end{aligned}$$

(10.5a)

Analogously, the two eigenvalues emanating from $t_2$ are negative if and only if

$$\begin{aligned} \tau _1+\tau _2 < -1, \end{aligned}$$

(10.5b)

and those originating from $t_3$ are negative if and only if

$$\begin{aligned} m_2 > -\frac{(\alpha _1+\beta _1+m_1+\rho )(\alpha _2+\beta _2+\rho )}{\alpha _1+\beta _1+\rho }. \end{aligned}$$

(10.5c)

Conditions (10.5a) and (10.5b) yield (3.10).

Concerning (10.5c), we observe that it is always satisfied if $\rho >-(\alpha _2+\beta _2)$ because then the right-hand side is negative. Next we show, that (10.5c) is also satisfied if $\rho >-\alpha _2$. Because the right-hand side of (10.5c) is strictly monotone decreasing in $\rho$, it is sufficient to prove that (10.5c) holds if $\rho =-\alpha _2$. Then simple rearrangement of (10.5c) leads to the condition

$$\begin{aligned} \frac{m_2(\alpha _1+\beta _1-\alpha _2)}{\beta _1\beta _2} + \frac{\alpha _1+\beta _1-\alpha _2+m_1}{\beta _1} <0, \end{aligned}$$

(10.6)

which can be rewritten as

$$\begin{aligned} \tau _1+\tau _2+1 + \frac{\alpha _1-\alpha _2}{\beta _1}(1+\tau _2)<0. \end{aligned}$$

(10.7)

This is satisfied if (10.5b) holds because this also implies $1+\tau _2<0$. One shows similarly that (10.5c) is satisfied if $\rho \ge -\beta _2$. Therefore, we have proved that $\mathsf M _1$ is asymptotically stable if (3.10) and (3.11) hold.

1.3 Calculation of equilibria with two polymorphic loci if $\rho =0$

As shown in the main text, by Corollary 3.9 of Nagylaki and Lou (2007) it is sufficient to assume that $A_1B_2$ is absent, which implies $D_k=p_k(1-q_k)$ and $p_k\le q_k$. Setting $\rho =0$, we find from the equations $\dot{p}_1=0$ and $\dot{q}_1=0$ (2.7) that

$$\begin{aligned} p_2&= p_1[m_1-\alpha _1(1-p_1)-\beta _1(1-q_1)]/m_1,\end{aligned}$$

(10.8a)

$$\begin{aligned} q_2&= 1 - (1-q_1)(m_1+\alpha _1p_1+\beta _1q_1)/m_1 \end{aligned}$$

(10.8b)

holds at equilibrium. Substituting (10.8) into $\dot{p}_2$ and $\dot{q}_2$, we obtain at equilibrium,

$$\begin{aligned} 0&= p_1[g_1(p_1,q_1)-\alpha _1^2\alpha _2p_1^3-\beta _1^2\beta _2q_1^3]/m_1^2, \end{aligned}$$

(10.9a)

$$\begin{aligned} 0&= (1-q_1)[g_2(p_1,q_1)-\alpha _1^2\alpha _2p_1^3-\beta _1^2\beta _2q_1^3]/m_1^2, \end{aligned}$$

(10.9b)

where $g_1$ and $g_2$ are quadratic polynomials in $(p_1,q_1)$. The obvious substitution results in the equilibrium condition

$$\begin{aligned} 0&= \alpha _1\alpha _2(\alpha _1+\beta _1)p_1^2 + \beta _1\beta _2(\alpha _1+\beta _1)q_1^2 + (\alpha _1+\beta _1)(\alpha _1\beta _2+\alpha _2\beta _1)p_1q_1 \nonumber \\&\qquad + [m_1(\alpha _1(2\alpha _2+\beta _2)+\alpha _2\beta _1) -(\alpha _1+\beta _1)(\alpha _2\beta _1+\alpha _1(\alpha _2+\beta _2))]p_1\nonumber \\&\qquad +[m_1(\beta _2(\alpha _1+2\beta _2)+\alpha _2\beta _1) -\beta _1\beta _2(\alpha _1+\beta _1)]q_1 \nonumber \\&\qquad + m_1[m_1(\alpha _2+\beta _2)+m_2(\alpha _1+\beta _1)-(\alpha _1+\beta _1)(\alpha _2+\beta _2)]. \end{aligned}$$

(10.10a)

It is easy to check that $\mathsf{F _0}$ always fulfills this condition and it is the only solution satisfying $0\le p_1=q_1\le 1$. Hence, unless there is curve $(p_1,q_1)$ of solutions of (10.10) that passes through $\mathsf{F _0}$ and through either a point on $p_1=0$ with $0< q_1\le 1$ or on $q_1=1$ with $0\le p_1<1$, $\mathsf{F _0}$ is the unique admissible solution of (10.10).

Because $\mathsf{F _0}$ has an eigenvalue 0 only if either (3.26) is satisfied or if $|\kappa _1+\kappa _2|=1$ (which occurs if and only if $\mathsf{F _0}$ collides with either $\mathsf M _1$ or $\mathsf M _4$), $\mathsf{F _0}$ is the only equilibrium with both loci polymorphic, except when (3.26) is satisfied. In the latter case, a line of equilibria exists, as we show now.

We calculate $m_2$ from (3.26) and substitute into (10.10). The right-hand side factorizes into two linear terms. Only one of them gives rise to admissible equilibria and, in fact, yields the manifold:

$$\begin{aligned} p_1 = \frac{\theta [\beta _1(1-q_1)-m_1]-\alpha _1\beta _1(\alpha _2+\beta _2)}{\alpha _1\theta }, \end{aligned}$$

(10.11)

where $0\le q_1\le 1$. The allele frequencies in the other deme are obtained from (10.8). It is straightforward to check that not only $\mathsf{F _0}$, but also the equilibria $\mathsf{P _\mathsf{A ,1}}$ and $\mathsf{P _\mathsf{B ,2}}$ lie on this manifold. In terms of the gamete frequencies, this manifold is a straight line.

1.4 Stability of $\mathsf{F _0}$

In this section we derive the stability of $\mathsf{F _0}$.

As $A_1B_2$ is lost if $\rho =0$ and (2.3) hold, it is sufficient to consider the dynamics (2.5) in $S_3\times S_3$. In this case, the characteristic polynomial at $\mathsf{F _0}$ factors into two quadratic polynomials, $P(\lambda )=t_1(\lambda )t_2(\lambda )$. These are given by

$$\begin{aligned} t_1(\lambda )&= \lambda ^2+\left[ (\alpha _1+\beta _1-\alpha _2-\beta _2)\sqrt{1-\kappa _1\kappa _2}-(m_1+m_2) \right] \lambda \nonumber \\&\quad +(\alpha _1+\beta _1)(\alpha _2+\beta _2)\left[ 1+(\kappa _1-\kappa _2)\sqrt{1-\kappa _1\kappa _2}\right] ,\end{aligned}$$

(10.12a)

$$\begin{aligned} t_2(\lambda )&= \lambda ^2+\frac{1}{2}\left[ \alpha _1+\alpha _2-\beta _1-\beta _2+(\alpha _1-\alpha _2+\beta _1-\beta _2)\sqrt{1-\kappa _1\kappa _2}\right] \lambda \nonumber \\&\quad +\frac{1}{2}\left[ -\alpha _1\beta _2(1+\sqrt{1-\kappa _1\kappa _2})-\alpha _2\beta _1(1-\sqrt{1-\kappa _1\kappa _2})\right] . \end{aligned}$$

(10.12b)

The polynomial $t_1$ determines the stability with respect to the (effectively one-locus) system where only ’alleles’ $A_1B_1$ and $A_2B_2$ are present. It is convex with $t_1(0)\ge 0$ if and only if $|\kappa _1+\kappa _2|\le 1$ (where the equalities correspond), i.e., whenever $\mathsf{F _0}$ is admissible, cf. (3.23). If $|\kappa _1+\kappa _2|<1$, $t^{^{\prime }}_1(0)>0$ and $t_1$ attains a negative value at its minimum (as can be shown easily). Therefore, all eigenvalues emanating from $t_1$ are real and negative whenever $\mathsf{F _0}$ is admissible.

The polynomial $t_2$ determines stability with respect to the interior of $S_3\times S_3$. It is convex and attains its minimum at

$$\begin{aligned} \lambda _\text{ min }=\frac{1}{4}\left[ (\alpha _1+\beta _2-\alpha _2-\beta _2) (1-\sqrt{1-\kappa _1\kappa _2})\right] \end{aligned}$$

(10.13)

where $\lambda _\text{ min }<0$ by (2.3a) and (3.22). As $t_2(\lambda _\text{ min })<0$, the eigenvalues emanating from $t_2$ are real. As

$$\begin{aligned} t_2(0)\ge 0 \iff m_1m_2\le {\tilde{m}}, \end{aligned}$$

(10.14)

where the equalities correspond and ${\tilde{m}}$ is defined in (3.27), and because $t^{^{\prime }}_2(0)>0$, we conclude that the two eigenvalues emanating from $t_2$ are negative if and only if (3.28) holds.

1.5 Stability of SLPs if $\rho =0$

For $\rho =0$ it is sufficient to study the dynamics (2.5) in $S_3\times S_3$. SLPs where $\hat{x}_{k,2}>0$ ($k=1,2$), i.e., $\mathsf{P _\mathsf{A ,2}}$ and $\mathsf{P _\mathsf{B ,1}}$, are unstable. It remains to study the stability of $\mathsf{P _\mathsf{A ,1}}$ and $\mathsf{P _\mathsf{B ,2}}$.

We present the analysis for $\mathsf{P _\mathsf{A ,1}}$ in detail, as results for $\mathsf{P _\mathsf{B ,2}}$ follow analogously.

At $\mathsf{P _\mathsf{A ,1}}$ the characteristic polynomial factors into two quadratic polynomials, $P(\lambda )=t_1(\lambda )t_2(\lambda )$, given by

$$\begin{aligned} t_1(\lambda )&= \lambda ^2+\left[ \alpha _1(\sqrt{1-4\sigma _1\sigma _2}-\sigma _1)- \alpha _2(\sqrt{1-4\sigma _1\sigma _2}+\sigma _2)\right] \lambda \nonumber \\&+\alpha _1\alpha _2\left[ (\sigma _1-\sigma _2)\sqrt{1-4\sigma _1\sigma _2}- (1-4\sigma _1\sigma _2)\right] ,\end{aligned}$$

(10.15a)

$$\begin{aligned} t_2(\lambda )&= \lambda ^2+\frac{1}{2}\left[ 2\beta _1+2\beta _2+\alpha _1 (1-\sqrt{1-4\sigma _1\sigma _2})+\alpha _1(1+\sqrt{1-4\sigma _1\sigma _2})\right] \lambda \nonumber \\&+\frac{1}{2}\left[ \beta _1(\beta _2+\alpha _2)+\beta _2(\alpha _1+\beta _1) +\theta \sqrt{1-\sigma _1\sigma _2}\right] . \end{aligned}$$

(10.15b)

$t_1$ determines stability with respect to the one-locus system where $B_1$ is fixed. $t_1(0)=0$ if and only if $|\sigma _1+\sigma _2|=1$, i.e., whenever $\mathsf{P _\mathsf{A ,1}}$ collides with a ME according to (3.9a) and (3.9b). Whenever $|\sigma _1+\sigma _2|<1$, i.e., $\mathsf{P _\mathsf{A ,1}}$ is admissible (3.7), $t_1(0)>0$ and $t_1^{^{\prime }}(0)>0$. As $t_1^{^{\prime \prime }}(\lambda )>0$ for every $\lambda$, $t_1$ attains a minimum, where it is straightforward to show that $t_1$ takes a negative value at its minimum. Thus, all eigenvalues emanating from $t_1$ are real and negative whenever $\mathsf{P _\mathsf{A ,1}}$ is admissible.

$t_2$ determines stability with respect to the interior of $S_3\times S_3$. $t_2(0)\ge 0$, if and only if $m_1m_2\ge {\tilde{m}}$, cf. (3.27), where the equalities correspond. Whenever $m_1m_2>{\tilde{m}}$, $t_2^{^{\prime }}(0)>0$. As $t_2^{^{\prime \prime }}(\lambda )>0$ for every $\lambda$, $t_2$ attains a minimum, where it is straightforward to show that $t_2$ takes a negative value at its minimum. Thus, all eigenvalues emanating from $t_1$ are real and negative whenever $m_1m_2>{\tilde{m}}$ holds. Otherwise, at least one eigenvalue is positive.

Combining the results obtained for $t_1$ and $t_2$ it follows that $\mathsf{P _\mathsf{A ,1}}$ is asymptotically stable if and only if

$$\begin{aligned} -1<\sigma _1+\sigma _2<1\quad \text{ and } \quad m_1m_2>{\tilde{m}}\end{aligned}$$

(10.16)

hold. We note that $m_1m_2>{\tilde{m}}$ is equivalent to $(\sigma _1\tau _2-\sigma _2\tau _1)^2<-(\sigma _1+\tau _1)(\sigma _2+\tau _2)$, and our general assumption (2.3) implies $\tau _1<\sigma _1$ and $\sigma _1\tau _2-\sigma _2\tau _1<0$. Using these relations we can show with the help of Mathematica that (10.16) is incompatible with $-1<\tau _1+\tau _2$. Consequently, $\mathsf{P _\mathsf{B ,2}}$ is not admissible if $\mathsf{P _\mathsf{A ,1}}$ is asymptotically stable.

1.6 The super-symmetric case

We prove that in the super-symmetric case of Sect. 3.10, all SLPs are unstable.

We assume symmetric migration rates ($m_1=m_2=m$), equivalent loci ($\alpha _k=\beta _k=a$), and selection in deme 2 mirrors that in deme 1 ($\alpha _k=-\alpha _{k^*}$). Thus, $\theta =0$. Equilibria may collide (thus leave or enter the state space) if and only if at least one of their eigenvalues is zero. Eigenvalues are zeros of the characteristic polynomial, which has the form $P(\lambda ) = c_6\lambda ^6 + \dots + c_1 \lambda + c_0$. If zero is an eigenvalue at an equilibrium, i.e., $P(0)=0$, the constant term $c_0$ must vanish. In the super-symmetric case every characteristic polynomials at an SLP has the same constant term

$$\begin{aligned} c_0\!=\!-a^2 \rho \left( 2a^2 \sqrt{a^2\!+\!m^2}\!+\! m (3m\!-\!\rho ) \sqrt{a^2\!+\!m^2} \!-\! (a^2\!+\!m^2)(3m\!-\!\rho ) \right) .\qquad \end{aligned}$$

(10.17)

One can show that $c_0=0$ is impossible if $m > 0$.

1.7 Important quantities and relations

The following section complements Sect. 4.1. Here, we derive all relations of $\phi ^\mathsf X$ (4.2) and $m^\mathsf X$ (4.3) needed inSects. 4.2 to 4.8 and in the proofs of the theorems there.

Using (4.9a), (4.9b), (4.5a), (4.5b), (4.12) and (4.13), we derive all possible inequalities between $m^\mathsf{A }$ and $m^\mathsf{B }$:

$$\begin{aligned} 0<m^\mathsf{A }<m^\mathsf{B }&\iff \phi ^\mathsf{B }\le \phi , \end{aligned}$$

(10.18a)

$$\begin{aligned} 0<-m^\mathsf{A }<-m^\mathsf{B }&\iff \alpha _2 > \beta _2 \quad \text{ and } \, \phi <\tilde{\phi }^\mathsf{AB }, \end{aligned}$$

(10.18b)

$$\begin{aligned} 0<-m^\mathsf{B }<-m^\mathsf{A }&\iff {\left\{ \begin{array}{ll} \alpha _2 \le \beta _2 \quad \text{ and } \,\phi <\phi ^\mathsf{A }, \text{ or } \\ \alpha _2 > \beta _2 \quad \text{ and } \, \tilde{\phi }^\mathsf{AB }<\phi <\phi ^\mathsf{A }, \end{array}\right. } \end{aligned}$$

(10.18c)

$$\begin{aligned} 0 < -m^\mathsf{B }< m^\mathsf{A }&\iff \phi ^\mathsf{A }\le \phi <\phi ^\mathsf{AB }, \end{aligned}$$

(10.18d)

$$\begin{aligned} 0 < m^\mathsf{A }< -m^\mathsf{B }&\iff \phi ^\mathsf{AB }<\phi <\phi ^\mathsf{B }, \end{aligned}$$

(10.18e)

where

$$\begin{aligned} 0<m^\mathsf{B }\!\le \! m^\mathsf{A },\; 0 \!<\! -m^\mathsf{A }\!\le \! m^\mathsf{B }, \quad \text{ and } \, 0 < m^\mathsf{B }\!\le \! -m^\mathsf{A }\;\text{ are } \text{ infeasible }.\qquad \end{aligned}$$

(10.18f)

Using (4.6), (4.5), and (4.12)–(4.15) we obtain the following inequalities for $m^{*}$:

$$\begin{aligned} 0<m^{*}<-m^{\mathsf{F _0}}&\iff \phi ^\mathsf{M _1}(\rho =0)<\phi <\phi ^{\mathsf{F _0}}, \end{aligned}$$

(10.19a)

$$\begin{aligned} 0<-m^{\mathsf{F _0}}<m^{*}&\iff \phi <\phi ^\mathsf{M _1}(\rho =0), \end{aligned}$$

(10.19b)

$$\begin{aligned} 0<m^{*}<m^{\mathsf{F _0}}&\iff \phi ^{\mathsf{F _0}}<\phi <\phi ^\mathsf{M _4}(\rho =0), \end{aligned}$$

(10.19c)

$$\begin{aligned} 0<m^{\mathsf{F _0}}<m^{*}&\iff \phi ^\mathsf{M _4}(\rho =0)<\phi , \end{aligned}$$

(10.19d)

$$\begin{aligned} 0<m^{*}<-m^\mathsf{A }&\iff \phi ^\mathsf{M _1}(\rho =0)<\phi <\phi ^\mathsf{A }, \end{aligned}$$

(10.19e)

$$\begin{aligned} 0<m^{*}<-m^\mathsf{B }&\iff \phi ^\mathsf{AB }<\phi <\phi ^\mathsf{B }, \end{aligned}$$

(10.19f)

$$\begin{aligned} 0<m^{*}<m^\mathsf{A }&\iff \phi ^\mathsf{A }<\phi <\phi ^\mathsf{AB }, \end{aligned}$$

(10.19g)

$$\begin{aligned} 0<m^{*}<m^\mathsf{B }&\iff \phi ^\mathsf{B }<\phi <\phi ^\mathsf{M _4}(\rho =0), \end{aligned}$$

(10.19h)

$$\begin{aligned} 0<-m^\mathsf{A }<m^{*}&\iff \phi <\phi ^\mathsf{M _4}(\rho =0), \end{aligned}$$

(10.19i)

$$\begin{aligned} 0<-m^\mathsf{B }<m^{*}&\iff \phi <\phi ^\mathsf{AB }, \end{aligned}$$

(10.19j)

$$\begin{aligned} 0<m^\mathsf{A }<m^{*}&\iff \phi ^\mathsf{AB }<\phi , \end{aligned}$$

(10.19k)

$$\begin{aligned} 0<m^\mathsf{B }<m^{*}&\iff \phi ^\mathsf{M _4}(\rho =0)<\phi . \end{aligned}$$

(10.19l)

From (4.12), (4.13), (10.18), and (10.19e)–(10.19l) we infer

$$\begin{aligned} \min \{|m^\mathsf{A }|,|m^\mathsf{B }|\} \le m^{*}. \end{aligned}$$

(10.20)

Next, we derive the relations between $m^{\mathsf{F _0}}$ and $m^\mathsf{A }$ or $m^\mathsf{B }$ needed in the proof of Theorem 4.6. As their derivation is lengthy, the reader may wish to skip the proof and go immediately to the results given by (10.37) and (10.38).

Our approach to derive the possible relations between $m^{\mathsf{F _0}}$ and $m^\mathsf{A }$ or $m^\mathsf{B }$ is as follows: First, we derive all relevant relations of $\phi ^\mathsf X$ (4.2) for arbitrary recombination $\rho$. We use these relations to determine the required relations between $m^\mathsf{A }$, $m^\mathsf{B }$, $m^\mathsf{M _1}$ and $m^\mathsf{M _4}$ for arbitrary $\rho$. By setting $\rho =0$ in the results obtained and by the equivalence given in (4.10), the possible relations between $m^{\mathsf{F _0}}$ and $m^\mathsf{A }$ or $m^\mathsf{B }$ follow immediately.

By definition, the values $\phi ^\mathsf{A }$, $\phi ^\mathsf{B }$, $\phi ^{\mathsf{F _0}}$, $\tilde{\phi }^\mathsf{AB }$, $\phi ^\mathsf{AB }$, $\phi ^\mathsf{A \mathsf{F _0}}$, and $\phi ^\mathsf{B \mathsf{F _0}}$ (4.2) are independent of the recombination rate $\rho$. Their relations under (2.3) are given in (4.12), (4.13), and (4.14).

The values $\phi ^\mathsf{M _1}$, $\tilde{\phi }^\mathsf{M _1}$, $\phi ^\mathsf{M _4}$, and $\tilde{\phi }^\mathsf{M _4}$ (4.2) depend on $\rho$, and we analyze this dependence in the following. The conditions which determine the admissibility of $\phi ^\mathsf{M _\mathsf i }$ and $\tilde{\phi }^\mathsf{M _\mathsf i }$ ($i=1,4$) are:

$$\begin{aligned} 0 < \phi ^\mathsf{M _1}< 1&\iff 0\le \rho <-\beta _2 \text{ or } \rho >-\alpha _2-\beta _2, \end{aligned}$$

(10.21a)

$$\begin{aligned} 0 < \tilde{\phi }^\mathsf{M _1}< 1&\iff 0\le \rho <-\alpha _2 \text{ or } \rho >-\alpha _2-\beta _2, \end{aligned}$$

(10.21b)

$$\begin{aligned} 0 < \phi ^\mathsf{M _4}< 1&\iff 0\le \rho <\alpha _1 \text{ or } \rho >\alpha _1+\beta _1, \end{aligned}$$

(10.21c)

$$\begin{aligned} 0 < \tilde{\phi }^\mathsf{M _4}< 1&\iff 0\le \rho <\beta _1 \text{ or } \rho >\alpha _2+\beta _2, \end{aligned}$$

(10.21d)

with the relations

$$\begin{aligned} 0\le \rho <-\beta _2&\Longrightarrow&0 < \phi ^\mathsf{M _1}< \phi ^\mathsf{A }, \end{aligned}$$

(10.22a)

$$\begin{aligned} 0\le \rho < \alpha _1&\Longrightarrow&\phi ^\mathsf{B }< \phi ^\mathsf{M _4}< 1. \end{aligned}$$

(10.22b)

To determine further relations of $\phi ^\mathsf{M _1}$, $\tilde{\phi }^\mathsf{M _1}$, $\phi ^\mathsf{M _4}$, and $\tilde{\phi }^\mathsf{M _4}$, we define the following critical recombination rates:

$$\begin{aligned} \rho ^\mathsf{M _1}\!&= \! \frac{\alpha _2\beta _1(\alpha _2\!+\!\beta _1)\!-\!\alpha _1\beta _2(\alpha _1\!+\!\beta _2)}{2\theta } \nonumber \\&+\,\frac{\sqrt{(\alpha _2\beta _1(\alpha _2\!+\!\beta _1)\!-\!\alpha _1\beta _2(\alpha _1\!+\!\beta _2))^2\!-\!4 \theta ^2 (\alpha _2\beta _1\!+\!\alpha _1\beta _2)}}{2\theta }, \end{aligned}$$

(10.23a)

$$\begin{aligned} \tilde{\rho }^\mathsf{M _1}\!&= \! \frac{-\theta (\alpha _1\!+\!\alpha _2)\!+\!\alpha _2\beta _1^2\!-\!\alpha _1\beta _2^2}{2\theta } \nonumber \\&+\,\frac{\sqrt{(\!-\!\theta (\alpha _1\!+\!\alpha _2)\!+\!\alpha _2\beta _1^2\!-\!\alpha _1\beta _2^2)^2\!-\!4\theta ^2(\alpha _2\beta _1\!+\!\alpha _1(\alpha _2\!+\!\beta _2))}}{2\theta }, \end{aligned}$$

(10.23b)

$$\begin{aligned} \rho ^\mathsf{M _4}\!&= \! \frac{\alpha _1\beta _2(\alpha _1\!+\!\beta _2)\!-\!\alpha _2\beta _1(\alpha _2\!+\!\beta _1)}{2\theta } \nonumber \\&+ \frac{\sqrt{(\alpha _1\beta _2(\alpha _1\!+\!\beta _2)\!-\!\alpha _2\beta _1(\alpha _2\!+\!\beta _1))^2\!-\!4\theta ^2(\alpha _2\beta _1\!+\!\alpha _1\beta _2)}}{2\theta }, \end{aligned}$$

(10.23c)

$$\begin{aligned} \tilde{\rho }^\mathsf{M _4}\!&= \! \frac{\theta (\beta _1\!+\!\beta _2)\!+\!\alpha _1^2\beta _2\!-\!\alpha _2^2\beta _1}{2\theta } \nonumber \\&+ \frac{\sqrt{(-\theta (\beta _1\!+\!\beta _2)\!+\!\alpha _2^2 \beta _1\!-\!\alpha _1^2\beta _2)^2-4\theta ^2(\alpha _2\beta _1\!+\!\beta _2(\alpha _1+\beta _1))}}{2\theta }.\qquad \end{aligned}$$

(10.23d)

Next, we determine the admissibility of $\rho ^\mathsf X$ and $\tilde{\rho }^X$ defined in (10.23). Therefore, we partition the selection parameters satisfying (2.3) and $\beta _2>\alpha _2$ according to

$$\begin{aligned} \beta _2 > \alpha _2 +\frac{\alpha _1\beta _2(\beta _2-\alpha _1)}{\beta _1(\alpha _1+\beta _1-\beta _2)} > \alpha _2 \end{aligned}$$

(10.24a)

and

$$\begin{aligned} \alpha _2 +\frac{\alpha _1\beta _2(\beta _2-\alpha _1)}{\beta _1(\alpha _1+\beta _1-\beta _2)}>\beta _2 > \alpha _2. \end{aligned}$$

(10.24b)

Analogously, the selection parameters satisfying (2.3) and $\beta _1>\alpha _1$ can be partitioned according to

$$\begin{aligned} \beta _1 > \alpha _1+\frac{\alpha _1\alpha _2(\alpha _1-\alpha _2)}{\beta _2(\alpha _1-\alpha _2-\beta _2)}>\alpha _1 \end{aligned}$$

(10.25a)

and

$$\begin{aligned} \alpha _1+\frac{\alpha _1\alpha _2(\alpha _1-\alpha _2)}{\beta _2(\alpha _1-\alpha _2-\beta _2)}>\beta _1>\alpha _1. \end{aligned}$$

(10.25b)

Using these partitions, we obtain that $\rho ^\mathsf X$ and $\tilde{\rho }^X$ satisfy the following relations (as can be checked with Mathematica):

$$\begin{aligned} \tilde{\rho }^\mathsf{M _1}< \rho ^\mathsf{M _1}< -\alpha _2 <-\beta _2&\iff \beta _2 < \alpha _2, \end{aligned}$$

(10.26a)

$$\begin{aligned} \tilde{\rho }^\mathsf{M _1}< \rho ^\mathsf{M _1}= -\alpha _2 = -\beta _2&\iff \beta _2 = \alpha _2,\end{aligned}$$

(10.26b)

$$\begin{aligned} -\beta _2<\tilde{\rho }^\mathsf{M _1}< -\alpha _2<\rho ^\mathsf{M _1}<-\alpha _2-\beta _2&\iff (10.24\text{ a }) \text{ holds },\end{aligned}$$

(10.26c)

$$\begin{aligned} \tilde{\rho }^\mathsf{M _1}<-\beta _2< -\alpha _2<\rho ^\mathsf{M _1}<-\alpha _2-\beta _2&\iff (10.24\text{ b }) \text{ holds },\end{aligned}$$

(10.26d)

$$\begin{aligned} \alpha _1<\tilde{\rho }^\mathsf{M _4}<\beta _1<\rho ^\mathsf{M _4}<\alpha _1+\beta _1&\iff (10.25\text{ a }) \text{ holds }, \end{aligned}$$

(10.26e)

$$\begin{aligned} \tilde{\rho }^\mathsf{M _4}<\alpha _1<\beta _1<\rho ^\mathsf{M _4}<\alpha _1+\beta _1&\iff (10.25\text{ b }) \text{ holds }. \end{aligned}$$

(10.26f)

As $\rho \ge 0$ and (2.3) hold, we obtain that

$$\begin{aligned} \phi ^\mathsf{M _1}= \tilde{\phi }^\mathsf{M _1}&\iff \rho = -\alpha _2-\beta _2 \text{ or } \rho = \rho ^\mathsf{M _1},\qquad \end{aligned}$$

(10.27a)

$$\begin{aligned} \tilde{\phi }^\mathsf{AB }= \phi ^\mathsf{M _1}\;\iff \; \tilde{\phi }^\mathsf{AB }= \tilde{\phi }^\mathsf{M _1}&\iff \rho = \rho ^\mathsf{M _1},\qquad \end{aligned}$$

(10.27b)

$$\begin{aligned} \phi ^\mathsf{A }=\tilde{\phi }^\mathsf{M _1}&\iff \rho = \tilde{\rho }^\mathsf{M _1},\qquad \end{aligned}$$

(10.27c)

and

$$\begin{aligned} \phi ^\mathsf{M _4}= \tilde{\phi }^\mathsf{M _4}&\iff \rho = \alpha _1+\beta _1 \text{ or } \rho = \rho ^\mathsf{M _4}, \end{aligned}$$

(10.28a)

$$\begin{aligned} \phi ^\mathsf{B }=\tilde{\phi }^\mathsf{M _4}&\iff \rho = \tilde{\rho }^\mathsf{M _4}, \end{aligned}$$

(10.28b)

where

$$\begin{aligned} \phi ^\mathsf{A }=\phi ^\mathsf{M _1} \text{ and } \, \phi ^\mathsf{B }=\phi ^\mathsf{M _4}\; \text{ are } \text{ infeasible }. \end{aligned}$$

(10.28c)

We derive the following relations additional to (10.22a) and (10.22b), using (10.26) and (10.27):

$$\begin{aligned} 0<\tilde{\phi }^\mathsf{M _1}<\phi ^\mathsf{A }&\iff \tilde{\rho }^\mathsf{M _1}<\rho <-\alpha _2,\end{aligned}$$

(10.29a)

$$\begin{aligned} \phi ^\mathsf{A }<\tilde{\phi }^\mathsf{M _1}<1&\;\;\Longleftarrow \;&0\le \rho < \tilde{\rho }^\mathsf{M _1}, \end{aligned}$$

(10.29b)

and

$$\begin{aligned} \phi ^\mathsf{B }<\tilde{\phi }^\mathsf{M _4}<1&\iff \tilde{\rho }^\mathsf{M _4}<\rho <\beta _1, \end{aligned}$$

(10.30a)

$$\begin{aligned} 0<\tilde{\phi }^\mathsf{M _4}<\phi ^\mathsf{B }&\;\;\Longleftarrow \;&0\le \rho <\tilde{\rho }^\mathsf{M _4}, \end{aligned}$$

(10.30b)

and by recalling (4.13):

$$\begin{aligned} \tilde{\phi }^\mathsf{AB }< \phi ^\mathsf{M _1}<\phi ^\mathsf{A }&\; \iff \;&\beta _2 < \alpha _2 \quad \text{ and }\; 0\le \rho < \rho ^\mathsf{M _1},\end{aligned}$$

(10.31a)

$$\begin{aligned} 0 < \phi ^\mathsf{M _1}< \tilde{\phi }^\mathsf{AB }&\; \iff \;&\beta _2 < \alpha _2 \quad \text{ and }\; \rho ^\mathsf{M _1}< \rho < -\alpha _2. \end{aligned}$$

(10.31b)

Now we derived all relations between $\phi ^X$ necessary to deduce the relevant relations between $m^\mathsf{M _1}$ ($m^\mathsf{M _4}$) and $m^\mathsf{A }$, $m^\mathsf{B }$.

First, we note that under (2.3), $m^\mathsf{M _1}>0$ if $\phi <\phi ^\mathsf{A }$ and $m^\mathsf{M _4}>0$ if $\phi ^\mathsf{B }<\phi$.

In the following, we derive all possible relations between $m^\mathsf{A }$, $m^\mathsf{B }$, and $m^\mathsf{M _1}$ where we assume that $\phi <\phi ^\mathsf{A }$ (otherwise $\mathsf M _1$ is unstable, cf. Proposition 4.1, and $m^\mathsf{M _1}$ is not of particular interest). By (4.5a), (4.5b), (4.12), (4.13), (10.22a), (10.27), (10.29), and (10.31), we obtain that

$$\begin{aligned}&0<-m^\mathsf{B }<-m^\mathsf{A }<m^\mathsf{M _1}\nonumber \\&\quad \iff \; {\left\{ \begin{array}{ll} \beta _2<\alpha _2 \quad \text{ and } \, \rho < \rho ^\mathsf{M _1}\quad \text{ and } \, \tilde{\phi }^\mathsf{AB }< \phi < \phi ^\mathsf{M _1}, \text{ or }\\ \beta _2\ge \alpha _2 \quad \text{ and } \, \rho < -\beta _2 \quad \text{ and } \, \phi < \phi ^\mathsf{M _1}, \end{array}\right. } \end{aligned}$$

(10.32a)

$$\begin{aligned}&0<-m^\mathsf{A }<-m^\mathsf{B }<m^\mathsf{M _1}\nonumber \\&\quad \iff \; \beta _2<\alpha _2 \quad \text{ and } \, {\left\{ \begin{array}{ll} \rho < \rho ^\mathsf{M _1}\quad \text{ and } \, \phi <\tilde{\phi }^\mathsf{AB }, \text{ or } \\ \rho ^\mathsf{M _1}< \rho <-\alpha _2 \quad \text{ and } \,\phi <\tilde{\phi }^\mathsf{M _1}, \end{array}\right. } \end{aligned}$$

(10.32b)

$$\begin{aligned}&0\!<\!-m^\mathsf{A }\!=\!-m^\mathsf{B }<m^\mathsf{M _1}\iff \beta _2<\alpha _2 \quad \text{ and } \, \rho \!<\! \rho ^\mathsf{M _1}\quad \text{ and } \,\phi \!=\!\tilde{\phi }^\mathsf{AB }, \nonumber \\ \end{aligned}$$

(10.32c)

and

$$\begin{aligned}&0<-m^\mathsf{B }< m^\mathsf{M _1}<-m^\mathsf{A }\nonumber \\&\quad \iff \; {\left\{ \begin{array}{ll} \beta _2 \le \alpha _2 \quad \text{ and } \, \rho < \tilde{\rho }^\mathsf{M _1}\; \text{ and } \, \phi ^\mathsf{M _1}<\phi <\phi ^\mathsf{A }, \text{ or } \\ \beta _2 \le \alpha _2 \quad \text{ and } \, \tilde{\rho }^\mathsf{M _1}<\rho <\rho ^\mathsf{M _1}\; \text{ and } \, \phi ^\mathsf{M _1}<\phi <\tilde{\phi }^\mathsf{M _1}, \text{ or } \\ (10.24\text{ a }) \quad \text{ and } \, \rho <-\beta _2 \; \text{ and } \, \phi ^\mathsf{M _1}<\phi <\phi ^\mathsf{A }, \text{ or } \\ (10.24\text{ a }) \quad \text{ and } \, -\beta _2< \rho <\tilde{\phi }^\mathsf{M _1}\; \text{ and } \, \phi <\phi ^\mathsf{A }, \text{ or } \\ (10.24\text{ a }) \quad \text{ and } \, \tilde{\rho }^\mathsf{M _1}< \rho <-\alpha _2 \; \text{ and } \, \phi <\tilde{\phi }^\mathsf{M _1}, \text{ or } \\ (10.24\text{ b }) \quad \text{ and } \, \rho <\tilde{\rho }^\mathsf{M _1}\; \text{ and } \, \phi ^\mathsf{M _1}<\phi <\phi ^\mathsf{A }, \text{ or } \\ (10.24\text{ b }) \quad \text{ and } \, \tilde{\rho }^\mathsf{M _1}<\rho <-\beta _2 \; \text{ and } \, \phi ^\mathsf{M _1}<\phi <\tilde{\phi }^\mathsf{M _1}, \text{ or } \\ (10.24\text{ b }) \quad \text{ and } \, -\beta _2<\rho <-\alpha _2 \; \text{ and } \, \phi <\tilde{\phi }^\mathsf{M _1}, \end{array}\right. } \end{aligned}$$

(10.33a)

$$\begin{aligned}&0< -m^\mathsf{A }< m^\mathsf{M _1}< -m^\mathsf{B }\nonumber \\&\quad \iff \; \beta _2<\alpha _2 \quad \text{ and } \, {\left\{ \begin{array}{ll} \rho ^\mathsf{M _1}< \rho < -\alpha _2 \quad \text{ and } \, \tilde{\phi }^\mathsf{M _1}< \phi < \phi ^\mathsf{M _1}, \text{ or }\quad \\ -\alpha _2 < \rho < -\beta _2 \quad \text{ and } \, \phi < \phi ^\mathsf{M _1}, \end{array}\right. } \end{aligned}$$

(10.33b)

$$\begin{aligned}&0< m^\mathsf{M _1}< -m^\mathsf{B }<-m^\mathsf{A }\nonumber \\&\quad \iff \; {\left\{ \begin{array}{ll} \beta _2<\alpha _2 \quad \text{ and } \, \tilde{\rho }^\mathsf{M _1}<\rho <\rho ^\mathsf{M _1}\; \text{ and } \, \tilde{\phi }^\mathsf{M _1}<\phi <\phi ^\mathsf{A }, \text{ or } \\ \beta _2<\alpha _2 \quad \text{ and } \, \rho ^\mathsf{M _1}<\rho \; \text{ and } \, \tilde{\phi }^\mathsf{AB }<\phi <\phi ^\mathsf{A }, \text{ or } \\ \beta _2 \ge {\alpha _2} \quad \text{ and } \, \tilde{\rho }^\mathsf{M _1}< \rho <-\beta _2 \; \text{ and } \, \tilde{\phi }^\mathsf{M _1}<\phi <\phi ^\mathsf{A }, \text{ or } \\ \beta _2 \ge {\alpha _2} \quad \text{ and } \, -\beta _2<\rho \; \text{ and } \, \phi <\phi ^\mathsf{A }, \text{ or } \end{array}\right. } \end{aligned}$$

(10.33c)

$$\begin{aligned}&0<m^\mathsf{M _1}<-m^\mathsf{A }<-m^\mathsf{B }\nonumber \\&\quad \iff \; \beta _2<\alpha _2 \quad \text{ and } \, {\left\{ \begin{array}{ll} \rho ^\mathsf{M _1}<\rho <-\beta _2 \quad \text{ and } \, \phi ^\mathsf{M _1}<\phi <\tilde{\phi }^\mathsf{AB }, \text{ or } \\ -\beta _2<\rho \quad \text{ and } \, \phi <\tilde{\phi }^\mathsf{AB }. \end{array}\right. } \end{aligned}$$

(10.33d)

From (10.18f) it follows that

$$\begin{aligned}&0<-m^\mathsf{A }<-m^\mathsf{B }<m^\mathsf{M _1}\quad \text{ occurs } \text{ never } \text{ if } \; \beta _2\ge \alpha _2, \end{aligned}$$

(10.34a)

$$\begin{aligned}&0< -m^\mathsf{A }< m^\mathsf{M _1}< -m^\mathsf{B }\quad \text{ occurs } \text{ never } \text{ if } \; \beta _2\ge \alpha _2,\end{aligned}$$

(10.34b)

$$\begin{aligned}&0<m^\mathsf{M _1}<-m^\mathsf{A }<-m^\mathsf{B }\quad \text{ occurs } \text{ never } \text{ if } \; \beta _2\ge \alpha _2. \end{aligned}$$

(10.34c)

To derive all possible relations between $m^\mathsf{A }$, $m^\mathsf{B }$, and $m^\mathsf{M _4}$ we assume $\phi >\phi ^\mathsf{B }$ (cf. Proposition 4.1). By (4.5a), (4.5b), (4.12), (4.13), (10.22b), (10.28), and (10.30), we obtain that

$$\begin{aligned}&0<m^\mathsf{A }<m^\mathsf{B }<m^\mathsf{M _4}\;\iff \; \rho <\alpha _1 \; \text{ and } \, \phi ^\mathsf{M _4}<\phi , \end{aligned}$$

(10.35a)

$$\begin{aligned}&0<m^\mathsf{A }<m^\mathsf{M _4}<m^\mathsf{B }\nonumber \\&\quad \iff \; {\left\{ \begin{array}{ll} (10.25\text{ a }) \quad \text{ and } \, \rho <\alpha _1 \quad \text{ and } \, \phi ^\mathsf{B }<\phi <\phi ^\mathsf{M _4}, \text{ or }\\ (10.25\text{ a }) \quad \text{ and } \, \alpha _1<\rho <\tilde{\rho }^\mathsf{M _4}\quad \text{ and } \, \phi ^\mathsf{B }<\phi , \text{ or }\\ (10.25\text{ a }) \quad \text{ and } \, \tilde{\rho }^\mathsf{M _4}<\rho <\beta _1 \quad \text{ and } \, \tilde{\phi }^\mathsf{M _4}<\phi , \text{ or }\\ (10.25\text{ b }) \quad \text{ and } \, \rho <\tilde{\rho }^\mathsf{M _4}\quad \text{ and } \, \phi ^\mathsf{B }<\phi <\phi ^\mathsf{M _4}, \text{ or }\\ (10.25\text{ b }) \quad \text{ and } \, \tilde{\rho }^\mathsf{M _4}<\rho <\alpha _1 \quad \text{ and } \, \tilde{\phi }^\mathsf{M _4}<\phi <\phi ^\mathsf{M _4}, \text{ or }\\ (10.25\text{ b }) \quad \text{ and } \, \alpha _1<\rho <\beta _1 \quad \text{ and } \, \tilde{\phi }^\mathsf{M _4}<\phi , \end{array}\right. } \end{aligned}$$

(10.35b)

$$\begin{aligned}&0<m^\mathsf{M _4}<m^\mathsf{A }<m^\mathsf{B }\;\iff \; {\left\{ \begin{array}{ll} \tilde{\rho }^\mathsf{M _4}<\rho <\beta _1 \; \text{ and } \, \phi ^\mathsf{B }<\phi <\tilde{\phi }^\mathsf{M _4}, \text{ or } \\ \beta _1<\rho \; \text{ and } \, \phi ^\mathsf{B }<\phi . \end{array}\right. }\nonumber \\ \end{aligned}$$

(10.35c)

From (10.18f) it follows that

$$\begin{aligned}&0<m^\mathsf{B }<m^\mathsf{A }<m^\mathsf{M _4}\; \text{ occurs } \text{ never } \text{ if } \; \alpha _1<\beta _1, \end{aligned}$$

(10.36a)

$$\begin{aligned}&0<m^\mathsf{B }<m^\mathsf{M _4}<m^\mathsf{A }\; \text{ occurs } \text{ never } \text{ if } \; \alpha _1<\beta _1, \end{aligned}$$

(10.36b)

$$\begin{aligned}&0<m^\mathsf{M _4}<m^\mathsf{B }<m^\mathsf{A }\; \text{ occurs } \text{ never } \text{ if } \; \alpha _1<\beta _1. \end{aligned}$$

(10.36c)

If $\phi <\phi ^\mathsf{A }$ or $\phi >\phi ^\mathsf{B }$, the relations involving $m^{\mathsf{F _0}}$, $m^\mathsf{A }$ and $m^\mathsf{B }$ follow immediately by (4.10), i.e., by setting $\rho =0$ in the relevant formulas in (10.32)–(10.35). The remaining cases where $\phi ^\mathsf{A }<\phi <\phi ^\mathsf{B }$ can be calculated easily using $\tilde{\theta }=\alpha _1\alpha _2-\beta _1\beta _2$, (4.5), (4.15), and (10.18). Then, all admissible relations are:

$$\begin{aligned} 0<-m^\mathsf{A }<-m^\mathsf{B }<-m^{\mathsf{F _0}}&\iff \alpha _2>\beta _2 \; \text{ and } \, \phi <\tilde{\phi }^\mathsf{AB }, \end{aligned}$$

(10.37a)

$$\begin{aligned} 0<-m^\mathsf{B }<-m^\mathsf{A }<-m^{\mathsf{F _0}}&\iff {\left\{ \begin{array}{ll} \alpha _2>\beta _2 \; \text{ and } \, \tilde{\phi }^\mathsf{AB }<\phi <\phi ^\mathsf{M _1}, \text{ or } \quad \\ \alpha _2\le \beta _2 \; \text{ and } \, \phi <\phi ^\mathsf{M _1},\quad \end{array}\right. } \end{aligned}$$

(10.37b)

$$\begin{aligned} 0<m^\mathsf{A }<-m^\mathsf{B }<-m^{\mathsf{F _0}}&\iff \tilde{\theta }>0 \; \text{ and } \, \phi ^\mathsf{AB }<\phi <\phi ^{\mathsf{F _0}}, \end{aligned}$$

(10.37c)

$$\begin{aligned} 0<-m^\mathsf{B }<m^\mathsf{A }<-m^{\mathsf{F _0}}&\iff {\left\{ \begin{array}{ll} \tilde{\theta }<0 \; \text{ and } \, \phi ^\mathsf{A \mathsf{F _0}}<\phi <\phi ^{\mathsf{F _0}}, \text{ or }\quad \\ \tilde{\theta }\ge 0 \; \text{ and } \, \phi ^\mathsf{A \mathsf{F _0}}<\phi <\phi ^\mathsf{AB }, \end{array}\right. }\end{aligned}$$

(10.37d)

$$\begin{aligned} 0<m^\mathsf{A }<-m^\mathsf{B }<m^{\mathsf{F _0}}&\iff {\left\{ \begin{array}{ll} \tilde{\theta }<0 \; \text{ and } \, \phi ^\mathsf{AB }<\phi <\phi ^\mathsf{B \mathsf{F _0}}, \text{ or }\quad \\ \tilde{\theta }\ge 0 \; \text{ and } \,\phi ^{\mathsf{F _0}}<\phi <\phi ^\mathsf{B \mathsf{F _0}}, \end{array}\right. }\end{aligned}$$

(10.37e)

$$\begin{aligned} 0<-m^\mathsf{B }<m^\mathsf{A }<m^{\mathsf{F _0}}&\iff \tilde{\theta }<0 \; \text{ and } \, \phi ^{\mathsf{F _0}}<\phi <\phi ^\mathsf{AB }, \end{aligned}$$

(10.37f)

$$\begin{aligned} 0<m^\mathsf{A }<m^\mathsf{B }<m^{\mathsf{F _0}}&\iff \phi ^\mathsf{M _4}(\rho =0)<\phi , \end{aligned}$$

(10.37g)

$$\begin{aligned} 0<-m^\mathsf{B }<-m^{\mathsf{F _0}}<-m^\mathsf{A }&\iff \phi ^\mathsf{M _1}(\rho =0)<\phi <\phi ^\mathsf{A }, \end{aligned}$$

(10.37h)

$$\begin{aligned} 0<-m^\mathsf{B }<-m^{\mathsf{F _0}}<m^\mathsf{A }&\iff \phi ^\mathsf{A }<\phi <\phi ^\mathsf{A \mathsf{F _0}}, \end{aligned}$$

(10.37i)

$$\begin{aligned} 0<m^\mathsf{A }<m^{\mathsf{F _0}}<-m^\mathsf{B }&\iff \phi ^\mathsf{B \mathsf{F _0}}<\phi <\phi ^\mathsf{B }, \end{aligned}$$

(10.37j)

$$\begin{aligned} 0<m^\mathsf{A }<m^{\mathsf{F _0}}<m^\mathsf{B }&\iff \phi ^\mathsf{B }<\phi <\phi ^\mathsf{M _4}(\rho =0). \end{aligned}$$

(10.37k)

Because other strict inequalities between $m^\mathsf{A }$, $m^\mathsf{B }$, and $m^{\mathsf{F _0}}$ do not occur, we infer

$$\begin{aligned} \min \{|m^\mathsf{A }|,|m^\mathsf{B }|\} \le |m^{\mathsf{F _0}}|. \end{aligned}$$

(10.38)