# Fixation probabilities in populations under demographic fluctuations

## Abstract

We study the fixation probability of a mutant type when introduced into a resident population. We implement a stochastic competitive Lotka–Volterra model with two types and intra- and interspecific competition. The model further allows for stochastically varying population sizes. The competition coefficients are interpreted in terms of inverse payoffs emerging from an evolutionary game. Since our study focuses on the impact of the competition values, we assume the same net growth rate for both types. In this general framework, we derive a formula for the fixation probability \(\varphi \) of the mutant type under weak selection. We find that the most important parameter deciding over the invasion success of the mutant is its death rate due to competition with the resident. Furthermore, we compare our approximation to results obtained by implementing population size changes deterministically in order to explore the parameter regime of validity of our method. Finally, we put our formula in the context of classical evolutionary game theory and observe similarities and differences to the results obtained in that constant population size setting.

## Keywords

Demographic stochasticity Fixation probability Diffusion theory Evolutionary games Weak selection## Mathematics Subject Classification

60J60 91A22 92D25## 1 Introduction

The evolutionary dynamics of a mutant strain in a resident population is a well-studied topic in the field of population dynamics. Results concerning the fixation probability, the average fixation time or coexistence behavior can be applied in various biological fields, e.g. population genetics, bacterial evolution, viral dynamics or cancer initiation (Nowak 2006; Altrock et al. 2015; Ewens 2004). While the first theoretical analysis of such processes relied on deterministic differential equations, over the course of time more detailed models were studied describing the stochasticity of microscopic processes on the individual level. These individual based models can be approximated and studied by the replicator equation (in the large population size limit), by branching processes (in case of a small number of invading mutants) or by multi-type birth-death processes (Nowak 2006; Hofbauer and Sigmund 1998; Sandholm 2010; Haccou et al. 2005; Ewens 2004). However, the ecological evolution of the entire population is mostly neglected in these kinds of models and a constant population size is assumed instead. On the other hand, in population genetics and theoretical ecology, studies focused more on the effect that population dynamics have on the fixation probability rather than the concrete interaction mechanisms between the mutant and wild-type individuals (Ewens 1967; Kimura and Ohta 1974; Otto and Whitlock 1997). More recently, researchers started investigating models connecting the stochastic interaction between individuals and stochastic population dynamics from a theoretical point of view (Lambert 2005, 2006; Champagnat and Lambert 2007; Parsons and Quince 2007a, b; Melbinger et al. 2010; Cremer et al. 2011; Gabel et al. 2013; Constable et al. 2016; Chotibut and Nelson 2017). For a historical overview on the calculation of fixation probabilities, see Patwa and Wahl (2008).

To our knowledge, the first analytical approximation of fixation probabilities under stochastically varying population sizes is due to Lambert (2005, 2006). In these papers, the author analyzes models of interacting species by considering the corresponding diffusion equations under the constraint of weak selection. Going one step back on the descriptive scale and analyzing the Kolmogorov forward equation instead of its diffusion approximation, Champagnat and Lambert study the effect of various model parameters on the fixation probabilities and extend the previous results (Champagnat and Lambert 2007). In parallel to these studies, Parsons and Quince examined the effect of variable growth rates on the fixation probability and mean fixation time in a two species system with stochastically varying population size (Parsons and Quince 2007a, b). These results were later complemented and refined in Parsons et al. (2010). Instead of focusing on variable growth rates, in this paper we concentrate on the effect of variable competition coefficients on the fixation probability.

The model we will work with was introduced in Huang et al. (2015). It is a generalized stochastic Lotka–Volterra-model which connects an evolutionary game with the competition coefficients of the model. Individuals reproduce at constant rates and die spontaneously or based on competition within and between species. This leads to stochastically induced demographic fluctuations driven by interactions within the population. We focus on two types only and our goal is to calculate the probability that a mutant takes over such a population of changing size.

Recently, further models have been studied which connected game theoretical dynamics with exogenous population growth (Melbinger et al. 2010; Tao and Cressman 2007; Traulsen et al. 2012; Cremer et al. 2011; McAvoy et al. 2018). For instance, Ashcroft et al. (2017) consider a model with deterministic cell growth defined by a power law and stochastic species interactions derived from an evolutionary game. The authors rely on simulation results suggesting that the evolutionary outcome not only depends on the game played by the species, but also on the growth exponent of the power law governing the population growth.

In the context of public goods dynamics (Constable et al. 2016) study the invasion probability of producers and non-producers of the public good under varying population sizes. Using a time-scale separation under a weak selection approximation, they find that producers can successfully invade a colony of non-producers even though they have a lower fitness than the resident type.

The present paper is structured as follows: In Sect. 2 we describe the generalized Lotka–Volterra-model and restate some basic properties of the system, which were already described in Huang et al. (2015). In Sect. 3 we apply tools developed by Lambert (2006) in order to derive a formula for the fixation probability in the weak selection limit. This allows us to interpret the impact of the competition coefficients separately. Furthermore, we compare the results for various competition matrices induced by different games with each other, i.e. the differences between coordination, coexistence and dominance games. Finally, in Sect. 4 we compare our results to similar models derived in the physics and population genetics literature dealing with deterministic ecological evolution of the model. Furthermore, we examine the fixation probability of a single mutant in a wild-type population, which allows us to compare our findings with those obtained in classical evolutionary game theory, i.e. in finite but fixed population sizes.

## 2 Model

The model we consider is a competitive Lotka–Volterra system consisting of two types, the mutant *X* and the wild-type *Y*. We assume a well-mixed population, i.e. dynamics do not depend on the spatial arrangement of individuals, and a discrete state space describing the number of individuals of the two types, *X* and *Y*.

*M*controls the total population size in stationarity.

*X*(

*t*) and

*Y*(

*t*) be the number of mutant and wild-type individuals at time

*t*, respectively, the differential equations of the deterministic model that emerges in the large population limit read

*M*indeed characterizes the scale of the total population size in equilibrium. In the following, we will work with the rescaled variables \(x(t)=\frac{X(t)}{M}\) and \(y(t)=\frac{Y(t)}{M}\) while the fraction of mutants in the population is given by \(p=\frac{x}{x+y}\). We denote the steady state of this value by

*X*in a population of

*Y*individuals. The techniques we use rely on the theory of stochastic diffusions, see e.g. Ewens (2004). Hence, we will work with the diffusion approximation of the above system; for a detailed derivation see “Appendix A”. We find the stochastic differential equations

*X*.

## 3 Fixation probabilities

The main result of this paper is the approximation of the probability of a mutant strain to reach fixation in a resident population of randomly fluctuating size under weak selection. Note first that due to the competition processes neither of the two species is able to go to \(\infty \) and hence each of them will die out at a (finite) random time (Lambert 2006). We define fixation of the mutant *X* as follows (see also Lambert 2006, Definition 1.1 for a similar definition):

### Definition 1

(*Fixation*) Species *X* becomes fixed if for some \(t\ge 0\) we have \(y(t)=0\) and \(x(t)>0\).

*X*if the initial type-frequencies are \(x_0\) and \(y_0\). Then diffusion theory, see also Ewens (2004) and Gardiner (2004) or “Appendix B”, implies that \(\varphi \) solves

*X*-individuals in the population and the whole rescaled population size, respectively. Given the same birth and death rates for both species, i.e. \(\beta _X=\beta _Y=\beta \) and \(\gamma _X=\gamma _Y=\gamma \), and noting that

Our goal is to approximate the solution of Eq. (8). Therefore, we start with the neutral setting which forms the basis of the subsequent calculations.

### 3.1 Neutral model

### 3.2 Fixation probability under weak selection

- (i)
\((1-\frac{d}{b})^2\ll 1\),

- (ii)
\((1-\frac{d}{b})(2-\frac{d}{c}-\frac{d}{b}) \ll 1\),

- (iii)
\((1-\frac{d}{b})(1+\frac{d}{a}-2\frac{d}{b}) \ll 1\) and

- (iv)
\((1-\frac{d}{b})(\frac{1}{p^*}-2) \ll 1\).

*d*and

*b*are close together such that their squared difference is negligible. Due to conditions (ii) and (iii) it follows that also

*a*and

*c*are close to

*d*and thus

*b*which then implies that the whole competition matrix is just a small perturbation of the neutral matrix in which all entries are the same. Finally, condition (iv) is needed to ensure that there indeed is an internal fixed point in the deterministic system which is close to 1 / 2. For the case that there exists no internal fixed point we refer to Sect. 3.4.

We note again, that we claim that the product of two values which are already small is negligible, the single terms separately however are important for our approximation formula. We see in the derivation of the following theorem that the condition on the equilibrium \(p^*\) is necessary for the separation of the variables *p* and *z* reflecting a separation of the ecological and the evolutionary variable. For a extended discussion on the choice of fixed points in these kind of models see also Czuppon and Gokhale (2018).

### Theorem 1

*p*and solves

### Remark 1

We note that this is basically a linearization around the neutral fixation probability and reduces to the neutral model if all payoff coefficients are equal, i.e. \(\varphi (p,z)=p\) due to \(1-\frac{d}{b}=0\).

The proof of the Theorem is given in “Appendix D”. The idea is to apply the infinitesimal generator \(\widetilde{G}\) from Eq. (7) to the formula stated in Eq. (9). Inserting conditions (i)–(iv) then gives the result.

*b*and

*d*can change the sign of the first order effect under weak selection. An interesting application is to consider fixation out of the neighbourhood of the internal steady state. Precisely at that point, we find \(\varphi (p^*,z) = p^*\) under weak selection. This is also implicit in the formulas derived in Lambert (2006), Champagnat and Lambert (2007), and Constable and McKane (2015) but has not been stated explicitly. For \(p=p^*+\varepsilon \), we have

*p*and larger than neutral for negative deviations in

*p*. On the other hand, for an unstable steady state \(p^*\) (\(d>b\)), the fixation probability out of the neighborhood is larger than neutral for positive deviations in

*p*and smaller than neutral for negative deviations in

*p*. For a detailed study of fixation probabilities when leaving the deterministic steady state see also Park and Traulsen (2017).

- (a)
coexistence game – \(a<c\) and \(b>d\);

- (b)
coordination game – \(a>c\) and \(b<d\);

- (c)
dominance game – \(a>c, b>d\) or \(a<c,b<d\).

A qualitatively different picture arises in case (c). Here, either type *X* or type *Y* strictly dominates the other species in a game theoretic sense. This implies that the deterministic model only allows for single species equilibria where the stationary point of the dominant (inferior) type is stable (unstable). Thus, Theorem 1 does not hold in this case since \(p^*\) does not tend to \(\frac{1}{2}\). In fact, \(p^*\) does not even exist. Instead we will replace condition (iv) by an adapted version which then gives a similar approximation, see Eq. (14) in Sect. 3.4 below.

### 3.3 Coexistence and coordination games

In this section, we compare the resulting fixation probabilities in a coexistence and coordination game with the neutral fixation probability \(\varphi _{neu}(p,z)=p\). In order to do so we need a Lemma characterizing the impact of the initial population size which we prove in “Appendix E”.

### Lemma 1

The solution \(\psi (z)\) of Eq. (10) is positive for all \(z>0\).

### Remark 2

In fact the function \(\psi \) is a growing function in *z* as can be seen in Fig. 1. This can be explained as follows: since the only difference between the strains are the corresponding competition parameters, regions where these parameters play a dominant role should have a larger impact on the fixation probability. Translating this to initial population sizes yields that larger initial population sizes, being affected mostly by competition processes, also have a larger influence on the evolutionary outcome and thus a larger value for \(\psi \).

Now, we can state some immediate consequences of the fixation probability which follow from Eq. (9).

### Corollary 1

- 1.
For arbitrary \(a,c>0\) and \(p<p^*\) we have that \(\varphi >\varphi _{neu}\) iff \(b>d\).

- 2.
The probability of fixation is an increasing function in the competition parameter

*a*.

### Proof

*a*which gives

### Remark 3

The first statement of the Corollary has the obvious implication that for a mutant to invade a resident population it is important to perform well against the wild-type. This also implies that species with lower single species equilibria (i.e. \(a<d\)) can have a higher chance of fixating than neutral. This can end up in a decrease of the overall population size. But still, as the second part of the Corollary shows, species with higher single-species equilibria also have a higher chance to reach fixation.

Before turning to dominance games we take a brief look at some special cases in the context of coexistence and coordination games.

#### 3.3.1 Symmetric and asymmetric games

*b*closer to

*d*improves the analytical prediction due to the weak selection approximation in conditions (i)–(iii).

*c*does play a role. To be more precise, whenever \(b>c\) we have \(\varphi _{asym}>\varphi _{sym}\). This condition resembles the shifting of the internal equilibrium of the deterministic system towards the mutant-axis, i.e.

### 3.4 Dominance game

In contrast to coexistence and coordination games the dominance game does not have an internal equilibrium in its deterministic counterpart. This is due to one strategy strictly dominating the other. In terms of parameters this means that either \(a>c\) and \(b>d\) or \(a<c\) and \(b<d\) hold. As already mentioned, for the analysis of the fixation probability it does not make sense to assume condition (iv) which states that the internal equilibrium should be close to \(\frac{1}{2}\). Hence, we can already infer that the analytical solution will not intersect with the neutral line due to one strategy being favored independently of its frequency (Fig. 4).

- (v)
\(\left( 1-\frac{d}{b}\right) \left( 1+\frac{db}{ac}\frac{c-a}{b-d}\right) \ll 1\).

### Theorem 2

The proof is an imitation of the proof of Theorem 1 and therefore spared out.

*b*being larger or smaller than

*d*, respectively. This finding is rather trivial, since we consider a dominance game and \(b>d\) ensures the mutant being advantageous. More importantly, Eq. (14) allows to calculate the first order approximation of the neutral result.

## 4 Discussion

Now that we have derived the fixation probabilities in all possible scenarios captured by our model, we can compare them to previously derived results. Since our model takes into account stochastic fluctuations in population size, it is especially interesting to see how that affects the fixation probability in comparison to deterministically varying or constant population sizes. Concerning constant population sizes we also compare our results to the \(\frac{1}{3}\)-rule which deals with the invasion of a single mutant into a wild-type population.

### 4.1 Comparison to models with deterministically varying population size

Fixation probabilities under deterministically changing population sizes have been extensively studied in the population genetics literature (Uecker and Hermisson 2011; Otto and Whitlock 1997; Kimura and Ohta 1974; Waxman 2011; Wahl and Gerrish 2001). It is possible to, at least partially, compare our results to these models. In order to do so, we consider the case of a dominating trait with frequency independent selection, i.e. we are in the context of Theorem 2 with parameters being equal along the rows of the competition matrix, i.e. \(a=b\) and \(c=d\). This transforms the competition being only dependent on the population density and not on the population composition. Then typically \(s=a-c\) represents the fitness advantage of a mutant when compared to a wild-type resulting in a higher replacement (or reproduction) rate of wild-type individuals. This effect cannot easily be translated to our mechanistic model in (1) and (2). In our model selection happens due to competition and thus affects the death rates of individuals, see (Baar et al. 2017) for a similar model in the adaptive dynamics framework. To account for that we should set \(s=\frac{1}{d}-\frac{1}{b}\) which is already part of the first order term in our derived approximations in equations (9) and (11), see also Lambert (2006) for a similar definition of selection in a similar model.

Using the selection coefficient *s* and the population size *M*, a classical assumption in population genetics is \(Ms\gg 1\) which is violated in our setting since our approximation is applicable when \(Ms<1\) which emerges from the selection interaction modeled by the microscopic competition reactions stated in (2). Furthermore, once an individual dies due to a competition event it does not get replaced by another one, allowing for fluctuating population sizes. This is, as far as we can judge, different to models with varying population size in existing population genetics literature where the population size dynamics are often separated from the evolutionary dynamics. These two differences prevent a rigorous comparison between results obtained in the population genetics context and our microscopic model.

Employing a mechanistic individual based model as we do has both advantages and disadvantages. As a drawback, one could argue that externally driven fluctuations in population size, for example seasonal fluctuations or bottleneck experiments as analyzed in Uecker and Hermisson (2011), and Wahl and Gerrish (2001), cannot be dealt with—at least not in our studied model. On the other hand the microscopic model allows for concrete interpretation of the selection dynamics and allows us to explore the stochastic effects in small populations in more detail. That is, as opposed to a general selection value *s* affecting the birth rate, we can study the specific impact of varying the different parameters corresponding to single reactions much like it was performed in Lambert (2006), Champagnat and Lambert (2007), Parsons and Quince (2007a, b), Parsons et al. (2010), and Czuppon and Gokhale (2018). This microscopic approach has recently received some attention, see Doebeli et al. (2017).

We now proceed to compare our approximation with those obtained in the literature considering similar scenarios.

#### 4.1.1 Deterministic logistic growth—Otto and Whitlock (1997), Kimura and Ohta (1974)

*s*is the selection advantage,

*K*the carrying capacity and

*r*the logistic growth parameter. These parameters can be translated to our setting. As the selection strength we set \(s=\frac{1}{d}-\frac{1}{b}\) which is the selective advantage of the wild-type in the frequency-independent case, i.e. \(a=b\) and \(c=d\). The carrying capacity is set to \(K=(\beta -\gamma )aM\) and \(r=\beta -\gamma \) is the effective growth.

The equation derived by Otto and Whitlock is valid for \(s,r< 0.1\) and large population sizes *zM*. Since neither \(r<0.1\) nor \(zM\gg 1\) are met in our simulations it is not surprising to see deviations between our approximation and (16), see also Fig. 5.

*Q*is given by (see Eq. (A8) in Otto and Whitlock 1997)

#### 4.1.2 Deterministic logistic growth—Uecker and Hermisson (2011)

*s*is the selective advantage of the mutant,

*r*the logistic growth rate,

*b*the birth rate of individuals, \(\xi \) the neutral evolutionary replacement rate and

*K*the carrying capacity. Translating our modeling parameters into this setting we obtain \(s=\frac{1}{d}-\frac{1}{b}\), \(r=\beta -\gamma \), \(b=\beta \), \(\xi =0\) and \(K=raM\). We note, that the choice of \(\xi \) is to some extent arbitrary. We chose to set it to 0 since we do not have any comparable mechanism in our model.

The authors note that for \(\varphi _{UH}(1/(zM)) M \gtrsim 10\) the approximation in Eq. (18) fits excellently to the simulation results. However, again this does not belong to the range covered by our approximation such that we find a disagreement between the approximations, see Fig. 5.

*p*. Their equation (14) together with (24) then give

However, the branching process approach is, as the approximations by Otto and Whitlock and Kimura and Ohta, limited to the case of beneficial mutations, i.e. \(s>0\). We therefore compute only the case \(a=b>c=d\) for this approximation in the figures below.

### Remark 4

It might seem odd that both branching process estimates obtained by Otto/Whitlock and Uecker/Hermisson overestimate the actual fixation probability. Usually, branching process approaches tend to underestimate fixation probabilities since they do not account for the absorbing state at \(p=1\). However, the overestimation in these cases comes from the distinct implementation of the selection processes. While in our model selection acts implicitly through the competition processes, in the mentioned papers it is explicitly added to the birth rate of the mutant. The complicated intertwining of selection and population ecology in our model causes difficulties in computing the fixation probabilities in the same vein as suggested in the branching process approaches. On the other hand the added selective advantage to the birth rate causes the mutant to have a higher reproductive rate independent of the population composition and size which does not hold either in our case. Hence the overestimation of the results obtained in Otto and Whitlock (1997) and Uecker and Hermisson (2011).

#### 4.1.3 Nearly constant population size—Constable and McKane (2015)

We now want to compare our results to a model with nearly constant population size, i.e. basically constant population size with small fluctuations around the ecological equilibrium. Therefore, we use a time-scale separation argument, developed by Constable and McKane (2015) in the case of Lotka–Volterra dynamics.

*O*(1) the process first grows until it reaches size

*O*(

*M*) since until then the competition terms, which are of order \(O(M^{-1})\), can be neglected. A similar reasoning shows that for initial population sizes of order \(O(M^\alpha )\) for \(\alpha >1\) the population decreases to order

*O*(

*M*) since the linear birth and death terms are negligible, at least when compared to the competition rates. Thus, competition is the key driver in this case. This reasoning is true as long as

*M*is sufficiently large such that the birth-death and competition processes can be separated. As we see in Fig. 6, choosing \(M=100\) is already large enough to observe this time-scale separation.

Summarizing the comparison between the different models we observe that for lower population sizes (\(M\sim 100\)) where stochastic fluctuations have a strong impact on the model dynamics, our approximation which explicitly accounts for these fluctuations, provides the best fit. For intermediate population sizes (\(M\sim 1000\)) our approximation seems to overestimate the effect of demographic stochasticity and thus methods like the projection of Constable and McKane describe the evolutionary trajectories best. However both, our approximation as well as the time-scale separation work with very weak selection strengths (\(s=1-\frac{d}{b}\)) such that \(Ms=O(1)\). Other scalings of *Ms* yield different limiting models such that other techniques are necessary for analyzing the fixation probability. Therefore, e.g. Uecker and Hermisson use branching processes to study a system with a larger population size (or selection strength) shifting *Ms* to larger values. Their results work best for parameter choices which yield \(Ms>10\) and provide better fits than the previously mentioned approximations.

### 4.2 Fixation of a single mutant under frequency-dependent selection

*zM*. As can be seen in Fig. 9, the probability of fixation is a decreasing function of the initial population sizes. This translates to the already observed fact that fixation of a mutant strain is more likely in a smaller population and therefore initially growing population than in a larger one which is initially decreasing, cf. Kimura and Ohta (1974). We can also infer this from the formula describing the fixation probability: since we are working in the weak selection limit the governing part of \(\varphi (p,z)\) is the initial frequency of mutants, here \(\frac{1}{zM}\), which is decreasing for increasing

*z*which means that \(\varphi \) is a decreasing function in

*z*.

#### 4.2.1 Comparison to models with constant population sizes

In models with a constant population size *N*, the fixation probability of a mutant strain under weak selection can be translated to the location of the mixed equilibrium of the corresponding replicator equation. More specifically, in the case of bistability (coordination game) a mutant has a higher probability than neutral, i.e. \(\frac{1}{N}\), to invade a resident population if the basin of attraction of the wild-type is smaller than \(\frac{1}{3}\). This is referred to as the \(\frac{1}{3}\)-rule first derived in Nowak et al. (2004) and later generalized in scope in Lessard and Ladret (2007); Pfaffelhuber and Wakolbinger (2018). An equivalent approximation has also been made for multiplayer games (Kurokawa and Ihara 2009; Gokhale and Traulsen 2010; Lessard 2011). However, as noted in Traulsen et al. (2006) and Sample and Allen (2017) the order of limits is relevant for the rule to hold. One first needs to take the weak selection limit and then perform an approximation for large population sizes *N*.

Just by looking at our conditions for the approximation to hold, especially condition (iv) stating that \(p^*\approx \frac{1}{2}\), we would not expect the \(\frac{1}{3}\)-law to hold. Still, we are able to draw some more general conclusions from our approximation and those obtained in Constable and McKane (2015) which suggests that a similar rule may not hold in populations with fluctuating size.

*b*and

*d*describing the competition pressure of the species due to the resident type. However, these do not imply any properties of equilibria in the corresponding deterministic system. Even more, the choice of the parameters

*a*and

*c*does not affect the qualitative behavior of the fixation probability when compared to the neutral case as was already pointed out in Corollary 1. Thus, in the present model the difference between the neutral fixation probability and the probability of fixation under weak selection can be entirely described by the parameter \(\frac{d}{b}\) and is independent of \(\frac{a}{c}\) as can be seen in Fig. 10.

*N*(or rather

*M*) be finite which results in the first term \(\frac{1}{zM}\) being non-zero in contrast to being 0 for \(M\rightarrow \infty \). Additionally, our first order term also involves the deterministic equilibrium \(p^*\) which is again reminiscent of the formulas derived under weak selection. Hence, our result can be seen as a connection between the two limit orderings since it involves both, ecological and evolutionary terms in the first order approximation. Thus, our approach does not belong to the class of “first population limit, then selection limit” models as for instance analyzed in Sample and Allen (2017) nor does it belong to the “first weak selection, then large population limit” class but is rather somewhere in between the two limit orderings (even though formally according to Definition 2 provided in Sample and Allen (2017) our limit belongs to the category of \(\omega N\)-limits). This can also be related to the fact that our approximation holds neither in case \(M(1-d/b)\rightarrow \infty \) (typically associated with the \(N\omega \)-limit) nor for \(M(1-d/b)\rightarrow 0\) (associated with the \(\omega N\)-limit).

Moreover, this sort of breakdown of the \(\frac{1}{3}\)-rule under demographic fluctuations has also been observed by Ashcroft et al. (2017). Instead, at least in our case, the fixation behavior simply depends on the stability of the internal fixed point of the deterministic system. For a asymptotically stable fixed point the fixation probability of a single mutant is larger than neutral, while for an unstable fixed point it is smaller. This similarity between the general replicator equation, i.e. the deterministic evolutionary behavior in the fixed population size model and the fixation probability holds for both, our stochastically varying population size model, as well as for the model with constant population size derived in Constable and McKane (2015), see also Fig. 7 in Sect. 4.1. Furthermore, the similarity between the fixation probability and its corresponding replicator dynamics is generalizable to models with multiple player interactions under demographic fluctuations (Czuppon and Gokhale 2018). These observations lead us to our conjecture that the qualitative behavior of the fixation probability under weak selection and varying population sizes, at least in a mechanistic modeling approach, is largely influenced by the deterministic replicator dynamics as opposed to the stochastic birth-death dynamics which led to the \(\frac{1}{3}\)-rule. The main reason for it being the conditions on the fixed point under which ecological and evolutionary processes can be separated.

## 5 Conclusion

The goal of this manuscript is the analysis of the invasion of a mutant strain when introduced into a wild-type population. Assuming a constant population size or deterministically varying population sizes this quantity has already been studied extensively. Here, we extend the analysis to systems including stochastic demographic fluctuations. Dealing with a competitive Lotka–Volterra model we are able to approximate the fixation probability under the weak selection assumption, i.e. the interaction rates between individuals just differ slightly. This mechanistic implementation of the model allows for a straightforward interpretation of the selection effects in terms of competition rates, much in the spirit of Doebeli et al. (2017).

In order to obtain an expression for the fixation probability, we approximate the model by stochastic diffusions and apply tools from stochastic differential equation theory. We observe that the evolutionary success of a mutant mainly depends on its interspecific competition rate *b*. This is due to the resident type being more frequent initially yielding a higher probability for a mutant individual to interact with a resident type. This implies that a larger payoff for the mutant interacting with the wild-type ensures an enhanced fixation probability. This can be seen explicitly by the factor \((1-\frac{d}{b})\) occurring in Eq. (9) which is the only term in the formula that can switch signs given an initially rare mutant, i.e. \(p<p^*\).

Still, the values *a* and *c* play a role in the overall evolutionary picture. While lowering *c* increases \(p^*\) and thus the region where the invading type has a selective advantage, the parameter *a* has an impact on the overall population size after fixation of the mutant strain. This might end in a decrease of the total number of individuals if \(a<d\), even though the mutant has a selective advantage over the wild-type due to \(b>d\).

Furthermore, we studied the fixation probability of exactly one mutant in the initial population. Non-surprisingly and as already observed in systems with deterministic population growth/decrease, cf. Kimura and Ohta (1974), we see that the fixation probability monotonically decreases for increasing initial population sizes. Additionally, we find that in our system due to the varying population size the famous \(\frac{1}{3}\)-rule for fixed population sizes, see Nowak et al. (2004), does not hold anymore. However, we can relate the deterministic internal equilibrium to the intersection of the neutral fixation probability and its counterpart including selection, i.e. \(\varphi _{neu}(p^*,z)=\varphi (p^*,z)=p^*\). This has already been observed in Constable and McKane (2015) and Lambert (2006) but not been stated explicitly

We also explored the parameter range under which our modeling assumptions are valid. As it turns out the regime is different when compared to results from population genetics. There, usually \(Ms\gg 1\) is assumed while in our framework the approximation holds for \(Ms<1\). This is due to the mechanistic implementation of the population dynamics while in population genetics the former condition is usually assumed in order to derive a reasonable stochastic diffusion limit.

The evolutionary result of populations under stochastically fluctuating population sizes has been studied in various scenarios over the last few years (Melbinger et al. 2010; Chotibut and Nelson 2015; Constable et al. 2016; Chotibut and Nelson 2017). The stochasticity of the system as opposed to a deterministic modeling approach allows for different asymptotic behaviors and especially can reverse the deterministic behavior. This triggers the question for calculating fixation probabilities. We added new insight on the impact of the competition parameters on the fixation probability. Furthermore, we showcase a method from stochastic diffusion theory and developed in Lambert (2006) for approximating this quantity at least in the weak selection limit. Even though it is limited to the study of two interacting species, it is adaptable to many other models (and not only Lotka–Volterra-type systems) which include stochastic variation on the population size level.

## Notes

### Acknowledgements

Open access funding provided by Max Planck Society. The authors thank Hildegard Uecker and George Constable for helpful discussion about the models described in Sects. 4.1.2 and 4.1.3. Furthermore, we thank Joachim Hermisson for carefully reading the manuscript. His comments significantly improved the discussion, especially in Sect. 4.1. Furthermore, we appreciate the effort of an anonymous referee, who helped to improve the discussion in Sect. 4.2. Both authors acknowledge generous funding from the Max Planck Society.

## References

- Altrock PM, Liu LL, Michor F (2015) The mathematics of cancer: integrating quantitative models. Nat Rev Cancer 15(12):730–745CrossRefGoogle Scholar
- Ashcroft P, Smith C, Garrod M, Galla T (2017) Effects of population growth on the success of invading mutants. J Theor Biol 420:232–240MathSciNetCrossRefGoogle Scholar
- Baar M, Bovier A, Champagnat N (2017) From stochastic, individual-based models to the canonical equation of adaptive dynamics in one step. Ann Appl Probab 27(2):1093–1170MathSciNetCrossRefGoogle Scholar
- Champagnat N, Lambert A (2007) Evolution of discrete populations and the canonical diffusion of adaptive dynamics. Ann Appl Probab 17(1):102–155MathSciNetCrossRefGoogle Scholar
- Chotibut T, Nelson DR (2015) Evolutionary dynamics with fluctuating population sizes and strong mutualism. Phys Rev E 92(2):022718MathSciNetCrossRefGoogle Scholar
- Chotibut T, Nelson DR (2017) Population genetics with fluctuating population sizes. J Stat Phys 167(3):777–791MathSciNetCrossRefGoogle Scholar
- Constable G, McKane A (2015) Models of genetic drift as limiting forms of the Lotka–Volterra competition model. Phys Rev Lett 114:038101CrossRefGoogle Scholar
- Constable GWA, McKane AJ (2017) Mapping of the stochastic Lotka–Volterra model to models of population genetics and game theory. Phys Rev E 96(2):022416MathSciNetCrossRefGoogle Scholar
- Constable GWA, McKane AJ (2018) Exploiting fast-variables to understand population dynamics and evolution. J Stat Phys 172(1):3–43MathSciNetCrossRefGoogle Scholar
- Constable GWA, Rogers T, McKane AJ, Tarnita CE (2016) Demographic noise can reverse the direction of deterministic selection. Proc Natl Acad Sci 113(32):E4745–E4754CrossRefGoogle Scholar
- Cremer J, Melbinger A, Frey E (2011) Evolutionary and population dynamics: a coupled approach. Phys Rev E 84:051921CrossRefGoogle Scholar
- Czuppon P, Gokhale CS (2018) Disentangling eco-evolutionary effects on trait fixation. bioRxiv. https://doi.org/10.1101/259069
- Doebeli M, Ispolatov Y, Simon B (2017) Towards a mechanistic foundation of evolutionary theory. eLife 6:e23804CrossRefGoogle Scholar
- Ethier SN, Kurtz TG (1986) Markov processes: characterization and convergence. Wiley series in probability and mathematical statistics. Wiley, New YorkCrossRefGoogle Scholar
- Ewens WJ (1967) The probability of survival of a new mutant in a fluctuating enviroment. Heredity 22:438–443CrossRefGoogle Scholar
- Ewens WJ (2004) Mathematical population genetics. I. Theoretical introduction. Springer, New YorkCrossRefGoogle Scholar
- Gabel A, Meerson B, Redner S (2013) Survival of the scarcer. Phys Rev E 87:010101CrossRefGoogle Scholar
- Gardiner CW (2004) Handbook of stochastic methods, 3rd edn. Springer, NYCrossRefGoogle Scholar
- Gokhale CS, Traulsen A (2010) Evolutionary games in the multiverse. Proc Natl Acad Sci USA 107:5500–5504MathSciNetCrossRefGoogle Scholar
- Haccou P, Jagers P, Vatutin VA (2005) Branching processes: variation, growth, and extinction of populations, vol 5. Cambridge University Press, CambridgeCrossRefGoogle Scholar
- Hofbauer J, Sigmund K (1998) Evolutionary games and population dynamics. Cambridge University Press, CambridgeCrossRefGoogle Scholar
- Huang W, Hauert C, Traulsen A (2015) Stochastic game dynamics under demographic fluctuations. Proc Natl Acad Sci USA 112:9064–9069CrossRefGoogle Scholar
- Jones E, Oliphant T, Peterson P et al. (2001) SciPy: open source scientific tools for python. http://www.scipy.org/
- Kallenberg O (2002) Foundations of modern probability. Springer, BerlinCrossRefGoogle Scholar
- Kimura M, Ohta T (1974) Probability of gene fixation in an expanding finite population. Proc Natl Acad Sci USA 71:3377–3379CrossRefGoogle Scholar
- Kurokawa S, Ihara Y (2009) Emergence of cooperation in public goods games. Proc R Soc B 276:1379–1384CrossRefGoogle Scholar
- Lambert A (2005) The branching process with logistic growth. Ann Appl Probab 15(2):1506–1535MathSciNetCrossRefGoogle Scholar
- Lambert A (2006) Probability of fixation under weak selection: a branching process unifying approach. Theor Popul Biol 69:419–441CrossRefGoogle Scholar
- Lessard S (2011) On the robustness of the extension of the one-third law of evolution to the multi-player game. Dyn Games Appl 1:408–418MathSciNetCrossRefGoogle Scholar
- Lessard S, Ladret V (2007) The probability of fixation of a single mutant in an exchangeable selection model. J Math Biol 54:721–744MathSciNetCrossRefGoogle Scholar
- McAvoy A, Fraiman N, Hauert C, Wakeley J, Nowak MA (2018) Public goods games in populations with fluctuating size. Theor Popul Biol 121:72–84CrossRefGoogle Scholar
- Melbinger A, Cremer J, Frey E (2010) Evolutionary game theory in growing populations. Phys Rev Lett 105(17):178101CrossRefGoogle Scholar
- Nowak MA (2006) Evolutionary dynamics. Harvard University Press, CambridgezbMATHGoogle Scholar
- Nowak MA, Sasaki A, Taylor C, Fudenberg D (2004) Emergence of cooperation and evolutionary stability in finite populations. Nature 428:646–650CrossRefGoogle Scholar
- Otto SP, Whitlock MC (1997) The probability of fixation in populations of changing size. Genetics 146:723–733Google Scholar
- Park HJ, Traulsen A (2017) Extinction dynamics from meta-stable coexistences. Phys Rev E 96:042412CrossRefGoogle Scholar
- Parsons TL, Quince C (2007) Fixation in haploid populations exhibiting density dependence I: the non-neutral case. Theor Popul Biol 72:121–135CrossRefGoogle Scholar
- Parsons TL, Quince C (2007) Fixation in haploid populations exhibiting density dependence II: the quasi-neutral case. Theor Popul Biol 72:468–479CrossRefGoogle Scholar
- Parsons TL, Quince C, Plotkin JB (2010) Some consequences of demographic stochasticity in population genetics. Genetics 185:1345–1354CrossRefGoogle Scholar
- Patwa Z, Wahl LM (2008) The fixation probability of beneficial mutations. J R Soc Interface 5:1279–1289CrossRefGoogle Scholar
- Pfaffelhuber P, Wakolbinger A (2018) Hitting probabilities and expected hitting times under a weak drift: on the 1/3-rule and beyond. arXiv:1801.01584
- Roberts AJ (1989) Appropriate initial conditions for asymptotic descriptions of the long term evolution of dynamical systems. J Aust Math Soc Ser B Appl Math 31(1):48–75MathSciNetCrossRefGoogle Scholar
- Sample C, Allen B (2017) The limits of weak selection and large population size in evolutionary game theory. J Math Biol 75(5):1285–1317MathSciNetCrossRefGoogle Scholar
- Sandholm WH (2010) Population games and evolutionary dynamics. MIT Press, CambridgezbMATHGoogle Scholar
- Tao Y, Cressman R (2007) Stochastic fluctuations through intrinsic noise in evolutionary game dynamics. Bull Math Biol 69:1377–1399MathSciNetCrossRefGoogle Scholar
- Traulsen A, Pacheco JM, Imhof LA (2006) Stochasticity and evolutionary stability. Phys Rev E 74:021905MathSciNetCrossRefGoogle Scholar
- Traulsen A, Claussen JC, Hauert C (2012) Stochastic differential equations for evolutionary dynamics with demographic noise and mutations. Phys Rev E 85:041901CrossRefGoogle Scholar
- Uecker H, Hermisson J (2011) On the fixation process of a beneficial mutation in a variable environment. Genetics 188(4):915–930CrossRefGoogle Scholar
- van Kampen NG (1997) Stochastic processes in physics and chemistry, 2nd edn. Elsevier, AmsterdamGoogle Scholar
- Wahl LM, Gerrish PJ (2001) The probability that beneficial mutations are lost in populations with periodic bottlenecks. Evolution 55(12):2606–2610CrossRefGoogle Scholar
- Waxman D (2011) A unified treatment of the probability of fixation when population size and the strength of selection change over time. Genetics 188(4):907–913CrossRefGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.