## Abstract

We consider the evolution of populations under the joint action of mutation and differential reproduction, or selection. The population is modelled as a finite-type Markov branching process in continuous time, and the associated genealogical tree is viewed both in the forward and the backward direction of time. The stationary type distribution of the reversed process, the so-called ancestral distribution, turns out as a key for the study of mutation–selection balance. This balance can be expressed in the form of a variational principle that quantifies the respective roles of reproduction and mutation for any possible type distribution. It shows that the mean growth rate of the population results from a competition for a maximal long-term growth rate, as given by the difference between the current mean reproduction rate, and an asymptotic decay rate related to the mutation process; this tradeoff is won by the ancestral distribution. We then focus on the case when the type is determined by a sequence of letters (like nucleotides or matches/mismatches relative to a reference sequence), and we ask how much of the above competition can still be seen by observing only the letter composition (as given by the frequencies of the various letters within the sequence). If mutation and reproduction rates can be approximated in a smooth way, the fitness of letter compositions resulting from the interplay of reproduction and mutation is determined in the limit as the number of sequence sites tends to infinity. Our main application is the quasispecies model of sequence evolution with mutation coupled to reproduction but independent across sites, and a fitness function that is invariant under permutation of sites. In this model, the fitness of letter compositions is worked out explicitly. In certain cases, their competition leads to a phase transition.

This is a preview of subscription content, access via your institution.

## References

Akin E. (1979): The Geometry of Population Genetics. Springer, Berlin Heidelberg New York

Athreya K.B., Ney P.E. (1972): Branching Processes. Springer, Berlin Heidelberg New York

Baake E., Baake M., Bovier A., Klein M. (2005): An asymptotic maximum principle for essentially nonlinear evolution models. J. Math. Biol. 50, 83–114; ArXiv:q-bio.PE/0311020

Baake E., Wagner H. (2001): Mutation–selection models solved exactly with methods from statistical mechanics. Genet. Res. 78, 93–117

Bürger R. (2000): The Mathematical Theory of Selection, Recombination, and Mutation. Wiley, Chichester

Crow J. F., Kimura M. (1970): An Introduction to Population Genetics Theory. Harper & Row, New York

Dembo A., Zeitouni O. (1998): Large Deviations: Techniques and Applications. Springer, Berlin Heidelberg New York

Edwards A. (2002): The fundamental theorem of natural selection. Theor. Pop. Biol. 61, 335–337

Eigen M. (1971): Selforganization of matter and the evolution of biological macromolecules. Naturwiss. 58, 465–523

Eigen M., McCaskill J., Schuster P. (1989): The molecular quasi-species. Adv. Chem. Phys. 75, 149–263

Ewens W. (2004): Mathematical Population Genetics, 2nd edn. Springer, Berlin Heidelberg New York

Ewens W., Grant G. (2005): Statistical Methods in Bioinformatics, 2nd edn. Springer, Berlin Heidelberg New York

Garske, T.: Error thresholds in a mutation-selection model with Hopfield-type fitness. Bull. Math. Biol. (in press) arXiv:q-bio.PE/0505056. DOI 10.1007/s11538-006-9072-1

Garske T., Grimm U. (2004): A maximum principle for the mutation-selection equilibrium of nucleotide sequences. Bull. Math. Biol. 66, 397–421; arXiv:physics/0303053v2

Georgii H.-O., Baake E. (2003): Multitype branching processes: the ancestral types of typical individuals. Adv. Appl. Prob. 35, 1090–1110;arXiv:math.PR/0302049

Gerland U., Hwa T. (2002): On the selection and evolution of regulatory DNA motifs. J. Mol. Evol. 55, 386–400

Hein J., Schierup M., Wiuf C. (2005): Gene genealogies, variation and evolution : a primer in coalescent theory. Oxford University Press, Oxford

Hermisson J., Redner O., Wagner H., Baake E. (2002): Mutation–selection balance: Ancestry, load, and maximum principle. Theor. Pop. Biol. 62, 9–46; arXiv:cond-mat/0202432

Hofbauer J. (1985): The selection mutation equation. J. Math. Biol. 23, 41–53

den Hollander F. (2000). Large Deviations. AMS, Providence, RI

Jagers P., Nerman O. (1984): The stable doubly infinite pedigree process of supercritical branching populations. Z für Wahrscheinlichkeitstheorie und verwandte Gebiete 65, 445–460

Jagers P. (1989): General branching processes as Markov fields. Stoch. Proc. Appl. 32, 183–242

Jagers P. (1992): Stabilities and instabilities in population dynamics. J. Appl. Prob. 29, 770–780

Kamp C. (2003): A quasispecies approach to viral evolution in the context of an adaptive immune system. Microbes Infect. 5, 1397–1405

Karlin K.S., Taylor H.M. (1975): A first course in stochastic processes, 2nd edn. Academic Press, San Diego

Kemeny J.G., Snell J.L. (1981): Finite Markov Chains. Springer, Berlin Heidelberg New York

Kesten H., Stigum B.P. (1966): A limit theorem for multidimensional Galton-Watson processes. Ann. Math. Statist. 37, 1211–1233

Kurtz T., Lyons R., Pemantle R., Peres Y. (1997): A conceptual proof of the Kesten-Stigum theorem for multi-type branching processes. In: Athreya K.B., Jagers P (eds) Classical and Modern Branching Processes. pp. 181–185, Springer, New York

Lindvall T. (1992): Lectures on the Coupling Method. Wiley, New York

Lyons R., Pemantle R., Peres Y. (1995): Conceptual proofs of LlogL criteria for mean behaviour of branching processes. Ann. Prob. 23, 1125–1138

Mitrinovic D. (1970): Analytic inequalities. Springer, Berlin Heidelberg New York

Rockafellar R.T. (1970): Convex Analysis. Princeton University Press, Princeton

Stannat W. (2004): On the convergence of genetic algorithms—a variational approach. Probab. Theory Relat. Fields 129, 113–132

## Author information

### Authors and Affiliations

### Corresponding author

## Rights and permissions

## About this article

### Cite this article

Baake, E., Georgii, HO. Mutation, selection, and ancestry in branching models: a variational approach.
*J. Math. Biol.* **54, **257–303 (2007). https://doi.org/10.1007/s00285-006-0039-5

Received:

Revised:

Published:

Issue Date:

DOI: https://doi.org/10.1007/s00285-006-0039-5

### Keywords

- Mutation–selection models
- Branching processes
- Quasispecies model
- Variational analysis
- Large deviations

### Mathematics Subject Classification (2000)

- 92D15
- 60J80
- 60F10
- 90C46
- 15A18