Bulletin of Mathematical Biology

, Volume 70, Issue 4, pp 1115–1139 | Cite as

Mixed-up Trees: the Structure of Phylogenetic Mixtures

  • Frederick A. MatsenEmail author
  • Elchanan Mossel
  • Mike Steel
Original Article


In this paper, we apply new geometric and combinatorial methods to the study of phylogenetic mixtures. The focus of the geometric approach is to describe the geometry of phylogenetic mixture distributions for the two state random cluster model, which is a generalization of the two state symmetric (CFN) model. In particular, we show that the set of mixture distributions forms a convex polytope and we calculate its dimension; corollaries include a simple criterion for when a mixture of branch lengths on the star tree can mimic the site pattern frequency vector of a resolved quartet tree. Furthermore, by computing volumes of polytopes we can clarify how “common” non-identifiable mixtures are under the CFN model. We also present a new combinatorial result which extends any identifiability result for a specific pair of trees of size six to arbitrary pairs of trees. Next we present a positive result showing identifiability of rates-across-sites models. Finally, we answer a question raised in a previous paper concerning “mixed branch repulsion” on trees larger than quartet trees under the CFN model.


Phylogenetics Model identifiability Mixture model Polytope Discrete Fourier analysis 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Allman, E.S., Rhodes, J.A., 2006. The identifiability of tree topology for phylogenetic models, including covarion and mixture models. J. Comput. Biol. 13(5), 1101–1113. CrossRefMathSciNetGoogle Scholar
  2. Bandelt, H.J., Dress, A.W.M., 1992. A canonical decomposition theory for metrics on a finite set. Adv. Math. 92, 47–105. CrossRefMathSciNetzbMATHGoogle Scholar
  3. Felsenstein, J., 2004. Inferring Phylogenies. Sinauer Press, Sunderland. Google Scholar
  4. Gawrilow, E., Joswig, M., 2005. Geometric reasoning with polymake. arXiv:math.CO/0507273.
  5. Grünbaum, B., 2003. Convex Polytopes. Springer, Berlin. Google Scholar
  6. Kaibel, V., Pfetsch, M.E., 2003. Some algorithmic problems in polytope theory. In: Algebra, Geometry, and Software Systems, pp. 23–47. Springer, Berlin. Google Scholar
  7. Matsen, F.A., Steel, M., 2007. Phylogenetic mixtures on a single tree can mimic a tree of another topology. arXiv:0704.2260v1 [q-bio.PE]. Google Scholar
  8. Meacham, C.A., 1983. Theoretical and computational considerations of the compatibility of qualitative taxonomic characters. In: J. Felsenstein (Ed.), Numerical taxonomy, NATO ASI Series, vol. G1, pp. 304–314. Springer, Berlin. Google Scholar
  9. Mossel, E., Steel, M., 2004. A phase transition for a random cluster model on phylogenetic trees. Math. Biosci. 187(4), 189–203. CrossRefMathSciNetzbMATHGoogle Scholar
  10. Ochman, H., Lawrence, J.G., Groisman, E.A., 2000. Lateral gene transfer and the nature of bacterial innovation. Nature 405(6784), 299–304. CrossRefGoogle Scholar
  11. Rokas, A., Williams, B.L., King, N., Carroll, S.B., 2003. Genome-scale approaches to resolving incongruence in molecular phylogenies. Nature 425(6960), 798–804. CrossRefGoogle Scholar
  12. Semple, C., Steel, M., 2003. Phylogenetics. Oxford Lecture Series in Mathematics and its Applications, vol. 24. Oxford University Press, Oxford. zbMATHGoogle Scholar
  13. Simon, C., Nigro, L., Sullivan, J., Holsinger, K., Martin, A., Grapputo, A., Franke, A., McIntosh, C., 1996. Large differences in substitutional pattern and evolutionary rate of 12S ribosomal RNA genes. Mol. Biol. Evol. 13(7), 923–932. Google Scholar
  14. Steel, M.A., Szekely, L.A., Hendy, M.D., 1994. Reconstructing trees when sequence sites evolve at variable rates. J. Comput. Biol. 1(2), 153–163. CrossRefGoogle Scholar
  15. Štefankovič, D., Vigoda, E., 2007. Phylogeny of mixture models: robustness of maximum likelihood and non-identifiable distributions. J. Comput. Biol. 14(2), 156–189. CrossRefMathSciNetGoogle Scholar
  16. Ziegler, G.M., 1994. Lectures on Convex Polytopes. Springer, Berlin. Google Scholar

Copyright information

© Society for Mathematical Biology 2008

Authors and Affiliations

  • Frederick A. Matsen
    • 1
    Email author
  • Elchanan Mossel
    • 2
  • Mike Steel
    • 1
  1. 1.Biomathematics Research CentreUniversity of CanterburyCanterburyNew Zealand
  2. 2.StatisticsUC BerkeleyBerkeleyUSA

Personalised recommendations