Bulletin of Mathematical Biology

, Volume 70, Issue 4, pp 1115–1139

Mixed-up Trees: the Structure of Phylogenetic Mixtures

  • Frederick A. Matsen
  • Elchanan Mossel
  • Mike Steel
Original Article

DOI: 10.1007/s11538-007-9293-y

Cite this article as:
Matsen, F.A., Mossel, E. & Steel, M. Bull. Math. Biol. (2008) 70: 1115. doi:10.1007/s11538-007-9293-y


In this paper, we apply new geometric and combinatorial methods to the study of phylogenetic mixtures. The focus of the geometric approach is to describe the geometry of phylogenetic mixture distributions for the two state random cluster model, which is a generalization of the two state symmetric (CFN) model. In particular, we show that the set of mixture distributions forms a convex polytope and we calculate its dimension; corollaries include a simple criterion for when a mixture of branch lengths on the star tree can mimic the site pattern frequency vector of a resolved quartet tree. Furthermore, by computing volumes of polytopes we can clarify how “common” non-identifiable mixtures are under the CFN model. We also present a new combinatorial result which extends any identifiability result for a specific pair of trees of size six to arbitrary pairs of trees. Next we present a positive result showing identifiability of rates-across-sites models. Finally, we answer a question raised in a previous paper concerning “mixed branch repulsion” on trees larger than quartet trees under the CFN model.


Phylogenetics Model identifiability Mixture model Polytope Discrete Fourier analysis 

Copyright information

© Society for Mathematical Biology 2008

Authors and Affiliations

  • Frederick A. Matsen
    • 1
  • Elchanan Mossel
    • 2
  • Mike Steel
    • 1
  1. 1.Biomathematics Research CentreUniversity of CanterburyCanterburyNew Zealand
  2. 2.StatisticsUC BerkeleyBerkeleyUSA

Personalised recommendations