Who Watches the Watchmen? An Appraisal of Benchmarks for Multiple Sequence Alignment

  • Stefano Iantorno
  • Kevin Gori
  • Nick Goldman
  • Manuel Gil
  • Christophe Dessimoz
Part of the Methods in Molecular Biology book series (MIMB, volume 1079)


Multiple sequence alignment (MSA) is a fundamental and ubiquitous technique in bioinformatics used to infer related residues among biological sequences. Thus alignment accuracy is crucial to a vast range of analyses, often in ways difficult to assess in those analyses. To compare the performance of different aligners and help detect systematic errors in alignments, a number of benchmarking strategies have been pursued. Here we present an overview of the main strategies—based on simulation, consistency, protein structure, and phylogeny—and discuss their different advantages and associated risks. We outline a set of desirable characteristics for effective benchmarking, and evaluate each strategy in light of them. We conclude that there is currently no universally applicable means of benchmarking MSA, and that developers and users of alignment tools should base their choice of benchmark depending on the context of application—with a keen awareness of the assumptions underlying each benchmarking strategy.

Key words

Multiple sequence alignment Benchmarking Phylogenetic Protein structure Sequence evolution Consistency Homology 



The authors thank Julie Thompson for helpful feedback on the manuscript. CD is supported by SNSF advanced researcher fellowship #136461. This article started as assignment for the graduate course “Reviews in Computational Biology” at the Cambridge Computational Biology Institute, University of Cambridge.


  1. 1.
    Kemena C, Notredame C (2009) Upcoming challenges for multiple sequence alignment methods in the high-throughput era. Bioinformatics 25(19):2455–2465PubMedCrossRefGoogle Scholar
  2. 2.
    Aniba MR, Poch O, Thompson JD (2010) Issues in bioinformatics benchmarking: the case study of multiple sequence alignment. Nucleic Acids Res 38(21):7353–7363PubMedCrossRefGoogle Scholar
  3. 3.
    Edgar RC (2010) Quality measures for protein alignment benchmarks. Nucleic Acids Res 38(7):2145–2153PubMedCrossRefGoogle Scholar
  4. 4.
    Thompson JD, Linard B, Lecompte O, Poch O (2011) A comprehensive benchmark study of multiple sequence alignment methods: current challenges and future perspectives. PLoS One 6(3):e18093PubMedCrossRefGoogle Scholar
  5. 5.
    Löytynoja A (2012) Alignment methods: strategies, challenges, benchmarking, and comparative overview. Methods Mol Biol 855:203–235PubMedCrossRefGoogle Scholar
  6. 6.
    Thompson JD, Higgins DG, Gibson TJ (1994) CLUSTAL W: improving the sensitivity of progressive multiple sequence alignment through sequence weighting, position-specific gap penalties and weight matrix choice. Nucleic Acids Res 22(22):4673–4680PubMedCrossRefGoogle Scholar
  7. 7.
    Morrison DA (2009) Why would phylogeneticists ignore computerized sequence alignment? Syst Biol 58(1):150–158PubMedCrossRefGoogle Scholar
  8. 8.
    Mardis ER (2008) The impact of next-generation sequencing technology on genetics. Trends Genet 24(3):133–141. doi: 10.1016/j.tig.2007.12.007 PubMedCrossRefGoogle Scholar
  9. 9.
    Anisimova M, Cannarozzi G, Liberles D (2010) Finding the balance between the mathematical and biological optima in multiple sequence alignment. Trends Evol Biol 2(1):e7CrossRefGoogle Scholar
  10. 10.
    Stebbings LA, Mizuguchi K (2004) HOMSTRAD: recent developments of the Homologous Protein Structure Alignment Database. Nucleic Acids Res 32(Database issue):D203–D207PubMedCrossRefGoogle Scholar
  11. 11.
    Thompson JD, Koehl P, Ripp R, Poch O (2005) BAliBASE 3.0: latest developments of the multiple sequence alignment benchmark. Proteins 61:127–136PubMedCrossRefGoogle Scholar
  12. 12.
    Stoye J, Evers D, Meyer F (1998) Rose: generating sequence families. Bioinformatics 14(2):157–163PubMedCrossRefGoogle Scholar
  13. 13.
    Cartwright RA (2005) DNA assembly with gaps (Dawg): simulating sequence evolution. Bioinformatics 21(Suppl 3):iii31–iii38PubMedCrossRefGoogle Scholar
  14. 14.
    Hall BG (2008) Simulating DNA coding sequence evolution with EvolveAGene 3. Mol Biol Evol 25(4):688–695PubMedCrossRefGoogle Scholar
  15. 15.
    Fletcher W, Yang Z (2009) INDELible: a flexible simulator of biological sequence evolution. Mol Biol Evol 26(8):1879–1888PubMedCrossRefGoogle Scholar
  16. 16.
    Sipos B, Massingham T, Jordan GE, Goldman N (2011) PhyloSim – Monte Carlo simulation of sequence evolution in the R statistical computing environment. BMC Bioinformatics 12(1):104PubMedCrossRefGoogle Scholar
  17. 17.
    Koestler T, Av H, Ebersberger I (2012) REvolver: modeling sequence evolution under domain constraints. Mol Biol Evol 29(9):2133–2145PubMedCrossRefGoogle Scholar
  18. 18.
    Dalquen DA, Anisimova M, Gonnet GH, Dessimoz C (2012) ALF-a simulation framework for genome evolution. Mol Biol Evol 29(4):1115–1123PubMedCrossRefGoogle Scholar
  19. 19.
    Thompson JD, Plewniak F, Poch O (1999) A comprehensive comparison of multiple sequence alignment programs. Nucleic Acids Res 27(13):2682–2690, gkc432 [pii]PubMedCrossRefGoogle Scholar
  20. 20.
    Blackburne BP, Whelan S (2012) Measuring the distance between multiple sequence alignments. Bioinformatics 28(4):495–502. doi: 10.1093/bioinformatics/btr701 PubMedCrossRefGoogle Scholar
  21. 21.
    Löytynoja A, Goldman N (2008) Phylogeny-aware gap placement prevents errors in sequence alignment and evolutionary analysis. Science 320(5883):1632–1635. doi: 10.1126/science.1158395 PubMedCrossRefGoogle Scholar
  22. 22.
    Golubchik T, Wise MJ, Easteal S, Jermiin LS (2007) Mind the gaps: evidence of bias in estimates of multiple sequence alignments. Mol Biol Evol 24(11):2433–2442PubMedCrossRefGoogle Scholar
  23. 23.
    Huelsenbeck JP (1995) Performance of phylogenetic methods in simulation. Syst Biol 44(1):17–48Google Scholar
  24. 24.
    Do CB, Mahabhashyam MS, Brudno M, Batzoglou S (2005) ProbCons: probabilistic consistency-based multiple sequence alignment. Genome Res 15(2):330–340. doi: 10.1101/gr.2821705 PubMedCrossRefGoogle Scholar
  25. 25.
    Notredame C, Higgins DG, Heringa J (2000) T-Coffee: a novel method for fast and accurate multiple sequence alignment. J Mol Biol 302(1):205–217. doi: 10.1006/jmbi.2000.4042 PubMedCrossRefGoogle Scholar
  26. 26.
    Lassmann T, Sonnhammer ELL (2005) Automatic assessment of alignment quality. Nucleic Acids Res 33(22):7120–7128PubMedCrossRefGoogle Scholar
  27. 27.
    Landan G, Graur D (2007) Heads or tails: a simple reliability check for multiple sequence alignments. Mol Biol Evol 24(6):1380–1383PubMedCrossRefGoogle Scholar
  28. 28.
    Hall BG (2008) How well does the HoT score reflect sequence alignment accuracy? Mol Biol Evol 25(8):1576–1580PubMedCrossRefGoogle Scholar
  29. 29.
    Chothia C, Lesk AM (1986) The relation between the divergence of sequence and structure in proteins. EMBO J 5(4):823PubMedGoogle Scholar
  30. 30.
    Mizuguchi K, Deane CM, Blundell TL, Overington JP (1998) HOMSTRAD: a database of protein structure alignments for homologous families. Protein Sci 7(11):2469–2471. doi: 10.1002/pro.5560071126 PubMedCrossRefGoogle Scholar
  31. 31.
    Thompson JD, Plewniak F, Poch O (1999) BAliBASE: a benchmark alignment database for the evaluation of multiple alignment programs. Bioinformatics 15(1):87–88, btc017 [pii]PubMedCrossRefGoogle Scholar
  32. 32.
    Van Walle I, Lasters I, Wyns L (2005) SABmark – a benchmark for sequence alignment that covers the entire known fold space. Bioinformatics 21(7):1267–1268. doi: 10.1093/bioinformatics/bth493 PubMedCrossRefGoogle Scholar
  33. 33.
    Edgar RC (2004) MUSCLE: multiple sequence alignment with high accuracy and high throughput. Nucleic Acids Res 32(5):1792–1797. doi: 10.1093/nar/gkh340 PubMedCrossRefGoogle Scholar
  34. 34.
    Gardner P, Wilm A, Washietl S (2005) A benchmark of multiple sequence alignment programs upon structural RNAs. Nucleic Acids Res 33(8):2433–2439PubMedCrossRefGoogle Scholar
  35. 35.
    Kim J, Sinha S (2010) Towards realistic benchmarks for multiple alignments of non-coding sequences. BMC Bioinformatics 11:54PubMedCrossRefGoogle Scholar
  36. 36.
    Mathews DH (2005) Predicting a set of minimal free energy RNA secondary structures common to two sequences. Bioinformatics 21(10):2246–2253. doi: 10.1093/bioinformatics/bti349 PubMedCrossRefGoogle Scholar
  37. 37.
    Havgaard JH, Lyngso RB, Stormo GD, Gorodkin J (2005) Pairwise local structural alignment of RNA sequences with sequence similarity less than 40%. Bioinformatics 21(9):1815–1824. doi: 10.1093/bioinformatics/bti279 PubMedCrossRefGoogle Scholar
  38. 38.
    Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ (1990) Basic local alignment search tool. J Mol Biol 215:403–410PubMedGoogle Scholar
  39. 39.
    Thompson JD, Fdr P, Ripp R, Thierry J-C, Poch O (2001) Towards a reliable objective function for multiple sequence alignments1. J Mol Biol 314(4):937–951. doi: 10.1006/jmbi.2001.5187 PubMedCrossRefGoogle Scholar
  40. 40.
    Raghava GP, Searle SM, Audley PC, Barber JD, Barton GJ (2003) OXBench: a benchmark for evaluation of protein multiple sequence alignment accuracy. BMC Bioinformatics 4:47. doi: 10.1186/1471-2105-4-47 PubMedCrossRefGoogle Scholar
  41. 41.
    Russell RB, Barton GJ (1992) Multiple protein sequence alignment from tertiary structure comparison: assignment of global and residue confidence levels. Proteins 14(2):309–323. doi: 10.1002/prot.340140216 PubMedCrossRefGoogle Scholar
  42. 42.
    Pop M, Salzberg SL (2008) Bioinformatics challenges of new sequencing technology. Trends Genet 24(3):142–149. doi: 10.1016/j.tig.2007.12.006 PubMedCrossRefGoogle Scholar
  43. 43.
    Berger SA, Stamatakis A (2011) Aligning short reads to reference alignments and trees. Bioinformatics 27(15):2068–2075. doi: 10.1093/bioinformatics/btr320 PubMedCrossRefGoogle Scholar
  44. 44.
    Dessimoz C, Gil M (2010) Phylogenetic assessment of alignments reveals neglected tree signal in gaps. Genome Biol 11(4):R37PubMedCrossRefGoogle Scholar
  45. 45.
    Jordan G, Goldman N (2011) The effects of alignment error and alignment filtering on the sitewise detection of positive selection. Mol Biol Evol 29:1125. doi: 10.1093/molbev/msr272 PubMedCrossRefGoogle Scholar
  46. 46.
    Blackshields G, Wallace IM, Larkin M, Higgins DG (2006) Analysis and comparison of benchmarks for multiple sequence alignment. In Silico Biol 6(4):321–339PubMedGoogle Scholar
  47. 47.
    Lassmann T, Sonnhammer EL (2002) Quality assessment of multiple alignment programs. FEBS Lett 529(1):126–130, S0014579302031897 [pii]PubMedCrossRefGoogle Scholar
  48. 48.
    Strope CL, Abel K, Scott SD, Moriyama EN (2009) Biological sequence simulation for testing complex evolutionary hypotheses: indel-Seq-Gen version 2.0. Mol Biol Evol 26(11):2581–2593. doi: 10.1093/molbev/msp174 PubMedCrossRefGoogle Scholar
  49. 49.
    Lassmann T, Sonnhammer EL (2006) Kalign, Kalignvu and Mumsa: web servers for multiple sequence alignment. Nucleic Acids Res 34(Web Server issue):W596–W599. doi: 10.1093/nar/gkl191 PubMedCrossRefGoogle Scholar
  50. 50.
    Kemena C, Taly JF, Kleinjung J, Notredame C (2011) STRIKE: evaluation of protein MSAs using a single 3D structure. Bioinformatics 27(24):3385–3391. doi: 10.1093/bioinformatics/btr587 PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2014
Chapter 4 was created within the capacity of an US governmental employment. US copyright protection does not apply.

Authors and Affiliations

  • Stefano Iantorno
    • 1
    • 2
  • Kevin Gori
    • 3
  • Nick Goldman
    • 3
  • Manuel Gil
    • 4
  • Christophe Dessimoz
    • 3
  1. 1.Wellcome Trust Sanger InstituteCambridgeUK
  2. 2.National Institute of Allergy and Infectious Diseases, National Institutes of HealthBethesdaUSA
  3. 3.EMBL-European Bioinformatics InstituteCambridgeUK
  4. 4.Max F. Perutz Laboratories, Center for Integrative Bioinformatics Vienna, Medical University ViennaUniversity of ViennaViennaAustria

Personalised recommendations