Skip to main content
SpringerLink
Log in

Clustering Near-Identical Sequences for Fast Homology Search

  • Conference paper
Research in Computational Molecular Biology (RECOMB 2006)

Part of the book series: Lecture Notes in Computer Science ((LNBI,volume 3909))

  • 1267 Accesses

Abstract

We present a new approach to managing redundancy in sequence databanks such as GenBank. We store clusters of near-identical sequences as a representative union-sequence and a set of corresponding edits to that sequence. During search, the query is compared to only the union-sequences representing each cluster; cluster members are then only reconstructed and aligned if the union-sequence achieves a sufficiently high score. Using this approach in BLAST results in a 27% reduction is collection size and a corresponding 22% decrease in search time with no significant change in accuracy. We also describe our method for clustering that uses fingerprinting, an approach that has been successfully applied to collections of text and web documents in Information Retrieval. Our clustering approach is ten times faster on the GenBank nonredundant protein database than the fastest existing approach, CD-HIT. We have integrated our approach into FSA-BLAST, our new Open Source version of BLAST, available from http://www.fsa-blast.org/. As a result, FSA-BLAST is twice as fast as NCBI-BLAST with no significant change in accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Altschul, S., Madden, T., Schaffer, A., Zhang, J., Zhang, Z., Miller, W., Lipman, D.: Gapped BLAST and PSI–BLAST: A new generation of protein database search programs. Nucleic Acids Research 25, 3389–3402 (1997)

    Article  Google Scholar 

  2. Li, W., Jaroszewski, L., Godzik, A.: Sequence clustering strategies improve remote homology recognitions while reducing search times. Protein Engineering 15, 643–649 (2002)

    Article  Google Scholar 

  3. Park, J., Holm, L., Heger, A., Chothia, C.: RSDB: representative sequence databases have high information content. Bioinformatics 16, 458–464 (2000)

    Article  Google Scholar 

  4. Altschul, S., Gish, W., Miller, W., Myers, E., Lipman, D.: Basic local alignment search tool. Journal of Molecular Biology 215, 403–410 (1990)

    Google Scholar 

  5. Pearson, W., Lipman, D.: Improved tools for biological sequence comparison. Proceedings of the National Academy of Sciences USA 85, 2444–2448 (1988)

    Article  Google Scholar 

  6. Pearson, W., Lipman, D.: Rapid and sensitive protein similarity searches. Science 227, 1435–1441 (1985)

    Article  Google Scholar 

  7. Li, W., Jaroszewski, L., Godzik, A.: Tolerating some redundancy significantly speeds up clustering of large protein databases. Bioinformatics 18, 77–82 (2001)

    Article  MATH  Google Scholar 

  8. Smith, T., Waterman, M.: Identification of common molecular subsequences. Journal of Molecular Biology 147, 195–197 (1981)

    Article  Google Scholar 

  9. Burke, J., Davison, D., Hide, W.: d2_cluster: A validated method for clustering EST and full-length DNA sequences. Genome Research 9, 1135–1142 (1999)

    Article  Google Scholar 

  10. Bleasby, A.J., Wootton, J.C.: Construction of validated, non-redundant composite protein sequence databases. Protein Engineering 3, 153–159 (1990)

    Article  Google Scholar 

  11. Kallberg, Y., Persson, B.: KIND — a non-redundant protein database. Bioinformatics 15, 260–261 (1999)

    Article  Google Scholar 

  12. Grillo, G., Attimonelli, M., Liuni, S., Pesole, G.: CLEANUP: a fast computer program for removing redundancies from nucleotide sequence databases. Computer Applications in the Biosciences 12, 1–8 (1996)

    Google Scholar 

  13. Holm, L., Sander, C.: Removing near-neighbour redundancy from large protein sequence collections. Bioinformatics 14, 423–429 (1998)

    Article  Google Scholar 

  14. Li, W., Jaroszewski, L., Godzik, A.: Clustering of highly homologous sequences to reduce the size of large protein databases. Bioinformatics 17, 282–283 (2001)

    Article  Google Scholar 

  15. Malde, K., Coward, E., Jonassen, I.: Fast sequence clustering using a suffix array algorithm. Bioinformatics 19, 1221–1226 (2003)

    Article  Google Scholar 

  16. Gracy, J., Argos, P.: Automated protein sequence database classification. i. integration of compositional similarity search, local similarity search, and multiple sequence alignment. Bioinformatics 14, 164–173 (1998)

    Article  Google Scholar 

  17. Gusfield, D.: Algorithms on Strings, Trees, and Sequences. Cambridge University Press, Cambridge (1997)

    Book  MATH  Google Scholar 

  18. Manber, U., Myers, G.: Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing 22, 935–948 (1993)

    Article  MATH  MathSciNet  Google Scholar 

  19. Cheung, C.F., Yu, J.X., Lu, H.: Constructing suffix tree for gigabyte sequences with megabyte memory. IEEE Transactions on Knowledge and Data Engineering 17, 90–105 (2005)

    Article  Google Scholar 

  20. Lee, C., Grasso, C., Sharlow, M.F.: Multiple sequence alignment using partial order graphs. Bioinformatics 18, 452–464 (2002)

    Article  Google Scholar 

  21. Bernstein, Y., Cameron, M.: Fast discovery of similar sequences in large genomic collections. In: Proc. European Conference on Information Retrieval (to appear, 2006)

    Google Scholar 

  22. Manber, U.: Finding similar files in a large file system. In: Proceedings of the USENIX Winter 1994 Technical Conference, San Fransisco, CA, USA, pp. 1–10 (1994)

    Google Scholar 

  23. Heintze, N.: Scalable document fingerprinting. In: 1996 USENIX Workshop on Electronic Commerce, Oakland, California, USA, pp. 191–200 (1996)

    Google Scholar 

  24. Brin, S., Davis, J., García-Molina, H.: Copy detection mechanisms for digital documents. In: Carey, M., Schneider, D. (eds.) Proceedings of the ACM SIGMOD Annual Conference, San Jose, California, United States, pp. 398–409. ACM Press, New York (1995)

    Google Scholar 

  25. Broder, A.Z., Glassman, S.C., Manasse, M.S., Zweig, G.: Syntactic clustering of the web. Computer Networks and ISDN Systems 29, 1157–1166 (1997)

    Article  Google Scholar 

  26. Bernstein, Y., Zobel, J.: A scalable system for identifying co-derivative documents. In: Apostolico, A., Melucci, M. (eds.) SPIRE 2004. LNCS, vol. 3246, pp. 55–67. Springer, Heidelberg (2004)

    Chapter  Google Scholar 

  27. Bernstein, Y., Zobel, J.: Redundant documents and search effectiveness. In: CIKM 2005: Proceedings of the 14th ACM international conference on Information and knowledge management, pp. 736–743. ACM Press, New York (2005)

    Chapter  Google Scholar 

  28. Johnson, S.: Hierarchical clustering schemes. Psychometrika 32, 241–254 (1967)

    Article  Google Scholar 

  29. Robinson, A., Robinson, L.: Distribution of glutamine and asparagine residues and their near neighbors in peptides and proteins. Proceedings of the National Academy of Sciences USA 88, 8880–8884 (1991)

    Article  Google Scholar 

  30. Henikoff, S., Henikoff, J.: Amino acid substitution matrices from protein blocks. Proceedings of the National Academy of Sciences USA 89, 10915–10919 (1992)

    Article  Google Scholar 

  31. Taylor, W.: The classification of amino-acid conservation. Journal of Theoretical Biology 119, 205–218 (1986)

    Article  Google Scholar 

  32. Murzin, A., Brenner, S., Hubbard, T., Chothia, C.: SCOP: a structural classification of proteins database for the investigation of sequences and structures. Journal of Molecular Biology 247, 536–540 (1995)

    Google Scholar 

  33. Andreeva, A., Howorth, D., Brenner, S., Hubbard, T., Chothia, C., Murzin, A.: SCOP database in 2004: refinements integrate structure and sequence family data. Nucleic Acids Research 32, D226–D229 (2004)

    Google Scholar 

  34. Brenner, S., Chothia, C., Hubbard, T.: Assessing sequence comparison methods with reliable structurally identified distant evolutionary relationships. Proceedings of the National Academy of Sciences USA 95, 6073–6078 (1998)

    Article  Google Scholar 

  35. Park, J., Karplus, K., Barrett, C., Hughey, R., Haussler, D., Hubbard, T., Chothia, C.: Sequence comparisons using multiple sequences detect three times as many remote homologues as pairwise methods. Journal of Molecular Biology 284, 1201–1210 (1998)

    Article  Google Scholar 

  36. Chandonia, J., Hon, G., Walker, N., Conte, L.L., Koehl, P., Levitt, M., Brenner, S.: The ASTRAL compendium in 2004. Nucleic Acids Research 32, D189–D192 (2004)

    Google Scholar 

  37. Gribskov, M., Robinson, N.: Use of receiver operating characteristic (ROC) analysis to evaluate sequence matching. Computers & Chemistry 20, 25–33 (1996)

    Article  Google Scholar 

  38. Karlin, S., Altschul, S.: Methods for assessing the statistical significance of molecular sequence features by using general scoring schemes. Proceedings of the National Academy of Sciences USA 87, 2264–2268 (1990)

    Article  MATH  Google Scholar 

  39. Cameron, M., Williams, H.E., Cannane, A.: Improved gapped alignment in BLAST. IEEE Transactions on Computational Biology and Bioinformatics 1, 116–129 (2004)

    Article  Google Scholar 

  40. Cameron, M., Williams, H.E., Cannane, A.: A deterministic finite automaton for faster protein hit detection in BLAST. Journal of Computational Biology (to appear, 2005)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Information Technology, RMIT University, GPO Box 2476V, Melbourne, 3001, Australia

    Michael Cameron & Yaniv Bernstein

  2. Microsoft Corporation, One Microsoft Way, Redmond, Washington, 98052, USA

    Hugh E. Williams

Authors
  1. Michael Cameron
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yaniv Bernstein
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hugh E. Williams
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Georgia Institute of Technology and Università di Padova,  

    Alberto Apostolico

  2. Topic Chairs, P.O. Box

    Concettina Guerra

  3. Center for Molecular Biology and Computer Sciecne Department, Brown University, 115 Waterman St., 02912, Providence, RI, USA

    Sorin Istrail

  4. University of California, San Diego, USA

    Pavel A. Pevzner

  5. Department of Molecular and Computational Biology, University of Southern California, 1050 Childs Way, 90089-2910, Los Angeles, CA, USA

    Michael Waterman

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Cameron, M., Bernstein, Y., Williams, H.E. (2006). Clustering Near-Identical Sequences for Fast Homology Search. In: Apostolico, A., Guerra, C., Istrail, S., Pevzner, P.A., Waterman, M. (eds) Research in Computational Molecular Biology. RECOMB 2006. Lecture Notes in Computer Science(), vol 3909. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11732990_16

Download citation

  • DOI: https://doi.org/10.1007/11732990_16

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-33295-4

  • Online ISBN: 978-3-540-33296-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation