Clustering Near-Identical Sequences for Fast Homology Search

  • Michael Cameron
  • Yaniv Bernstein
  • Hugh E. Williams
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3909)


We present a new approach to managing redundancy in sequence databanks such as GenBank. We store clusters of near-identical sequences as a representative union-sequence and a set of corresponding edits to that sequence. During search, the query is compared to only the union-sequences representing each cluster; cluster members are then only reconstructed and aligned if the union-sequence achieves a sufficiently high score. Using this approach in BLAST results in a 27% reduction is collection size and a corresponding 22% decrease in search time with no significant change in accuracy. We also describe our method for clustering that uses fingerprinting, an approach that has been successfully applied to collections of text and web documents in Information Retrieval. Our clustering approach is ten times faster on the GenBank nonredundant protein database than the fastest existing approach, CD-HIT. We have integrated our approach into FSA-BLAST, our new Open Source version of BLAST, available from As a result, FSA-BLAST is twice as fast as NCBI-BLAST with no significant change in accuracy.


Search Time Cluster Member Candidate Pair Collection Size Nonredundant Protein Database 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Altschul, S., Madden, T., Schaffer, A., Zhang, J., Zhang, Z., Miller, W., Lipman, D.: Gapped BLAST and PSI–BLAST: A new generation of protein database search programs. Nucleic Acids Research 25, 3389–3402 (1997)CrossRefGoogle Scholar
  2. 2.
    Li, W., Jaroszewski, L., Godzik, A.: Sequence clustering strategies improve remote homology recognitions while reducing search times. Protein Engineering 15, 643–649 (2002)CrossRefGoogle Scholar
  3. 3.
    Park, J., Holm, L., Heger, A., Chothia, C.: RSDB: representative sequence databases have high information content. Bioinformatics 16, 458–464 (2000)CrossRefGoogle Scholar
  4. 4.
    Altschul, S., Gish, W., Miller, W., Myers, E., Lipman, D.: Basic local alignment search tool. Journal of Molecular Biology 215, 403–410 (1990)Google Scholar
  5. 5.
    Pearson, W., Lipman, D.: Improved tools for biological sequence comparison. Proceedings of the National Academy of Sciences USA 85, 2444–2448 (1988)CrossRefGoogle Scholar
  6. 6.
    Pearson, W., Lipman, D.: Rapid and sensitive protein similarity searches. Science 227, 1435–1441 (1985)CrossRefGoogle Scholar
  7. 7.
    Li, W., Jaroszewski, L., Godzik, A.: Tolerating some redundancy significantly speeds up clustering of large protein databases. Bioinformatics 18, 77–82 (2001)zbMATHCrossRefGoogle Scholar
  8. 8.
    Smith, T., Waterman, M.: Identification of common molecular subsequences. Journal of Molecular Biology 147, 195–197 (1981)CrossRefGoogle Scholar
  9. 9.
    Burke, J., Davison, D., Hide, W.: d2_cluster: A validated method for clustering EST and full-length DNA sequences. Genome Research 9, 1135–1142 (1999)CrossRefGoogle Scholar
  10. 10.
    Bleasby, A.J., Wootton, J.C.: Construction of validated, non-redundant composite protein sequence databases. Protein Engineering 3, 153–159 (1990)CrossRefGoogle Scholar
  11. 11.
    Kallberg, Y., Persson, B.: KIND — a non-redundant protein database. Bioinformatics 15, 260–261 (1999)CrossRefGoogle Scholar
  12. 12.
    Grillo, G., Attimonelli, M., Liuni, S., Pesole, G.: CLEANUP: a fast computer program for removing redundancies from nucleotide sequence databases. Computer Applications in the Biosciences 12, 1–8 (1996)Google Scholar
  13. 13.
    Holm, L., Sander, C.: Removing near-neighbour redundancy from large protein sequence collections. Bioinformatics 14, 423–429 (1998)CrossRefGoogle Scholar
  14. 14.
    Li, W., Jaroszewski, L., Godzik, A.: Clustering of highly homologous sequences to reduce the size of large protein databases. Bioinformatics 17, 282–283 (2001)CrossRefGoogle Scholar
  15. 15.
    Malde, K., Coward, E., Jonassen, I.: Fast sequence clustering using a suffix array algorithm. Bioinformatics 19, 1221–1226 (2003)CrossRefGoogle Scholar
  16. 16.
    Gracy, J., Argos, P.: Automated protein sequence database classification. i. integration of compositional similarity search, local similarity search, and multiple sequence alignment. Bioinformatics 14, 164–173 (1998)CrossRefGoogle Scholar
  17. 17.
    Gusfield, D.: Algorithms on Strings, Trees, and Sequences. Cambridge University Press, Cambridge (1997)zbMATHCrossRefGoogle Scholar
  18. 18.
    Manber, U., Myers, G.: Suffix arrays: a new method for on-line string searches. SIAM Journal on Computing 22, 935–948 (1993)zbMATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Cheung, C.F., Yu, J.X., Lu, H.: Constructing suffix tree for gigabyte sequences with megabyte memory. IEEE Transactions on Knowledge and Data Engineering 17, 90–105 (2005)CrossRefGoogle Scholar
  20. 20.
    Lee, C., Grasso, C., Sharlow, M.F.: Multiple sequence alignment using partial order graphs. Bioinformatics 18, 452–464 (2002)CrossRefGoogle Scholar
  21. 21.
    Bernstein, Y., Cameron, M.: Fast discovery of similar sequences in large genomic collections. In: Proc. European Conference on Information Retrieval (to appear, 2006)Google Scholar
  22. 22.
    Manber, U.: Finding similar files in a large file system. In: Proceedings of the USENIX Winter 1994 Technical Conference, San Fransisco, CA, USA, pp. 1–10 (1994)Google Scholar
  23. 23.
    Heintze, N.: Scalable document fingerprinting. In: 1996 USENIX Workshop on Electronic Commerce, Oakland, California, USA, pp. 191–200 (1996)Google Scholar
  24. 24.
    Brin, S., Davis, J., García-Molina, H.: Copy detection mechanisms for digital documents. In: Carey, M., Schneider, D. (eds.) Proceedings of the ACM SIGMOD Annual Conference, San Jose, California, United States, pp. 398–409. ACM Press, New York (1995)Google Scholar
  25. 25.
    Broder, A.Z., Glassman, S.C., Manasse, M.S., Zweig, G.: Syntactic clustering of the web. Computer Networks and ISDN Systems 29, 1157–1166 (1997)CrossRefGoogle Scholar
  26. 26.
    Bernstein, Y., Zobel, J.: A scalable system for identifying co-derivative documents. In: Apostolico, A., Melucci, M. (eds.) SPIRE 2004. LNCS, vol. 3246, pp. 55–67. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  27. 27.
    Bernstein, Y., Zobel, J.: Redundant documents and search effectiveness. In: CIKM 2005: Proceedings of the 14th ACM international conference on Information and knowledge management, pp. 736–743. ACM Press, New York (2005)CrossRefGoogle Scholar
  28. 28.
    Johnson, S.: Hierarchical clustering schemes. Psychometrika 32, 241–254 (1967)CrossRefGoogle Scholar
  29. 29.
    Robinson, A., Robinson, L.: Distribution of glutamine and asparagine residues and their near neighbors in peptides and proteins. Proceedings of the National Academy of Sciences USA 88, 8880–8884 (1991)CrossRefGoogle Scholar
  30. 30.
    Henikoff, S., Henikoff, J.: Amino acid substitution matrices from protein blocks. Proceedings of the National Academy of Sciences USA 89, 10915–10919 (1992)CrossRefGoogle Scholar
  31. 31.
    Taylor, W.: The classification of amino-acid conservation. Journal of Theoretical Biology 119, 205–218 (1986)CrossRefGoogle Scholar
  32. 32.
    Murzin, A., Brenner, S., Hubbard, T., Chothia, C.: SCOP: a structural classification of proteins database for the investigation of sequences and structures. Journal of Molecular Biology 247, 536–540 (1995)Google Scholar
  33. 33.
    Andreeva, A., Howorth, D., Brenner, S., Hubbard, T., Chothia, C., Murzin, A.: SCOP database in 2004: refinements integrate structure and sequence family data. Nucleic Acids Research 32, D226–D229 (2004)Google Scholar
  34. 34.
    Brenner, S., Chothia, C., Hubbard, T.: Assessing sequence comparison methods with reliable structurally identified distant evolutionary relationships. Proceedings of the National Academy of Sciences USA 95, 6073–6078 (1998)CrossRefGoogle Scholar
  35. 35.
    Park, J., Karplus, K., Barrett, C., Hughey, R., Haussler, D., Hubbard, T., Chothia, C.: Sequence comparisons using multiple sequences detect three times as many remote homologues as pairwise methods. Journal of Molecular Biology 284, 1201–1210 (1998)CrossRefGoogle Scholar
  36. 36.
    Chandonia, J., Hon, G., Walker, N., Conte, L.L., Koehl, P., Levitt, M., Brenner, S.: The ASTRAL compendium in 2004. Nucleic Acids Research 32, D189–D192 (2004)Google Scholar
  37. 37.
    Gribskov, M., Robinson, N.: Use of receiver operating characteristic (ROC) analysis to evaluate sequence matching. Computers & Chemistry 20, 25–33 (1996)CrossRefGoogle Scholar
  38. 38.
    Karlin, S., Altschul, S.: Methods for assessing the statistical significance of molecular sequence features by using general scoring schemes. Proceedings of the National Academy of Sciences USA 87, 2264–2268 (1990)zbMATHCrossRefGoogle Scholar
  39. 39.
    Cameron, M., Williams, H.E., Cannane, A.: Improved gapped alignment in BLAST. IEEE Transactions on Computational Biology and Bioinformatics 1, 116–129 (2004)CrossRefGoogle Scholar
  40. 40.
    Cameron, M., Williams, H.E., Cannane, A.: A deterministic finite automaton for faster protein hit detection in BLAST. Journal of Computational Biology (to appear, 2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Michael Cameron
    • 1
  • Yaniv Bernstein
    • 1
  • Hugh E. Williams
    • 2
  1. 1.School of Computer Science and Information TechnologyRMIT UniversityMelbourneAustralia
  2. 2.Microsoft CorporationRedmondUSA

Personalised recommendations