DRESS: dimensionality reduction for efficient sequence search
Similarity search in large sequence databases is a problem ubiquitous in a wide range of application domains, including searching biological sequences. In this paper we focus on protein and DNA data, and we propose a novel approximate method method for speeding up range queries under the edit distance. Our method works in a filter-and-refine manner, and its key novelty is a query-sensitive mapping that transforms the original string space to a new string space of reduced dimensionality. Specifically, it first identifies the \(t\) most frequent codewords in the query, and then uses these codewords to convert both the query and the database to a more compact representation. This is achieved by replacing every occurrence of each codeword with a new letter and by removing the remaining parts of the strings. Using this new representation, our method identifies a set of candidate matches that are likely to satisfy the range query, and finally refines these candidates in the original space. The main advantage of our method, compared to alternative methods for whole sequence matching under the edit distance, is that it does not require any training to create the mapping, and it can handle large query lengths with negligible losses in accuracy. Our experimental evaluation demonstrates that, for higher range values and large query sizes, our method produces significantly lower costs and runtimes compared to two state-of-the-art competitor methods.
KeywordsSimilarity search Alphabet reduction Biological sequences
The work of Vassilis Athitsos was partially supported by National Science Foundation grants IIS-1055062, CNS-1059235, CNS-1035913, and CNS-1338118. The work of Gautam Das was partially supported by National Science Foundation under grants 0812601, 0915834, 1018865 and grants from Microsoft Research.
Conflict of interest
The authors declare that they have no conflict of interest.
- Arasu A, Ganti V, Kaushik R (2006) Efficient exact set-similarity joins. In: Proceedings of very large database endowment (PVLDB), pp 918–929Google Scholar
- Behm A, Vernica R, Alsubaiee S, Ji S, Lu J, Jin L, Lu Y, Li C (2010) UCI Flamingo Package 4.0. http://flamingo.ics.uci.edu/releases/4.0/
- Burrows M, Wheeler DJ (1994) A block-sorting lossless data compression algorithm. Tech. Rep. 124, Systems Research Center, Palo Alto, http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.6774
- Jongeneel CV (2000) Searching the expressed sequence tag (est) databases: panning for genes. Bioinformatics 1:76–92Google Scholar
- Kent WJ (2002) Resource BLAT-The BLAST-like alignment tool. Genome ResGoogle Scholar
- Kim MS, Whang KY, Lee JG, Lee MJ (2005a) n-gram/2l: a space and time efficient two-level n-gram inverted index structure. In: Proceedings of the 31st international conference on very large data bases, VLDB Endowment, pp 325–336Google Scholar
- Li C, Wang B, Yang X (2007) Vgram: improving performance of approximate queries on string collections using variable-length grams. In: Proceedings of the 33rd international conference on Very large data bases, VLDB Endowment, pp 303–314Google Scholar
- Li C, Lu J, Lu Y (2008a) Efficient merging and filtering algorithms for approximate string searches. International conference on data engineering (ICDE)Google Scholar
- Li Y, Patel JM, Terrell A (2012) Wham: a high-throughput sequence alignment method. ACM Trans Database Syst (TODS) 37(4):28Google Scholar
- Litwin W, Mokadem R, Rigaux P, Schwarz T (2007) Fast ngram-based string search over data encoded using algebraic signatures. In: Proceedings of the very large database endowment (PVLDB), pp 207–218Google Scholar
- Meek C, Patel JM, Kasetty S (2003) Oasis: an online and accurate technique for local-alignment searches on biological sequences. In: Proceedings of very large database endowment (PVLDB), vol 29, pp 910–921Google Scholar
- Papapetrou P, Athitsos V, Kollios G, Gunopulos D (2009) Reference-based alignment in large sequence databases. Proc Very Large Database Endow (PVLDB) 2(1):205–216Google Scholar
- Traina C, Traina AJM, Seeger B, Faloutsos C (2000) Slim-trees: high performance metric trees minimizing overlap between nodes. International conference on extending database technology (EDBT), pp 51–65Google Scholar
- Venkateswaran J, Lachwani D, Kahveci T, Jermaine C (2006) Reference-based indexing of sequence databases. In: International conference on very large databases (VLDB), pp 906–917Google Scholar
- Vieira MR, Traina C, Chino FJT, Traina AJM (2004) Dbm-tree: a dynamic metric access method sensitive to local density data. Brazilian symposium on databases (SBBD), pp 163–177Google Scholar
- Wandelt S, Starlinger J, Bux M, Leser U (2013) Rcsi: scalable similarity search in thousand(s) of genomes. Proceedings of the VLDB Endowment (PVLDB) p (to appear)Google Scholar
- Yang X, Wang B, Li C (2008) Cost-based variable-length-gram selection for string collections to support approximate queries efficiently. In: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, ACM, pp 353–364Google Scholar