Abstract
Similarity search and similarity join on strings are important for applications such as duplicate detection, error detection, data cleansing, or comparison of biological sequences. Especially DNA sequencing produces large collections of erroneous strings which need to be searched, compared, and merged. However, current RDBMS offer similarity operations only in a very limited and inefficient form that does not scale to the amount of data produced in Life Science projects.
We present PETER, a prefix tree based indexing algorithm supporting approximate search and approimate joins. Our tool supports Hamming and edit distance as similarity measure and is available as C++ library, as Unix command line tool, and as cartridge for a commercial database. It combines an efficient implementation of compressed prefix trees with advanced pre-filtering techniques that exclude many candidate strings early. The achieved speed-ups are dramatic, especially for DNA with its small alphabet. We evaluate our tool on several collections of long strings containing up to 5,000,000 entries of length up to 3,500. We compare its performance to agrep, nrgrep, and user-defined functions inside a relational database. Our experiments reveal that PETER is faster by orders of magnitudes compared to the command-line tools. Compared to RDBMS, it computes similarity joins in minutes for which UDFs did not finish within a day and outperforms the built-in join methods even in the exact case.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Aghili, S.A., Agrawal, D., Abbadi, A.E.: Bft: Bit filtration technique for approximate string join in biological databases. In: Nascimento, M.A., de Moura, E.S., Oliveira, A.L. (eds.) SPIRE 2003. LNCS, vol. 2857, pp. 326–340. Springer, Heidelberg (2003)
Altschul, S.F., Madden, T.L., Schäffer, A.A., Zhang, J., Zhang, Z., Miller, W., Lipman, D.J.: Gapped blast and psi-blast: a new generation of protein database search programs. Nucleic acids research 25(17) (September 1997)
Fickett, J.W.: Fast optimal alignment. Nucleic Acids Research 12 (1984)
Gravano, L., Ipeirotis, P.G., Jagadish, H.V., Koudas, N., Muthukrishnan, S., Srivastava, D.: Approximate string joins in a database (almost) for free. In: VLDB (2001)
Gravano, L., Ipeirotis, P.G., Koudas, N., Srivastava, D.: Text joins in an rdbms for web data integration. In: WWW 2003 (2003)
Gusfield, D.: Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. Cambridge University Press, Cambridge (1997)
Han, J., Pei, J., Yin, Y., Mao, R.: Mining frequent patterns without candidate generation: A Frequent-Pattern tree approach. Data Mining and Knowledge Discovery 8(1) (2004)
Jampani, R., Pudi, V.: Using Prefix-Trees for efficiently computing set joins. In: Zhou, L.-z., Ooi, B.-C., Meng, X. (eds.) DASFAA 2005. LNCS, vol. 3453, pp. 761–772. Springer, Heidelberg (2005)
Kalyanaraman, A., Alaru, S.: Expressed sequence tags: Clustering and applications. In: Handbook of Computational Molecular Biology. Chapman & Hall CRC computer information science, Boca Raton (2006)
Koudas, N., Marathe, A., Srivastava, D.: Flexible string matching against large databases in practice. In: VLDB 2004 (2004)
Lee, T., Pouliot, Y., Wagner, V., Gupta, P., Calvert, D.S., Tenenbaum, J., Karp, P.: Biowarehouse: a bioinformatics database warehouse toolkit. BMC Bioinformatics 7 (2006)
Morrison, D.R.: PATRICIA - practical algorithm to retrieve information coded in alphanumeric. Journal of the ACM (JACM) 15(4) (1968)
Navarro, G.: A guided tour to approximate string matching. ACM Computing Surveys 33(1) (2001)
NCBI. dbEST (1992), http://www.ncbi.nlm.nih.gov/dbest
Schallehn, E., Sattler, K.-U., Saake, G.: Efficient similarity-based operations for data integration. Data & Knowledge Engineering 48 (2004)
Shang, H., Merrett, T.: Tries for approximate string matching. IEEE TKDE 8(4) (1996)
Sutinen, E., Tarhio, J.: Filtration with q-Samples in approximate string matching. In: Hirschberg, D.S., Meyers, G. (eds.) CPM 1996. LNCS, vol. 1075. Springer, Heidelberg (1996)
Sutinen, E., Tarhio, J.: On using q-Gram locations in approximate string matching. In: Spirakis, P.G. (ed.) ESA 1995. LNCS, vol. 979, Springer, Heidelberg (1995)
Wagner, R.A., Fischer, M.J.: The String-to-String correction problem. Journal of the ACM (JACM) 21(1) (1974)
Williams, H.E., Zobel, J.: Indexing and retrieval for genomic databases. IEEE TKDE 14(1) (2002)
Xiao, C., Wang, W., Lin, X.: Ed-Join: an efficient algorithm for similarity joins with edit distance constraints. In: VLDB 2008 (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Rheinländer, A., Knobloch, M., Hochmuth, N., Leser, U. (2010). Prefix Tree Indexing for Similarity Search and Similarity Joins on Genomic Data. In: Gertz, M., Ludäscher, B. (eds) Scientific and Statistical Database Management. SSDBM 2010. Lecture Notes in Computer Science, vol 6187. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-13818-8_36
Download citation
DOI: https://doi.org/10.1007/978-3-642-13818-8_36
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-13817-1
Online ISBN: 978-3-642-13818-8
eBook Packages: Computer ScienceComputer Science (R0)