Abstract
This paper presents a language identification algorithm using cosine similarity against a filtered and weighted subset of the most frequent n-grams in training data with optional inter-string score smoothing, and its implementation in an open-source program. When applied to a collection of strings in 1100 languages containing at most 65 characters each, an average classification accuracy of over 99.2% is achieved with smoothing and 98.2% without. Compared to three other open-source language identification programs, the new program is both much more accurate and much faster at classifying short strings given such a large collection of languages.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Brown, R.D.: Finding and Identifying Text in 900+ Languages. Digital Investigation 9, S34–S43 (2012)
Cavnar, W.B., Trenkle, J.M.: N-Gram-Based Text Categorization. In: Proceedings of SDAIR 1994, 3rd Annual Symposium on Document Analysis and Information Retrieval, UNLV Publications/Reprographics, pp. 161–175 (April 1994)
Ljubešić, N., Mikelić, N., Boras, D.: Language identification: How to distinguish similar languages. In: Lužar-Stifter, V., Hljuz Dobrić, V. (eds.) Proceedings of the 29th International Conference on Information Technology Interfaces, Zagreb, pp. 541–546. SRCE University Computing Centre (2007)
Ahmed, B., Cha, S.H., Tappert, C.: Language Identification from Text Using N-gram Based Cumulative Frequency Addition. In: Proceedings of Student/Faculty Research Day, CSIS, Pace University (May 2004)
Carter, S., Tsagkias, M., Weerkamp, W.: Semi-Supervised Priors for Microblog Language Identification. In: Proceedings of the Dutch-Belgian Information Retrieval Workshop (DIR 2011), Amsterdam (February 2011)
Damashek, M.: Gauging Similarity with n-grams: Language Independent Categorization of Text. Science 267(5199), 843–848 (1995)
Shuyo, N.: Language Detection Library - 99% over precision for 49 languages (December 2010), http://www.slideshare.net/shuyo/language-detection-library-for-java (accessed May 30, 2013)
Xia, F., Lewis, W.D., Poon, H.: Language ID in the Context of Harvesting Language Data off the Web. In: Proceedings of EACL 2009, pp. 870–878 (2009)
Likasoft: Polyglot 3000, http://polyglot3000.com (accessed May 30, 2013)
United Bible Societies: Scripture Language Report 2011 (2011), http://www.unitedbiblesocieties.org/wp-content/uploads/2012/04/-report-TABLE-I-2011-Rec-March-28.doc (accessed May 30, 2013)
Hugueney, B.: libtextcat 2.2-9: Faster Unicode-focused C++ reimplementation of libtextcat (2011), https://github.com/scientific-coder/libtextcat (accessed May 30, 2013)
Barkov, A.: mguesser version 0.4 (2008), http://www.mnogosearch.org/guesser/mguesser-0.4.tar.gz (accessed May 30, 2013)
Shuyo, N.: Language Detection Library for Java, http://code.google.com/p/language-detection/ (accessed May 30, 2013)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Brown, R.D. (2013). Selecting and Weighting N-Grams to Identify 1100 Languages. In: Habernal, I., Matoušek, V. (eds) Text, Speech, and Dialogue. TSD 2013. Lecture Notes in Computer Science(), vol 8082. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40585-3_60
Download citation
DOI: https://doi.org/10.1007/978-3-642-40585-3_60
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40584-6
Online ISBN: 978-3-642-40585-3
eBook Packages: Computer ScienceComputer Science (R0)