Abstract
Language identification an important task for web information retrieval. This paper presents the implementation of a tool for language identification in mono- and multi-lingual documents. The tool implements four algorithms for language identification. Furthermore, we present a n-gram approach for the identification of languages in multi-lingual documents. An evaluation for monolingual texts of varied length is presented. Results for eight languages including Ukrainian and Russian are shown. It could be shown that n-gram-based approaches outperform word-based algorithms for short texts. For longer texts, the performance is comparable. The evaluation for multi-lingual documents is based on both short synthetic documents and real world web documents. Our tool is able to recognize the languages present as well as the location of the language change with reasonable accuracy.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Artemenko, O., Shramko, M.: Entwicklung eines Werkzeugs zur Sprachidentifikation in mono- und multilingualen Texten. Master thesis. University of Hildesheim (2005), http://web1.bib.uni-hildesheim.de/2005/artemenko_shramko.pdf
Cavnar, W.B., Trenkle, J.M.: N-Gram-Based Text Categorization. Symposion on Document Analysis and Information Retrieval, pp. 161–176. Univ. of Nevada, Las Vegas
Cowie, J., Ludovik, E., Zacharski, R.: An Autonomous, Web-based, Multilingual Corpus Collection Tool. In: Proc. Intl. Conference on Natural Language Processing and Industrial Applications, Moncton, pp. 142–148 (1998)
Grefenstette, G.: Comparing two language identification schemes. In: JADT 1995. 3rd International conference on Statistical Analysis of Textual Data, Rome (1995)
Hollink, V., Kamps, J., Monz, C., de Rijke, M.: Monolingual Document Retrieval for European Languages. Information Retrieval 7(1-2), 33–52 (2004)
Kikui, G.: Identifying the Coding System and Language of On-line Documents on the Internet. In: Proc 16th Conf. on Computational linguistics, Denmark, vol. 2, pp. 652–657 (1996)
Kralisch, A., Mandl T.: Barriers of Information Access across Languages on the Internet: Network and Language Effects. In: Proc. Hawaii International Conference on System Sciences (HICSS-39) (2006)
Lins, R., Gonçalves, P.: Automatic Language Identification of Written Texts. In: Proc. ACM SAC Symposium on Applied Computing, Nicosia, Cyprus, pp. 1128–1133 (March 2004)
Martino, M., Paulsen, R.: Natural language determination using partial words, U.S. Patent No. 6216102 B1 (April 2001)
Martins, B., Silva, M.: Language Identification in Web Pages. In: Proc. ACM SAC Symposium on Applied Computing, Santa Fe, New Mexico, USA, March 13-17, 2005, pp. 764–768 (2005)
McNamee, P., Mayfield, J.: Character N-Gram Tokenization for European Language Text Retrieval. Information Retrieval 7(1/2), 73–98 (2004)
Prager, J.: Linguini: Language Identification for Multilingual Documents. In: Proc. 32nd Hawaii International Conf. on System Sciences (1999)
Sigurbjörnsson, B., Kamps, J., de Rijke, M.: Blueprint of a cross-lingual web retrieval collection. Journal on Digital Information Management 3(9-13) (2005)
Souter, C., Churcher, G., Hayes, J., Hughes, J., Johnson, S.: Natural Language Identification Using Corpus-Based Models. Hermes J. Linguistics 13, 183–203 (1994)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mandl, T., Shramko, M., Tartakovski, O., Womser-Hacker, C. (2006). Language Identification in Multi-lingual Web-Documents. In: Kop, C., Fliedl, G., Mayr, H.C., Métais, E. (eds) Natural Language Processing and Information Systems. NLDB 2006. Lecture Notes in Computer Science, vol 3999. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11765448_14
Download citation
DOI: https://doi.org/10.1007/11765448_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34616-6
Online ISBN: 978-3-540-34617-3
eBook Packages: Computer ScienceComputer Science (R0)