Language Identification in Multi-lingual Web-Documents

  • Thomas Mandl
  • Margaryta Shramko
  • Olga Tartakovski
  • Christa Womser-Hacker
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3999)

Abstract

Language identification an important task for web information retrieval. This paper presents the implementation of a tool for language identification in mono- and multi-lingual documents. The tool implements four algorithms for language identification. Furthermore, we present a n-gram approach for the identification of languages in multi-lingual documents. An evaluation for monolingual texts of varied length is presented. Results for eight languages including Ukrainian and Russian are shown. It could be shown that n-gram-based approaches outperform word-based algorithms for short texts. For longer texts, the performance is comparable. The evaluation for multi-lingual documents is based on both short synthetic documents and real world web documents. Our tool is able to recognize the languages present as well as the location of the language change with reasonable accuracy.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Artemenko, O., Shramko, M.: Entwicklung eines Werkzeugs zur Sprachidentifikation in mono- und multilingualen Texten. Master thesis. University of Hildesheim (2005), http://web1.bib.uni-hildesheim.de/2005/artemenko_shramko.pdf
  2. 2.
    Cavnar, W.B., Trenkle, J.M.: N-Gram-Based Text Categorization. Symposion on Document Analysis and Information Retrieval, pp. 161–176. Univ. of Nevada, Las VegasGoogle Scholar
  3. 3.
    Cowie, J., Ludovik, E., Zacharski, R.: An Autonomous, Web-based, Multilingual Corpus Collection Tool. In: Proc. Intl. Conference on Natural Language Processing and Industrial Applications, Moncton, pp. 142–148 (1998)Google Scholar
  4. 4.
    Grefenstette, G.: Comparing two language identification schemes. In: JADT 1995. 3rd International conference on Statistical Analysis of Textual Data, Rome (1995)Google Scholar
  5. 5.
    Hollink, V., Kamps, J., Monz, C., de Rijke, M.: Monolingual Document Retrieval for European Languages. Information Retrieval 7(1-2), 33–52 (2004)CrossRefGoogle Scholar
  6. 6.
    Kikui, G.: Identifying the Coding System and Language of On-line Documents on the Internet. In: Proc 16th Conf. on Computational linguistics, Denmark, vol. 2, pp. 652–657 (1996)Google Scholar
  7. 7.
    Kralisch, A., Mandl T.: Barriers of Information Access across Languages on the Internet: Network and Language Effects. In: Proc. Hawaii International Conference on System Sciences (HICSS-39) (2006) Google Scholar
  8. 8.
    Lins, R., Gonçalves, P.: Automatic Language Identification of Written Texts. In: Proc. ACM SAC Symposium on Applied Computing, Nicosia, Cyprus, pp. 1128–1133 (March 2004)Google Scholar
  9. 9.
    Martino, M., Paulsen, R.: Natural language determination using partial words, U.S. Patent No. 6216102 B1 (April 2001)Google Scholar
  10. 10.
    Martins, B., Silva, M.: Language Identification in Web Pages. In: Proc. ACM SAC Symposium on Applied Computing, Santa Fe, New Mexico, USA, March 13-17, 2005, pp. 764–768 (2005)Google Scholar
  11. 11.
    McNamee, P., Mayfield, J.: Character N-Gram Tokenization for European Language Text Retrieval. Information Retrieval 7(1/2), 73–98 (2004)CrossRefGoogle Scholar
  12. 12.
    Prager, J.: Linguini: Language Identification for Multilingual Documents. In: Proc. 32nd Hawaii International Conf. on System Sciences (1999)Google Scholar
  13. 13.
    Sigurbjörnsson, B., Kamps, J., de Rijke, M.: Blueprint of a cross-lingual web retrieval collection. Journal on Digital Information Management 3(9-13) (2005)Google Scholar
  14. 14.
    Souter, C., Churcher, G., Hayes, J., Hughes, J., Johnson, S.: Natural Language Identification Using Corpus-Based Models. Hermes J. Linguistics 13, 183–203 (1994)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Thomas Mandl
    • 1
  • Margaryta Shramko
    • 1
  • Olga Tartakovski
    • 1
  • Christa Womser-Hacker
    • 1
  1. 1.Information ScienceUniversität HildesheimHildesheimGermany

Personalised recommendations