Chapter

Natural Language Processing and Information Systems

Volume 2553 of the series Lecture Notes in Computer Science pp 76-84

Date:

Automatic Identification of European Languages

  • Anna V. ZhdanovaAffiliated withA.P. Ershov Institute of Informatics SystemsNovosibirsk State University

* Final gross prices may vary according to local VAT.

Get Access

Abstract

We describe our word-based implementation of a language identifying system for the text messages written in European languages. Specifically, we use and compare linguistic (based on functional words) and statistic (based on the word frequency) approaches to construction of the identifying vocabularies. Our version of the statistic approach copes with the differences in degrees of word overlap among languages and the problem of the small-size messages. In addition, it allows an user to choose the accuracy of language identification. At present, our system identifies 8 languages (Bulgarian, English, French, German, Italian, Russian, Spanish and Swedish) in various encodings. With the identifying vocabularies of limited size (less than 1500 keys per language), the accuracy of identification attains 99% even for the messages containing only one sentence.