Abstract
This chapter presents an unsupervised solution to language identification. The method sorts multilingual text corpora sentence-wise into different languages. In this attempt, the main difference to previous methods is that no training data for the different languages is provided and the number of languages does not have to be known beforehand. This application illustrates the benefits of a parameter-free graph clustering algorithm like Chinese Whispers, as the data–words and their statistical dependencies – are represented naturally in a graph, and the number of clusters (here: languages) as well as their size distribution is unknown. The feasibility and robustness of the approach for non-standard language data is demonstrated in a case study on Twitter data.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Biemann, C. (2012). Unsupervised Language Separation. In: Structure Discovery in Natural Language. Theory and Applications of Natural Language Processing. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-25923-4_5
Download citation
DOI: https://doi.org/10.1007/978-3-642-25923-4_5
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-25922-7
Online ISBN: 978-3-642-25923-4
eBook Packages: Computer ScienceComputer Science (R0)