SMS Normalization: Combining Phonetics, Morphology and Semantics

  • Jesús Oliva
  • José Ignacio Serrano
  • María Dolores del Castillo
  • Ángel Iglesias
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7023)

Abstract

The language used in electronic communications such as e-mails, chats and SMS texts presents special phenomena and important deviations from natural language. Typical machine translation approaches are difficult to adapt to SMS language due to the many irregularities this kind of language shows. This paper presents a new approach for SMS normalization that combines lexical and phonological translation techniques with disambiguation algorithms at two different levels: lexical and semantic. The results obtained by the system outperform some of the existing methods of SMS normalization despite the fact that the corpus created has some features that complicates the normalization task.

Keywords

Target Word Machine Translation Real Word Word Sense Disambiguation Word Error Rate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aw, A., Zhang, M., Xiao, J., Su, J.: A phrase-based statistical model for sms text normalization. In: Proceedings of the COLING/ACL on Main Conference Poster Sessions, pp. 33–40. Association for Computational Linguistics, Morristown (2006)CrossRefGoogle Scholar
  2. 2.
    Banerjee, S., Pedersen, T.: Extended gloss overlaps as a measure of semantic relatedness. In: Proceedings of the Eighteenth International Joint conference on Artificial Intelligence, Acapulco, pp. 805–810 (2003)Google Scholar
  3. 3.
    Baron, N.: Computer mediated communication as a force in language change. Vis. Lang., 118–141 (2004)Google Scholar
  4. 4.
    Carreras, X., Chao, I., Padró, L., Padró, M.: Freeling: An open-source suite of language analyzers. In: Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004 (2004)Google Scholar
  5. 5.
    Choudhury, M., Saraf, R., Jain, V., Mukherjee, A., Sarkar, S., Basu, A.: Investigation and modeling of the structure of texting language. Int. J. Doc. Anal. Recognit. 10(3), 157–174 (2007)CrossRefGoogle Scholar
  6. 6.
    Fellbaum, C.: WordNet: An Electronic Lexical Database. MIT Press (1998)Google Scholar
  7. 7.
    Ferri, S.: El fenómeno de economía linguística en el lenguaje SMS: breve estudio experimental en alumnos de 16 años, pp. 255–270 (2005)Google Scholar
  8. 8.
    Kobus, C., Yvon, F., Damnati, G.: Normalizing sms: are two metaphors better than one? In: COLING 2008 (2008)Google Scholar
  9. 9.
    Levenshtein, V.I.: Binary codes capable of correcting deletions, insertions, and reversals. Tech. Rep. 8 (1966)Google Scholar
  10. 10.
    Michelizzi, J.: Semantic relatedness applied to all word sense disambiguation. Master’s thesis, University of Minesotta (2005)Google Scholar
  11. 11.
    Guimier de Neef, É., Debeurme, A., Park, J.: Tilt correcteur de sms: évaluation et bilan qualitatif. Actes de TALN 2007 (2007)Google Scholar
  12. 12.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation. Tech. Rep. RC22176 (W0109-022), IBM Research Division (2001)Google Scholar
  13. 13.
    Pedersen, T., Banerjee, S., Patwardhan, S.: Maximizing semantic relatedness to perform word sense disambiguation (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Jesús Oliva
    • 1
  • José Ignacio Serrano
    • 1
  • María Dolores del Castillo
    • 1
  • Ángel Iglesias
    • 1
  1. 1.Bioengineering GroupSpanish National Research Council (CSIC)Arganda del ReySpain

Personalised recommendations