Standardizing Tweets with Character-Level Machine Translation
This paper presents the results of the standardization procedure of Slovene tweets that are full of colloquial, dialectal and foreign-language elements. With the aim of minimizing the human input required we produced a manually normalized lexicon of the most salient out-of-vocabulary (OOV) tokens and used it to train a character-level statistical machine translation system (CSMT). Best results were obtained by combining the manually constructed lexicon and CSMT as fallback with an overall improvement of 9.9% increase on all tokens and 31.3% on OOV tokens. Manual preparation of data in a lexicon manner has proven to be more efficient than normalizing running text for the task at hand. Finally we performed an extrinsic evaluation where we automatically lemmatized the test corpus taking as input either original or automatically standardized wordforms, and achieved 75.1% per-token accuracy with the former and 83.6% with the latter, thus demonstrating that standardization has significant benefits for upstream processing.
Keywordstwitterese standardization character-level machine translation
Unable to display preview. Download preview PDF.
- 3.Kaufmann, M., Kalita, J.: Syntactic Normalization of Twitter Messages. In: Proceedings of the 8th International Conference on Natural Language Processing, ICON 2010 (2010)Google Scholar
- 5.Pennell, D., Liu, Y.: Toward text message normalization: Modeling abbreviation generation. In: 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5364–5367 (2011)Google Scholar
- 6.Pennell, D., Liu, Y.: A Character-Level Machine Translation Approach for Normalization of SMS Abbreviations. In: Proceedings of 5th International Joint Conference on Natural Language Processing, pp. 974–982. Asian Federation of Natural Language Processing, Chiang Mai (November 2011)Google Scholar
- 7.De Clercq, O.E., Desmet, B., Schulz, S., Lefever, E., Hoste, V.: Normalization of Dutch user-generated content. In: Proceedings of Recent Advances in Natural Language Processing, INCOMA, pp. 179–188 (2013)Google Scholar
- 8.Scherrer, Y., Erjavec, T.: Modernizing historical Slovene words with character-based SMT. In: BSNLP 2013 - 4th Biennial Workshop on Balto-Slavic Natural Language Processing, Sofia, Bulgarie, pp. 2013–2014 (July 2013)Google Scholar
- 9.Arhar, Š.: Učni korpus SSJ in leksikon besednih oblik za slovenino. Jezik in slovstvo 54(3-4), 43–56 (2009)Google Scholar
- 11.Erjavec, T.: Automatic linguistic annotation of historical language: ToTrTaLe and XIX century Slovene. In: Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pp. 33–38. Association for Computational Linguistics, Portland (June 2011)Google Scholar