Advertisement

Towards Combining Multitask and Multilingual Learning

  • Matus PikuliakEmail author
  • Marian Simko
  • Maria Bielikova
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11376)

Abstract

Machine learning is an increasingly important approach to Natural Language Processing. Most languages however do not possess enough data to fully utilize it. When dealing with such languages it is important to use as much auxiliary data as possible. In this work we propose a combination of multitask and multilingual learning. When learning a new task we use data from other tasks and other languages at the same time. We evaluate our approach with a neural network based model that can solve two tasks – part-of-speech tagging and named entity recognition – with four different languages at the same time. Parameters of this model are partially shared across all data and partially they are specific for individual tasks and/or languages. Preliminary experiments show that this approach has its merits as we were able to beat baseline solutions that do not combine data from all the available sources.

Keywords

Transfer learning Multilingual learning Deep natural language processing 

Notes

Acknowledgements

This work was partially supported by the Slovak Research and Development Agency under the contract No. APVV-15-0508, and by the Scientific Grant Agency of the Slovak Republic, grants No. VG 1/0667/18 and No. VG 1/0646/15.

References

  1. 1.
    Agić, Ž., Johannsen, A., Plank, B., Alonso, H.M., Schluter, N., Søgaard, A.: Multilingual projection for parsing truly low-resource languages. Trans. Assoc. Comput. Linguist. 4, 301–312 (2016)CrossRefGoogle Scholar
  2. 2.
    Ammar, W., Mulcaire, G., Ballesteros, M., Dyer, C., Smith, N.: Many languages, one parser. Trans. Assoc. Comput. Linguist. 4, 431–444 (2016)CrossRefGoogle Scholar
  3. 3.
    Benikova, D., Biemann, C., Reznicek, M.: NoSta-D named entity annotation for German: guidelines and dataset. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation. LREC 2014, 26–31 May 2014, Reykjavik, Iceland, pp. 2524–2531 (2014)Google Scholar
  4. 4.
    Bos, J., Basile, V., Evang, K., Venhuizen, N.J., Bjerva, J.: The Groningen meaning bank. In: Ide, N., Pustejovsky, J. (eds.) Handbook of Linguistic Annotation, pp. 463–496. Springer, Dordrecht (2017).  https://doi.org/10.1007/978-94-024-0881-2_18CrossRefGoogle Scholar
  5. 5.
    Buys, J., Botha, J.A.: Cross-lingual morphological tagging for low-resource languages. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1954–1964. Association for Computational Linguistics (2016)Google Scholar
  6. 6.
    Conneau, A., Lample, G., Ranzato, M., Denoyer, L., Jégou, H.: Word translation without parallel data. In: 6th International Conference on Learning Representations, Vancouver, Canada, May 2018Google Scholar
  7. 7.
    Cotterell, R., Heigold, G.: Cross-lingual character-level neural morphological tagging. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 759–770. Association for Computational Linguistics (2017)Google Scholar
  8. 8.
    Gallay, L., Šimko, M.: Utilizing vector models for automatic text lemmatization. In: Freivalds, R., Engels, G., Catania, B. (eds.) SOFSEM 2016. LNCS, vol. 9587, pp. 532–543. Springer, Heidelberg (2016).  https://doi.org/10.1007/978-3-662-49192-8_43CrossRefGoogle Scholar
  9. 9.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  10. 10.
    Joty, S., Nakov, P., Màrquez, L., Jaradat, I.: Cross-language learning with adversarial neural networks. In: Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. 226–237. Association for Computational Linguistics (2017)Google Scholar
  11. 11.
    Kravalova, J., Zabokrtsky, Z.: Czech named entity corpus and SVM-based recognizer. In: Proceedings of the 2009 Named Entities Workshop: Shared Task on Transliteration (NEWS 2009), pp. 194–201. Association for Computational Linguistics (2009)Google Scholar
  12. 12.
    Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 260–270. Association for Computational Linguistics (2016)Google Scholar
  13. 13.
    Lin, Y., Yang, S., Stoyanov, V., Ji, H.: A multi-lingual multi-task architecture for low-resource sequence labeling. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 799–809. Association for Computational Linguistics (2018). http://aclweb.org/anthology/P18-1074
  14. 14.
    Liu, P., Qiu, X., Huang, X.: Adversarial multi-task learning for text classification. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1–10. Association for Computational Linguistics (2017)Google Scholar
  15. 15.
    Mikolov, T., Yih, W., Zweig, G.: Linguistic regularities in continuous space word representations. In: Human Language Technologies: Proceedings and Conference of the North American Chapter of the Association of Computational Linguistics, 9–14 June 2013, Westin Peachtree Plaza Hotel, Atlanta, Georgia, USA, pp. 746–751 (2013)Google Scholar
  16. 16.
    Nivre, J., et al.: Universal dependencies v1: a multilingual treebank collection. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation. LREC 2016, 23–28 May 2016, Portorož, Slovenia (2016)Google Scholar
  17. 17.
    Peng, N., Dredze, M.: Multi-task domain adaptation for sequence tagging. In: Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 91–100. Association for Computational Linguistics (2017)Google Scholar
  18. 18.
    Ruder, S.: A survey of cross-lingual embedding models. CoRR abs/1706.04902 (2017)Google Scholar
  19. 19.
    Sang, E.F.T.K., Meulder, F.D.: Introduction to the CoNLL-2003 shared task: language-independent named entity recognition. In: Proceedings of the Seventh Conference on Natural Language Learning. CoNLL 2003, 31 May–1 June 2003, Held in Cooperation with HLT-NAACL 2003, Edmonton, Canada, pp. 142–147 (2003)Google Scholar
  20. 20.
    Søgaard, A., Goldberg, Y.: Deep multi-task learning with low level tasks supervised at lower layers. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 231–235. Association for Computational Linguistics (2016)Google Scholar
  21. 21.
    Subramanian, S., Trischler, A., Bengio, Y., Pal, C.J.: Learning general purpose distributed sentence representations via large scale multi-task learning. In: 6th International Conference on Learning Representations, Vancouver, Canada, May 2018Google Scholar
  22. 22.
    Tiedemann, J.: Rediscovering annotation projection for cross-lingual parser induction. In: Proceedings of COLING 2014, The 25th International Conference on Computational Linguistics: Technical Papers, pp. 1854–1864. Dublin City University and Association for Computational Linguistics (2014)Google Scholar
  23. 23.
    Tieleman, T., Hinton, G.: Lecture 6.5-rmsprop: divide the gradient by a running average of its recent magnitude. COURSERA: Neural Netw. Mach. Learn. 4(2), 26–31 (2012)Google Scholar
  24. 24.
    Yang, Z., Salakhutdinov, R., Cohen, W.W.: Transfer learning for sequence tagging with hierarchical recurrent networks. In: 5th International Conference on Learning Representations, Toulon, France, April 2017Google Scholar
  25. 25.
    Zirikly, A., Hagiwara, M.: Cross-lingual transfer of named entity recognizers without parallel corpora. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 390–396. Association for Computational Linguistics (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Faculty of Informatics and Information TechnologiesSlovak University of Technology in BratislavaBratislavaSlovakia

Personalised recommendations