Skip to main content

Advanced Transfer Learning Approach for Improving Spanish Sentiment Analysis

Part of the Lecture Notes in Computer Science book series (LNAI,volume 11835)

Abstract

In the last years, innovative techniques like Transfer Learning have impacted strongly in Natural Language Processing, increasing massively the state-of-the-art in several challenging tasks. In particular, the Universal Language Model Fine-Tuning (ULMFiT) algorithm has proven to have an impressive performance on several English text classification tasks. In this paper, we aim at developing an algorithm for Spanish Sentiment Analysis of short texts that is comparable to the state-of-the-art. In order to do so, we have adapted the ULMFiT algorithm to this setting. Experimental results on benchmark datasets (InterTASS 2017 and InterTASS 2018) show how this simple transfer learning approach performs well when compared to fancy deep learning techniques.

Keywords

  • Sentiment analysis
  • Natural Language Processing
  • Language Model
  • Transfer learning

This work was funded by CONCYTEC-FONDECYT under the call E041-01 [contract number 34-2018-FONDECYT-BM-IADT-SE].

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-33749-0_10
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-33749-0
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.

References

  1. Arora, S., Liang, Y., Ma, T.: A simple but tough-to-beat baseline for sentence embeddings. In: International Conference on Learning Representations (2017)

    Google Scholar 

  2. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)

    CrossRef  Google Scholar 

  3. Brooke, J., Tofiloski, M., Taboada, M.: Cross-linguistic sentiment analysis: From English to Spanish. In: Proceedings of RANLP, vol. 2009, pp. 50–54 (2009)

    Google Scholar 

  4. Chiruzzo, L., Rosá, A.: RETUYT-InCo at TASS 2018: sentiment analysis in Spanish variants using neural networks and SVM. In: Proceedings of TASS 2018: Workshop on Sentiment Analysis at SEPLN, pp. 57–63 (2018)

    Google Scholar 

  5. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018)

    Google Scholar 

  6. Fastai: Fastai, May 2019. https://github.com/fastai/fastai

  7. Fastai: ULMFiT, May 2019. https://github.com/fastai/fastai/tree/master/courses/dl2/imdb_scripts

  8. Garcia, M., Martinez, E., Villena, J., Garcia, J.: Tass 2015 - the evolution of the spanish opinion mining systems. Procesamiento de Lenguaje Natural 56, 33–40 (2016)

    Google Scholar 

  9. Garcia-Cumbreras, M.A., Villena-Roman, J., Martinez-Camara, E., Diaz-Galiano, M., Martin-Valdivia, T., Ureña Lopez, A.: Overview of TASS 2016. In: Proceedings of TASS 2016: Workshop on Sentiment Analysis at SEPLN, pp. 13–21 (2016)

    Google Scholar 

  10. Gonzalez, J.A., Hurtado, L.F., Pla, F.: ELiRF-UPV at TASS 2018: sentiment analysis in twitter based on deep learning. In: Proceedings of TASS 2018: Workshop on Sentiment Analysis at SEPLN, pp. 37–44 (2018)

    Google Scholar 

  11. Graves, A.: Supervised Sequence Labelling with Recurrent Neural Networks. Studies in Computational Intelligence. Springer, Berlin (2012). https://doi.org/10.1007/978-3-642-24797-2. https://cds.cern.ch/record/1503877

    CrossRef  MATH  Google Scholar 

  12. Howard, J., Ruder, S.: Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328–339. Association for Computational Linguistics, Melbourne, July 2018. https://www.aclweb.org/anthology/P18-1031

  13. Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25–29 October 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1746–1751 (2014). http://aclweb.org/anthology/D/D14/D14-1181.pdf

  14. Liu, B.: Sentiment Analysis and Opinion Mining. Morgan and Claypool Publishers, San Rafael (2012)

    CrossRef  Google Scholar 

  15. Martinez-Camara, E., et al.: Overview of TASS 2018: opinions, health and emotions. In: Proceedings of TASS 2018: Workshop on Sentiment Analysis at SEPLN, pp. 13–27 (2018)

    Google Scholar 

  16. Martinez-Camara, E., Diaz-Galiano, M., Garcia-Cumbreras, M.A., Garcia-Vega, M., Villena-Roman, J.: Overview of TASS 2017. In: Proceedings of TASS 2017: Workshop on Sentiment Analysis at SEPLN, pp. 13–21 (2017)

    Google Scholar 

  17. McGlohon, M., Glance, N., Reiter, Z.: Star quality: aggregating reviews to rank products and merchants. In: Proceedings of Fourth International Conference on Weblogs and Social Media (ICWSM) (2010)

    Google Scholar 

  18. Merity, S., Keskar, N.S., Socher, R.: Regularizing and optimizing LSTM language models. CoRR abs/1708.02182 (2017). http://arxiv.org/abs/1708.02182

  19. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 26, pp. 3111–3119. Curran Associates, Inc. (2013). http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

  20. Montañes, R., Aznar, R., del Hoyo, R.: Application of a hybrid deep learning model for sentiment analysis in Twitter. In: Proceedings of TASS 2018: Workshop on Sentiment Analysis at SEPLN, pp. 51–56 (2018)

    Google Scholar 

  21. Ochoa-Luna, J., Ari, D.: Deep neural network approaches for spanish sentiment analysis of short texts. In: Simari, G.R., Fermé, E., Gutiérrez Segura, F., Rodríguez Melquiades, J.A. (eds.) IBERAMIA 2018. LNCS (LNAI), vol. 11238, pp. 430–441. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03928-8_35

    CrossRef  Google Scholar 

  22. Palomino, D.: ULMFit implementation for TASS dataset evaluation, May 2019. https://github.com/dpalominop/ULMFit

  23. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010). https://doi.org/10.1109/TKDE.2009.191

    CrossRef  Google Scholar 

  24. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014). http://www.aclweb.org/anthology/D14-1162

  25. Peters, M., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 2227–2237. Association for Computational Linguistics, New Orleans, Louisiana, June 2018. https://doi.org/10.18653/v1/N18-1202, https://www.aclweb.org/anthology/N18-1202

  26. Rother, K., Rettberg, A.: ULMFiT at GermEval-2018: a deep neural language model for the classification of hate speech in German Tweets. In: Proceedings of the GermEval 2018 Workshop, pp. 113–119 (2018)

    Google Scholar 

  27. Tang, D., Wei, F., Qin, B., Yang, N., Liu, T., Zhou, M.: Sentiment embeddings with applications to sentiment analysis. IEEE Trans. Knowl. Data Eng. 28(2), 496–509 (2016)

    CrossRef  Google Scholar 

  28. Wu, X., Lv, S., Zang, L., Han, J., Hu, S.: Conditional BERT contextual augmentation. CoRR abs/1812.06705 (2018). http://arxiv.org/abs/1812.06705

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Daniel Palomino .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Palomino, D., Ochoa-Luna, J. (2019). Advanced Transfer Learning Approach for Improving Spanish Sentiment Analysis. In: Martínez-Villaseñor, L., Batyrshin, I., Marín-Hernández, A. (eds) Advances in Soft Computing. MICAI 2019. Lecture Notes in Computer Science(), vol 11835. Springer, Cham. https://doi.org/10.1007/978-3-030-33749-0_10

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-33749-0_10

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-33748-3

  • Online ISBN: 978-3-030-33749-0

  • eBook Packages: Computer ScienceComputer Science (R0)