Joint Emoji Classification and Embedding Learning

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10367)

Abstract

Under conversation scenarios, emoji is widely used to express humans’ feelings, which greatly enriches the representation of plain text. Plentiful utterances with emoji are produced by humans manually in social media platforms every day, which make emoji great influence on the human life. For the academic community, researchers are always with the help of utterances including emoji as annotated data to work on sentiment analysis, yet lack of adequate attention to emoji itself. The challenges lie in how to discriminate so many different kinds of emoji, especially for those with similar meanings, which make this problem quite different from traditional sentiment analysis. In this paper, in order to gain an insight into emoji, we propose a matching architecture using deep neural networks to jointly learn emoji embeddings and make classification. In particular, we use a convolutional neural network to get the embedding of the utterance and match it with the embedding of the corresponding emoji, to obtain its best classification, and otherwise also train the emoji embeddings. Experiments based on a massive dataset demonstrate the effectiveness of our proposed approach better than traditional softmax methods in terms of p@1, p@5 and MRR evaluation metrics. Then a test of human experience shows the performance could meet the requirement of practice systems.

Keywords

Emoji classification Embedding learning Deep learning Neural networks 

Notes

Acknowledgements

This paper is partially supported by the National Natural Science Foundation of China (NSFC Grant Nos. 61472006 and 91646202) as well as the National Basic Research Program (973 Program No. 2014CB340405).

References

  1. 1.
    Tang, D., Wei, F., Qin, B., Zhou, M., Liu, T.: Building large-scale twitter-specific sentiment lexicon: a representation learning approach. In: Proceedings of the 25th International Conference on Computational Linguistics, pp. 172–182 (2014)Google Scholar
  2. 2.
  3. 3.
    Pak, A., Paroubek, P.: Twitter as a corpus for sentiment analysis and opinion mining. In: International Conference on Language Resources and Evaluation, pp. 1320–1326 (2010)Google Scholar
  4. 4.
    Yan, J.L.S., Turtle, H.R.: Exploring fine-grained emotion detection in tweets. In: Proceedings of NAACL-HLT, pp. 73–80 (2016)Google Scholar
  5. 5.
    Wu, F., Song, Y., Huang, Y.: Microblog sentiment classification with contextual knowledge regularization. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence, pp. 2332–2338 (2015)Google Scholar
  6. 6.
    Deng, L., Wiebe, J., Choi, Y.: Joint inference and disambiguation of implicit sentiments via implicature constraints. In: Proceedings of the 25th International Conference on Computational Linguistics, pp. 79–88 (2014)Google Scholar
  7. 7.
    Wilson, T., Wiebe, J., Hoffmann, P.: Recognizing contextual polarity in phrase-level sentiment analysis. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 347–354 (2005)Google Scholar
  8. 8.
    Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T., Qin, B.: Learning sentiment-specific word embedding for twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 1555–1565 (2014)Google Scholar
  9. 9.
    Socher, R., Perelygin, A., Wu, J.Y., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 1631–1642 (2013)Google Scholar
  10. 10.
    dos Santos, C.N., Gatti, M.: Deep convolutional neural networks for sentiment analysis of short texts. In: Proceedings of the 25th International Conference on Computational Linguistics, pp. 69–78 (2014)Google Scholar
  11. 11.
    Mou, L., Peng, H., Li, G., Xu, Y., Zhang, L., Jin, Z. Discriminative neural sentence modeling by tree-based convolution. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 2315–2325 (2015)Google Scholar
  12. 12.
    Beck, D., Cohn, T., Specia, L.: Joint emotion analysis via multi-task Gaussian processes. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 1798–1803 (2014)Google Scholar
  13. 13.
    Wang, Z., Lee, S.Y.M., Li, S., Zhou, G.: Emotion detection in code-switching texts via bilingual and sentimental information. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pp. 763–768 (2015)Google Scholar
  14. 14.
    Chang, Y.C., Chen, C.C., Hsieh, Y.L., Chen, C.C., Hsu, W.L.: Linguistic template extraction for recognizing reader-emotion and emotional resonance writing assistance. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pp. 775–780 (2015)Google Scholar
  15. 15.
    Turney, P.D.: Thumbs up or thumbs down?: semantic orientation applied to unsupervised classification of reviews. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 417–424 (2002)Google Scholar
  16. 16.
    Taboada, M., Brooke, J., Tofiloski, M., Voll, K., Stede, M.: Lexicon-based methods for sentiment analysis. Comput. Linguist. 37(2), 267–307 (2011)CrossRefGoogle Scholar
  17. 17.
    Kouloumpis, E., Wilson, T., Moore, J.D.: Twitter sentiment analysis: the good the bad and the omg!. In: Proceedings of the 15th International AAAI Conference on Web and Social Media, pp. 538–541 (2011)Google Scholar
  18. 18.
    Lambert, P.: Aspect-level cross-lingual sentiment classification with constrained SMT. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pp. 781–787 (2015)Google Scholar
  19. 19.
    Wang, X., Wei, F., Liu, X., Zhou, M., Zhang, M.: Topic sentiment analysis in twitter: a graph-based hashtag sentiment classification approach. Proceedings of the 20th ACM International Conference on Information and Knowledge Management, pp. 1031–1040 (2011)Google Scholar
  20. 20.
    Yang, M., Peng, B., Chen, Z., Zhu, D., Chow, K.P.: A topic model for building fine-grained domain-specific emotion lexicon. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 421–426 (2014)Google Scholar
  21. 21.
    Zhou, H., Chen, L., Shi, F., Huang, D.: Learning bilingual sentiment word embeddings for cross-language sentiment classification. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pp. 430–440 (2015)Google Scholar
  22. 22.
    Dong, L., Wei, F., Zhou, M., Xu, K.: Adaptive multi-compositionality for recursive neural models with applications to sentiment analysis. In: Proceedings of the 28th AAAI Conference on Artificial Intelligence, pp. 1537–1543 (2014)Google Scholar
  23. 23.
    Ren, Y., Zhang, Y., Zhang, M., Ji, D.: Improving twitter sentiment classification using topic-enriched multi-prototype word embeddings. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, pp. 3038–3044 (2016)Google Scholar
  24. 24.
    Zhang, M., Zhang, Y., Vo, D.T.: Gated neural networks for targeted sentiment analysis. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, pp. 3087–3093 (2016)Google Scholar
  25. 25.
    Vanzo, A., Croce, D., Basili, R.: A context-based model for sentiment analysis in twitter. In: Proceedings of the 25th International Conference on Computational Linguistics, pp. 2345–2354 (2014)Google Scholar
  26. 26.
    Ren, Y., Zhang, Y., Zhang, M., Ji, D.: Context-sensitive twitter sentiment classification using neural network. In: Proceedings of the 30th AAAI Conference on Artificial Intelligence, pp. 215–221 (2016)Google Scholar
  27. 27.
    Li, S., Huang, L., Wang, R., Zhou, G.: Sentence-level emotion classification with label and context dependence. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pp. 1045–1053 (2015)Google Scholar
  28. 28.
    Cappallo, S., Mensink, T., Snoek, C.G.: Image2emoji: Zero-shot emoji prediction for visual media. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 1311–1314 (2015)Google Scholar
  29. 29.
    Hu, B., Lu, Z., Li, H., Chen, Q.: Convolutional neural network architectures for matching natural language sentences. In: Annual Conference on Neural Information Processing Systems, pp. 2042–2050 (2014)Google Scholar
  30. 30.
    Yan, R., Song, Y., Wu, H.: Learning to Respond with Deep Neural Networks for Retrieval based Human-Computer Conversation System. In: Proceedings of SIGIR, pp. 55–64 (2016)Google Scholar
  31. 31.
    Socher, R., Huang, E.H., Pennin, J., Ng, A.Y., Manning, C.D.: Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In: Annual Conference on Neural Information Processing Systems, pp. 801–809 (2011)Google Scholar
  32. 32.
    Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, pp. 1724–1734 (2014)Google Scholar
  33. 33.
    Tang, Y.: Deep learning using linear support vector machines. arXiv preprint arXiv:1306.0239 (2013)
  34. 34.
    Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychol. Bull. 76(5), 378 (1971)CrossRefGoogle Scholar
  35. 35.
    Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3(Feb), 1137–1155 (2003)MATHGoogle Scholar
  36. 36.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)Google Scholar
  37. 37.
    Novak, P.K., Smailović, J., Sluban, B., Mozetič, I.: Sentiment of emojis. PloS One 10(12), e0144296 (2015)CrossRefGoogle Scholar
  38. 38.
    Babu, G.S., Zhao, P., Li, X.L.: Deep convolutional neural network based regression approach for estimation of remaining useful life. In: Proceedings of the 21st International Conference on Database Systems for Advanced Applications, pp. 214–228 (2016)Google Scholar
  39. 39.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)Google Scholar
  40. 40.
    Shang, L., Lu, Z., Li, H.: Neural responding machine for short-text conversation. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics, pp. 1577–1586 (2015)Google Scholar
  41. 41.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  42. 42.
    Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.Y., Gao, J., Dolan, B.: A neural network approach to context-sensitive generation of conversational responses. In: Proceedings of NAACL-HLT, pp. 196–205 (2015)Google Scholar
  43. 43.
    Mou, L., Song, Y., Yan, R., Li, G., Zhang, L., Jin, Z.: Sequence to backward and forward sequences: a content-introducing approach to generative short-text conversation. In: Proceedings of the 26th International Conference on Computational Linguistics, pp. 3349–3358 (2016)Google Scholar
  44. 44.
    Hogenboom, A., Bal, D., Frasincar, F., Bal, M., de Jong, F., Kaymak, U.: Exploiting emoticons in polarity classification of text. J. Web Eng. 14, 22–40 (2015)Google Scholar
  45. 45.
    Wang, Y., Feng, S., Wang, D., Yu, G., Zhang, Y.: Multi-label chinese microblog emotion classification via convolutional neural network. In: Li, F., Shim, K., Zheng, K., Liu, G. (eds.) APWeb 2016. LNCS, vol. 9931, pp. 567–580. Springer, Cham (2016). doi:10.1007/978-3-319-45814-4_46 CrossRefGoogle Scholar
  46. 46.
    Zhao, Z., Liu, T., Hou, X., Li, B., Du, X.: Distributed text representation with weighting scheme guidance for sentiment analysis. In: Li, F., Shim, K., Zheng, K., Liu, G. (eds.) APWeb 2016. LNCS, vol. 9931, pp. 41–52. Springer, Cham (2016). doi:10.1007/978-3-319-45814-4_4 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.School of EECSPeking UniversityBeijingChina
  2. 2.Institute of Computer Science and TechnologyPeking UniversityBeijingChina
  3. 3.Beijing Institute of Big Data ResearchBeijingChina

Personalised recommendations