Skip to main content
SpringerLink
Log in

Transfer learning techniques for emotion classification on visual features of images in the deep learning network

  • Published:
International Journal of Speech Technology Aims and scope Submit manuscript

Abstract

Emotion is subjective which convey rich semantics based on an image that induces different emotion based on each individual. A novel method is proposed for emotion classification by using deep learning network with transfer learning method. Transfer learning techniques are the predictive model that reuses the model trained on related predictive problems. The purpose of the proposed work is to classify the emotion perception from images based on visual features. Image augmentation and segmentation is performed to build powerful classifier. The performance of deep convolution neural network (CNN) is improved with transfer learning techniques in large scale Image-Emotion-dataset effectively. The experiments conducted on this dataset and result shows that proposed method achieve promising significant effect on emotion classification with good accuracy and PDA value, when compared with other state-of-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  • Barros, P., Parisi, G. I., & Wermter, S. (2019). A personalized affective memory neural model for improving emotion recognition. arXiv preprint arXiv:1904.12632.

  • Baziotis, C., Athanasiou, N., Chronopoulou, A., Kolovou, A., Paraskevopoulos, G., Ellinas, N., & Potamianos, A. (2018). NTUA-SLP at SemEval-2018 task 1: Predicting affective content in tweets with deep attentive RNNs and transfer learning. arXiv preprint arXiv:1804.06658.

  • Chen, M., Hao, Y., Li, Y., Wu, D., & Huang, D. (2015). LIVES: Learning through interactive video and emotion-aware system. In Proceedings of the 16th ACM international symposium on mobile ad hoc networking and computing, June 2015 (pp. 399–400). ACM.

  • Corchs, S., Fersini, E., & Gasparini, F. (2017). Ensemble learning on visual and textual data for social image emotion classification. International Journal of Machine Learning and Cybernetics, 10(8), 2057–2070.

    Article  Google Scholar 

  • Deng, J., Xia, R., Zhang, Z., Liu, Y., & Schuller, B. (2014). Introducing shared-hidden-layer autoencoders for transfer learning and their application in acoustic emotion recognition. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), May 2014 (pp. 4818–4822). IEEE.

  • Dong, X. L., & De Melo, G. (2018). A helping hand: Transfer learning for deep sentiment analysis. In Proceedings of the 56th annual meeting of the Association for Computational Linguistics, July 2018 (Vol. 1: Long Papers, pp. 2524–2534).

  • Felbo, B., Mislove, A., Sogaard, A., Rahwan, I., & Lehmann, S. (2017). Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. arXiv preprint arXiv:1708.00524.

  • Fong, B., & Westerink, J. (2012). Affective computing in consumer electronics. IEEE Transactions on Affective Computing,3(2), 129–131.

    Article  Google Scholar 

  • Gideon, J., Khorram, S., Aldeneh, Z., Dimitriadis, D., & Provost, E. M. (2017). Progressive neural networks for transfer learning in emotion recognition. arXiv preprint arXiv:1706.03256.

  • Haggag, M., Fathy, S., & Elhaggar, N. (2015). Ontology-based textual emotion detection. International Journal of Advanced Computer Science and Applications,6(9), 239–245.

    Article  Google Scholar 

  • Han, J., Zhang, Z., Schmitt, M., Pantic, M., & Schuller, B. (2017). From hard to soft: Towards more human-like emotion recognition by modelling the perception uncertainty. In Proceedings of the 25th ACM international conference on multimedia, October 2017 (pp. 890–897). ACM.

  • He, Y., & Ding, G. (2019). Deep transfer learning for image emotion analysis: Reducing marginal and joint distribution discrepancies together. Neural Processing Letters. https://doi.org/10.1007/s11063-019-10035-7.

    Article  Google Scholar 

  • Huang, Z., Pan, Z., & Lei, B. (2017). Transfer learning with deep convolutional neural network for SAR target classification with limited labelled data. Remote Sensing,9(9), 907.

    Article  Google Scholar 

  • Islam, J., & Zhang, Y. (2016). Visual sentiment analysis for social images using transfer learning approach. In 2016 IEEE international conferences on big data and cloud computing (BDCloud), social computing and networking (SocialCom), sustainable computing and communications (SustainCom) (BDCloud_SocialComSustainCom), October 2016 (pp. 124–130). IEEE.

  • Liu, X., Li, N., & Xia, Y. (2019). Affective image classification by jointly using interpretable art features and semantic annotations. Journal of Visual Communication and Image Representation,58, 576–588.

    Article  Google Scholar 

  • Ng, H. W., Nguyen, V. D., Vonikakis, V., & Winkler, S. (2015). Deep learning for emotion recognition on small datasets using transfer learning. In Proceedings of the 2015 ACM on international conference on multimodal interaction, November 2015 (pp. 443–449). ACM.

  • Nguyen Tien, D., Kien, N., Sridharan, S., Abbasnejad, I., Dean, D., & Fookes, C. (2018). Meta transfer learning for facial emotion recognition.

  • Ouyang, X., Kawaai, S., Goh, E. G. H., Shen, S., Ding, W., Ming, H., & Huang, D. Y. (2017). Audio–visual emotion recognition using deep transfer learning and multiple temporal models. In Proceedings of the 19th ACM international conference on multimodal interaction, November 2017 (pp. 577–582). ACM.

  • Pan, S. J., & Yang, Q. (2009). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering,22(10), 1345–1359.

    Article  Google Scholar 

  • Parthasarathy, S., & Busso, C. (2017). Jointly predicting arousal, valence and dominance with multi-task learning. In INTERSPEECH, August 2017 (pp. 1103–1107).

  • Pavlinek, M., & Podgorelec, V. (2017). Text classification method based on self-training and LDA topic models. Expert Systems with Applications,80, 83–93.

    Article  Google Scholar 

  • Peng, K. C., Chen, T., Sadovnik, A., & Gallagher, A. C. (2015). A mixed bag of emotions: Model, predict, and transfer emotion distributions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 860–868).

  • Peng, K. C., Chen, T., Sadovnik, A., & Gallagher, A. A mixed bag of emotions: Model, predict, and transfer emotion distributions supplementary material.

  • Rao, Y., Xie, H., Li, J., Jin, F., Wang, F. L., & Li, Q. (2016). Social emotion classification of short text via topic-level maximum entropy model. Information and Management,53(8), 978–986.

    Article  Google Scholar 

  • Rassadin, A., Gruzdev, A., & Savchenko, A. (2017). Group-level emotion recognition using transfer learning from face identification. In Proceedings of the 19th ACM international conference on multimodal interaction, November 2017 (pp. 544–548). ACM.

  • Shoumy, N. J., Ang, L. M., & Rahaman, D. M. (2019). Multimodal big data affective analytics. Multimodal analytics for next-generation big data technologies and applications (pp. 45–71). Cham: Springer.

    Chapter  Google Scholar 

  • Tamil Priya, D., & Divya Udayan, J. (2019a). A comprehensive survey on various semantic based video/image retrieval techniques. International Journal of Innovative Technology and Exploring Engineering,8(6), 185–196.

    Google Scholar 

  • Tamil Priya, D., & Divya Udayan, J. (2019b). A semantic classification of images by predicting emotional concepts from visual features. TEST Engineering and Management,59(6), 42.

    Google Scholar 

  • Vadicamo, L., Carrara, F., Cimino, A., Cresci, S., Dell’Orletta, F., Falchi, F., & Tesconi, M. (2017). Cross-media learning for image sentiment analysis in the wild. In Proceedings of the IEEE international conference on computer vision (pp. 308–317).

  • Wang, X., Jia, J., Tang, J., Wu, B., Cai, L., & Xie, L. (2015). Modeling emotion influence in image social networks. IEEE Transactions on Affective Computing,6(3), 286–297.

    Article  Google Scholar 

  • Yang, Y., Jia, J., Wu, B., & Tang, J. (2016). Social role-aware emotion contagion in image social networks. In Thirtieth AAAI conference on artificial intelligence, February 2016.

  • Yang, J., She, D., & Sun, M. (2017). Joint image emotion classification and distribution learning via deep convolutional neural network. In IJCAI (pp. 3266–3272).

  • Yang, J., She, D., Sun, M., Cheng, M. M., Rosin, P. L., & Wang, L. (2018). Visual sentiment prediction based on automatic discovery of affective regions. IEEE Transactions on Multimedia, 20(9), 2513–2525.

    Article  Google Scholar 

  • You, Q., Luo, J., Jin, H., & Yang, J. (2015). Robust image sentiment analysis using progressively trained and domain transferred deep networks. In Twenty-ninth AAAI conference on artificial intelligence, February 2015.

  • You, Q., Luo, J., Jin, H., & Yang, J. (2016). Building a large scale dataset for image emotion recognition: The fine print and the benchmark. In Thirtieth AAAI conference on artificial intelligence, February 2016.

  • Zhao, S., Yao, H., Gao, Y., Ding, G., & Chua, T. S. (2016a). Predicting personalized image emotion perceptions in social networks. IEEE Transactions on Affective Computing,9(4), 526–540.

    Article  Google Scholar 

  • Zhao, S., Yao, H., Gao, Y., Ji, R., Xie, W., Jiang, X., & Chua, T. S. (2016b). Predicting personalized emotion perceptions of social images. In Proceedings of the 24th ACM international conference on multimedia, October 2016 (pp. 1385–1394). ACM.

  • Zhao, S., Lin, C., Xu, P., Zhao, S., Guo, Y., Krishna, R., Ding, G., & Keutzer, K. (2019). CycleEmotionGAN: Emotional semantic consistency preserved cycleGAN for adapting image emotions. In AAAI conference on artificial intelligence (pp. 2620–2627).

  • Zhu, X., Liu, Y., Qin, Z., & Li, J. (2017). Data augmentation in emotion classification using generative adversarial networks. arXiv preprint arXiv:1711.00648.

  • Zhu, X., Liu, Y., Li, J., Wan, T., & Qin, Z. (2018). Emotion classification with data augmentation using generative adversarial networks. In Pacific–Asia conference on knowledge discovery and data mining, June 2018 (pp. 349–360). Cham: Springer.

Download references

Acknowledgements

The authors thank Vellore Institute of Technology for providing “VIT SEED GRANT” for carrying out this research work.

Author information

Authors and Affiliations

  1. School of Information Technology and Engineering, Vellore Institute of Technology, Vellore, Tamil Nadu, India

    D. Tamil Priya & J. Divya Udayan

Authors
  1. D. Tamil Priya
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. J. Divya Udayan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to J. Divya Udayan.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tamil Priya, D., Divya Udayan, J. Transfer learning techniques for emotion classification on visual features of images in the deep learning network. Int J Speech Technol 23, 361–372 (2020). https://doi.org/10.1007/s10772-020-09707-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10772-020-09707-w

Keywords

Search

Navigation