Skip to main content
SpringerLink
Log in

Deep learning neural networks for emotion classification from text: enhanced leaky rectified linear unit activation and weighted loss

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Accurate emotion classification for online reviews is vital for business organizations to gain deeper insights into markets. Although deep learning has been successfully implemented in this area, accuracy and processing time are still major problems preventing it from reaching its full potential. This paper proposes an Enhanced Leaky Rectified Linear Unit activation and Weighted Loss (ELReLUWL) algorithm for enhanced text emotion classification and faster parameter convergence speed. This algorithm includes the definition of the inflection point and the slope for inputs on the left side of the inflection point to avoid gradient saturation. It also considers the weight of samples belonging to each class to compensate for the influence of data imbalance. Convolutional Neural Network (CNN) combined with the proposed algorithm to increase the classification accuracy and decrease the processing time by eliminating the gradient saturation problem and minimizing the negative effect of data imbalance, demonstrated on a binary sentiment problem. All work was carried out using supervised deep learning. The results for accuracy and processing time are obtained by using different datasets and different review types. The results show that the proposed solution achieves better classification performance in different data scenarios and different review types. The proposed model takes less convergence time to achieve model optimization with seven epochs against the current convergence time of 11.5 epochs on average. The proposed solution improves accuracy and reduces the processing time of text emotion classification. The solution provides an average class accuracy of 96.63% against a current average accuracy of 91.56%. It also provides a processing time of 23.3 milliseconds compared to the current average processing time of 33.2 milliseconds. Finally, this study solves the issues of gradient saturation and data imbalance. It enhances overall average class accuracy and decreases processing time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Bengio Y (2000) Gradient-based optimization of Hyperparameters. Neural Comput 12(8):1889–1900. https://doi.org/10.1162/089976600300015187

    Article  MathSciNet  Google Scholar 

  2. Bose R (2011) Discovering business intelligence from the subjective web data. Int J Bus Intell Res 2:1–16

    Article  Google Scholar 

  3. Chatterjee A, Gupta U, Chinnakotla MK, Srikanth R, Galley M, Agrawal P (2019) Understanding emotions in text using deep learning and big data. Comput Hum Behav 93:309–317. https://doi.org/10.1016/j.chb.2018.12.029

    Article  Google Scholar 

  4. Chen T, Xu R, He Y, Wang X (2017) Improving sentiment analysis via sentence type classification using BiLSTM-CRF and CNN. Expert Syst Appl 72(C):221–230. https://doi.org/10.1016/j.eswa.2016.10.065

    Article  Google Scholar 

  5. Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P (2011) Natural Language Processing (Almost) from Scratch. J. Mach. Learn. Res. 12:2493–2537

    MATH  Google Scholar 

  6. Colnerič N, Demšar J (2020) Emotion recognition on twitter: comparative study and training a unison model. IEEE Trans Affect Comput 11(3):433–446. https://doi.org/10.1109/TAFFC.2018.2807817

    Article  Google Scholar 

  7. Dhaoui C, Webster Cynthia M, Tan Lay P (2017) Social media sentiment analysis: lexicon versus machine learning. J Consum Mark 34(6):480–488. https://doi.org/10.1108/JCM-03-2017-2141

    Article  Google Scholar 

  8. Gu X, Gu Y, Wu H (2017) Cascaded convolutional neural networks for aspect-based opinion summary. Neural Process Lett 46(2):581–594. https://doi.org/10.1007/s11063-017-9605-7

    Article  Google Scholar 

  9. Hanafy M, Khalil MI, and Abbas HM (2018) "Combining Classical and Deep Learning Methods for Twitter Sentiment Analysis," in Artificial Neural Networks in Pattern Recognition, Cham, L. Pancioni, F. Schwenker, and E. Trentin, Eds. Springer International Publishing, pp. 281–292.

  10. Hara K, Saitoh D, and Shouno H (2015) "Analysis of function of rectified linear unit used in deep learning. [Online]. Available: https://doi.org/10.1109/IJCNN.2015.7280578.

  11. Hinton G, Srivastava N, Krizhevsky A, Sutskever I and Salakhutdinov R (2012) Improving neural networks by preventing co-adaptation of feature detectors. arXiv.org

  12. IMDB Dataset (n.d.) [Online]. Available: https://www.kaggle.com/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews

  13. Jing Y, Guanci Y (2018) Modified Convolutional Neural Network Based on Dropout and the Stochastic Gradient Descent Optimizer. Algorithms 11(3):28. https://doi.org/10.3390/a11030028

    Article  MathSciNet  Google Scholar 

  14. Kalchbrenner N, Grefenstette E, Blunsom P (2014) A convolutional neural network for modelling sentences

  15. Kiritchenko S, Zhu X, Mohammad SM (2014) Sentiment analysis of short informal texts. J Artif Int Res 50(1):723–762

    Google Scholar 

  16. Kratzwald B, Ilic S, Kraus M, Feuerriegel S, and Prendinger H (2018) Deep learning for affective computing: text-based emotion recognition in decision support. arXiv.org. 115. https://doi.org/10.1016/j.dss.2018.09.002.

  17. Kratzwald B, Ilić S, Kraus M, Feuerriegel S, Prendinger H (2018) Deep learning for affective computing: Text-based emotion recognition in decision support. Decis Supp Syst 115:24–35. https://doi.org/10.1016/j.dss.2018.09.002

    Article  Google Scholar 

  18. Kraus M, Feuerriegel S (2017) Decision support from financial disclosures with deep neural networks and transfer learning. Decis Support Syst 104:38–48

    Article  Google Scholar 

  19. Lin D, Li L, Cao D, Lv Y, Ke X (2018) Multi-modality weakly labeled sentiment learning based on explicit emotion signal for Chinese microblog. Neurocomputing 272:258–269. https://doi.org/10.1016/j.neucom.2017.06.078

    Article  Google Scholar 

  20. Liu B (2020) Text sentiment analysis based on CBOW model and deep learning in big data environment. J Ambient Intell Human Comput 11(2):451–458. https://doi.org/10.1007/s12652-018-1095-6

    Article  Google Scholar 

  21. Macêdo D, Zanchettin C, Oliveira ALI, Ludermir T (2019) Enhancing batch normalized convolutional networks using displaced rectifier linear units: a systematic comparative study. Expert Syst Appl 124:271–281. https://doi.org/10.1016/j.eswa.2019.01.066

    Article  Google Scholar 

  22. Mahmoudi N, Docherty P, Moscato P (2018) Deep neural networks understand investors better. Decis Support Syst 112:23–34

    Article  Google Scholar 

  23. Minglei L, Qin L, Yunfei L, Lin G (2017) Inferring affective meanings of words from word embedding. T-AFFC 8(4):443–456. https://doi.org/10.1109/TAFFC.2017.2723012

    Article  Google Scholar 

  24. Mohammad SM, Kiritchenko S, Zhu X (2013) "NRC-Canada : building the state-of-the-art in sentiment analysis of tweets," presented at the proceedings of the seventh international workshop on semantic evaluation exercises, 2013/07/01

  25. Mou L et al. (2016) "How Transferable are Neural Networks in NLP Applications?," arXiv.org

  26. Mundra S, Sen A, Sinha M, Mannarswamy S, Dandapat S, and Roy S (2017) "Fine-Grained Emotion Detection in Contact Center Chat Utterances," in Advances in Knowledge Discovery and Data Mining, Cham, J. Kim, K. Shim, L. Cao, J.-G. Lee, X. Lin, and Y.-S. Moon, Eds.: Springer International Publishing, pp. 337–349

  27. Pang B, Lee L (2004) "A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts," presented at the Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics, Barcelona, Spain Online Available. https://doi.org/10.3115/1218955.1218990

  28. Phan D-A, Matsumoto Y, Shindo H (2018) Autoencoder for Semisupervised multiple emotion detection of conversation transcripts. T-AFFC 12:1–691. https://doi.org/10.1109/TAFFC.2018.2885304

    Article  Google Scholar 

  29. Phillips P, Sui Y, Liu B, Yang M, Cheng H (2018) Classification of Alzheimer’s disease based on eight-layer convolutional neural network with leaky rectified linear unit and max pooling. J Med Syst 42(5):1–11. https://doi.org/10.1007/s10916-018-0932-7

    Article  Google Scholar 

  30. Poernomo A, Kang D-K (2018) Biased dropout and Crossmap dropout: learning towards effective dropout regularization in convolutional neural network. Neural Netw 104:60–67. https://doi.org/10.1016/j.neunet.2018.03.016

    Article  Google Scholar 

  31. Qian S, Liu H, Liu C, Wu S, Wong HS (2018) Adaptive activation functions in convolutional neural networks. Neurocomputing 272:204–212. https://doi.org/10.1016/j.neucom.2017.06.070

    Article  Google Scholar 

  32. Randhawa S, Alsadoon A, Prasad PWC, Al-Dala’in T, Dawoud A, Alrubaie A (2021) Deep learning for liver tumour classification: enhanced loss function. Multimedia Tools Appl 80(3):4729–4750. https://doi.org/10.1007/s11042-020-09900-8

    Article  Google Scholar 

  33. Rao G, Huang W, Feng Z, Cong Q (2018) LSTM with sentence representations for document-level sentiment classification. Neurocomputing 308:49–57. https://doi.org/10.1016/j.neucom.2018.04.045

    Article  Google Scholar 

  34. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958

    MathSciNet  MATH  Google Scholar 

  35. Steno P, Alsadoon A, Prasad PWC, Al-Dala’in T, Alsadoon OH (2020) A novel enhanced region proposal network and modified loss function: threat object detection in secure screening using deep learning. J Supercomput. https://doi.org/10.1007/s11227-020-03418-4

  36. Stojanovski D, Strezoski G, Madjarov G, Dimitrovski I, Chorbev I (2018) Deep neural network architecture for sentiment analysis and emotion identification of twitter messages. Multimed Tools Appl 77(24):32213–32242. https://doi.org/10.1007/s11042-018-6168-1

    Article  Google Scholar 

  37. Sun X (2020) A novel approach to generate a large scale of supervised data for short text sentiment analysis. Multimed Tools Appl 79(9–10):5439–5459. https://doi.org/10.1007/s11042-018-5748-4

    Article  Google Scholar 

  38. Sun X, Zhang C (2020) Detecting anomalous emotion through big data from social networks based on a deep learning method. Multimed Tools Appl 79(13–14):9687–9687. https://doi.org/10.1007/s11042-018-5665-6

    Article  Google Scholar 

  39. Sun X, Li C, Ren F (2016) Sentiment analysis for Chinese microblog based on deep neural networks with convolutional extension features. Neurocomputing 210:227–236. https://doi.org/10.1016/j.neucom.2016.02.077

    Article  Google Scholar 

  40. Tang D, Qin B, Feng X, Liu T (2015) Effective LSTMs for target-dependent sentiment classification

  41. Tang Y (2015) "Deep Learning using Linear Support Vector Machines," arXiv.org

  42. Vinay Kumar J, Shishir K, Prabhat M (2018) Sentiment recognition in customer reviews using deep learning. Int J Enterprise Inform Syst (IJEIS) 14(2):77–86. https://doi.org/10.4018/IJEIS.2018040105

    Article  Google Scholar 

  43. Wu H, Gu X, Gu Y (2017) Balancing between over-weighting and under-weighting in supervised term weighting. Inf Process Manage 53(2):547–557. https://doi.org/10.1016/j.ipm.2016.10.003

    Article  Google Scholar 

  44. Xiong S, Lv H, Zhao W, Ji D (2018) Towards twitter sentiment classification by multi-level sentiment-enriched word embeddings. Neurocomputing 275:2459–2466. https://doi.org/10.1016/j.neucom.2017.11.023

    Article  Google Scholar 

  45. Xu R, Chen T, Xia Y, Lu Q, Liu B, Wang X (2015) Word Embedding Composition for Data Imbalances in Sentiment and Emotion Classification. Cogn Comput 7(2):226–240. https://doi.org/10.1007/s12559-015-9319-y

    Article  Google Scholar 

  46. Yoo S, Song J, Jeong O (2018) Social media contents based sentiment analysis and prediction system. Expert Syst Appl 105:102–111. https://doi.org/10.1016/j.eswa.2018.03.055

    Article  Google Scholar 

  47. Yoon K (2014) "convolutional neural networks for sentence classification," ed. Cornell University Library, arXiv.org, Ithaca

  48. Yu L, Zhang W, Wang J, Yu Y (2016) SeqGAN: sequence generative adversarial nets with policy gradient

    Google Scholar 

  49. Zhang Y and Wallace B (2016) A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification. arXiv.org.

  50. Zhang Y, Zhang Z, Miao D, Wang J (2019) Three-way enhanced convolutional neural networks for sentence-level sentiment classification. Inf Sci 477:55–64. https://doi.org/10.1016/j.ins.2018.10.030

    Article  Google Scholar 

  51. Zhang Z, Zou Y, Gan C (2018) Textual sentiment analysis via three different attention convolutional neural networks and cross-modality consistent regression. Neurocomputing 275:1407–1415. https://doi.org/10.1016/j.neucom.2017.09.080

    Article  Google Scholar 

  52. Zhao H, Liu F, Li L, Luo C (2018) A novel softplus linear unit for deep convolutional neural networks. Appl Intell 48(7):1707–1720. https://doi.org/10.1007/s10489-017-1028-7

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computing Mathematics and Engineering, Charles Sturt University (CSU), Sydney, Australia

    Hui Yang, Abeer Alsadoon, P. W. C. Prasad, Thair Al-Dala’in & Angelika Maag

  2. School of Computer Data and Mathematical Sciences, Western Sydney University (WSU), Sydney, Australia

    Abeer Alsadoon, P. W. C. Prasad & Thair Al-Dala’in

  3. Kent Institute Australia, Sydney, Australia

    Abeer Alsadoon, P. W. C. Prasad & Thair Al-Dala’in

  4. Asia Pacific International College (APIC),, Sydney, Australia

    Abeer Alsadoon & Thair Al-Dala’in

  5. Computer Science and Engineering, University of Kurdistan Hewler, Erbil, KRG, Iraq

    Tarik A. Rashid

  6. Department of Islamic Sciences, Al Iraqia University, Baghdad, Iraq

    Omar Hisham Alsadoon

Authors
  1. Hui Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abeer Alsadoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. P. W. C. Prasad
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thair Al-Dala’in
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tarik A. Rashid
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Angelika Maag
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Omar Hisham Alsadoon
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abeer Alsadoon.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, H., Alsadoon, A., Prasad, P.W.C. et al. Deep learning neural networks for emotion classification from text: enhanced leaky rectified linear unit activation and weighted loss. Multimed Tools Appl 81, 15439–15468 (2022). https://doi.org/10.1007/s11042-022-12629-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12629-1

Keywords

Search

Navigation