Skip to main content

Novel OGBEE-based feature selection and feature-level fusion with MLP neural network for social media multimodal sentiment analysis

Abstract

Numerous public networks, namely Instagram, YouTube, Facebook, Twitter, etc., share their own feelings and idea as videotapes, posts, and pictures. In future research, adapting to such data and mining valuable information from it will be an undeniably troublesome errand. This paper proposes a novel audio–video–textual-based multimodal sentiment analysis approach. The proposed approach investigates the sentiments that are collected from the web recordings that utilize audio, video, and textual modalities for further extraction. A feature-level fusion technique is employed in fusing the extracted features from different modalities. Therefore, the extracted features are optimally chosen by using a novel oppositional grass bee optimization (OGBEE) algorithm to obtain the best optimal feature set. Here, 12 benchmark functions are developed to validate the numerical efficiency and the effectiveness of a novel OGBEE algorithm for various aspects. Moreover, our proposed approach utilizes multilayer perceptron-based neural network (MLP-NN) for sentiment classification. The experimental analysis reveals that the proposed approach provides better classification accuracy of about 95.2% with less computational time.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

References

  1. Bhattarai B, Lee J (2019) Automatic music mood detection using transfer learning and multilayer perceptron. Int J Fuzzy Logic Intell Syst 19(2):88–96

    Article  Google Scholar 

  2. Cai H, Zhidiao Q, Li Z, Zhang Y, Xiping H, Bin H (2020) Feature-level fusion approaches based on multi-modal EEG data for depression recognition. Inf Fusion 59:127–138

    Article  Google Scholar 

  3. Cambria D, Feraco B (2017) A practical guide to sentiment analysis. Springer, Cham

    Book  Google Scholar 

  4. Chaturvedi I, Satapathy R, Cavallari S, Cambria E (2019) Fuzzy commonsense reasoning for multi-modal sentiment analysis. Pattern Recognit Lett 125:264–270

    Article  Google Scholar 

  5. Chen F, Ji R, Jinsong S, Cao D, Gao Y (2017) Predicting microblog sentiments via weakly supervised multi-modal deep learning. IEEE Trans Multimedia 20(4):997–1007

    Article  Google Scholar 

  6. Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the international conference on Multimedia, ACM, pp 1459–1462

  7. Gaidhane PJ, Nigam MJ (2018) A hybrid grey wolf optimizer and artificial bee colony algorithm for enhancing the performance of complex systems. J Comput Sci 27:284–302

    Article  Google Scholar 

  8. Gupta Roy R, Ghoshal D (2019a) Adaptive second-order sliding-mode controller for shank-foot orthosis system. Int J Control 92(7):1580–1589

    MathSciNet  Article  Google Scholar 

  9. Gupta Roy R, Ghoshal D (2019b) Grey wolf optimization-based second order sliding mode control for inchworm robot, Robotica. Cambridge University Press, Cambridge

    Google Scholar 

  10. Huang F, Zhang X, Zhao Z, Xu J, Li Z (2019) Image–text sentiment analysis via deep multi-modal attentive fusion. Knowl-Based Syst 167:26–37

    Article  Google Scholar 

  11. Huddar MG, Sannakki SS, Rajpurohit VS (2020) Multi-level feature optimization and multi-modal contextual fusion for sentiment analysis and emotion classification. Comput Intell 36(2):861–881

    MathSciNet  Article  Google Scholar 

  12. Intisar O, Hussien Y, Hasan J (2018) Multimodal sentiment analysis: a comparison study. J Comput Sci 14(6):804–818

    Article  Google Scholar 

  13. Ji R, Chen F, Cao L, Gao Y (2018) Cross-modality microblog sentiment prediction via bi-layer multi-modal hypergraph learning. IEEE Trans Multimedia 21(4):1062–1075

    Article  Google Scholar 

  14. Karaboga D, Akay B (2009) Artificial bee colony (ABC), harmony search and bees algorithms on numerical optimization. In: Innovative production machines and systems virtual conference

  15. Karaboga D, Akay B (2009b) A comparative study of artificial bee colony algorithm. Appl Math Comput 214(1):108–132

    MathSciNet  MATH  Google Scholar 

  16. Kaushik L, Abhijeet S, John HLH (2013) Automatic sentiment extraction from YouTube videos. In: 2013 IEEE workshop on automatic speech recognition and understanding, IEEE, pp 239–244

  17. Khan A, Ko DK, Lim SC, Kim HS (2019) Structural vibration-based classification and prediction of delamination in smart composite laminates using deep learning neural network. Compos Part B-Eng 161:586–594

    Article  Google Scholar 

  18. Kim B, Lee J (2018) A deep-learning based model for emotional evaluation of video clips. Int J Fuzzy Logic Intell Syst 18(4):245–253

    MathSciNet  Article  Google Scholar 

  19. Kim N-R, Kim K, Lee J-H (2017) Sentiment analysis in microblogs using HMMs with syntactic and sentimental information. Int J Fuzzy Logic Intell Syst 17(4):329–336

    MathSciNet  Article  Google Scholar 

  20. Luo L, Zhang B, Zhang G, Yue X (2020) Rapid prediction of cured shape types of composite laminates using a FEM-ANN method. Compos Struct 238:111980

    Article  Google Scholar 

  21. Majumder N, Hazarika D, Gelbukh A, Cambria E, Poria S (2018) Multi-modal sentiment analysis using hierarchical fusion with context modelling. Knowl-Based Syst 161:124–133

    Article  Google Scholar 

  22. Poria S, Cambria E, Howard N, Huang G-B, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multi-modal content. Neurocomputing 174:50–59

    Article  Google Scholar 

  23. Poria S, Peng H, Hussain A, Howard N, Cambria E (2017) Ensemble application of convolutional neural networks and multiple kernel learning for multi-modal sentiment analysis. Neurocomputing 261:217–230

    Article  Google Scholar 

  24. Poria M, Hazarika C, Gelbukh A, Hussain A (2018) Multi-modal sentiment analysis: addressing key issues and setting up the baselines. IEEE Intell Syst 33(6):17–25

    Article  Google Scholar 

  25. Radhakrishnan V, Joseph C (2018) Sentiment extraction from naturalistic video. Procedia Comput Sci 143:626–634

    Article  Google Scholar 

  26. Rejeesh MR (2019) Interest point based face recognition using adaptive neuro fuzzy inference system. Multimed Tools Appl 78(16):22691–22710

    Article  Google Scholar 

  27. Rogers SM, Matheson T, Despland E, Dodgson T, Burrows M, Simpson SJ (2003) Mechanosensory-induced behavioural gregarization in the desert locust Schistocerca gregaria. J Exp Biol 206:3991–4002

    Article  Google Scholar 

  28. Rosas VP, Mihalcea R, Morency L-P (2013) Multi-modal sentiment analysis of Spanish online videos. IEEE Intell Syst 28(3):38–45

    Article  Google Scholar 

  29. Saremi S, Mirjalili S, Lewis A (2017) Grasshopper optimisation algorithm: theory and application. Adv Eng Softw 105:30–47

    Article  Google Scholar 

  30. Sundararaj V (2016) An efficient threshold prediction scheme for wavelet based ECG signal noise reduction using variable step size firefly algorithm. Int J Intell Eng Syst 9(3):117–126

    Google Scholar 

  31. Sundararaj V (2019a) Optimal task assignment in mobile cloud computing by queue based ant-bee algorithm. Wireless Pers Commun 104(1):173–197

    Article  Google Scholar 

  32. Sundararaj V (2019b) Optimised denoising scheme via opposition-based self-adaptive learning PSO algorithm for wavelet-based ECG signal noise reduction. Int J Biomed Eng Technol 31(4):325–345

    Article  Google Scholar 

  33. Tjondronegoro D, Xiaohui T, Johannes S, Cher HL (2011) Multi-modal summarization of key events and top players in sports tournament videos. In: 2011 IEEE workshop on applications of computer vision (WACV), IEEE, pp 471–478

  34. Topaz CM, Bernoff AJ, Logan S, Toolson W (2008) A model for rolling swarms of locusts. Eur Phys J Spec Top 157:93–109

    Article  Google Scholar 

  35. Ustun D, Akdagli A (2018) Design of band–notched UWB antenna using a hybrid optimization based on ABC and DE algorithms. AEU Int J Electron Commun 87:10–21

    Article  Google Scholar 

  36. Vinu S, Muthukumar S, Kumar RS (2018) An optimal cluster formation based energy efficient dynamic scheduling hybrid MAC protocol for heavy traffic load in wireless sensor networks. Comput Secur 77:277–288

    Article  Google Scholar 

  37. Xing J, Xiaofeng L (2019) Feature extraction algorithm of audio and video based on clustering in sports video analysis. J Vis Commun Image Represent, p 02694

  38. Xu J, Huang F, Zhang X, Wang S, Li C, Li Z, He Y (2019) Sentiment analysis of social images via hierarchical deep fusion of content and links. Appl Soft Comput 80:387–399

    Article  Google Scholar 

  39. Yang J, She D, Sun M, Cheng MM, Rosin PL, Wang L (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans Multimedia 20(9):2513–2525

    Article  Google Scholar 

  40. Yuan H, Li J, Lai LL, Tang YY (2020) Low-rank matrix regression for image feature extraction and feature selection. Inf Sci 522:214–226

    MathSciNet  Article  Google Scholar 

  41. Zhang Y, Song D, Zhang P, Wang P, Li J, Li X, Wang B (2018a) A quantum-inspired multi-modal sentiment analysis framework. Theoret Comput Sci 752:21–40

    MathSciNet  Article  Google Scholar 

  42. Zhang Z, Zou Y, Gan C (2018b) Textual sentiment analysis via three different attention convolutional neural networks and cross-modality consistent regression. Neurocomputing 275:1407–1415

    Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to S. Bairavel.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Human and animal rights

This article does not contain any studies with human or animal subjects performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by V. Loia.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Bairavel, S., Krishnamurthy, M. Novel OGBEE-based feature selection and feature-level fusion with MLP neural network for social media multimodal sentiment analysis. Soft Comput 24, 18431–18445 (2020). https://doi.org/10.1007/s00500-020-05049-6

Download citation

Keywords

  • Sentiment analysis
  • OMSANN
  • OGBEE
  • MLP
  • Feature extraction
  • Optimal solution
  • Accuracy