Numerous public networks, namely Instagram, YouTube, Facebook, Twitter, etc., share their own feelings and idea as videotapes, posts, and pictures. In future research, adapting to such data and mining valuable information from it will be an undeniably troublesome errand. This paper proposes a novel audio–video–textual-based multimodal sentiment analysis approach. The proposed approach investigates the sentiments that are collected from the web recordings that utilize audio, video, and textual modalities for further extraction. A feature-level fusion technique is employed in fusing the extracted features from different modalities. Therefore, the extracted features are optimally chosen by using a novel oppositional grass bee optimization (OGBEE) algorithm to obtain the best optimal feature set. Here, 12 benchmark functions are developed to validate the numerical efficiency and the effectiveness of a novel OGBEE algorithm for various aspects. Moreover, our proposed approach utilizes multilayer perceptron-based neural network (MLP-NN) for sentiment classification. The experimental analysis reveals that the proposed approach provides better classification accuracy of about 95.2% with less computational time.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
Tax calculation will be finalised during checkout.
Bhattarai B, Lee J (2019) Automatic music mood detection using transfer learning and multilayer perceptron. Int J Fuzzy Logic Intell Syst 19(2):88–96
Cai H, Zhidiao Q, Li Z, Zhang Y, Xiping H, Bin H (2020) Feature-level fusion approaches based on multi-modal EEG data for depression recognition. Inf Fusion 59:127–138
Cambria D, Feraco B (2017) A practical guide to sentiment analysis. Springer, Cham
Chaturvedi I, Satapathy R, Cavallari S, Cambria E (2019) Fuzzy commonsense reasoning for multi-modal sentiment analysis. Pattern Recognit Lett 125:264–270
Chen F, Ji R, Jinsong S, Cao D, Gao Y (2017) Predicting microblog sentiments via weakly supervised multi-modal deep learning. IEEE Trans Multimedia 20(4):997–1007
Eyben F, Wöllmer M, Schuller B (2010) Opensmile: the munich versatile and fast open-source audio feature extractor. In: Proceedings of the international conference on Multimedia, ACM, pp 1459–1462
Gaidhane PJ, Nigam MJ (2018) A hybrid grey wolf optimizer and artificial bee colony algorithm for enhancing the performance of complex systems. J Comput Sci 27:284–302
Gupta Roy R, Ghoshal D (2019a) Adaptive second-order sliding-mode controller for shank-foot orthosis system. Int J Control 92(7):1580–1589
Gupta Roy R, Ghoshal D (2019b) Grey wolf optimization-based second order sliding mode control for inchworm robot, Robotica. Cambridge University Press, Cambridge
Huang F, Zhang X, Zhao Z, Xu J, Li Z (2019) Image–text sentiment analysis via deep multi-modal attentive fusion. Knowl-Based Syst 167:26–37
Huddar MG, Sannakki SS, Rajpurohit VS (2020) Multi-level feature optimization and multi-modal contextual fusion for sentiment analysis and emotion classification. Comput Intell 36(2):861–881
Intisar O, Hussien Y, Hasan J (2018) Multimodal sentiment analysis: a comparison study. J Comput Sci 14(6):804–818
Ji R, Chen F, Cao L, Gao Y (2018) Cross-modality microblog sentiment prediction via bi-layer multi-modal hypergraph learning. IEEE Trans Multimedia 21(4):1062–1075
Karaboga D, Akay B (2009) Artificial bee colony (ABC), harmony search and bees algorithms on numerical optimization. In: Innovative production machines and systems virtual conference
Karaboga D, Akay B (2009b) A comparative study of artificial bee colony algorithm. Appl Math Comput 214(1):108–132
Kaushik L, Abhijeet S, John HLH (2013) Automatic sentiment extraction from YouTube videos. In: 2013 IEEE workshop on automatic speech recognition and understanding, IEEE, pp 239–244
Khan A, Ko DK, Lim SC, Kim HS (2019) Structural vibration-based classification and prediction of delamination in smart composite laminates using deep learning neural network. Compos Part B-Eng 161:586–594
Kim B, Lee J (2018) A deep-learning based model for emotional evaluation of video clips. Int J Fuzzy Logic Intell Syst 18(4):245–253
Kim N-R, Kim K, Lee J-H (2017) Sentiment analysis in microblogs using HMMs with syntactic and sentimental information. Int J Fuzzy Logic Intell Syst 17(4):329–336
Luo L, Zhang B, Zhang G, Yue X (2020) Rapid prediction of cured shape types of composite laminates using a FEM-ANN method. Compos Struct 238:111980
Majumder N, Hazarika D, Gelbukh A, Cambria E, Poria S (2018) Multi-modal sentiment analysis using hierarchical fusion with context modelling. Knowl-Based Syst 161:124–133
Poria S, Cambria E, Howard N, Huang G-B, Hussain A (2016) Fusing audio, visual and textual clues for sentiment analysis from multi-modal content. Neurocomputing 174:50–59
Poria S, Peng H, Hussain A, Howard N, Cambria E (2017) Ensemble application of convolutional neural networks and multiple kernel learning for multi-modal sentiment analysis. Neurocomputing 261:217–230
Poria M, Hazarika C, Gelbukh A, Hussain A (2018) Multi-modal sentiment analysis: addressing key issues and setting up the baselines. IEEE Intell Syst 33(6):17–25
Radhakrishnan V, Joseph C (2018) Sentiment extraction from naturalistic video. Procedia Comput Sci 143:626–634
Rejeesh MR (2019) Interest point based face recognition using adaptive neuro fuzzy inference system. Multimed Tools Appl 78(16):22691–22710
Rogers SM, Matheson T, Despland E, Dodgson T, Burrows M, Simpson SJ (2003) Mechanosensory-induced behavioural gregarization in the desert locust Schistocerca gregaria. J Exp Biol 206:3991–4002
Rosas VP, Mihalcea R, Morency L-P (2013) Multi-modal sentiment analysis of Spanish online videos. IEEE Intell Syst 28(3):38–45
Saremi S, Mirjalili S, Lewis A (2017) Grasshopper optimisation algorithm: theory and application. Adv Eng Softw 105:30–47
Sundararaj V (2016) An efficient threshold prediction scheme for wavelet based ECG signal noise reduction using variable step size firefly algorithm. Int J Intell Eng Syst 9(3):117–126
Sundararaj V (2019a) Optimal task assignment in mobile cloud computing by queue based ant-bee algorithm. Wireless Pers Commun 104(1):173–197
Sundararaj V (2019b) Optimised denoising scheme via opposition-based self-adaptive learning PSO algorithm for wavelet-based ECG signal noise reduction. Int J Biomed Eng Technol 31(4):325–345
Tjondronegoro D, Xiaohui T, Johannes S, Cher HL (2011) Multi-modal summarization of key events and top players in sports tournament videos. In: 2011 IEEE workshop on applications of computer vision (WACV), IEEE, pp 471–478
Topaz CM, Bernoff AJ, Logan S, Toolson W (2008) A model for rolling swarms of locusts. Eur Phys J Spec Top 157:93–109
Ustun D, Akdagli A (2018) Design of band–notched UWB antenna using a hybrid optimization based on ABC and DE algorithms. AEU Int J Electron Commun 87:10–21
Vinu S, Muthukumar S, Kumar RS (2018) An optimal cluster formation based energy efficient dynamic scheduling hybrid MAC protocol for heavy traffic load in wireless sensor networks. Comput Secur 77:277–288
Xing J, Xiaofeng L (2019) Feature extraction algorithm of audio and video based on clustering in sports video analysis. J Vis Commun Image Represent, p 02694
Xu J, Huang F, Zhang X, Wang S, Li C, Li Z, He Y (2019) Sentiment analysis of social images via hierarchical deep fusion of content and links. Appl Soft Comput 80:387–399
Yang J, She D, Sun M, Cheng MM, Rosin PL, Wang L (2018) Visual sentiment prediction based on automatic discovery of affective regions. IEEE Trans Multimedia 20(9):2513–2525
Yuan H, Li J, Lai LL, Tang YY (2020) Low-rank matrix regression for image feature extraction and feature selection. Inf Sci 522:214–226
Zhang Y, Song D, Zhang P, Wang P, Li J, Li X, Wang B (2018a) A quantum-inspired multi-modal sentiment analysis framework. Theoret Comput Sci 752:21–40
Zhang Z, Zou Y, Gan C (2018b) Textual sentiment analysis via three different attention convolutional neural networks and cross-modality consistent regression. Neurocomputing 275:1407–1415
Conflict of interest
The authors declare that they have no conflict of interest.
Human and animal rights
This article does not contain any studies with human or animal subjects performed by any of the authors.
Informed consent was obtained from all individual participants included in the study.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Communicated by V. Loia.
About this article
Cite this article
Bairavel, S., Krishnamurthy, M. Novel OGBEE-based feature selection and feature-level fusion with MLP neural network for social media multimodal sentiment analysis. Soft Comput 24, 18431–18445 (2020). https://doi.org/10.1007/s00500-020-05049-6
- Sentiment analysis
- Feature extraction
- Optimal solution