Abstract
In this paper, we hypothesize that sarcasm detection is closely associated with the emotion present in memes. Thereafter, we propose a deep multitask model to perform these two tasks in parallel, where sarcasm detection is treated as the primary task, and emotion recognition is considered an auxiliary task. We create a large-scale dataset consisting of 7416 memes in Hindi, one of the widely spoken languages. We collect the memes from various domains, such as politics, religious, racist, and sexist, and manually annotate each instance with three sarcasm categories, i.e., i) Not Sarcastic, ii) Mildly Sarcastic or iii) Highly Sarcastic and 13 fine-grained emotion classes. Furthermore, we propose a novel Knowledge Infusion (KI) based module which captures sentiment-aware representation from a pre-trained model using the Memotion dataset. Detailed empirical evaluation shows that the multitasking model performs better than the single-task model. We also show that using this KI module on top of our model can boost the performance of sarcasm detection in both single-task and multi-task settings even further. Code and dataset are available at this link: https://www.iitp.ac.in/ ai-nlp-ml/resources.html#Sarcastic-Meme-Detection.
D. Bandyopadhyay and G. Kumari—Equal Contribution.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
Each meme in Memotion. dataset is annotated with both sarcasm and sentiment classes.
References
Bayerl, P.S., Paul, K.I.: What determines inter-coder agreement in manual annotations? A meta-analytic investigation. Comput. Linguist. 37(4), 699–725 (2011). https://doi.org/10.1162/COLI_a_00074, https://aclanthology.org/J11-4004
Bernadt, M., Emmanuel, J.: Diagnostic agreement in psychiatry. Br. J. Psychiatry: J. Mental Sci. 163, 549–50 (1993). https://doi.org/10.1192/S0007125000034012
Boland, K., Wira-Alam, A., Messerschmidt, R.: Creating an annotated corpus for sentiment analysis of German product reviews. GESIS-Technical reports, 2013/05 (2013)
Bouazizi, O.: Tomoaki: a pattern-based approach for sarcasm detection on twitter. IEEE Access 4, 5477–5488 (2016). https://doi.org/10.1109/ACCESS.2016.2594194
Castro, S., Hazarika, D., Pérez-Rosas, V., Zimmermann, R., Mihalcea, R., Poria, S.: Towards multimodal sarcasm detection (an _obviously_ perfect paper). CoRR abs/1906.01815 (2019). http://arxiv.org/abs/1906.01815
Chauhan, D.S., Dhanush, S.R., Ekbal, A., Bhattacharyya, P.: Sentiment and emotion help sarcasm? A multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4351–4360. Association for Computational Linguistics, Online, July 2020. https://doi.org/10.18653/v1/2020.acl-main.401, https://aclanthology.org/2020.acl-main.401
Davidson, T., Bhattacharya, D., Weber, I.: Racial bias in hate speech and abusive language detection datasets. In: Proceedings of the Third Workshop on Abusive Language Online, pp. 25–35. Association for Computational Linguistics, Florence, August 2019. https://doi.org/10.18653/v1/W19-3504, https://aclanthology.org/W19-3504
Dong, X., Li, C., Choi, J.D.: Transformer-based context-aware sarcasm detection in conversation threads from social media. CoRR abs/2005.11424 (2020). https://arxiv.org/abs/2005.11424
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale (2020). https://doi.org/10.48550/ARXIV.2010.11929, https://arxiv.org/abs/2010.11929
Dror, R., Baumer, G., Shlomov, S., Reichart, R.: The Hitchhiker’s guide to testing statistical significance in natural language processing. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1383–1392. Association for Computational Linguistics, Melbourne, Australia, July 2018. https://doi.org/10.18653/v1/P18-1128, https://aclanthology.org/P18-1128
Ekman, P., Cordaro, D.T.: What is meant by calling emotions basic. Emot. Rev. 3, 364–370 (2011)
Ghosal, D., Akhtar, M.S., Chauhan, D., Poria, S., Ekbal, A., Bhattacharyya, P.: Contextual inter-modal attention for multi-modal sentiment analysis. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3454–3466. Association for Computational Linguistics, Brussels, October–November 2018. https://doi.org/10.18653/v1/D18-1382, https://aclanthology.org/D18-1382
Ghosh, D., Fabbri, A.R., Muresan, S.: The role of conversation context for sarcasm detection in online interactions. CoRR abs/1707.06226 (2017). http://arxiv.org/abs/1707.06226
Hasan, M.K., et al.: UR-FUNNY: a multimodal language dataset for understanding humor. CoRR abs/1904.06618 (2019). http://arxiv.org/abs/1904.06618
He, S., Zheng, X., Wang, J., Chang, Z., Luo, Y., Zeng, D.: Meme extraction and tracing in crisis events. In: 2016 IEEE Conference on Intelligence and Security Informatics (ISI), pp. 61–66 (2016). https://doi.org/10.1109/ISI.2016.7745444
Joshi, A., Bhattacharyya, P., Carman, M.J.: Automatic sarcasm detection: a survey. ACM Comput. Surv. 50(5), 73:1–73:22 (2017). https://doi.org/10.1145/3124420
Joshi, A., Tripathi, V., Bhattacharyya, P., Carman, M.J.: Harnessing sequence labeling for sarcasm detection in dialogue from TV series ‘Friends’. In: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 146–155. Association for Computational Linguistics, Berlin, August 2016. https://doi.org/10.18653/v1/K16-1015, https://aclanthology.org/K16-1015
Kiela, D., Firooz, H., Mohan, A., Goswami, V., Singh, A., Ringshia, P., Testuggine, D.: The hateful memes challenge: detecting hate speech in multimodal memes. CoRR abs/2005.04790 (2020). https://arxiv.org/abs/2005.04790
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR abs/1412.6980 (2015)
Kosti, R., Alvarez, J.M., Recasens, A., Lapedriza, A.: Emotic: emotions in context dataset. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2309–2317 (2017). https://doi.org/10.1109/CVPRW.2017.285
krippendorff, k.: Computing Krippendorff’s alpha-reliability, January 2011
Kumar, A., Joshi, A.: Striking a balance: alleviating inconsistency in pre-trained models for symmetric classification tasks. In: Findings of the Association for Computational Linguistics: ACL 2022, pp. 1887–1895. Association for Computational Linguistics, Dublin, May 2022. https://doi.org/10.18653/v1/2022.findings-acl.148, https://aclanthology.org/2022.findings-acl.148
Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: VisualBERT: a simple and performant baseline for vision and language (2019). https://doi.org/10.48550/ARXIV.1908.03557, https://arxiv.org/abs/1908.03557
Liu, L., Priestley, J.L., Zhou, Y., Ray, H.E., Han, M.: A2text-net: a novel deep neural network for sarcasm detection. In: 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI), pp. 118–126 (2019). https://doi.org/10.1109/CogMI48466.2019.00025
Liu, Y., et al.: Multilingual denoising pre-training for neural machine translation (2020). https://doi.org/10.48550/ARXIV.2001.08210, https://arxiv.org/abs/2001.08210
Majumder, N., Poria, S., Peng, H., Chhaya, N., Cambria, E., Gelbukh, A.: Sentiment and sarcasm classification with multitask learning. IEEE Intell. Syst. 34, 38–43 (2019). https://doi.org/10.1109/MIS.2019.2904691
McNemar, Q.: Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12(2), 153–157 (1947). https://doi.org/10.1007/bf02295996
Öhman, E.: Emotion annotation: rethinking emotion categorization. In: DHN Post-Proceedings (2020)
Pires, T., Schlinger, E., Garrette, D.: How multilingual is multilingual BERT? (2019). https://doi.org/10.48550/ARXIV.1906.01502, https://arxiv.org/abs/1906.01502
Poria, S., Cambria, E., Hazarika, D., Vij, P.: A deeper look into sarcastic tweets using deep convolutional neural networks. CoRR abs/1610.08815 (2016). http://arxiv.org/abs/1610.08815
Radford, A., et al.: Learning transferable visual models from natural language supervision. CoRR abs/2103.00020 (2021). https://arxiv.org/abs/2103.00020
Rhanoui, M., Mikram, M., Yousfi, S., Barzali, S.: A CNN-BILSTM model for document-level sentiment analysis. Mach. Learn. Knowl. Extract. 1(3), 832–847 (2019). https://doi.org/10.3390/make1030048, https://www.mdpi.com/2504-4990/1/3/48
Ribeiro, M.T., Singh, S., Guestrin, C.: “why should I trust you?”: Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August 2016, pp. 1135–1144 (2016)
Rockwell, P., Theriot, E.M.: Culture, gender, and gender mix in encoders of sarcasm: a self-assessment analysis. Commun. Res. Rep. 18(1), 44–52 (2001). https://doi.org/10.1080/08824090109384781, https://doi.org/10.1080/08824090109384781
Sharma, C., et al.: SemEval-2020 task 8: memotion analysis - the visuo-lingual metaphor! CoRR abs/2008.03781 (2020). https://arxiv.org/abs/2008.03781
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2015)
Suryawanshi, S., Chakravarthi, B.R., Arcan, M., Buitelaar, P.: Multimodal meme dataset (MultiOFF) for identifying offensive content in image and text. In: Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pp. 32–41. European Language Resources Association (ELRA), Marseille, France, May 2020. https://aclanthology.org/2020.trac-1.6
Tan, H., Bansal, M.: LXMERT: learning cross-modality encoder representations from transformers (2019). https://doi.org/10.48550/ARXIV.1908.07490, https://arxiv.org/abs/1908.07490
Tsur, O., Rappoport, A.: RevRank: a fully unsupervised algorithm for selecting the most helpful book reviews. In: ICWSM (2009)
Venkatesan, R., Er, M.J.: Multi-label classification method based on extreme learning machines. In: 2014 13th International Conference on Control Automation Robotics Vision (ICARCV), pp. 619–624 (2014). https://doi.org/10.1109/ICARCV.2014.7064375
Wilson, P.A., Lewandowska, B.: The Nature of Emotions. In: Cambridge University Press (2012)
Zhu, H., Mak, D., Gioannini, J., Xia, F.: NLPStatTest: a toolkit for comparing NLP system performance. In: Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 40–46. Association for Computational Linguistics, Suzhou, China, December 2020. https://aclanthology.org/2020.aacl-demo.7
Acknowledgement
The research reported in this paper is an outcome of the project “HELIOS-Hate, Hyperpartisan, and Hyperpluralism Elicitation and Observer System”, sponsored by Wipro.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bandyopadhyay, D., Kumari, G., Ekbal, A., Pal, S., Chatterjee, A., BN, V. (2023). A Knowledge Infusion Based Multitasking System for Sarcasm Detection in Meme. In: Kamps, J., et al. Advances in Information Retrieval. ECIR 2023. Lecture Notes in Computer Science, vol 13980. Springer, Cham. https://doi.org/10.1007/978-3-031-28244-7_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-28244-7_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-28243-0
Online ISBN: 978-3-031-28244-7
eBook Packages: Computer ScienceComputer Science (R0)