Skip to main content
SpringerLink
Log in

Performance analysis of transformer-based architectures and their ensembles to detect trait-based cyberbullying

  • Original Article
  • Published:
Social Network Analysis and Mining Aims and scope Submit manuscript

Abstract

The influence of social media is one of the most dominating characteristics of the current era, and this has led cyberbullying to grow into a more serious social issue. As a result, automated cyberbullying detection systems need to be an integral part of almost all social media platforms. Past studies on this domain have primarily focused on hand-picked features and traditional machine learning approaches for cyberbullying detection from user comments on social media. Recently, transformers have been proved to be quite effective in various language-related tasks; however, their effectiveness has not been extensively explored in this particular domain. In this study, we evaluate the individual performance of several well-known transformer-based architectures and aim to contribute to the development of automated cyberbullying detection systems by proposing our own transformer-based ensemble framework. Our proposed framework is evaluated on a balanced and an imbalanced dataset, both of which are constructed from a large collection of Twitter comments and are publicly available. Our proposed architecture outperforms all the baseline models, as well the individual standalone classifier networks that are used in our ensemble, obtaining an average F1-score of 95.92% on the balanced dataset and an average F1-score of 87.51% on the imbalanced dataset. We further investigate the cases where our proposed architecture misclassifies samples from both datasets, preventing it from achieving a perfect score. Our models and code have been made publicly available (https://github.com/tasnim7ahmed/Extended-Cyberbullying-Detection/).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. https://www.wikipedia.org/.

  2. https://www.reddit.com/.

  3. https://colab.research.google.com/.

  4. https://pytorch.org/docs/.

References

  • Afsharizadeh M, Ebrahimpour-Komleh H, Bagheri, A (2018) Query-oriented text summarization using sentence extraction technique. In: 2018 4th International Conference on Web Research (ICWR): 128–132 IEEE

  • Agrawal S, Awekar A (2018) Deep learning for detecting cyberbullying across multiple social media platforms. In: European Conference on Information Retrieval: 141–153 Springer

  • Ahmed T, Kabir M, Ivan S, Mahmud H, Hasan K (2021) Am i being bullied on social media? an ensemble approach to categorize cyberbullying. In: 2021 IEEE International Conference on Big Data (Big Data): 2442–2453 https://doi.org/10.1109/BigData52589.2021.9671594

  • Ali WNHW, Mohd M, Fauzi F (2018) Cyberbullying detection: an overview. In: 2018 Cyber Resilience Conference (CRC): 1–3 IEEE

  • Anand A, Pugalenthi G, Fogel GB, Suganthan P (2010) An approach for classification of highly imbalanced data using weighting and undersampling. Amino acids 39(5):1385–1391

    Article  Google Scholar 

  • Badjatiya P, Gupta S, Gupta M, Varma, V Deep (2017) learning for hate speech detection in tweets. In: Proceedings of the 26th International Conference on World Wide Web Companion. WWW ’17 Companion: 759–760. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva CHE https://doi.org/10.1145/3041021.3054223

  • Baldwin T, Cook P, Lui M, MacKinlay A, Wang, L (2013) How noisy social media text, how diffrnt social media sources? In: Proceedings of the Sixth International Joint Conference on Natural Language Processing: 356–364

  • Bayzick J, Kontostathis A, Edwards L (2011) Detecting the presence of cyberbullying using computer software

  • Bretschneider U, Wöhner T, Peters R. (2014) Detecting online harassment in social networks

  • Burnap P, Williams ML (2015) Cyber hate speech on twitter: An application of machine classification and statistical modeling for policy and decision making. Policy Int 7(2):223–242

    Article  Google Scholar 

  • Chapin J (2016) Adolescents and cyber bullying: The precaution adoption process model. Educat inform technol 21(4):719–728

    Google Scholar 

  • Chatzakou, D., Leontiadis, I., Blackburn, J., Cristofaro, E.D., Stringhini, G., Vakali, A., Kourtellis, N.(2019): Detecting cyberbullying and cyberaggression in social media. ACM Trans Web 13(3) doi.org/10.1145/3343484

  • Cheng L, Li J, Silva YN, Hall DL, Liu H Xbully (2019): Cyberbullying detection within a multi-modal context. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining. WSDM ’19: 339–347 New York. USA:https://doi.org/10.1145/3289600.3291037

  • Cheng L, Guo R, Silva YN, Hall D, Liu H (2021) Modeling temporal patterns of cyberbullying detection with hierarchical attention networks. ACM/IMS Transact Data Sci 2(2):1–23

    Article  Google Scholar 

  • Dani H, Li J, Liu H (2017) Sentiment informed cyberbullying detection in social media. In: Ceci M, Hollmén J, Todorovski L, Vens C, Džeroski S (eds) Machine Learning and Knowledge Discovery in Database. Springer, Cham, pp 52–67

    Chapter  Google Scholar 

  • Davidson T, Warmsley D, Macy M, Weber I (2017) Automated hate speech detection and the problem of offensive language. Proceed Intern AAAI Conf Web Soc Media 11(1):512–515

    Article  Google Scholar 

  • De la Pena Sarracén GL, Pons RG, Cuza CEM, Rosso P (2018) Hate speech detection using attention-based lstm. EVALITA Eval NLP Speech Tools Italian 12:235

    Google Scholar 

  • Devlin, J., Chang, M.-W., Lee, K., Toutanova, K 2019 : BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: 14171–4186https://doi.org/10.18653/v1/N19-1423.

  • Djuric N, Zhou J, Morris R, Grbovic M, Radosavljevic V, Bhamidipati, N (2015) Hate speech detection with comment embeddings. In: Proceedings of the 24th International Conference on World Wide Web. WWW ’15 Companion, pp. 29–30. Association for Computing Machinery, New York, USA https://doi.org/10.1145/2740908.2742760

  • Elsafoury F, Katsigiannis S, Wilson SR, Ramzan N, (2021) Does bert pay attention to cyberbullying? In: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval: 1900–1904

  • Fang Y, Yang S, Zhao B, Huang C (2021) Cyberbullying detection in social networks using bi-gru with self-attention mechanism. Information 12(4):171

    Article  Google Scholar 

  • Gambäck B, Sikdar, (2017). UK Using convolutional neural networks to classify hate-speech. In: Proceedings of the First Workshop on Abusive Language Online: 85–90

  • Gupta S, Bolden S, Kachhadia J, Korsunska A, Stromer-Galley J, Polibert (2020) : Classifying political social media messages with bert. In: Social, Cultural and Behavioral Modeling (SBP-BRIMS 2020) Conference. Washington DC

  • Keskar NS, Mudigere D, Nocedal J, Smelyanskiy M, Tang (2016) PTP On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836

  • Korovesis K, Alexandridis G, Caridakis G, Polydoras P, Tsantilas P. (2020) Leveraging aspect-based sentiment prediction with textual features and document metadata. In: 11th Hellenic Conference on Artificial Intelligence: 168–174

  • Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V, Roberta (2019) : A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

  • Li J, Cheng K, Wang S, Morstatter F, Trevino RP, Tang J, Liu H (2017) Feature selection: A data perspective. ACM Comput. Surv. 50(6) https://doi.org/10.1145/3136625

  • Loshchilov I, Hutter ,F (2017) Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101

  • Lun J, Zhu J, Tang Y, Yang M (2020) Multiple data augmentation strategies for improving performance on automatic short answer scoring. Proceed AAAI Conf Artificial Intell 34:13389–13396

    Google Scholar 

  • Malmasi S, Zampieri M (2018) Challenges in discriminating profanity from hate speech. J Exp Theor Art Intell 30(2):187–202. https://doi.org/10.1080/0952813X.2017.1409284

    Article  Google Scholar 

  • Mittos A, Zannettou S, Blackburn J, De Cristofaro, E (2020) and we will fight for our race!” a measurement study of genetic testing conversations on reddit and 4chan. Proceedings of the International AAAI Conference on Web and Social Media 14(1):452–463

  • Nguyen H, Zhang C, Xia C, Yu PS (2020) Dynamic semantic matching and aggregation network for few-shot intent detection. arXiv preprint arXiv:2010.02481

  • Nobata C, Tetreault J, Thomas A, Mehdad Y, Chang Y (2016) Abusive language detection in online user content. In: Proceedings of the 25th International Conference on World Wide Web. WWW ’16, pp. 145–153. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE https://doi.org/10.1145/2872427.2883062.

  • Ottoni R, Cunha E, Magno G, Bernardina P, Meira Jr, W, Almeida V (2018) Analyzing right-wing youtube channels: Hate, violence and discrimination. In: Proceedings of the 10th ACM Conference on Web Science WebSci ’18: 323–332. Association for Computing Machinery, New York.USA https://doi.org/10.1145/3201064.3201081.

  • Park JH, Fung P(2017) One-step and Two-step Classification for Abusive Language Detection on Twitter

  • Pradhan, A., Yatam, V.M., Bera, P (2020) : Self-attention for cyberbullying detection. In: 2020 International Conference on Cyber Situational Awareness, Data Analytics and Assessment: 1–6 IEEE

  • Qiu S, Xu B, Zhang J, Wang Y, Shen X, De Melo G, Long C, Li X (2020) Easyaug: An automatic textual data augmentation platform for classification tasks. Companion Proceed Web Conf 2020:249–252

    Google Scholar 

  • Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I et al (2019) Language models are unsupervised multitask learners. OpenAI blog 1(8):9

    Google Scholar 

  • Rajpurkar P, Zhang J, Lopyrev K, Liang P (2016) Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250

  • Rasmy L, Xiang Y, Xie Z, Tao C, Zhi D (2021) Med-bert: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction. NPJ digital med 4(1):1–13

    Article  Google Scholar 

  • Reynolds K, Kontostathis A, Edwards, L. (2011). Using machine learning to detect cyberbullying. In: 2011 10th International Conference on Machine Learning and Applications and Workshops 2: 241–244 https://doi.org/10.1109/ICMLA.2011.152

  • Rios A, Kavuluru R (2018) Few-shot and zero-shot multi-label learning for structured label spaces. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing 2018: 3132 NIH Public Access

  • Sanh V, Debut L, Chaumond J, Wolf T Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108

  • Sobhani P, Viktor H, Matwin S (2014) Learning from imbalanced data using ensemble methods and cluster-based undersampling. In: International Workshop on New Frontiers in Mining Complex Patterns: 69–83 Springer

  • Squicciarini A, Rajtmajer S, Liu Y, Griffin C (2015) Identification and characterization of cyberbullying dynamics in an online social network. In: Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2015. ASONAM ’15: 280–285. Association for Computing Machinery, New York. USA https://doi.org/10.1145/2808797.2809398

  • Van Hee C, Lefever E, Verhoeven B, Mennes J, Desmet B, De Pauw G, Daelemans W, Hoste V (2015) Automatic detection and prevention of cyberbullying. In: International Conference on Human and Social Analytics (HUSO 2015): 13–18 IARIA

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L (2017) Polosukhin I attention is all you need. In: Adv Neural Inform ProcessSyst: 5998–6008

  • Wang J, Fu K, Lu C-T (2020). Sosnet: A graph convolutional network approach to fine-grained cyberbullying detection. In: 2020 IEEE International Conference on Big Data (Big Data): 1699–1708 https://doi.org/10.1109/BigData50022.2020.9378065

  • Wang J, Fu K, Lu C. (2020) Fine-grained balanced cyberbullying dataset

  • Waseem Z Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In: Proceedings of the First Workshop on NLP and Computational Social Science: 138–142

  • Waseem Z, Hovy D(2016) Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In: Proceedings of the NAACL Student Research Workshop: 88–93

  • Xu J-M, Jun K-S, Zhu X, Bellmore(2012) A Learning from bullying traces in social media. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: 656–666

  • Xu H, Liu B, Shu L, Yu P (2019) Open-world learning and application to product classification. In: The World Wide Web Conference: 3413–3419

  • Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R.R., Le, Q.V (2019): Xlnet: Generalized autoregressive pretraining for language understanding. Adv neural inform process syst32

  • Zhang Z, Luo L (2019) Hate speech detection: A solved problem? the challenging case of long tail on twitter. Semantic Web 10(5):925–945

    Article  Google Scholar 

  • Zhu Y, Kiros R, Zemel RS, Salakhutdinov R, Urtasun R, Torralba A, Fidler S (2015) Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. IEEE Inter Conf Comput Vision (ICCV) 2015:19–27

    Google Scholar 

Download references

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Author information

Author notes

  1. Tasnim Ahmed, Shahriar Ivan, Mohnisul Kabir these authors contributed equally to this work.

Authors and Affiliations

  1. Department of Computer Science and Engineering, Islamic University of Technology, Board Bazar, Gazipur, 1704, Dhaka, Bangladesh

    Tasnim Ahmed, Shahriar Ivan, Mohsinul Kabir, Hasan Mahmud & Kamrul Hasan

Authors
  1. Tasnim Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shahriar Ivan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohsinul Kabir
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hasan Mahmud
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kamrul Hasan
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TA took part in conceptualization, methodology, software, formal analysis, investigation, resources, data curation, validation, writing—original draft, visualization. SI involved in conceptualization, formal analysis, investigation, resources, writing—Original draft, visualization. MK took part in conceptualization, methodology, formal analysis, investigation, resources, data curation, writing—original draft. HM involved in supervision. KH took part in supervision.

Corresponding author

Correspondence to Tasnim Ahmed.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmed, T., Ivan, S., Kabir, M. et al. Performance analysis of transformer-based architectures and their ensembles to detect trait-based cyberbullying. Soc. Netw. Anal. Min. 12, 99 (2022). https://doi.org/10.1007/s13278-022-00934-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13278-022-00934-4

Keywords

Search

Navigation