Weak Supervision and Machine Learning for Online Harassment Detection

Part of the Human–Computer Interaction Series book series (HCIS)


Automated detection tools can enable both the study of online harassment and technology that mitigates its harm. Machine learning methods allow these tools to adapt and improve using data. Yet current well-established machine learning approaches require amounts of data that are often unmanageable by practitioners aiming to train harassment detectors. Emerging methods that learn models from weak supervision represent one important avenue to address this challenge. In contrast to the full supervision used in most traditional machine learning methods, weak supervision does not require annotators to label individual examples of the target concept. Instead, annotators provide approximate descriptions of the target concept, sucsh as rule-of-thumb indicators. In this chapter, we describe the weak supervision paradigm and some general principles that drive emerging methods. And we detail a weakly supervised method for detection of online harassment that uses key-phrase indicators as the form of weak supervision. This method considers multiple aspects of the online harassment phenomenon, using interplay between these aspects to bolster the weak supervision into a useful model. We describe experimental results demonstrating this approach on detecting harassment in social media data. Finally, we discuss the ongoing challenges for using machine learning methods to build harassment detectors.


Online Harassment Weak Supervision Target Concept Harassment Score Victim Score 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Ashktorab Z, Vitak J (2016) Designing cyberbullying mitigation and prevention solutions through participatory design with teenagers. In: Proceedings of the CHI conference on human factors in computing systems, pp 3895–3905Google Scholar
  2. Bellmore A, Calvin AJ, Xu JM, Zhu X (2015) The five W’s of bullying on Twitter: who, what, why, where, and when. Comput. Hum. Behav. 44:305–314CrossRefGoogle Scholar
  3. Bifet A, Frank E (2010) Sentiment knowledge discovery in Twitter streaming data. In: Proceedings of the international conference on discovery science, pp 1–15Google Scholar
  4. Blackburn J, Kwak H (2014) STFU NOOB!: predicting crowdsourced decisions on toxic behavior in online games. In: Proceedings of the 23rd international conference on world wide web, ACM, WWW ’14, pp 877–888.
  5. Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: Proceedings of the conference on computational learning theory, ACM, pp 92–100Google Scholar
  6. Bunescu RC, Mooney R (2007) Learning to extract relations from the web using minimal supervision. Proc. Ann. Meet. Assoc. Comput. Linguist. 45:576–583Google Scholar
  7. Chen LC, Fidler S, Yuille AL, Urtasun R (2014) Beat the mturkers: automatic image labeling from weak 3D supervision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3198–3205Google Scholar
  8. Chen Y, Zhou Y, Zhu S, Xu H (2012) Detecting offensive language in social media to protect adolescent online safety. In: Proceedings of the international conference on social computing, pp 71–80Google Scholar
  9. Dadvar M, de Jong F, Ordelman R, Trieschnigg D (2012) Improved cyberbullying detection using gender information. In: Dutch-Belgian information retrieval workshop, pp 23–25Google Scholar
  10. Dinakar K, Reichart R, Lieberman H (2011) Modeling the detection of textual cyberbullying. In: ICWSM workshop on social mobile webGoogle Scholar
  11. Ditch the Label (2013) The annual cyberbullying survey. http://www.ditchthelabelorg/
  12. Djuric N, Zhou J, Morris R, Grbovic M, Radosavljevic V, Bhamidipati N (2015) Hate speech detection with comment embeddings. In: International conference on world wide web, pp 29–30Google Scholar
  13. Duggan M (2017) Online harassment. Technical report, Pew Research CenterGoogle Scholar
  14. Halpern Y, Horng S, Sontag D (2016) Clinical tagging with joint probabilistic models. In: Proceedings of the conference on machine learning for healthcare, pp 209–225Google Scholar
  15. Hoffmann R, Zhang C, Ling X, Zettlemoyer L, Weld DS (2011) Knowledge-based weak supervision for information extraction of overlapping relations. In: Proceedings of the annual meeting of the association for computational linguistics: human language technologies. Association for Computational Linguistics, pp 541–550Google Scholar
  16. Hosseinmardi H, Ghasemianlangroodi A, Han R, Lv Q, Mishra S (2014a) Towards understanding cyberbullying behavior in a semi-anonymous social network. In: Proceedings of the IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM), pp 244–252Google Scholar
  17. Hosseinmardi H, Li S, Yang Z, Lv Q, Rafiq RI, Han R, Mishra S (2014b) A comparison of common users across Instagram and to better understand cyberbullying. In: Proceedings of the IEEE international conference on big data and cloud computingGoogle Scholar
  18. Hosseinmardi H, Mattson SA, Rafiq RI, Han R, Lv Q, Mishra S (2015a) Analyzing labeled cyberbullying incidents on the Instagram social network. In: International conference on social informatics, pp 49–66Google Scholar
  19. Hosseinmardi H, Mattson SA, Rafiq RI, Han R, Lv Q, Mishra S (2015b) Detection of cyberbullying incidents on the Instagram social network. Association for the Advancement of Artificial IntelligenceGoogle Scholar
  20. Huang Q, Singh VK (2014) Cyber bullying detection using social and textual analysis. In: Proceedings of the international workshop on socially-aware multimedia, pp 3–6Google Scholar
  21. Koren Y, Bell R, Volinsky C (2009) Matrix factorization techniques for recommender systems. Computer 42(8):30–37CrossRefGoogle Scholar
  22. Krishnamurthy J, Mitchell TM (2012) Weakly supervised training of semantic parsers. In: Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning. Association for Computational Linguistics, pp 754–765Google Scholar
  23. Lavrenko V, Croft WB (2001) Relevance based language models. In: Proceedings of the international ACM SIGIR conference on research and development in information retrieval, pp 120–127Google Scholar
  24. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444CrossRefGoogle Scholar
  25. Lenhart A, Ybarra M, Zickurh K, Price-Feeney M (2016) Online harassment, digital abuse, and cyberstalking in america. Technical report, Data & Society Research InstituteGoogle Scholar
  26. Margono H, Yi X, Raikundalia GK (2014) Mining Indonesian cyber bullying patterns in social networks. In: Proceedings of the Australasian computer science conference, vol 147Google Scholar
  27. McGhee I, Bayzick J, Kontostathis A, Edwards L, McBride A, Jakubowski E (2011) Learning to identify internet sexual predation. Int J Electron Commer 15(3):103–122CrossRefGoogle Scholar
  28. Mintz M, Bills S, Snow R, Jurafsky D (2009) Distant supervision for relation extraction without labeled data. In: Proceedings of the joint conference of the 47th annual meeting of the ACL and the 4th international joint conference on natural language processing of the AFNLP. Association for Computational Linguistics, pp 1003–1011Google Scholar
  29. Nahar V, Li X, Pang C (2013) An effective approach for cyberbullying detection. Commun. Inf. Sci. Manag. Eng. 3(5):238–247Google Scholar
  30. Nigam K, McCallum A, Mitchell T (2006) Semi-supervised text classification using EM. Semi-Supervised Learning, pp 33–56Google Scholar
  31. NoSwearing (2016) List of swear words & curse words. http://wwwnoswearingcom/dictionaryGoogle Scholar
  32. Patton DU, McKeown K, Rambow O, Macbeth J (2016) Using natural language processing and qualitative analysis to intervene in gang violence. arXiv:160908779
  33. Ptaszynski M, Dybala P, Matsuba T, Masui F, Rzepka R, Araki K (2010) Machine learning and affect analysis against cyber-bullying. In: Linguistic and cognitive approaches to dialog agents symposium, pp 7–16Google Scholar
  34. Raisi E, Huang B (2017a) Co-trained ensemble models for weakly supervised cyberbullying detection. In: NIPS 2017 workshop on learning with limited labeled dataGoogle Scholar
  35. Raisi E, Huang B (2017b) Cyberbullying detection with weakly supervised machine learning. In: Proceedings of the IEEE/ACM international conference on social networks analysis and mining, pp 409–416Google Scholar
  36. Ramakrishnan N, Butler P, Self N, Khandpur R, Saraf P, Wang W, Cadena J, Vullikanti A, Korkmaz G, Kuhlman C, Marathe A, Zhao L, Ting H, Huang B, Srinivasan A, Trinh K, Getoor L, Katz G, Doyle A, Ackermann C, Zavorin I, Ford J, Summers K, Fayed Y, Arredondo J, Gupta D, Mares D (2014) ‘Beating the news’ with EMBERS: forecasting civil unrest using open source indicators. In: ACM SIGKDD conference on knowledge discovery and data mining, pp 1799–1808Google Scholar
  37. Ratner AJ, De Sa CM, Wu S, Selsam D, Ré C (2016) Data programming: creating large training sets, quickly. Adv Neural Inf Process Syst 29:3567–3575Google Scholar
  38. Ravi S, Diao Q (2015) Large scale distributed semi-supervised learning using streaming approximation. arXiv:151201752
  39. Reynolds K, Kontostathis A, Edwards L (2011) Using machine learning to detect cyberbullying. In: Proceedings of the international conference on machine learning and applications and workshops (ICMLA), vol 2. pp 241–244Google Scholar
  40. Riedel S, Yao L, McCallum A (2010) Modeling relations and their mentions without labeled text. In: Joint European conference on machine learning and knowledge discovery in databases. Springer, pp 148–163Google Scholar
  41. Shin J, Wu S, Wang F, De Sa C, Zhang C, Ré C (2015) Incremental knowledge base construction using DeepDive. Proc VLDB Endow 8(11):1310–1321CrossRefGoogle Scholar
  42. Silva TH, de Melo PO, Almeida JM, Salles J, Loureiro AA (2013) A picture of Instagram is worth more than a thousand words: workload characterization and application. DCOSS, pp 123–132Google Scholar
  43. Warner W, Hirschberg J (2012) Detecting hate speech on the world wide web. In: Workshop on language in social media, pp 19–26Google Scholar
  44. Xu J, Schwing AG, Urtasun R (2014) Tell me what you see and I will show you where it is. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3190–3197Google Scholar
  45. Yao L, Riedel S, McCallum A (2010) Collective cross-document relation extraction without labelled data. In: Proceedings of the 2010 conference on empirical methods in natural language processing. Association for Computational Linguistics, pp 1013–1023Google Scholar
  46. Yin D, Xue Z, Hong L, Davison BD, Kontostathis A, Edwards L (2009) Detection of harassment on Web 2.0. Content Analysis in the WEB 20Google Scholar
  47. Zhu J, Mao J, Yuille AL (2014) Learning from weakly supervised data by the expectation loss SVM (e-SVM) algorithm. In: Advances in neural information processing systems, pp 1125–1133Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Virginia TechBlacksburgUSA

Personalised recommendations