Keywords

1 Introduction

This paper describes SOCYTI, which is a research and innovation project currently engaging with users, gathering requirements, and writing initial technical specifications and which deals with assessing key behaviors and risks to local communities and mitigating the identified risks by developing advanced and real-time methods for monitoring and preventing such malicious behavior on the social cyber-space. Online violence-inducing behaviors (e.g., hate speech) and information disorder have become a global threat for information integrity and are driving distrust towards individuals, communities, and governments worldwide [1]. Hate speech is not a new phenomenon, but social and technological developments, comprising the persistent spread of social media, malicious social and political discourse, political polarization and deepening economic inequality have determined both an increase in its incidence and the ease with which it spreads. This is an alarming trend that is undermining democratic discourse, fueling discrimination, and stirring violence across the world [1].

Community resilience [2, 3] (the ability of a community to cope with disasters) has increasingly become a priority for local to national governments, especially in the times of COVID-19 pandemic when even a small-scale hazard could threaten the already resource-scarce community/municipality services. Community resilience is defined in many ways in the literature across multiple disciplines [2, 3]; in simple terms, it is the ability of a community to bounce back together [3]. A whole community approach to resilience requires efforts from all the stakeholders of a community including both individual members of the public and government services [4]. Further, research literature on community resilience indicates that an important yet underexplored factor to building community resilience is social connectedness, harmony, and cohesion among community members [2, 5]. Any attempt for community resilience assessment by local community services needs to include the factors for social atmosphere of the community. Some reports show that existing tools to support resilience initiatives of the local community services lack the ability to assess and mitigate the dynamic risks associated with the social atmosphere through malicious behavior online such as spreading hate and misinformation on social media [6, 7]. Such behaviors significantly harm the response efforts of community members during disasters, and thus, local community services require tools to timely monitor risks to resilience from social atmosphere [8]. A variety of large-scale social media datasets, collaborative mapping technologies, and data science approaches have emerged that can facilitate computational social science research to gain a better understanding of community resilience processes accounting for risks associated with social behavior of community members and design actionable tools for local community services. The SOCYTI project will analyze the big data sources at scale, by taking in account all ethical, social, and legal (privacy and data protection) challenges and considerations (e.g., compliance with human rights, respect right of privacy, compliance with data protection principles such as transparency, necessity, and proportionality etc.), in contrast to only existing approaches of small-scale human observations, or survey-based analytical approaches.

In order to achieve the objectives a SOCYTI system will be developed, which will act as a toolset for further analysis.

2 Objectives of SOCYTI Project

The SOCYTI has three main objectives.

First, we aim to analyze the types of violence-inducing social behaviors expressed by the public online that are critical to understanding the risk factors associated with social atmosphere of a community that harm the community resilience, in order to advance the existing community resilience index models.

Second, we aim to infer violence-inducing social behavior from the multimodal and multilingual posts shared by public on social media considering an appropriate trade-off between individual interests (maintaining individual privacy) and the legitimate concerns of achieving and sustaining community resilience.

Third, we aim to design and develop a real-time violence-inducing social behavior detection system for online social media content that could inform the risks to local community services for conducting proactive intervention such as through crisis communication strategies and campaigns.

3 Background and Challenges

Many methods and tools have been developed for the early detection of online malicious activities and actors, for instance, by using natural language processing and social network analysis, or by identifying bots and various patterns [9,10,11]. But there is a lot more that needs to be done. The SOCYTI is trying to address four most important challenges relevant to the research domain.

Lack of Human Behavior Indicators in Community Resilience Models:

There is an increased interest in community resilience by the local government and community/municipality services. Recent studies show that community resilience is a highly local culture-bound phenomenon and related to the kind and frequency of hazards experienced by a local community [2, 3]. Further, research literature on community resilience indicates that an important but underexplored factor to building community resilience is social connectedness, harmony, and cohesion among community members, which have often been seen to help people deal with uncertainty after a disaster [2,3,4,5]. However, any threats (e.g., spreading hate speech and disinformation against target demographics online) that break such social harmony present risks to the wholistic community resilience. Thus, any attempt for community resilience assessment should be based on both the historic factors of local hazard experiences in the community as well as the emerging risks in the social atmosphere of the community. Resilience indicators from local to global levels have been developed by the U.S. Federal Emergency Management Agency (FEMA) [12] and the United Nations Office for Disaster Risk Reduction (UNISDR) [13]. New research initiated by the EU Horizon 2020 projects such as Resiloc also addresses the issue of developing resilience methods [14]. However, models or indexes as well as tools developed in these projects and initiatives do not factor in the dynamic risks associated with the human/social behavior of the local community members, especially in the era of social media where public could be manipulated [5, 12]. We will address this challenge of first characterizing the types of risks caused due to the violence-inducing behavior in social media that are harmful for the local community resilience, by taking a social cybersecurity approach. In this process, we will construct extendible artefacts of such characterization of malicious behaviors via an ontology and defining novel indexes with the resulting risks from social atmosphere for community resilience to empower community services.

Limited Studies on Social Cybersecurity and Violence-inducing Behaviors Online: Online content platforms, primarily social networking sites are observing an increasing trend of various types of aggressive behavior such as racism and sexism in the shared content, often manifesting through offensive or malicious language, or multimedia [9, 11, 12, 15]. Such behaviors pose a risk to the civil foundation of our communities, by promoting negative social construction of the diverse social identities such as race and gender that, in turn, divides our society and harms the social connectedness of local community members during disasters [5, 9, 10, 12].

Social cybersecurity has emerged as a new area of applied computational social science research to study malicious online behavior with two objectives [5, 12]: “(a) characterize, understand, and forecast cyber-mediated changes in human behavior and in social, cultural, and political outcomes; and (b) build a social cyber infrastructure that will allow the essential character of a society to persist in a cyber-mediated information environment that is characterized by changing conditions, actual or imminent social cyberthreats, and cyber-mediated threats.” In the context of our proposed research, we investigate how to characterize, understand, and detect malicious human behavior with implications to community resilience. While hate speech [1] is one type of relevant social behavior to our research, our aim is to specifically identify the comprehensive set of behavior classes [16], which could be associated to the varied types of risks posed to the community resilience. Such malicious behavior could lead to harmful implications for the local community services and their limited resources for interventions.

There is a growing interest in automatically detecting malicious social behavior online in social cybersecurity field, however, there are a variety of related conceptual definitions investigated in the literature [1, 9,10,11], such as ‘cyberbullying’, and ‘online harassment’. Schmidt & Wiegand [9] summarizes the diverse definitions of such malicious behaviors studied in the last two decades as follows: abusive messages, hostile messages or flames, cyberbullying, hate speech, insults and profanity, offensive language, and teasing messages. Therefore, the challenge to advance the research studying such behaviors in online social spaces from the community resilience perspective has been the definition of what is a violence-inducing behavior with risk implications for local community services. It is, therefore, crucial to develop corresponding novel datasets while preserving the privacy of individuals for building advanced computational methods to detect a comprehensive set of violence-inducing behavior, whether offensive, abusive, insulting, or other malicious behavior, as summarized above. We will focus on the content, rather than individual profile, for analyzing the malicious social behavior patterns. We scope the malicious social behaviors online to the most severe types of violence-inducing behaviors and focus on developing an ontology of such behaviors as well as the automated methods to detect them.

Next, we have identified the following two primary technical challenges that limit the performance of the AI methods for the detection of such malicious behaviors.

Sparsity of labelled datasets and major focus on single language:

AI methods for online malicious behaviors are primarily dependent on the labelled datasets for employing supervised or semi-supervised machine learning techniques to infer such behaviors from the online message content. It is difficult to capture all types of malicious behaviors in the labelled data despite the large-scale annotation tasks, due to the sparsity of the presence of such malicious behavior in the content and multiple ways in which these behaviors are expressed. For instance, Founta et al. [10] found the presence of only about 7% content containing the malicious behavior versus normal/harmless behavior in a large-scale annotation task conducted for the social media posts from Twitter. Additionally, recent literature has shown that performance of hate speech detection models trained on one targeted identity does not generalize to other targeted identities [17]. Therefore, datasets that contain a limited set of targeted identities may not be sufficient to train models that are capable of general hate speech detection. Furthermore, it could significantly affect the training process of models for low-resource languages, as the limited data may not represent all target identities. Furthermore, the existing labelled datasets [18] are primarily available for a few languages and only recent efforts include multilingual annotation tasks but focusing only on hateful behavior detection [19, 20]. It is due to the complexity of inferring diverse malicious behaviors from the natural language content that has led researchers to focus on a single language, mainly English [21]. Yet to the best of our knowledge, there is no effort made to develop Norwegian language datasets to encourage the design of AI methods for online violence-inducing behavior detection. This not only requires just an annotated dataset development but rather, a careful multidisciplinary understanding of qualitative context of malicious social behavior in Norwegian culture and society.

Proliferation of multimodal malicious behaviors and lack of AI detection methods:

The online content sharing has increasingly become multimodal in nature over the last few years—as per the leading network provider Cisco, online multimedia will make up more than 82% of all consumer internet traffic by 2022 [20]. Thus, we expect the likelihood of malicious behavior being expressed via multimodal content will increase over time. Currently, natural language text is the dominant modality to share content online across geographies, given the ease of sharing text across any online platform whether on social networking sites, news comment sections, or forums. Therefore, the existing literature has primarily focused on exploiting textual content using NLP methods [1, 11] for malicious behavior detection. There is a critical need to design AI methods for exploiting the multimodal content (e.g., Twitter posts with both text and images) to detect diverse malicious social behaviors that could easily get viral across the language boundaries of geography and culture, presenting a greater risk to the local communities across multiple geographical areas.

4 SOCYTI Approach

Fig. 1.
figure 1

Overview of the SOCYTI approach.

Full size image

The project aims at reaching its objectives by thoroughly studying various aspects of community resilience involving social connectedness and risks to social atmosphere through big data analytics, AI methods and by innovating on several scientific/technical components. Figure 1 illustrates the overarching approach used in the SOCYTI project. Specifically, we propose the following novel scientific advancement aligned to our three objectives:

Social scientists in SOCYTI team have conducted several questionnaires and interviews with the end users to identify scenarios that drive the next research activities. Furthermore, they have performed a qualitative analysis of the results of the questionnaires and interviews. This analysis and learnings from past research have provided us with the necessary knowledge about the risk factors associated with the social atmosphere to develop a community resilience index and an ontology for violence-inducing behaviors.

Next task involves creating a system to detect potentially violent social behavior from social media posts. More precisely, we are developing models to identify various forms of violent social behavior observed in online social media such as hate speech. As previously noted, there are numerous datasets available for detecting violent social behavior. Unfortunately, no such datasets are available in Norwegian, one of the core languages we aim to support in the SOCYTI project. Hence, we will undertake the annotation of social media data in the Norwegian language to broaden the linguistic coverage of our proposed system. This will enable us to train and assess our multilingual models. To do this, we have gathered social media data from Twitter posts that contain keywords associated with hate (as identified in Hurtlex [22]). We will label these posts as either hateful or non-hateful through a combination of manual review and predictions generated by a Large Language Model (LLM) [23].

Next, we will train hate speech detection models with the help of all collected datasets. Once we’ve trained these models, we’ll apply them for ongoing, real-time detection of hate speech in online social media. Following that, we will employ visualizations and reports to present a more comprehensible summary of these real-time instances of violence-inducing behaviors. This will empower the local community services with a real-time violence-inducing behavior system for online social media content. It allows better risk management for the public safety community services to improve their crisis communication strategies and campaigns to counter violence implications.

Fig. 2.
figure 2

The architectural approach to SOCYTI.

Full size image

5 The Proposed SOCYTI System

Our team proposed to develop the SOCYTI system as a toolset comprising various components for detecting and monitoring violence-inducing social behavior, specifically through hate speech detection methods as the foundation. Figure 2 illustrates the high-level architectural approach proposed for the system design.

We note that detecting hate speech using static models has the disadvantage of not being robust to domain shifts particularly when behaviors such as hate speech are targeted at different identity groups [17]. By leveraging knowledge bases (such as WordNet) we can augment the existing datasets to contain a substantial number of training examples for all identities. For example, we can find synonyms of a given identity from WordNet and replace the occurrences of the identity term in the examples with its synonyms to create more examples of similar ground truth label. By such augmentation, we aim to enhance the performance of hate speech detection for all identities. Nonetheless, a limitation of WordNet is its static nature. To address this issue, we will develop our own knowledge base, which supplements WordNet with additional information obtained through automated data mining techniques, leveraging the assistance of large language models. Furthermore, with multilingual WordNet and multilingual language models, we can extend the aforementioned process to support multiple languages. Here, a multilingual knowledge graph such as Open Multilingual Wordnet (OMW) [24], and DBpedia [25] provide cross-lingual interpretations for similar entity mentions in multilingual social media content, which improves explicit context representation (e.g., a religious group of a violence-target mention in a text) to let the algorithms learn patterns of violence-inducing behavior across languages efficiently. Furthermore, we will use those knowledge bases and continuous training of models to adapt to evolving social discourse. Apart from the textual content related to violence-inducing behavior, algorithms can perform better for multilingual online content when exploiting the multimodal information from both text and multimedia objects, which helps augment feature space and capture the context of malice more effectively than the approach of inferring the behavior using features from only text.

Furthermore, the SOCYTI system will provide customizable components for detecting violence-inducing behavior expressed through hate speech in different modalities of data including text and multimedia data. To facilitate multimodal hate speech detection, our proposal involves training a model that encodes various types of inputs, such as images, into their respective vector representations. These representations are then concatenated, and classification is performed on the combined input.

6 Conclusion

In conclusion, this work provides an overview of ongoing SOCYTI project, outlining its scope and objectives, and emphasizing the significance of hate speech detection in reducing violence-inducing behaviors within online communities. We also explore the background literature for this research and highlight the limitations and challenges associated with the existing methods. Lastly, we explain the approach employed in the SOCYTI project and provide insights into the proposed hate speech detection system that will be utilized for identifying violence-inducing behaviors in social media.