Keywords:

1 Introduction

The digitalization and related socio-technological innovations enable online users to actively engage in the creation and distribution of online content. Alongside this development, the scholarly interest in user-generated content has established a flourishing and growing field of research (Naab and Sehl 2017). One of the most widespread forms of public user-generated content are user comments (Newman et al. 2019; Stroud et al. 2016). User comments originate when users take the opportunity to post written messages stimulated by a news item (e.g., news article or video). They are published either in comment sections on websites or on the social media pages of news outlets (e.g., Facebook page of Frankfurter Allgemeine Zeitung). As soon as more users engage in commenting on the same news item and referencing others’ comments, an online discussion develops (Ziegele and Quiring 2013). The news outlets provide a selection of topics along journalistic criteria that are relevant to society and reach a potentially large audience. At the same time, they retain the decision-making power over the user comments published in their comment sections and can moderate or delete undesired comments or limit the comment sections to selected topics (Weber 2014).

Research examines user comments from a broad variety of perspectives. It investigates content characteristics (see below), comment authors (e.g., Springer 2014), regulation practices of harmful content (e.g., Heldt 2019; Stroud et al. 2015; Ziegele et al. 2019), and journalists’ and editors’ engagement with user comments (e.g., Meyer and Carey 2014). In addition, scholars examine a multitude of effects of user comments on the readers’ attitudes towards topics (e.g., Sikorski 2016) and social issues (e.g., Hwang 2018), their evaluations of journalistic content and behaviors (e.g., Naab et al. 2020). Lastly, scholars in many subfields of communication science examine user comments on specific topics of their interests. This reflects the relevance of user-generated content in the users’ media repertoires (Newman et al. 2019) and the mounting evidence on the effects of user comments on attitudes and behaviors. Among these are studies in the realm of political communication (e.g., Blassnig et al. 2019), health communication (e.g., Holton et al. 2014), science communication (e.g., Koteyko et al. 2013; Kraker et al. 2014; Walter et al. 2018), and many others. Although user comments barely represent the opinion climate on issues, they are often used to assess the „pulse of the public debate“ (Douai and Nofal 2012, p. 269).

2 Common Research Designs of Content Analyses and Combinations of Methods

Mostly, scholars fathom user comments’ content characteristics with standardized methods: We find a multitude of standardized content analyses (see below) and a number of qualitative text analyses (e.g., on opinion dynamics on socially relevant topics, Al-Saggaf 2006; on discussion quality, Graham and Witschge 2003; on linguistic features applied during discussions, Küchler and Naab 2018; Neurauter-Kessels 2011). What is more, a growing number of studies investigates user comments with automated and semi-automated approaches (e.g., Gardiner et al. 2016; Stoll et al. 2020). The increasing importance of computational methods in communication research in recent years and the comparably easy access to large data sets of public content from social media sites contribute to this development (Possler et al. 2019). In the field of user comments, automated detection of incivility and other forms of offensive speech in user comments receive utmost attention. There are no systematic reviews of content analyses on user comments. However, in their scoping review of studies on user-generated content in general, Naab and Sehl (2017) reveal that standardized content analyses dominate over discourse analyses, text analyses, and further qualitative approaches.

Rarely, studies apply a combined approach (similarly, Naab and Sehl, 2017). For example, Graham and Wright (2015) combine a content analysis of online discussions with qualitative interviews with journalists. Ziegele et al. (2014) combine a content analysis of discussion factors in user comments with qualitative interviews with users.

Cross-sectional analyses clearly dominate over longitudinal analyses (similarly, Naab and Sehl 2017), but exceptions provide valuable insights (e.g., Gardiner et al., 2016; Kraker et al., 2014; Wright et al. 2020). Comparisons between cultures, countries, or language regions constitute a minority within the field of content analyses (Naab and Sehl 2017; for an exception see Ruiz et al. 2011).

In content analytic studies of user comments, several researchers investigate contextual influences. For instance, they examine the effects of the discourse architecture of the platform hosting the comment section. For this reason, they compare comment sections on news websites with those on news outlets’ social media pages (Rowe 2014; Ziegele et al. 2014) or compare comment sections with differences in user registration, anonymity, moderation strategies, ethical frameworks, and various technological features (Freelon 2015; Ksiazek 2015; Ruiz et al. 2011; Santana 2014). Another context factor of interest is the news item that attracts the comments. Yet most studies restrain themselves to the influence of journalistic features on the amount of comments (e.g., on the influence of news topics, Almgren and Olsson, 2015; Liu et al. 2015; Tenenboim and Cohen 2015; of news factors in the articles, Weber 2014; of the deliberative quality of the articles, Marzinkowski and Engelmann 2018).

With regard to sampling it is fair to state that user comments on Facebook pages catch overwhelming scholarly attention (Jünger, in prep.). This can be attributed to the development that many news outlets have shifted their comment sections to Facebook (Su et al. 2018) and that Facebook is still among the most widely used social media platforms for news consumption (Pew Research Center 2015). Furthermore, Facebook has provided comparably easy access to its public content in the past. Additional studies (often undertaken by journalism scholars) focus on comment sections on the websites of news outlets. Be that as it may, other platforms have received much less attention (see for exceptions, e.g., analyses of YouTube comments, Djerf-Pierre et al. 2019).

Very rarely, content analyses are conducted as part of experimental designs. For example, in a field experiment, Stroud et al. (2015) varied, who engaged with the users of a comment section. They analyzed the effects on the quality of the succeeding comments. In a few laboratory experiments, scholars manipulated comments in a thread and content analyzed the responses of their participants (e.g., Chen and Lu 2017; Naab 2020).

3 Theoretical Frameworks and Main Constructs

Regarding content characteristics of user comments, the most prevalent theoretical framework is deliberation theory (Freelon 2015). This strand of research analyzes how far online discussions come close to the normative ideal of deliberative discussions (often referring to the notions of deliberation by Barber 1984; Gastil 2008; Habermas 1989). It claims that deliberative online discussions should include diverse viewpoints; participants should present well-reasoned arguments on the substantial issue; discussions should provide equal opportunity to speak for all; participants should refer to one another reciprocally and consider the others’ arguments while being respectful and authentic.

Studies investigate the heterogeneity of views in comment sections and views complement to the standpoints offered in the news articles they accompany. While some researchers claim that comments complement the journalistic products (Baden and Springer 2014; Douai and Nofal 2012; Jakobs 2014), others are more skeptical and find that comments rarely contain alternative opinions or complementary standpoints (Noci et al. 2010; Toepfl and Piwoni 2015).

Analyses of user comments’ valence reveal that comments on websites and Facebook pages of news media outlets are predominantly adverse and critical (Coe et al. 2014). They criticize aspects such as a lack of facticity and impartiality (Prochazka and Schweiger, 2016).

Further, research indicates that online discussions can present well-reasoned arguments and facts (e.g., by citing sources to back up claims, Graham and Wright 2015). However, several studies suggest that emotions dominate online discussions (Jakobs 2014; Lilienthal et al. 2014; Loveland and Popescu 2011; Taddicken and Bund 2010).

The interactive character of online discussion through reciprocal exchange between participants is a further matter of many analyses: Scholars examine the amount of reply comments, references to other users, journalists, or the content of other comments (Graham and Wright 2015). Whether other users’ views are reflected, is also examined by analyzing if comments pose questions (Graham and Wright 2015; Manosevitch and Walker 2009). Interactivity is rather low, since comment authors exchange with others only to a limited extent and dialogues between authors end after a few comments (Jakobs 2014; Loveland and Popescu 2011; Noci et al. 2010; Singer 2009; Ziegele et al. 2014).

Overwhelming scholarly attention is devoted to the analysis of uncivil and otherwise disrespectful speech in user comments (on the term incivility, Papacharissi 2004). This is in line with public debates about detrimental effects of offensive content in social media and about opportunities to regulate such content. Several empirical investigations show, depending on their study design, their measurement of incivility, and sampling technique, that up to 50 percent of the comments include personal attacks, obscenities, racism, sexism, and other forms of offensive speech (Coe et al. 2014; Diakopoulos and Naaman 2011a, 2011b; Ksiazek 2015; Noci et al. 2010; Ruiz et al. 2011). Yet, researchers support that, at least at the beginning of comment threads, discussions are coherent and focus on the issue under debate (Belli et al. 2000; Graham and Wright 2015; Noci et al. 2010; Ruiz et al. 2011). User comments mostly are assessed as mediocrely comprehensible and correct in language, grammar, and style (Jakobs 2014).

Besides analyses that are (more or less explicitly) tied to the normative concept of deliberation, studies also consider so-called discussion factors in user comments. They follow the idea of news factors in journalists’ news selection and presentation (Galtung and Ruge 1965). Ziegele et al. (2014) argue that “specific message characteristics in user comments stimulate response comments from subsequent users” (p. 1114). Among these characteristics are controversy, personalization, uncertainty, unexpectedness, comprehensibility, and negativity.

Aside from the actual content of the user comments themselves, scholars also investigate assumptions about the authors of such comments. For example, they code the gender of authors (e.g., in studies on the relationship between author gender and writing or receiving incivility, Gardiner et al. 2016; Küchler et al. 2019) or the professional status of participants (e.g., in studies on the engagement of journalists in the comment sections, Jakobs 2014; Wright et al. 2020).

4 Research Desiderata

Despite the extensive empirical literature, it is advisable to point out some desiderata of the existing field of research: Most studies refer to online discussions in Western cultures and only rarely do scholars take the effort to compare findings across countries or cultures. In their sample selection, most studies refer to user comments in the comment sections of the most read news outlets. While this is understandable considering that these comments will also have greater reach and potential effects, it disregards potential deviations in comment sections of less popular outlets. These might also have different discourse architectures, readers, and commenting patterns. The generalizability of the findings is further limited because user comments on Facebook pages of news outlets receive overproportionate attention while other social media platforms like YouTube, TikTok, or Instagram are barely examined.

With regard to the dimensions under investigation, Freelon (2015) argues for a closer examination of normative frameworks besides deliberation theory such as liberal individualism or communitarianism. Additionally, many researchers focus on detecting and describing forms of incivility and related forms of disrespectful speech in comments. While the social significance of these studies is uncontested, it should not divert scholarly attention from further characteristics of online discussions.

Relevant Variables in DOCA – Database of Variables for Content Analysis

Arguments: https://doi.org/10.34778/5d

Clarity: https://doi.org/10.34778/5e

Number of reply comments: https://doi.org/10.34778/5f