Introduction

With the growing enthusiasm for sharing scholarly information via Twitter, tweets mentioning scientific papers (hereinafter “scholarly tweets”) are widespread. In the altmetrics realm, Twitter has arguably become one of the most crucial data sources, with more than one third of recent scientific papers being tweeted (Fang et al., 2020a). Scholarly tweets, therefore, have long been seen as measureable traces possibly capturing the impact of research outputs in a broader sense (Bornmann & Haunschild, 2016; Eysenbach, 2011).

Instead of merely serving as countable information carriers bringing scientific papers to the attention of Twitter users, scholarly tweets per se are also informative in terms of the content incorporated, the characteristics of users involved, as well as the possible user engagement triggered. In other words, the creation of scholarly tweets stands not only for an outcome of Twitter reception of science by users who posted them, but also a prologue of another narrative about how other users might interact with them in the Twitter universe, being relevant to quantitative elaboration of science–social media interactions (Costas et al., 2021).

Scholarly tweets as the objects of study

Díaz-Faes et al. (2019) proposed the umbrella term secondary social media metrics to conceptualize metrics taking “social media objects” (i.e., social media users and their online activities) as the objects of study, distinguishing them from primary social media metrics which focus on “research objects” (e.g., publications, datasets, journals, and individual scholars), in particular “the use and visibility of publications on social media”. To date, in the direction of secondary social media metrics, many research efforts centering on scholarly tweets have been made to characterize the mechanisms of how Twitter users process, circulate, and engage with scholarly information from different perspectives.

At the tweet level, tweet content analyses provide straightforward insights into the tweeting behavior of users who are disseminating scholarly information. For example, by scrutinizing the content of scholarly tweets received by the top ten most tweeted papers in the field of dentistry, Robinson-Garcia et al. (2017) exemplified the scarce existence of original thought but more mechanical nature of the bulk of tweet content. Similarly, with a case study containing 270 tweets, Thelwall et al. (2013) reported that the majority of the observed scholarly tweets only echoed a paper title or presented a brief summary. Regarding the sentiment of tweet texts, scholarly tweets were found to be generally neutral, with limited share showing positive or negative sentiment expressed by users (Friedrich et al., 2015; Thelwall et al., 2013). Besides, the use of some tweet features in scholarly tweets, such as hashtags (word or phrase prefixed with #) and user mentions (user’s handle name prefixed with @), was also of interest by some altmetric research (Haustein et al., 2014a; Xu et al., 2018), because it represents a particular form of user interactions enhancing the description and visibility of tweets and facilitating connections amongst users (Haustein, 2019; Holmberg et al., 2014).

At the user level, the presence of scholarly tweets makes it possible to recognize and characterize users discussing science on Twitter. Scholarly tweets, therefore, were drawn upon for identifying and classifying Twitter users participating in scholarly communication (Costas et al., 2020; Díaz-Faes et al., 2019; Vainio & Holmberg, 2017; Yu et al., 2019), and for further exploring how users by type performed differently while utilizing Twitter for scholarly communication (Didegah et al., 2018; Holmberg & Thelwall, 2014; Mohammadi et al., 2018; Yu, 2017). Besides the research on the characterization of individual users who posted scholarly tweets, the Twitter uptake and usage of various scholarly entities, such as scholarly book publishers (Wang & Zuccala, 2021), scholarly journals (Zheng et al., 2019), and universities (Yolcu, 2013), is also an important topic in the direction of secondary social media metrics. Moreover, the aforementioned objects derived from scholarly tweets, either at the tweet or user level, were not only studied separately, but sometimes networked in different ways to map the contexts in which Twitter interactions with science happened. The network methods include but not limited to co-occurrence of hashtags (Haunschild et al., 2019), co-occurrence of users and hashtags (Hellsten & Leydesdorff, 2020), user mentions network (Said et al., 2019), and follower/friend network of users (Alperin et al., 2019; Robinson-Garcia et al., 2018), which were collectively conceptualized as heterogeneous couplings by Costas et al. (2021).

User engagement behaviors around scholarly tweets

In addition to tweet content and user characteristics, user engagement behaviorFootnote 1 around scholarly tweets is also a focal point of secondary social media metrics. Conceptually speaking, scholarly tweets offer the possibility for a wider range of users to participate in science-focused discussions through many engagement behaviors enabled by Twitter. In the current platform version (2021), Twitter provides several engagement functionalities for users to interact with tweets on their own initiative. As illustrated with a tweet example in Fig. 1, there are four main types of engagement functionalities with corresponding metrics visible at the bottom of tweets and publicly retrievable through the Twitter API, including (1) like, (2) retweet, (3) quote tweet, and (4) reply. These engagement behaviors differ in both input and output. In terms of input, liking and retweeting are relatively basic and simple engagement behaviors because they are both devoid of extra original content added, whereas quoting and replying are comparatively more informative and conversational because they enable users to express original thought and content. As to output, except liking, the other three types of engagement behaviors can produce new tweets (i.e., retweets, quote tweets, and replies) which are accounted for users’ total number of tweets posted.

Fig. 1
figure 1

User engagement functionalities and metrics embedded in a tweet example

Full size image

In existing altmetric literature, retweeting is the most analyzed user engagement behavior. In 2010, Priem and Costello (2010) found that retweets, as the outcomes of retweeting behavior, only made up 19% of a sample of scholarly tweets posted by 28 academic users. However, in more recent research, it was generally reported that retweets accounted for close to or over half of scholarly tweets of papers (Alperin et al., 2019; Didegah et al., 2018; Haustein, 2019), being a key component of the data base of studies related to scholarly Twitter metrics. For a sample of tweets posted by the Twitter accounts of 25 U.S. health agencies, Bhattacharya et al. (2014) found that about one third of them had zero retweet while the rest were retweeted at least once. As a form of information diffusion in nature, retweets were often analyzed to help capture topics of the public’s interest in sharing (Bhattacharya et al., 2014; Kahle et al., 2016), or to construct Twitter dissemination networks of scientific knowledge across communities (Araujo, 2020; Hassan et al., 2019).

Besides retweeting, other types of user engagement behaviors, such as liking, replying, and clicking, were also studied to help yield insights into whether and how the public engages with scholarly information on Twitter. For instance, considering a spectrum of user engagement metrics (e.g., retweets, likes, replies, clicks on tweeted URLs), Kahle et al. (2016) studied the rates of user engagement with the tweets posted by the official Twitter accounts of the European Organization for Nuclear Research (CERN). Mohammadi et al. (2018) surveyed the motivations behind users’ liking and retweeting behaviors in scientific contexts and reported that most survey respondents liked a tweet to “inform the authors that their tweets were interesting” and retweeted for the sake of disseminating the tweets. Based on the click metrics data provided by Bitly for its generated short links, Fang et al. (2021) analyzed the click rates on Twitter of short links referring to scientific papers and concluded that nearly half of the studied scholarly short links were not clicked by Twitter users at all.

Objectives

Although some attempts so far have been made to enhance the understanding of how people react to scholarly tweets, existing literature generally focused on either a specific user engagement behavior or a specific discipline. Little is known about the overall picture of the coverage of diverse types of user engagement with science on Twitter. User engagement metrics are promising component of secondary social media metrics, because they have the potential to assess the effectiveness of scholarly tweets in disseminating scholarly information, thus offering the possibility to measure the attention received by scientific papers from wider audiences. Given that sufficient data presence is one of the key preconditions for applying metrics in practice (Fang et al., 2020a), it is important to unravel the overall presence of different user engagement metrics amongst scholarly tweets in advance. Against this background, on the basis of a large-scale and cross-disciplinary dataset, the main research objective of this study is to systematically unravel the extent to which scholarly tweets are related to different categories of user engagement. Specifically, this study sets out to address the following research questions (RQs):

RQ1:

To what extent do Twitter users engage with scholarly tweets through different types of behaviors (i.e., liking, retweeting, quoting, and replying)?

RQ2:

Which research fields tend to attract the highest levels of user engagement on Twitter?

RQ3:

How does user engagement (both presence and frequency) correlate with other types of science-based and Twitter-based factors, such as the scholarly impact of tweeted papers, use of tweet features on the platform, and user characteristics?

Data and methods

Dataset

We retrieved a total of 6,229,001 Web of Science-indexed (WoS) papers published between 2016 and 2018 from the CWTS in-house database, and searched their scholarly tweets recorded by Altmetric.com until October 2019. For the matching with Altmetric.com data, WoS papers are restricted to those with DOI or PubMed ID assigned. On the whole, there are 1,999,199 WoS papers (accounting for 32.1%) with at least one scholarly tweet received, totally generating 7,037,233 unique original scholarly tweets.Footnote 2 Note that to explore user engagement behaviors, in this study the analyzed scholarly tweets are limited to original tweets which can be engaged with through the engagement functionalities provided by Twitter.

For the approximately 7 million scholarly tweets in our dataset, we retrieved their engagement metrics (i.e., number of likes, retweets, quotes, and replies received) with the Twitter API in February 2021.

The CWTS publication-level classification

To compare the user engagement situations of scholarly tweets across subject fields of science, we applied the CWTS publication-level classification system (Waltman & Van Eck, 2012) to assign scholarly tweets with subject field information based on their mentioned scientific papers. The CWTS classification clusters WoS papers into micro-level fields based on their citation relationships. These micro-level fields are then algorithmically assigned to five main subject fields of science, including Social Sciences and Humanities (SSH), Biomedical and Health Sciences (BHS), Physical Sciences and Engineering (PSE), Life and Earth Sciences (LES), and Mathematics and Computer Science (MCS).Footnote 3 For our dataset, there are a total of 5,932,279 scholarly tweets (accounting for 84.3%) referring to scientific papers with the subject field information assigned by the CWTS classification system. This set of scholarly tweets was drawn as a subsample for studying the subject field differences of user engagement. Table 1 presents the distribution of the analyzed scientific papers and scholarly tweets across the five subject fields of science.Footnote 4

Table 1 Five subject fields of the CWTS publication-level classification system
Full size table

Science-based and Twitter-based factors of scholarly tweets

To explore how the presence of user engagement behaviors associates with scholarly, tweet, and user-related factors of the scholarly tweets, we extracted a total of ten factors from the following three dimensions: (1) scholarly impact of tweeted papers, (2) use of tweet features, and (3) user characteristics of those who posted scholarly tweets.

As listed in Table 2, in the dimension of scholarly impact of tweeted papers, we selected WoS citations and Mendeley readers to reflect the impact of tweeted papers in the science landscape. In the dimension of tweet features, we focused on the use of hashtags (a word or phrase preceded by the sign #) and user mentions (a user handle name preceded by the sign @). Number of hashtags used, and number of users mentioned in tweets were analyzed to reflect how users edit their scholarly tweets with such interactive tweet features. Last but not least, in the dimension of user characteristics, we studied six factors related to users’ demographics and behaviors on Twitter. Thus, number of followers and number of lists in which users are listedFootnote 5 represent social media capital held by users, because these two factors largely affect how broad the audiences can be reached for posted tweets. Number of friends and number of likes given tell the story of how active users interact with other users by following others or liking their tweets. Number of tweets posted by users and their science focus depict users’ overall tweeting behavior. The former indicates how frequent users post all kinds of tweets, while the latter implies how concentrated users are on tweeting scientific papers.

Table 2 Analyzed factors related to scholarly tweets
Full size table

In the “Results” section, the correlations between the four analyzed user engagement metrics and the ten factors were studied based on not only the Spearman correlation analysis of counts (performed by R), but also the visualized change trend of the coverage of user engagement among scholarly tweets aggregated at the different levels of each studied factor (coverage of user engagement refers to proportion of scholarly tweets with at least one specific user engagement received).

Regression analysis: hurdle model

To further investigate how the ten different factors listed in Table 2 can predict user engagement, a regression analysis was implemented by using these ten factors as explanatory variables, as well as the four user engagement metrics (i.e., likes, tweets, quotes, and replies) as response variables. As presented later in the Results section, in consideration of that all of the four user engagement metrics are count data and the data are over-dispersed (data with the variance much greater than the mean value) and zero-inflated (data with excess zero values), we adopted hurdle models (Mullahy, 1986) as our regression models. Given that social media engagement data were generally found to be count data with the characteristics of zero-inflation and over-dispersion, hurdle models have been applied by many previous research to model user engagement on different social media platforms like Twitter (Bhattacharya et al., 2014), Facebook (Bhattacharya et al., 2017; Bohn et al., 2014), and Weibo (Fu & Chau, 2013).

Hurdle model is a two-part regression model that processes the count data as two separate components: one is the zero portion modeling whether an observation takes zero value or non-zero value (typically a binary logit model), while the other is the count portion fitting those non-zero values (a zero-truncated negative binomial model used in this study in consideration of the over-dispersion of the count data). In our case, the zero portion in the hurdle models determines whether a scholarly tweet gets at least one specific user engagement or not, while the count portion models how many times it is engaged with through certain behavior. Therefore, the hurdle models of user engagement metrics allow for the simultaneous modeling of both the odds for scholarly tweets of being engaged with, as well as the frequency of being engaged with by users. We employed the pscl package in R (Zeileis et al., 2008) to construct four hurdle models (mode 1: likes; model 2: retweets; model 3: quotes; model 4: replies).

Results

The Results section consists of four parts. The first part exhibits the overall presence of the four types of user engagement (i.e., likes, retweets, quotes, and replies) among the 7 million scholarly tweets in our dataset. The second part compares the presence of user engagement across scholarly tweets in different subject fields. The third part investigates how both the presence and frequency of user engagement relates to the three dimensions (i.e., scholarly impact, tweet features, and user characteristics) and ten factors listed in Table 2. The last part focuses on the hurdle regression of user engagement metrics.

Overall user engagement with scholarly tweets

Figure 2 illustrates the coverage of the four types of user engagement among the 7 million scholarly tweets. About 52% have been engaged with through at least one of the four analyzed engagement behaviors, namely, the overall coverage of user engagement among scholarly tweets is 52%. Around 20% of scholarly tweets were engaged with by users through only one type of engagement behavior, while as low as 2% got all the four types of engagement. More specifically, the coverage of likes is 44%, followed by retweets 36%. Liking and retweeting appear to be the most prevalent engagement behaviors around scholarly information. In contrast, the coverage of both quotes and replies is relatively scarce. Only 9% of scholarly tweets got quoted by users, and as low as 7% received at least one reply.

Fig. 2
figure 2

Coverage of the four types of user engagement. Each square represents 1% of scholarly tweets in our dataset. A square tinted with specific color(s) indicates that its represented 1% of scholarly tweets got corresponding type(s) of user engagement. (Color figure online)

Full size image

Table 3 presents the descriptive statistics of the four engagement metrics to further reflect the extent to which scholarly tweets are engaged with. The coefficient of skewness and quartiles indicate that the distribution of all of the four types of engagement metrics is highly skewed. Only a few scholarly tweets got considerable user engagement, while the majority were never or rarely engaged with by Twitter users. Liking is the most widespread engagement behavior, contributing the most to user engagement metrics, followed by retweeting. On average, scholarly tweets in the dataset have been liked 2.95 times and retweeted 1.91 times. However, quoting and replying are more rare engagement behaviors, with only 1% of scholarly tweets being quoted for at least 3 times (99th percentile of quotes is 3) or replied for at least 2 times (99th percentile of replies is 2), suggesting that only a very limited share of scholarly tweets successfully aroused users’ interest in sharing and communicating their thought within Twitter conversations.

Table 3 Descriptive statistics of the four user engagement metrics
Full size table

User engagement across subject fields

Figure 3 shows how the coverage of the four types of user engagement varies across the five subject fields of science: Social Sciences and Humanities (SSH), Biomedical and Health Sciences (BHS), Physical Sciences and Engineering (PSE), Life and Earth Sciences (LES), and Mathematics and Computer Science (MCS). Overall, scholarly tweets mentioning SSH papers are more likely to be engaged with through any type of engagement behavior studied. For the field of SSH, the proportion of scholarly tweets with at least one engagement record always ranks first, suggesting the higher probability for SSH-related scholarly information to be engaged with by Twitter users over other subject fields. Besides, scholarly tweets from the fields of LES and BHS also present a relatively stronger potential in attracting different types of user engagement. In comparison, MCS and PSE are the two subject fields with sparser user engagement behaviors observed, showing the lowest coverage of all kinds of studied engagement metrics.

Fig. 3
figure 3

Proportion of scholarly tweets with different levels of A likes, B retweets, C quotes, and D replies across the five subject fields of science. (Color figure online)

Full size image

The descriptive statistics of user engagement metrics across subject fields presented in Table 4 reinforces the disciplinary differences observed in Fig. 3. Considering the greatest values of indicators highlighted in bold for each engagement metrics (if exists), SSH shows the most extensive distribution of all kinds of user engagement, thus acting as the most active subject field in giving rise to engagement with science on Twitter, followed by LES. BHS, as the subject field with the most scholarly tweets, contributes the most to the overall engagement metrics data due to the largest total number of corresponding engagement records. Besides, scholarly tweets of BHS papers also have a relatively higher presence of user engagement. However, user engagement is confirmed to be sparsely distributed among scholarly tweets in the fields of MCS and PSE.

Table 4 Descriptive statistics of the four engagement metrics across the five subject fields
Full size table

Correlation analysis of user engagement metrics

In this part, we examine the correlation between different types of user engagement metrics and the ten factors listed in Table 2. We aggregate scholarly tweets into different levels based on the values of each factor, then graphically show the coverage of specific user engagement for scholarly tweets aggregated at different levels of the corresponding factor. We also consider the results of the Spearman correlation analysis between the studied factors and user engagement metrics to further interpret the observed relationships.

From the perspective of scholarly impact of tweeted papers, Fig. 4 plots the change trend of the coverage of user engagement with the increase of (A) citations and (B) Mendeley readers of tweeted papers. Overall, the coverage of all kinds of user engagement is slightly higher for scholarly tweets mentioning papers with higher levels of citation counts and Mendeley readers accrued, although the uptrends are not that strong, particularly for citations. According to the Spearman correlations (see Fig. 7 in the “Appendix”), the four types of user engagement metrics are all positively but negligibly correlated with citations and readers (the coefficient \({r}_{s}\) ranges from 0.016 to 0.048 for citations, and ranges from 0.051 to 0.107 for readers).

Fig. 4
figure 4

Coverage of the four types of user engagement among scholarly tweets with different levels of A WoS citations and B Mendeley readers received by tweeted papers

Full size image

Regarding tweet features used in scholarly tweets, Fig. 5 shows the coverage of user engagement when different (A) numbers of hashtags are used, and different (B) numbers of users are mentioned in tweets. These two tweet feature factors present different patterns in their relationships with user engagement. As the number of hashtags per tweet increases, a slight rise can be observed in the coverage of likes, retweets, and quotes, but not for replies. This is confirmed by the positive and negligible correlations found between number of hashtags and number of likes, retweets, and quotes received by tweets (\({r}_{s}\) ranges between 0.042 and 0.113), whereas nearly no correlation found between number of hashtags and number of replies (\({r}_{s}\) = − 0.001). By comparison, findings show that when there is an increase in the number of users mentioned in tweets, the engagement trend is stronger, and this is especially for likes and retweets. When more than two users are mentioned in a tweet, the potential for being liked is over 80%, and the potential for being retweeted is 70%. Similarly, the coverage of both quotes and replies is relatively higher for scholarly tweets with more users mentioned than those without any mentioned users. Correspondingly, the Spearman correlations between user engagement metrics and number of mentioned users are comparatively stronger than other factors mentioned earlier, particularly for likes (\({r}_{s}\) = 0.237) and retweets (\({r}_{s}\) = 0.229).

Fig. 5
figure 5

Coverage of the four types of user engagement among scholarly tweets with different numbers of A hashtags used and B users mentioned in tweets

Full size image

In terms of user characteristics, Fig. 6 shows the coverage of user engagement for scholarly tweets posted by users with different characteristics. Specifically, Fig. 6(A) and (B) exhibit the change trend of the likelihood of being engaged with for scholarly tweets from users with different levels of followers and lists listed, respectively. These two factors, which to a large extent determine the audience size of posted tweets, are positively associated with the coverage of user engagement. The more followers that users accrue or the more lists that users are listed in positively correlate to the higher probability for their tweets of getting engagement. Based on the Spearman correlation analysis, number of followers of users is moderately correlated with both number of likes and retweets received (\({r}_{\mathrm{s}}\) = 0.349 and 0.368, respectively).

Fig. 6
figure 6

Coverage of the four types of user engagement among scholarly tweets posted by users with different levels of A followers, B lists listed, C friends, D likes given, E tweets posted, and F science focus

Full size image

Figure 6(C) and (D) show the relations between user engagement and the two factors about users’ interaction activity: number of friends and number of likes given. Similarly, these two factors also keep positive relationships with the coverage of user engagement. Overall, scholarly tweets posted by more active users (who interact more frequently with others by following other users and liking other users’ tweets) tend to show a relatively higher probability to be engaged with. The Spearman correlation analysis proves that there exist weak to moderate correlations between user engagement metrics and the two factors about users’ interaction activity (\({r}_{s}\) ranges from 0.086 for the correlation between number of friends and number of quotes up to 0.313 for the correlation between number of likes given and number of likes received).

Different from the patterns observed for the above factors, as shown in Fig. 6(E) and (F), number of tweets posted and science focus of users, which depict the overall tweeting activity of users, show negative relationships with the coverage of user engagement among scholarly tweets. In general, the greater number of tweets posted, as well as the stronger science focus of users, the lower levels of coverage of user engagement. These negative relationships are reinforced by the negative coefficients reported in the Spearman correlation analysis between these factors and user engagement metrics (\({r}_{s}\) ranges from − 0.147 to − 0.028).

Regression analysis using hurdle models

Given the statistically significant correlations existed amongst the studied factors, to further compare how different science-based and Twitter-based factors serve as predictors of user engagement to varying degrees, we performed regression analyses of the four types of user engagement using hurdle models. In order to avoid multicollinearity in the regression models, in each of the three dimensions discussed above, we selected several representative factors as the explanatory variables. For instance, in the dimension of scholarly impact of tweeted papers, since citations and Mendeley readers are strongly correlated with each other (\({r}_{s}\) = 0.712), we selected citations as one of the explanatory variables. In the dimension of tweet features, both number of hashtags used and number of mentioned users were included because they are weakly correlated (\({r}_{s}\) = 0.181). In the dimension of user characteristics, number of followers keeps moderate to strong correlations with other homogeneous user factors (\({r}_{s}\) ranges from 0.433 to 0.859) except science focus of users (\({r}_{s}\) = 0.015), so we adopted number of followers and science focus as representative variables in this dimension.

Table 5 reports the results of the zero portion of the four hurdle models of user engagement metrics (logit models): model 1 (likes), model 2 (retweets), model 3 (quotes), and model 4 (replies). Some variables were log-transformed for better model fitting. The zero portion of the models reflects how the selected explanatory variables relate to the odds for scholarly tweets of attracting at least one specific user engagement. The four models present a similar pattern in the zero portion, with citations, mentioned users and followers positively associated with the odds of getting at least one corresponding user engagement, whereas science focus presents a negative association. The exception is hashtags: in model 1 (likes), mode 2 (retweets) and model 3 (quotes), number of hashtags has a positive relationship with the odds that at least one like, retweet or quote occurs, however, in model 4 (replies), number of hashtags presents a negative association. The odds ratios (OR, exponent of regression coefficient in logit model) were calculated to help interpret the results. For example, in model 1 (likes), while all other variables remaining constant, a one-unit increase in the log-transformed number of mentioned users increases the odds of getting at least one like by 185.6% (OR 2.856). However, while holding all other variables constant in model 1 (likes), a unit increase in science focus decreases the odds of getting at least one like by 42.7% (OR 0.573).

Table 5 Results of the zero portion of the hurdle models of user engagement metrics
Full size table

Table 6 reports the results of the count portion of the hurdle models of user engagement metrics (zero-truncated negative binomial models). The count portion focuses on those scholarly tweets with at least one corresponding user engagement received and indicates how the explanatory variables associate with the increase of the frequency of user engagement. As is evident in all the four models, citations, mentioned users as well as followers are all positively associated with the frequency of user engagement, yet hashtags and science focus are negatively associated with the frequency of user engagement. Similarly, incidence rate ratios (IRR, exponent of regression coefficient in negative binomial model) were computed to help interpret the coefficient of a given variable while holding all other variables constant. For instance, in model 1 (likes), while all other variables remaining constant, a unit increase in the log-transformed number of followers increases the frequency of likes by a multiple of 1.484, while a unit increase in the log-transformed number of hashtags decreases the frequency of likes by a multiple of 0.908.

Table 6 Results of the count portion of the hurdle models of user engagement metrics
Full size table

Discussion

As discussed by Brossard and Scheufele (2013), in the era of mass media, science stories as well as their accuracy, importance and popularity are no longer just “presented in isolation but instead are embedded in a host of cues that accompany nearly all online news stories”, such as comments on blog posts and user engagement on social media. In the context of scholarly Twitter metrics, this argument, on the one side, highlights the importance of the examination of user engagement in studying science-social media interactions, but on the other side, poses a question about how many scholarly tweets indeed triggered user engagement which are believed to contain extra meaning added to science stories.

Although user engagement with scholarly tweets have long been seen valuable for characterizing the interactions between scholarly objects and social media (Wouters et al., 2019), there is still an overall lack of evidence which can be drawn upon to mirror how effectively scholarly tweets attract the public’s attention and further stimulate public engagement in Twitter conversations around science. Based on a large-scale and cross-disciplinary dataset, this study unravels the coverage of diverse types of user engagement among scholarly tweets, thus offering an answer to the question about the overall presence of public engagement with scholarly information on Twitter.

Overall presence of user engagement with scholarly tweets

In this study, with likes, retweets, quotes, and replies as the traces of user engagement, we found that likes and retweets were present for about 44% and 36% of the studied scholarly tweets, respectively. However, the coverage of quotes and replies is as low as 9% and 7%, respectively. The globally low presence of user engagement, particularly for the engagement behaviors with more informative outcomes generated, reveals the fact that the attention paid to scholarly tweets varied a lot on the one hand, and puts more emphasis on the significance of more in-depth measurement of Twitter reception of scientific papers on the other hand. For papers with exactly the same number of scholarly tweets accumulated, although the papers’ Twitter reception appears to be equal only based on their absolute number of scholarly tweets, those with scholarly tweets being widely engaged with might be disseminated and perceived on Twitter in a more effective way. This is because engagement behaviors provide concrete evidence that they reached out to audiences who also showed further interest.

It is notable that there are 2% of scholarly tweets that have accrued all the four types of user engagement, being the widely engaged tweets in our dataset. For these widely engaged scholarly tweets, it would be an important future research agenda to explore the mechanisms by which they succeed in attracting such extensive user engagement. From a theoretical point of view, such case study will help to unravel the driving force of Twitter engagement as well as the nature of Twitter attention towards scholarly information. From a practical point of view, it will shed some light on how to utilize Twitter as an effective tool to promote science communication, particularly for scholarly journals and science journalists.

Disciplinary patterns of user engagement with scholarly tweets

The presence of user engagement differs by subject field. As the subject fields found to be more frequently mentioned in the Twittersphere (Costas et al., 2015; Haustein et al., 2015), SSH, LES, and BHS also have their scholarly tweets more actively engaged with by users through liking, retweeting, quoting, and replying, outperforming the fields of PSE and MCS. Therefore, disciplinary patterns are confirmed for not only Twitter uptake of scientific papers as reported by previous research, but also Twitter user engagement as analyzed in this study.

There are multiple possible reasons for the consistent disciplinary patterns of Twitter activities towards SSH, BHS, and LES, such as the lay audiences’ preference for topics related to social issues, environmental problems, and healthcare (Haustein et al., 2014b, 2015), the higher degree of Twitter uptake by scholars from these fields (Costas et al., 2020; Mohammadi et al., 2018), and the stronger enthusiasm shown by scholars in these fields for communicating scholarly information with the public (Bentley & Kyvik, 2011; Kreimer et al., 2011). To further interpret the subject field differences, future research is needed to scrutinize the contexts in which user engagement takes place (e.g., engaging users’ identity and motivations) across subject fields.

Factors related to user engagement with scholarly tweets

Based on both correlation and regression analyses, we investigated the relationships between user engagement and a spectrum of science-based and Twitter-based factors, which still remain unverified on the basis of a large-scale and cross-disciplinary dataset. Through the lens of a series of other factors, a comprehensive analysis will help improve our understanding of the nature of user engagement and its position in the ecosystem of science-social media interactions. In addition, user engagement metrics exhibit the potential for inclusion in the altmetric toolkit, because they can measure social media attention received by scientific papers at a deeper level. Given the potential of user engagement metrics for research evaluation and more advanced social media metrics, a comprehensive analysis will be helpful to figure out whether these potential indicators can, for example, predict future scholarly impact of papers, or whether they can be predicted in advance by other Twitter-based factors.

In contrast to science-based factors, Twitter-based factors generally tend to be more related to user engagement. Specifically, from the perspective of tweet features, although hashtags and user mentions are both tweet features increasing the visibility of tweets, the former is utilized to label and broadcast tweets to potential users interested in the same topics, while the latter is targeted to specific users with notifications delivered to them, showing a more conversational nature than the former. As a result, number of users mentioned in tweets is more related to user engagement and more effective in predicting user engagement. From the perspective of user characteristics, both users’ social media capital (i.e., followers and lists listed) and interaction activity (i.e., friends and likes given) were positively correlated with user engagement around their tweets. Nevertheless, flooding the screen (i.e., too many tweets posted) and attaching to tweeting only scientific papers (i.e., too strong science focus) were found to be related to lower levels of user engagement. From a practical point of view, as suggested by Cheplygina et al. (2020) for scientists getting start on Twitter, building a community by interacting with others, as well as sharing something personal and non-academic can also be relevant to get support in science communication on Twitter.

This study took into account scholarly tweets written in all languages. With this we provide a relatively complete picture of user engagement regardless of the language of the tweets. However, more detailed tweet content analysis should be applied, considering different linguistic contexts, as well as more local topics and sentiment, which were not included in this study. Given that specific tweet content like those including awe-inspiring imagery and newsworthy items frequently attract high levels of engagement across social media platforms (Kahle et al., 2016), future research should also focus on how different tweet content might be related to subsequent user engagement in scholarly contexts.

Limitations

There are several limitations in this study. First, there are more than four types of engagement behaviors that users can take to interact with scholarly tweets, such as clicking on tweeted scholarly URLs, clicking on users’ profile, and adding tweets to bookmarks. However, these engagement metrics are currently not publicly accessible on a large scale, and they were not included in this study. Should these engagement metrics be made publicly retrievable in the future, a more complete picture of user interaction behaviors around science could be drawn. Second, although reply tweets and quote tweets, which are outcomes of replying and quoting behaviors, can be further engaged with through Twitter functionalities as well (e.g., liking or retweeting replies and quote tweets), they were not included in the analyzed dataset to avoid double counting. Lastly, we explored how the characteristics of engaged users (i.e., users whose tweets are engaged with) relate to user engagement, while the characteristics of engaging users (i.e., users who engaged with tweets) are also of relevance to the interpretation of the occurrence of engagement behaviors. However, due to the lack of detailed information of the engaging users, their demographics and behavioral patterns were not analyzed in this study.

Conclusions

This study contributes to the expansion of the idea of secondary social media metrics (Díaz-Faes et al., 2019) by presenting a large-scale and cross-disciplinary analysis of four types of user engagement (i.e., liking, retweeting, quoting, and replying) around scholarly tweets. The presence of user engagement amongst scholarly tweets is globally low, particularly for those behaviors with higher levels of engagement needed (e.g., quotes and replies). Scholarly tweets from the fields of SSH, LES, and BHS tend to have more frequent user engagement distributed. Finally, the presence of user engagement is more related to other Twitter-based factors (mentioned users in tweets and number of followers of users in particular) than with science-based factors of papers (e.g., citations and Mendeley readers), implying both the intrinsically connected dynamics of Twitter elements and the distinguishing focuses between scientific and tweeting activities. Overall, our findings provide a first overview of the extent to which scholarly tweets are related to broader public engagement with science on Twitter, thereby paving the way towards the measurement of Twitter reception of science in a more interactive and comprehensive manner.