Twitter Journal Club Impact on Engagement Metrics of the Neurocritical Care Journal

Twitter journal clubs are a modern way of highlighting articles published in a scientific journal. The Neurocritical Care journal (NCC) initiated a bimonthly, Twitter-based, online journal club in 2015 to increase the outreach of its published articles. We hypothesize that articles included in the Neurocritical Care Society Twitter Journal Club (NCSTJC) had greater engagement than other articles published during the same time period. We also investigated the relationship between number of citations and Altmetric score to assess whether the enhanced online activity resulted in higher citations. We gathered data in August 2020 on engagement metrics (number of downloads, Altmetric score, relative citation ratio, and number of citations) of all articles published in NCC between 2015 and 2018. Articles were analyzed into two groups: one featured in NCSTJC and the rest that were not (non-NCSTJC1), and the other comprised those that were not in NCSTJC but published under a similar category of articles as NCSTJC (non-NCSTJC2). Results were analyzed using descriptive statistics, and summary measures were used to report the spread. The groups were compared by using the Wilcoxon rank sum test, given that the data were not normally distributed. Spearman’s rank correlation was used to assess correlation between Altmetric score and citations for the articles in the NCSTJC and non-NCSTJC groups. For comparison, the top ten cited articles in NCC were analyzed for similar correlations. Between 2015 and 2018, NCC published 529 articles, 24 of which were included in the Twitter journal club. A total of 406 articles were published in the same category as the category of articles selected for NCSTJC. The articles discussed as a part of NCSTJC had a statistically significant trend toward a higher number of downloads, Altmetric score, relative citation ratio, and number of citations than rest of the articles published in the journal during the same time period and the rest of the articles published in same categories. Three NCSTJC articles were among the ten most-cited articles published by NCC between 2015 and 2018. We did not find a correlation between Altmetric scores and number of citations in the NCSTJC or non-NCSTJC1 or non-NCSTJC2 group, but there was a strong correlation between these two variables in high performing articles when the top ten cited articles were analyzed. Scientific journals are evolving their social media strategies in attempt to increase the outreach of their articles to the medical community. Platforms such as Twitter journal clubs can enhance such engagement. The long-term influence of such strategies on the impact factor of a medical journal and traditional engagement metrics, such as citations, calls for further research.

. Over the years, this practice became an essential component of academic medicine. Journal clubs continue to be an important source of emerging medical knowledge: they provide education in scientific method and encourage academic discourse in which participants not only learn about the scientific content of the chosen article but also how to critically appraise scientific literature [2].
Over the last few years, social media has become a major platform for networking in medicine. Twitter, in particular, has become a popular marketing tool for posting online open access information for scientific journals [3,4]. Tweets notify the community when academic journals publish new articles or seek new articles from the community. The concise format of 280 characters in each tweet provides brief, snippet-based discussions that have become the social media correlation of 10-s sound bites. As readers of scientific journals shift from print media to online browsing, many journals have shifted to using platforms such as Twitter to engage with their audience and advertise their publications. Emerging data suggest a positive trend in performance metrics for journals with an increasing social media footprint [5,6]. The Neurocritical Care Society (NCS) initiated an NCS Twitter Journal Club (NCSTJC) in February 2015 to promote engagement with the journal through social media.
Given the increasing role of Twitter in the academic medicine publication outreach, we evaluated the impact of the Twitter Journal Club as measured by four engagement metrics: number of downloads, Altmetric score, relative citation ratio, and number of citations. We also investigated the relationship between the number of citations and Altmetric score to assess whether enhanced online activity resulted in higher citations. We intended to use the results from social media engagement created by the Twitter journal club to inform future social media strategies for the journal.

Methods
We included all scientific articles published in the Neurocritical Care journal (NCC) from February 2015 to coincide with the initiation of NCSTJC in February 2015 until December 2018. Articles were identified as NCSTJC articles by reviewing the hashtag #NCSTJC using Twitter. The journal club discusses a key article selected from the latest issue of NCC by a subgroup of the Communications Committee in consultation with Chief Editor of the journal [7]. The articles are chosen based on their overall contribution to the literature and the interest bound to generate among the journal club attendees. The club occurs bimonthly over 2 days. To increase accessibility, each chosen article is openly accessible, free to download, 2 weeks before and after the scheduled date for the journal club, courtesy of the publisher (Springer). All posts regarding the journal club typically include the hashtag #NCSTJC, registered with the Symplur Healthcare Hashtag Project. The members of the Communications Committee of the NCS take lead in facilitating Twitter journal club discussions that are freely accessible online and have a wide range of participation, depending on the topic. We reviewed the issue of selection bias carefully but could not identify a way to adjust for this, given the nature of the selection process.
Engagement metrics on each article were obtained in August 2020 from Altmetrics and from the publisher (Springer) and included number of downloads, Altmetric score, relative citation ratio, and number of citations [8]. The number of downloads was calculated by the number of times the article full text was downloaded from the Springer Web site for NCC. Altmetric Attention score is a propriety score that measures the online activity of the article among all Internet-based online platforms. The relative citation ratio is a National Institute of Health metric representing a citation-based measure of scientific influence. It is calculated as the cites per year of each article, normalized to the citations per year received by National Institute of Health-funded articles in the same field and year. The number of citations data were drawn from PubMed Central, European PubMed Central, CrossRef, and Web of Science. August 2020 was the time chosen to allow a set time interval (approximately 18 months) for the last group of articles discussed in journal club to gather stable metrics. Results were analyzed initially by using descriptive statistics; summary measures were used to describe the amount of variability or spread in a set of data. To assess the influence of NCSTJC on engagement and citation metrics, we performed a bivariate analysis for articles from the two groups-those reviewed in NCSTJC and those that were not (non-NCSTJC1). Because non-NCSTJC articles contained many categories of articles, such as editorials, letters to editors, neuroimages and translational research, we compared the NCSTJC article with non-NCSTJC articles included only under these categories (non-NCSTJC2), yielding 406 articles. Given the nonnormal distribution and asymmetric sample size between the two groups, the Wilcoxon rank sum test was employed as a nonparametric alternative to the two sampled Student's t-test for comparing two independent group of samples. Spearman's rank correlation was used to assess for correlations between Altmetric score and citations for the articles in two groups. To address the issue of selection bias, we created a matched set of non-NCSTJC2 articles on the basis of the Altmetric score and downloads using the methodology described by Kosanke and Berstrahl [9] and compared the number of citations and relative citation index between the cases and controls by using a paired Student's t-test. A repeat analysis was done using this approach to adjust for promiscuity of journal club-related tweets contributing to the Altmetric score. The Altmetric score for NCSTJC articles was adjusted to reduce the contribution of NCSTJC-related tweets contributing to it by subtracting the fourth fraction of number of NCSTC-related tweets. This methodology was on the basis of the Altmetric algorithm of tweet and retweet contributions toward the final score. Finally, for comparison, a test for correlation was also performed for the top ten cited articles for NCC during that time period.

Results
Between February 2015 and December 2018, 529 articles were published by NCC. During this time, 27 articles were discussed using #NCSTJC; of these 2 were published in a journal other than NCC and one tagged but not formally reviewed in NCSTJC hence excluded to yield a total of 24 articles included in the final analysis (Table 1). To address the confounding factor of category that biases which articles are cited and which are chosen for the NCSTJC, we further analyzed the articles by category. Using the published categories in the journal, the NCSTJC articles were classified as original articles, guidelines, ethical matters, review articles, day in the life of a trainee, or practice pearls ( Table 2). No articles under the category of current concepts/opinions, editorials, letters of the editors or their response, history related articles, neuroimages, opinions, translational research or abstract supplement were included in the NCSTJC. Comparing this to articles published in the journal during the same period, 99 articles were published under these categories that were not included in the NCSTJC selections. Therefore, the analysis was repeated to compare the 24 articles discussed in NCSTJC with the 406 articles under comparable categories (non-NCSTJC2). The 24 articles discussed in NCSTJC had a range of 0-107 citations per article, median 10 citations per article; together, they contributed to 471 citations of the total 5754 citations generated from 2015 to 2018 from a total of 529 articles. Non-NCSTJC2 articles had a range of 0-280 citations with a median of 8 citations per article. While NCSTJC articles represented 4.5% of all NCC articles, they represented 8.1% of all citations during that time period. Three of these articles (including guidelines and a consensus-based statement) were among the ten most-cited articles for that time period (Table 3) with 58, 68 and 107 citations each [7][8][9]. Figure 1 visualizes the relative composition of articles each year: those discussed in NCSTJC compared to all articles published, including overlap with the top 10 cited articles. When four metrics (downloads, Altmetric score, relative citation ratio, and number of citations) were measured and analyzed for the two groups, the 24 articles included in NCSTJC trended  9 "Interrater Reliability of Pupillary Assessments" [29] toward a greater median than the 505 articles in the non-NCSTJC1 group and 406 articles in non-NCSTJC2 group (Tables 4, 5, Fig. 2). The median was reported given the large interquartile range in both groups. Spearman's rho for correlation between Altmetric score and citations generated a correlation coefficient of − 0.101 for the NCSTJC group and 0.28 for rest of the non-NCSTC1 articles and 0.22 for non-NCSTJC2 articles. When a matched set of non-NCSTJC2 articles was created based on Altmetric score and downloads to compare the number of citations and relative citation index between the   cases and controls, no significant difference was found in either outcome (Table 6). We further found in these outcomes when the NCSTJC articles Altmetric score was adjusted to counter the effect of number of tweets made through NCSTJC (Table 7). When the top 10 cited articles were analyzed, strong correlation was found between Altmetric score and number of citations (Spearman's rho = 0.802). Online Appendix provides results of these correlations. Figure 3 shows scatter plots for these correlations in the three groups.

Discussion
Twitter journal clubs are one of the various ways journals have used social media to enhance engagement. The NCSTJC in February 2015 to promote engagement with Neurocritical Care through social media. We observed that highlighting articles in this Twitter journal club using a specific hashtag #NCSTJC improved engagement metrics of these articles as measured by downloads, Altmetric scores, relative citation ratio, and times cited, signifying enhanced social media outreach. We did not find a correlation between Altmetric scores and number of citations in the NCSTJC or non-NCSTJC group but there was a strong correlation between these two variables in high performing articles when the top 10 cited articles were analyzed. For medical professionals looking to engage with the most current research, Twitter offers a curation of articles. As a result, academic journals like ours have begun to engage with Twitter and other social media platforms to reach a broader audience [4,10]. Emerging data overwhelmingly, though not without exception, suggest increasing influence of Twitter activity on journal impact factor, driving such trends [5,6]. One article reported that the number of Twitter followers of one academic journal's Twitter account increased by 0.78% for every 1% increase in the journal's impact factor and 0.62% for every 1% increase in the journal's overall citations [10]. Additionally, the simple presence of a Twitter feed was statistically significant for a rise in the journal's impact factor over 4 years [11]. Implementation of a focused social media strategy-such as regularly tweeting out new publications-showed an increase more than tenfold in the chances of an article being cited [12][13][14][15]. Some journals have explored additional social media strategies like the use of "influencers" or social media ambassadors and emails to the corresponding author directly to encourage promotion of their work via social media. Other have reported that including images (e.g., figures, tables, or formal visual abstracts) in a tweet significantly improved the number of article link clicks [16,17]. We found several other precedents of successful Twitter journal clubs, some in the form of Twitter chats [3]. The Journal of American College of Radiology reported that article views increased by 31.4% and website visits increased 25.5% as a result of focused Twitter chats [18]. Whether formats like a Twitter journal club offers additional advantage more than these other social media strategies like Twitter chats, Tweetorials or regular posting of article links is unclear. One journal reported their articles were downloaded three times more often when tweeted; however, other studies show no benefit of a journal's social media usage with no increase in median 30-day page views or number of article views [19][20][21]. Our observation showed a slight negative correlation between articles chosen for journal club and citations.
Twitter journal clubs represent a niche use of social media by academic journals and offer several unmeasured benefits to a journal as well. For one, based on our

Table 6 Comparison of number of citations and relative citation index between NCSTJC (cases) and non-NCSTJC2 articles (controls) after being matched by articles and downloads
The distribution of articles by the article type as well as a year publication included in the NCSTJC (cases) in comparison with non-NCSTJC2 articles (controls) when matched by articles and downloads to show the similarities are also shown. Comparisons were made with paired Student's t-test because the matching created pairs  experience, a low number of marketing resources is needed to launch the effort through inclusion of volunteer participants. Second, wider variety of participation is permitted to include professionals beyond physicians and sometimes the author of the article itself, in part because there is greater flexibility with regards to time and location. Finally, Twitter is a nonconfrontational platform that largely promotes a conversational nature, which is key to a successful journal club [3]. As such, Twitter journal clubs like NCSTJC can maintain the synchronous nature as well as casual environment of live discussion.
A meta-analysis in 2015 reported 24 Twitter journal clubs and reported the nephrology journal club (#NephJC) had the greatest reach reflected by the highest number of tweets and the greatest impressions per month [22]. Using the nephrology journal club as the , those articles that were featured in NCSTJC (middle panel), and the top ten cited articles from the same time period model, they described a process of hosting Twitter journal club in six distinct steps that require significant time and work both before and after the one-hour session. We modified the recommendations provided to incorporate our experience and create a revised model for Twitter journal club (Table 7). We found several other precedents of successful Twitter journal clubs, some in form of Twitter chats [3]. Journal of American College of Radiology reported that article views increased by 31.4% and website visits increased 25.5% as a result of focused Twitter chats [18]. Whether formats like a Twitter journal club offers additional advantage more than these other social media strategies like Twitter chats or Tweetorials or regular posting of article links is unclear.
There are, indeed, certain aspects of Twitter journal clubs that may be limiting. The 280-character limit and discontinuous nature of responses in a conversation might make discussion threads hard to follow for some. Experts may find the open-source nature inviting to discussions, but others may feel reluctant to share their opinion in an environment that is fast-paced as well as permanently recorded on the Internet. Strategies like #NCSTJC also require a considerable commitment of resources in the form of engaged volunteers that need intangible investment through outreach strategies.
Furthermore, there seems to be an unclear relationship between social media impact and traditional scientific impact, questioning the effort placed in such strategies like Twitter journal clubs as a strategy to enhance engagement metrics and provide a tangible return on investment. This uncertainty is consistent with our observations between Altmetric score and citations of articles included in journal club. In general, Altmetric scores, as a measure of social media engagement, represent immediate interaction while classic citation indices, as a measure of engagement by the academic community, make take years to manifest [23]. Similarly, it can also be argued that scientifically weak but attention-grabbing papers may generate a lot of social media interest early on, often in conjunction with the news cycle, but may not enumerate citations as time passes, thus making the comparison between social media engagement and scientific impact futile. Going forward, Altmetric scores and citations might best be viewed as complements to one another rather than surrogates when determining research dissemination and impact [24].
Our study has several limitations. First, a small number of articles were featured in the bimonthly NCSTJC and, by lacking rigorous selection criteria; they are vulnerable to selection bias. The articles were chosen by a small subgroup of engaged members of a committee that were deemed to be of clinical interest to society members portraying a selection bias. Second, since all articles were made freely accessible online around the time of the journal club, accessibility in itself could have contributed to increased engagement though it did not translate to increased citations. Similarly, the audience that engaged in social media activities like NCSTJC may represent trainees and clinical providers that primarily use social media for staying up to date with published literature due to lack of such opportunities in their vicinity and may represent a more engaged cohort. While these audience may impact Altmetric score, they may not be the demographic targeted to cite the articles more often. Members of the community involved in scientific writing that are likely to cite these articles may not necessarily be represented well on social media.
We did attempt to find controls for the articles featured in NCSTJC by looking for articles similar in content published in NCC; however, this did not result in enough articles that could be randomized by content, target audience and social media usage. Attempts at findings similar articles from other journal involved intellectual property with organization access limitations. We created a set of matched controls by matching articles based on Altmetric score and number of downloads and then comparing number of citations. All posts regarding the journal club were encouraged to include the #NCSTJC but it was virtually impossible to track individual posts about these articles that did not include the hashtag or link to the article, so the engagement metrics may be an underrepresentation of online activity. Lastly, we considered the possibility of promiscuity of tweets artificially elevating the Altmetric score of articles included in NCSTJC. There are several checks in place by Altmetric with proprietary adjustment of contribution of Twitter activity of posts including reducing contribution of retweets, capping contribution of tweets to the score at 200, adjusting for influence of people tweeting the content, modifiers applied for promiscuity of tweets and bias by the Twitter account making frequent posts from same journal (https:// www. altme tric. com/ blog/ score anddo nut/). Given these adjustments, we felt that promiscuity of having multiple tweets did not by itself contribute disproportionately to the score when compared to articles that could have been tweeted but with a less frequent number of tweets. Nevertheless, to counteract this bias to some extent, we reported the analysis of data after adjusting for the number of tweets and retweets contributing to Altmetric score and did not find any significant change in outcomes.
Despite these limitations, the NCSTJC continues to thrive with increasing engagement of journal audience observed through the last few years. Following the increasing engagement in response to the NCSTJC, the journal launched its official social media strategy in 2019 summer by recruiting a Social Media Editor and social media ambassadors to the journal with a goal to have a robust social media footprint for the journal.

Conclusions
Twitter is becoming an emerging platform for the dissemination of information in medical education-related academic activities. Although the exact impact of the NCSTJC initiative on member engagement, journal outreach, or journal impact factor or citations is hard to determine, we observed trends of enhanced engagement with articles that were discussed as a part of a discreet online journal club. In our observation, such activities did not have an impact on traditional engagement metrics such as citations and impact factor. This may be a representation of two complementary measures of engagement that do not warrant comparison. Going forward, social media is likely to play an ever-increasing role in the outreach of academic journals with strategies like Twitter journal clubs occupying their own niche in continuing medical education.