Context

The productivity and impact of academics has long been a topic of interest amongst members of funding, hiring, and promotion committees (Dowling, 2014). Similarly, research impact is also a key focus of academic institutions, with research-specific metrics such as citations being used to measure their reputation and research contributions (Linton et al., 2011). More recently, alternative metrics (such as the Altmetric attention score) have become commonly used to gauge the dissemination of research online via non-traditional channels as a proxy for social impact (Bornmann, 2014). While science has long been of interest to the public (Gauchat, 2012), its accessibility has traditionally been limited to those within academic circles. However, coinciding with an increased emphasis on alternative metrics, there has been a recent shift in the way the public access and interact with science, largely driven by social media becoming an important vehicle for science communication (Huber et al., 2019; Rowlands et al., 2011; Van Noorden, 2014). Social media provides a way of disseminating research findings broadly, and to audiences and institutions that may otherwise remain ignorant of the work (Büchi, 2017). As such, social media may increase the likelihood that research findings are taken up by the public, and ideally, influence practice in a positive manner. But over the last two decades there has been an exponential increase in the number of English-language peer-reviewed scientific articles, with approximately three million published in 2018 alone (Johnson et al., 2018). This rapid increase in published science has made it increasingly difficult for researchers to get their work noticed.

To improve research dissemination and engagement, academics are encouraged to improve the way they communicate their findings, from writing differently (Doubleday & Connell, 2017), improving how they produce scientific posters (Rossi et al., 2020), or by embracing visual abstracts. Visual abstracts can be defined as a visual representation of a studies key findings, which typically found in the abstract an article (Ibrahim et al., 2017). Within this, visual abstracts most often appear in the form of a graphical abstract (GA) (Hendges & Florek, 2019). As their name implies, GAs are image-based, and typically comprised of a simple panel-based layout and basic icons (Ramos & Concepcion, 2020). Like a movie poster or trailer, GAs summarises a study’s key findings to direct people to the published paper, potentially increasing relevant research and alternative metrics. Furthermore, if GAs can improve the online reach of research, it may see faster uptake in practice and have greater impact amongst practitioners in the field. This is particularly relevant in sport science, where the applied nature of the research means it can be implemented into practice quickly, and clinicians, coaches, athletes, and the public stand to benefit. While GAs have been used extensively in other fields, their use in sport science is comparably recent.

To the authors knowledge, three peer-reviewed studies, and one non-peer-reviewed pre-print, have investigated the effects of GAs on these types of metrics, with conflicting results. Contrary to the authors expectations, Pferschy-Wenzig et al. (2016) found that manuscripts published in the journal ‘Molecules’ without a GA performed significantly better in terms of full-text article downloads, abstract views, and total citations, than manuscripts published with a GA (Pferschy-Wenzig et al., 2016). Similarly, Aggarwal (2021) found no significant differences between articles published in the Journal of the American Medical Association, the British Medical Journal, and the New England Journal of Medicine, with and without GAs, with respect to Altmetric attention score or number of page views (Aggarwal, 2021). Within this, they also found that papers published without GAs received more citations than those published with GAs. The authors of the current study published a non-peer reviewed pre-print exploring whether citations and Altmetric attention scores differed between articles published with and without GAs in three quartile one (Q1) sport science journals (Bennett & Slattery, 2022). Like Aggarwal (2021), articles with a GA were associated with lower citations, and lower Altmetric attention scores, than those published without a GA (Bennett & Slattery, 2022). Conversely, Kim et al. (2022) found that articles published with a GA in the top ten journals in the fields of gastroenterology and hepatology were associated with higher Altmetric attention scores and citations than those published without (Kim et al., 2022). While the exact reasons for these findings are unclear, it could be partially related to the fields of research explored, where it may be that some topics are more appealing to academics and/or consumers than others (and vice versa). It may also be related to the fact that each of these four studies explored this topic using a cross sectional design, where it may be more insightful to explore this question longitudinally, whereby citations and Altmetric attention scores can be tracked over time.

Objectives

Therefore, the aim of this longitudinal study was to determine if papers published with a GA in the field of sport science were more likely to receive higher Altmetric attention scores and more citations than papers published without. It was hypothesized that articles published with a GA would receive more citations and higher Altmetric scores.

Methods

Three journals ranked within the top quartile of the category of “Sport Science” according to Scimago as of January 2022 and had published graphical abstracts since January 2019 or earlier (Journal of Sport and Health Sciences [JSHS]; Exercise and Sport Science Reviews [ESSR]; Psychology of Sport and Exercise [PSE]), were included in the analysis. Article specific data (article type; open access status; publication date), attention-based metrics (Altmetric attention score), and citations were extracted for all full-text articles published between January 2019 and December 2020 in the journals. This period was selected to provide a sufficient sample size of articles and ensure a minimum of 12 months since publication from the first timepoint of data extraction. A 12-month timeframe was deemed appropriate to provide an accurate indication of Altmetric attention score and citations considering there appears to be a quick uptake of attention-based indicators after publication (Thelwall et al., 2013), and that short-term citation counts are highly correlated with long-term citation counts in peer-reviewed research (Liu et al., 2015). Article specific data and Altmetric attention scores were extracted using the online Altmetric explorer software on the 11th of January 2022, 27th of May 2022, 15th of August 2022, and 21st of March 2023, to allow for longitudinal analysis. For all articles, day of publication was considered baseline, with Altmetric attention scores and citations assumed to be zero. GA status (yes/no) was identified manually. Citations extracted from Altmetric explorer are counted using Dimensions software, which has been shown to have comparable capture to Web of Science and Scopus (Harzing, 2019; Thelwall, 2018).

Descriptive information is presented as median (interquartile range) [range]. A negative binomial mixed effects model was used to establish whether Altmetric attention score or Citation counts (dependant variables) were associated with GA status, open access status, or article type (independent variables). Variables that displayed a significant association with the dependant variables were then included in a final multivariate analysis. For all analyses, the number of days since publication were included as a covariate, as was the journal of publication. Publication ID was included in all analysis as a random effect. Analysis was performed using Stata Statistical Software, release 17 (College Station, TX). For all outcomes, 95% confidence intervals are presented, and effect sizes were quantified using incidence rate ratios (IRR) and considered trivial (0.77–1.00 or 1.00–1.29), small (0.51–0.78 or 1.30–1.99), moderate (0.25–0.50 or 2.00–3.99), and large (≤ 0.24 or ≥ 4.00) (Hopkins, 2010).

Results

A total of 565 individual full text articles were identified for inclusion into the study. Three were excluded due to being “notes from the editor,” five were excluded due to being published with video abstracts, leaving 557 articles for analysis (Table 1).

Table 1 Overview of publication number by journal and by article type
Full size table

Of these, 295 were from PSE, 63 from ESSR, and 199 from JSHS. There was a total of 364 original articles, 130 reviews, 49 classified as an editorial, commentary, opinion, perspective, or consensus, and 14 letters to the editor. A total of 96 articles were published with a GA, and 370 articles were published open access.

Descriptive information for days since publication, citations, and Altmetric attention score for each data extraction point are provided in Table 2. Initial analysis indicated that articles with GAs were associated with a higher Altmetric attention score than those without (IRR 2.15 [95% CI 1.44–3.20]; p  ≤  0.001; moderate effect) and were associated with larger citations counts (IRR 1.44 [95% CI 1.07–1.94]; p = 0.016; small effect). Open access articles were associated higher Altmetric attention scores than those that were not open access (IRR 1.81 [95% CI 1.30–2.51]; p = 0.001; small effect) and were associated with larger citations counts (IRR 1.43 [95% CI 1.11–1.83]; p = 0.005; small effect). Reviews were associated with higher Altmetric attention scores than original articles (IRR 2.61 [95% CI 1.81–3.78]; p  ≤  0.001; moderate effect), although there was no difference between editorials (IRR 1.00 [95% CI 0.61–1.60]; p = 0.972; trivial effect) and letters to the editor (IRR 1.28 [95% CI 0.56–2.94]; p = 0.559; small effect). Similarly, reviews were associated with larger citations counts than original articles (IRR 2.96 [95% CI 2.28–3.85]; p ≤ 0.001; moderate effect), while editorials (IRR 0.64 [95% CI 0.45–0.91]; p ≤  0.012; small effect) and letters to the editor (IRR 0.47 [95% CI 0.26–0.88]; p = 0.017; moderate effect) were associated with less citations. Results of the multivariate analysis are presented in Table 3 and indicate that articles with GAs were associated with higher Altmetric attention score than those without, with no significant difference in citations counts (Fig. 1).

Table 2 Descriptive information for days since publication, citations, and Altmetric attention score for each data extraction point (presented as median (interquartile range) [range])
Full size table
Table 3 Multivariate negative binomial regression exploring the impact of article type, open access status, and abstract type (normal; graphical abstract; video abstract) on Altmetric attention scores and citation counts
Full size table
Fig. 1
figure 1

Mean predicted citation counts and Altmetric Attention Score in articles published with and without a graphical abstract (GA)

Full size image

Discussion

This study is the first to investigate the effect of GAs on article attention scores and citation counts using a longitudinal design. As hypothesised, including a GA with the publication of an article in three high-ranking sport science-related journals was associated with higher Altmetric attention scores. However, this finding was not repeated with respect to citations counts, where there were no differences observed between articles published with or without a GA.

In what appears to be the first empirical analysis of the effectiveness of GAs on relevant metrics, Pferschy-Wenzig et al. (2016) found conflicting results to the present study (Pferschy-Wenzig et al., 2016). Manuscripts published in Molecules between March 2014 and March 2015 without a GA performed significantly better in terms of article downloads, abstract views, and total citations than manuscripts with a GA. Aggarwal (2021) explored this topic across three medical journals, also finding no significant differences between articles with and without GAs on Altmetric attention score and number of page views, although articles without a GA attracted more citations (Aggarwal, 2021). A recently published pre-print using a cross-sectional design demonstrated that articles published with a GA were associated with lower Altmetric attention scores and citations counts than those published with a GA in three high ranking sport science journals (Bennett & Slattery, 2022). Lastly, Kim et al. (2022) found that articles published with a GA in the top ten journals in the fields of gastroenterology and hepatology were associated with higher Altmetric attention scores and citations than those published without (Kim et al., 2022).

The effect of GAs on citation counts is unclear, largely due to the conflicting results presented in the research, with some studies reporting a negative association (Aggarwal, 2021; Bennett & Slattery, 2022; Pferschy-Wenzig et al., 2016), the present study no association, and one a positive association (Kim et al., 2022). Why GAs would have a negative effect on citations is unclear, but Pferschy-Wenzig et al. (2016) provided potential explanations (Pferschy-Wenzig et al., 2016). Firstly, a generational divide. They hypothesised that graphical abstracts may be more common among early career researchers with a greater propensity to use social media. Conversely, more experienced researchers of greater renown, whose research may be more frequently, viewed, downloaded, and cited, may be less likely to use a GA and distribute it via social media. Secondly, the authors speculated well-designed GAs may produce a negative effect on traditional metrics if viewers can obtain all the relevant information they need from the GA alone, without proceeding to the full article. Another possible reason for this finding may be that while researchers use social media to disseminate their own research, when writing manuscripts, they are more likely to find relevant sources of information via peer reviewed sources, as recommended by most academic institutions. The findings of the current study partially support these suggestions, where publishing with a GA had no significant impact on citations, positive or negative. This may also be explained by the cross-sectional design used in previous studies, that did not follow changes in these outcomes over time. However, these findings run counter to those published by Kim et al. (2022), where articles published with a GA were associated with higher citations than those without (Kim et al., 2022). This study was the first to explore this topic in the field of gastroenterology and hepatology. The authors hypothesised that, as the number of journals adopting GAs in these fields have recently increased, readers may have become more familiar with their use and therefore more likely to invest in articles published with them, while looking past those without (Kim et al., 2022). The body of research thus far suggests that different fields may consider GAs differently, which may affect their potential to impact relevant article metrics.

Like citations, there is little agreement when looking at effect of GAs on Altmetric attention score in the published research, with one study reporting a negative association (Bennett & Slattery, 2022), one no association (Aggarwal, 2021), and a positive association (Kim et al., 2022), which aligns with the findings of the present study. Although it is important to highlight that while there are only three published studies examining the association between of GAs with Altmetric attention scores, there is small body of research exploring their effect on native Twitter-based metrics, unanimously finding positive effects. Ibrahim et al. (2017) conducted a prospective case–control crossover study of 44 original research articles published between July 2016 and December of 2016 in the Annals of Surgery (Ibrahim et al., 2017). Compared to tweeting the article title alone, tweets featuring a visual abstract experienced a more than sevenfold increase in impressions, an eightfold increase in retweets, and a twofold increase in article visits (Ibrahim et al., 2017). Later, Chapman et al. (2019) conducted a randomized controlled trial comparing visual abstracts, plain English abstracts, and standard tweets (Chapman et al., 2019). They found that visual abstracts attracted a significantly greater number of engagements (a composite of tweets, replies and likes) by health care professionals on Twitter compared to plain English written abstracts (Chapman et al., 2019). In another randomized crossover trial including 50 articles, Hoffberg et al. (2020) found that tweets with a visual abstract were associated with a significantly higher number of impressions, retweets, and link clicks compared with text-only abstract tweets (Hoffberg et al., 2020). Additional studies have found further evidence of positive performance of visual abstracts on Twitter compared to text tweets or figures from the paper in the fields of urology (Koo et al., 2019), nephrology (Oska et al., 2020), and orthopaedics (Chisari et al., 2021). Based on this small body of research, it appears that graphical abstracts do outperform text-based posts on Twitter, which may explain the current findings. Although it is important to note that twitter engagement only provides a small contribution to an article’s Altmetric attention score, where mentions in news articles, blog posts, policy documents, peer reviewed research, and patents all contribute more meaningfully (Altmetric, 2021). Nonetheless, the results of the present study do support these findings. It may be that GAs encourage more engagement and dissemination amongst practitioners and the public, who are less likely to find research through traditional means. This may be more likely in fields like sport science, where employees are not required to have completed traditional academic pathways to work in the field. This is somewhat supported by research demonstrating that medical professionals use social media to support their own education (Khan et al., 2021), highlighting that even in “more academic” professions, practitioners regularly use social media to support their practice.

When considering the above research, is important to note there are some notable differences that should be acknowledged. Each of the four previous studies (Aggarwal, 2021; Bennett & Slattery, 2022; Kim et al., 2022; Pferschy-Wenzig et al., 2016) exploring this topic did so using a cross sectional design, which does not provide insight into how GAs may impact relevant metrics over longer time periods. Additionally, neither Pferschy-Wenzig et al. (2016) or Kim et al. (2022) accounted for the time since publication in their analysis (Kim et al., 2022; Pferschy-Wenzig et al., 2016). Considering both citations and Altmetric attention score are likely to increase over time, this could have had a considerable impact on the reported findings and may explain the conflicting results observed. Furthermore, the research undertaken by Bennett and Slattery (2022) has only been published on a pre-print server and has not gone through formal peer review (Bennett & Slattery, 2022), which does reduce confidence in their findings. Lastly, each of the above peer-reviewed papers have all explored the effect of GAs on citations and Altmetric attention scores in different fields. It is likely that the way GAs are used by both researchers and practitioners are field-dependant. This could explain some of the variations observed across the literature thus far. Future research should aim to explore how researchers from different fields use and interpret GAs, and whether they are considered when seeking out new research.

The present study focused on the effect of publication of an article with or without a GA and its effect on metrics of interest to academics. However, there are many important factors which may affect these outcomes which were not within the scope of this work. Firstly, there have been no investigations on how the effectiveness of GA are altered by their quality: either in an artistic sense, or regarding how well they relay information to their target audience. There are many factors to consider when designing an effective graphical abstract including layout, use of text and graphic representations (icons or other visuals, for example) (Hullman & Bach, 2018). Poorly produced GAs may not have the same level of effectiveness as those produced by skilled researchers, or professional designers and animators. Previous research has shown that approximately 50 percent of graphical abstracts are duplicated from the manuscript’s existing visual components (i.e., figures) (Yoon & Chung, 2017). Given that figures within an article often isolate a particular finding, and are nested within the context provided by the article, this may not be the best method of summarising a research study as a GA. This is partially supported by research demonstrating that articles published with a professionally designed video abstract (VA) had significantly higher citation rates than those published using more “amateur” production methods (Ferreira et al., 2021). A further factor worthy of examination is the distribution of GAs and how this affects outcomes. Moreover, a key factor that is difficult to account for is the content of the research paper, and how impactful the author perceives its findings. For example, if an author does not consider their paper a great contribution to the field, they may put more effort to create a GA to increase its impact. Conversely, it is also possible that the reverse is true, where authors may be more likely to use well-designed GAs for papers that they believe to have the most impact. Additional factors which may influence research of this nature also include the effect of the author(s) renown or reputation, as well as the content, significance, or appeal of the research itself, which were not addressed in the present study.

It is also important to note that this study did not examine the effect of publishing with a VA on relevant article metrics. Prior research looking at articles published in the ‘New Journal of Physics’ demonstrated that those published with a VA were associated with higher citation counts than those without (Zong et al., 2019), suggesting that they may impact research dissemination differently to GAs. Taking this into consideration, there are limitations that should be considered with these findings. Given the small volume of articles published with VA in sport science, they were not considered in the present study. There is a possibility that VAs impact research uptake and dissemination different to GAs in this field. Moreover, the quality of the GA was not accounted for in the analysis. It would be plausible to suggest that those articles published with a well-designed GA would be more likely to improve Altmetric attention scores and citation counts than those that are either poorly designed, or simply based upon a figure within the paper. Additionally, there are other variables that could be associated with higher article metrics including the number of authors, the type of collaboration, the number of cited references, the length of the publications, the length of their titles, author affiliations, and specific topics of the paper that were not accounted for in the present analysis. For example, and collaborative study with many authors might experience a higher Altmetric attention score due to more self-tweeting by those authors, which may in turn also influence future citations. Finally, although this paper observed no differences in citation counts over time based irrespective of GA status, it is possible that this may not be the case if given a longer period of observation, such as several years. This is particularly relevant when considering the rate an article can accumulate citations is likely to be much slower than Altmetric attention scores, and having a larger citation window (i.e., time period after publication in which an article can be cited) is known to influence citation counts (Waltman, 2016).

Conclusions

The results of this longitudinal study suggest that graphical and video abstracts are associated with larger Altmetric attention scores, but not citation counts, in the field of sports science. This may suggest that publishing with a GA may help dissemination amongst relevant end-users, but not other researchers. However, further research is needed that addresses factors such as design quality, distribution, and research importance, to improve the strength of this finding.