Background

Macular disease is a major cause of morbidity in the UK. Age-related macular degeneration (AMD) is the commonest cause of visual loss in people aged over 65. Owen and colleagues have estimated the UK prevalence and incidence of late AMD, in the population aged 50 and over, at 2.4 % in 2007–2009 [1]. The prevalence increased to 4.8 % in those aged 65 years or more. They estimated that in 2020 the number of AMD cases will be 679,000.

Hereditary retinal dystrophies, including Stargardt disease, were the commonest single cause of visual loss in working age adults in 2009–2010, accounting for 20.2 % of blindness certifications in people of working age in England and Wales [2].

There have been recent advances in treatment of wet (neovascular) AMD but not in dry AMD or Stargardts and more research is needed. The Sight Loss and Vision Priority Setting Partnership (SLV-PSP) developed a “top 10” list of research priorities for AMD in 2013 [3].

Bibliometric measures of research performance

Bibliometric indicators are based on measuring patterns of authorship, publication and citations, and are widely used to evaluate the research performance of researchers (both individually and as a group) and of institutions. Such indicators form a key part of the global ranking and league tables which are important to Universities. For example, one such ranking, the Times Higher Education (THE) World University Rankings 2015–16, used 13 performance indicators, grouped into five areas [4]. Research influence, measured by citations, is weighted at 30 percent of the overall score. Another indicator was research productivity, accounting for 6 % of the overall score. This is based on a count of the number of papers per person (scaled for institutional size and normalised for subject) which were indexed in peer-reviewed journals in Elsevier’s Scopus database. Another indicator, accounting for 2.5 % of the total score, is international collaboration; that is, the proportion of a university’s total research journal publications that have at least one international co-author.

Rather than looking at citations to the total output of a department, institution or country, some evaluations judge the quality of research based on the number of highly-cited papers produced. This is based on the assumption that the most highly-cited articles are likely to have made the greatest contribution within their field, or to be about important innovations. Highly cited is variously defined as being in the top 20 % [5], or top 10 % [6], or top 1 % [7] of citations.

Journal Impact Factors (JIFs)

The JIF of a journal for a particular year is the average number of times articles from the journal have been cited in the past 2 years [8]. The JIF is often used to indicate the prestige of the journal, and publication in journals with a high JIF is often regarded as an indication of the quality of the paper itself. Consequently, there is considerable pressure on academics to publish in high JIF journals [9, 10].

However, some researchers and authors have called for a stop to the practice of JIFs being used as a surrogate measure of the quality of individual research articles when assessing an individual scientist’s contributions, or in hiring, promotion, or funding decisions, and for research to be assessed on its own merits [11].

Research needs and prioritisation of macular research

The James Lind Alliance (JLA) Priority Setting Partnerships (PSPs) are part of the National Institute for Health Research (NIHR) [12]. The aim of the PSPs is to identify and prioritise unanswered scientific questions that they agree are most important, and to help ensure that funders of health research are aware of the priorities of patients and clinicians.

In 2012, the Sight Loss and Vision Priority Setting Partnership (SLV-PSP) surveyed patients, carers and health care professionals to identify and prioritise unanswered questions in the prevention, diagnosis and treatment of sight loss and eye conditions [13]. This process resulted in the identification of the top questions for research for each of the 12 eye disease/condition categories, including the top 10 priorities for AMD.

Aims

The aims of this report are: (1) to use bibliometric indicators to review the UK’s research performance in macular disease conditions from 2011 to 2014 and to compare it with other countries producing major output in the area, and (2) to compare the UK’s publications with the SLV-PSP’s top 10 priorities for AMD.

Methods

Database searching

We compiled a database of journal articles on age-related macular disease (AMD) and Stargardt disease. The latter was chosen to represent the retinal dystrophies because it is the commonest (there are over 100 inherited retinal diseases and we were not resourced to include them all in this study) [14]. For the sake of brevity we will refer to AMD and Stargardt collectively as macular disease, though as will be seen, the great majority of research articles relates to AMD.

Database searching for macular disease articles from the UK, USA, China and Germany

We searched Elsevier’s Scopus database on February 18, 2015 using the search terms “stargardt* or age related macular degeneration” in the Article Title, Keyword or Abstract fields and limited it to publication years from 2011 to 2014. Only document types indexed as ‘articles’ or ‘reviews’ were downloaded. Editorials, letters, comments and notes were excluded. The search was not restricted to human research, but could include, for example, stem cell research in animals.

All documents, without any country limit, were downloaded. The search was then repeated and refined using the Country field limits (in turn) to: (1) United Kingdom (2) United States (3) Germany and (4) China. This limited the searches to those articles with at least one author with an address in these countries. The searches were downloaded separately into Excel. The fields downloaded included; Authors, Title, Year, Title of Journal, Volume, Cited by (number of citations), Affiliations of authors to institutions (including institution of the author for correspondence), Abstract and keywords and Document Type (article or review).

Database searching for macular disease articles from the UK

An additional search, for UK authored articles only, of the Thomson Reuters Web of Science™ (WoS) database was done to check whether any relevant records were missed by the Scopus search (possibly due to differences in indexing practices and journal coverage between the two databases). The WoS search strategy was: TOPIC: (age related macular degeneration OR stargardt*) AND ADDRESS: (north* ireland or scotland or england or wales) and limited to Document Types: (article or review) and Timespan: 2011–2014.

The results from the UK authored articles for Scopus and WoS, were combined and duplicates removed.

Classification of UK macular disease research into AMD SLV-PSP priorities

All 421 articles in the UK dataset were allocated where possible to the ten SLV-PSP priorities by one author (NW), initially based on title and abstract. In case of doubt the full text of articles were obtained and checked. Articles corresponding to the SLV-PSP priority 1 were sub-divided into reviews or primary research.

Percentile ranking of citations and JIFs

We obtained the top 10 % of world citations separately for 2011, 2012 and 2013 by downloading citations to macular disease articles and ranking the articles on the basis of citations in Excel and then selecting those articles in the top 10 % of journals.

To obtain the articles in the 10 % of the top ranked journals for 2014, we downloaded all the 2014 macular disease articles, obtained the JIFs of the journals they were published in, ranked them by their JIFs, and selected those articles in the 90th percentile. We used the 2013 Journal Impact Factors, obtained from the Journal Citation Reports®, published in June 2014.

Statistical analyses

We used non-parametric statistical tests to analyse data, due to the non-normal distribution of the dependent variables, citations and JIFs. The Wilcoxon Mann–Whitney Test was used to analyse the difference in citations between two independent groups, the Kruskal–Wallis test was used to analyse the differences in citations between three or more groups. Spearman’s correlation coefficient was used to compute the correlations. Stata 14.0 (Stata Corporation, College Station, TX, USA) was used for the analyses.

Results

Bibliometric analysis of UK macular disease publications from 2011 to 2014

We retrieved bibliographic records of 431 articles in Scopus when limiting the search to the United Kingdom. Thirty-two articles were deleted, as macular disease was not the prime focus of the article, and this left 399 articles. The WoS search identified another 22 articles, giving a total of 421 in the final UK data set: 81 % were indexed as articles and 19 % as reviews.

We analysed citations only to 2011, 2012 and 2013 articles, because many 2014 articles would not have had time to accumulate sufficient citations at the time we did the searches. The citations were analysed separately for each of the 3 years (to account for the fact that each year had a different citation window).

The numbers of citations to the UK articles from 2011 to 2013 are shown in Table 1.

Table 1 Citations to UK publications for 2011, 2012 and 2013
Full size table

The highly skewed nature of the citation distribution is evident from the data. A small number of articles received a large number of citations and the vast majority received relatively low numbers. To adjust for the fact that the older articles had more time to accumulate citations, the mean number of citations per article per year was calculated by dividing the mean citations per article by the number of years between publication and collection of the citation data. The mean number of citations per year was 3.9 % for 2011 articles, and 4.8 and 3.8 per year for 2012 and 2103 respectively. The percentage of articles that received no citations (6.3, 8.3 and 13.4 % for 2011, 2012 and 2013 respectively) was low (when considering the citation windows for each year), and will be likely to decrease over time.

Journals in which UK authors publish

The 421 UK macular disease articles were published in 141 different journals.

Table 2 shows the top 4 journals, which published 27.8 % of the articles. The first and third ranked journals (British Journal of Ophthalmology and Eye respectively) are from the UK and the other two (Investigative Ophthalmology & Visual Science and Ophthalmology) are based in the USA.

Table 2 Top 4 journals in which UK MD articles were published
Full size table

International collaboration

Forty-nine percent of articles were from international collaborations, with the number of countries ranging from 2 to 12. The most frequent collaborating country was the USA (22.3 %) followed by Germany (11.6 %), and Australia (7.1 %). Seventy-two percent of articles had the corresponding author based in the UK. The highest proportion of non-UK corresponding authors were the USA (10.9 %) and Germany (4.3 %).

Correlation between JIFs and citations

We examined the Spearman’s correlation between citations and JIFs for 2011, 2012 and 2013. In those years, 92 % (285 of 311) articles were in journals with a JIF. The results in Table 3 show that the strength of the relationship between citations and JIFs is highly statistically significant for all years, but varies over the 3 years. There is a weak correlation for 2012 (rho = 0.3526), and moderate correlations for 2011 (rho = 0.4635), and 2013 (rho = 0.6036).

Table 3 Correlation between citations and JIFs
Full size table

Comparing UK macular disease output and citations with other countries

The Scopus searches for macular disease articles worldwide from 2011 to 2013 retrieved 3262 articles. The four countries with the highest output (USA, UK, China and Germany) produced 64 % of the world’s output of macular disease research that we identified from 2011 to 2013. The world share of citations for these four countries was calculated and compared to their share of total citations, shown in. It can be seen that the USA, UK and Germany have higher percentage of the world share of citations than their share of the world output of articles, and are performing above the world average in terms of citations. China has a lower proportion of citations than articles, and on this measure is performing less well (Table 4).

Table 4 Citations versus output from 2011 to 2013 in macular disease research for USA, UK, Germany and China
Full size table

Proportion of UK Authors publications in highly cited publications

We looked at the proportion of UK authored papers included in the top 10 % of the world’s most cited macular disease articles. The total numbers of macular disease articles in the Scopus database were 1007 in 2011, 1121 in 2012, and 1139 in 2013 respectively. The numbers of citations needed for an article to be in the top 10 % of world citations in macular disease for these years were 28, 20 and 11 respectively.

Table 5 shows the proportion of highly cited publications with a UK author for 2011, 2012 and 2013, the UK’s overall article share for each year, and the ratio of highly cited articles to article share. It shows that in 2011, 2012 and 2013, UK authors’ share of the highly cited publications were 1.37, 1.88 and 1.78 times respectively greater than their share of world articles in macular disease.

Table 5 Proportion of UK authors of highly cited publications compared to article share
Full size table

The highly cited articles with UK authors comprised 76 % (41/54) primary research (mainly concerning treatment and pathophysiology) and 24 % review articles.

Articles in the top 10 % of JIFs in 2014 from UK authors

The 1135 worldwide macular disease articles published in 2014 journals were ranked by JIF. Eighty percent (906 of 1135) were in journals with a JIF, and 229 were in journals with no JIF. If we assume that in all articles with no JIF, that JIF = 0, and rank them on their percentiles based on JIF we find that the JIF needed to be in the top 10 % of journals 5.939. The mean JIF was 2.802 and median was 2.512.

We found that in 2014, 13.2 % of articles in journals ranked in the top 10 % of JIFs in macular disease had a UK author. UK authors’ world share of macular disease articles for 2014 was 9.9 %.

Matching UK macular research to the sight loss and vision priority setting partnerships (SLV-PSP)

We matched the set of 421 UK publications on macular disease from 2011 to 2104 against the SLV-PSP priorities for AMD. The results are in Table 6 (note that the numbers added up to 423 as there were 2 articles that belonged to both the categories ranked 5 and 6). Thirty-four percent of the publications did not fit any of the SLV-PSP priorities, as they did not include invasive treatment of wet AMD in their list. Only 8 % of the publications were on the top priority of the SLV-PSP list, which was treatment to stop dry AMD progressing and/or developing into wet AMD. Forty-four percent of publications addressing the top priority were primary research and 66 % were reviews.

Table 6 Comparison of published research and SLV-PSP priorities
Full size table

The second priority, ‘What is the cause of AMD?’, had 23 % of the publications, largely because of the amount of research into the genetics of AMD. This category was defined by the SLV-PSP as including whether the genetic factors responsible for the development or progression are known. Studies of genes comprised 45 articles, the biggest component of this group. It could be argued that genes are not the cause of AMD, but merely a factor resulting in susceptibility to it. The real causes remain to be determined. Similarly we included 19 studies on the role of complement under group 2, but this could be more a mechanism of disease triggered by the primary cause. It could be argued that the gene studies belong under group 5 (predictors) or group 7 (factors influencing progression).

If we take out the gene and complement studies from group 2, and take into account that some of the remaining articles are reviews, the body of literature on causes of AMD in the dataset begins to look rather sparse.

The third and fourth ranked PSP priorities, prevention of AMD and ways of restoring sight loss, matched only two and one publications respectively.

Discussion

We performed a bibliometric analysis of UK macular research from 2011 to 2014, and compared UK performance to the other leading countries in macular research, i.e. the USA, Germany and China. Also, we compared the 421 UK macular disease research publications published between 2011 and 2014 with the SLV-PSP’s top 10 priorities for AMD. The UK, USA, and Germany all performed above the world average in terms of citations compared to their article share, whereas China had a lower proportion of citations than articles. The mean citations per article per year for UK macular disease articles from 2011 to 2013 was 4.15. Forty-nine percent of articles were internationally collaborative. The UK was shown to be performing well, producing 9.7 % of the world’s output of macular disease articles from 2011 to 2013, but receiving 14.2 % of the world’s share citations. Also, 16.2 % of the top 10 % of macular disease publications, ranked by citations, had a UK author. In 2014, 13.2 % of articles in journals ranked in the top 10 % of JIFs were from UK authors; the overall UK article share was 9.9 %.

When matching UK macular research published from 2011 to 2014 (but performed earlier) against the priorities chosen in 2013 by the Sight Loss and Vision Priority Setting Partnerships (SLV-PSP), we found a considerable difference. Only 8 % of the publications were on the top priority of the SLV-PSP list and 34 % of the publications did not fit any of the SLV-PSP priorities.

Strengths of this study

Comprehensive searches of Scopus and the WoS were done to identify as much as possible of the world’s scholarly output on macular disease, and all abstracts were checked by both authors. We set our bibliometric performance of UK macular research in a world context, by comparing it to the other three top producers of research in this area. We also included examination of the mostly highly cited publications in the field, which has been recognised as a robust approach to research assessment [7, 15].

We measured citations in early 2015 to articles published between 2011 and 2013, thereby giving a citation window ranging between 2 and 4 years. Although articles will still be accumulating citations, in most fields a paper is thought to reach its citation peak at 3 years and for citations to decrease quickly thereafter [6, 16]. Liu and colleagues [17] examined time to citations of articles in 28 ophthalmology journals. They reported that there were few citations in the first year after publication, but a rapid increase later, reaching a peak in year 3.

Finally, we compared UK macular research with SLV-PSP priorities to see how well matched the research needs that had been identified by consensus amongst patients, carers and clinicians.

Limitations of this study

Our searches done using the terms ‘age-related macular disease or Stargardt*’ may have missed relevant research if neither term was used in title, keywords or abstract of the article. However this would not affect international comparisons. We did not look at other retinal dystrophies.

Although citations are a good indicator of research quality, they are not perfect. While there has been much debate about their meaning, they are still widely used in the assessment of academic performance of individuals and institutions [5, 7, 18, 19]. Some bibliometric studies exclude self-citations, but we included them in our analysis, as we took the view that this is a valid scientific practice (e.g. authors providing background and building on their previous work) [16].

Bibliometric analysis of macular research

In this study of macular research, we found that from 2011 to 2013 the UK produced 9.7 % of the world’s output, but received on average 14.2 % of the world’s share citations. Overall for these years the mean number of citations per article was 4.15. This is compared to the aggregate impact factor (AIF) of 2.357 for journals in the Journal Citation Reports® subject category of Ophthalmology for 2014 (the AIF is calculated in the same way as the JIF but at the subject level). This reflects well on UK macular research and shows that it is performing above average compared to the world-wide average for articles in ophthalmology journals.

In terms of the ratio of its share of world citations compared to output, for UK’s ratio of 1.46 compared favourably to the USA (1.49) and Germany (1.29). However, China’s ratio (0.73) was the lowest of the four countries. This concurs with findings from a recent study by Huang et al. who analysed articles published in 53 ophthalmic journals from 2000 to 2011, and found that despite the rapid increase in output, China ranked low in terms of citations to articles in ophthalmic journals [20].

UK authors and highly cited publications

From 2011 to 2013, UK authors had an average of 16.2 % of their macular disease articles in the top 10 % of the world’s most cited publications in this area, with only 9.7 % of the share of articles. This over representation of UK authors (including those with international collaboration) in the world’s top cited publications reflects well on UK macular disease researchers.

Our findings are consistent with those found in a 2013 report by Elsevier on all UK research. The Elsevier report was commissioned by UK’s Department of Business Innovation and Skills (BIS) to examine how the UK research base compares internationally and what trends may affect the UK’s future standing as a world-leading research economy [7]. They found that while the UK produced 6.4 % of global articles in 2012, its share of global citations was 11.6. Also, the UK produced 15.9 % of the world’s most highly cited articles (defined as being in the top 1 %). It therefore concluded that the UK punches above its weight in terms of citations.

UK authors and JIFs

In 2014, 13.2 % of articles in journals in the top 10 %, ranked by JIF, were from UK authors, compared to the overall UK article share of 9.9 %, showing that UK authors were over-represented in these journals s, in comparison with their article share. Despite objections to JIFs being used as a surrogate measure of the paper itself [21], and the fact many articles in journals with high JIFs are never cited [6, 22, 23], publishing in a journals with a high JIF is nevertheless considered prestigious, and very important for an academic’s reputation and career advancement [9].

Uncited articles

Another indicator of performance is the proportion of papers that are never cited, which a report by Thomson-Reuters uses to identify publications with no or very little influence [24]. We found that on this measure UK macular research did well. Larivière et al. [25] estimated that the proportion of articles uncited in the medical field in the WoS database was 20 % after 2 years and 12 % after 5 years. In this study, we found that the percentage of UK articles uncited was 6.3 % for those with a citation window of just over 4 years, and 8.3 and 13.4 % for 3 and 2 year citation windows respectively. Therefore, UK macular research compares favourably on this indicator.

International collaboration

Our study found an international collaboration rate of 49 % in UK macular research. The BIS report also found that UK researchers are highly collaborative, and in 2012 47.6 % of all UK articles involved international collaboration [7]. They noted that international research collaboration and researcher mobility were core to the maintenance and further development of the UK’s world-leading position as a research nation, but that increased collaboration and research leads to a lessening of differences amongst countries [7].

Correlation between citations and JIFs

We found that the correlation between citations to the macular disease articles and the JIFs of the journals in which they appeared from 2011 to 2013 varied from low to moderate over the 3 years. This lack of a consistent and strong correlation is similar to other recent reports which have examined the correlation between JIFs and citations to articles [26–28]. Ramin et al. found that the number of citations correlated poorly with the impact factor of journals in the AMD field [28]. This provides further evidence against using JIFs as a proxy for citations to individual articles in the journal.

Comparison of research published in 2011–2014 and the SLV-PSP’s research priorities

We found a considerable difference between the topics in the research published in 2011–2014 and the SLV-PSP priorities. It should of course be noted that the SLV-PSP report was published in October 2013, by which time the research documented in this report would have been done. The report of the SLV-PSP envisages that it will help funders of research into AMD to allocate research funding to the top priorities [3]. This assumes that researchers will also target those priorities, but the report suggests that they will be encouraged to do so and that future applicants for research grants should say which of the priorities their proposed research will address. The 2015 Vision 2020 funders meeting concluded that researchers should be made more aware of the SLV-PSP priorities, and that researchers should state which of the priorities their research would address [29].

Time will tell whether the prioritisation exercise affects allocation of research funds and the priorities of researchers. There may sometimes be a tension between patient priorities and the sorts of research that universities want done, but one would expect that research into the top SLV-PSP priorities would be attractive to universities. The balance of research will reflect the availability of treatments. In wet AMD there are several treatments, with research underway looking at how best to use them. In dry AMD, research is at an earlier stage.

A bibliometric study by Ramin and colleagues looked further back in time, at 3235 articles on AMD published between 1993 and 2013, identified via the Web of Science [28]. They found that most of the highly-cited papers were on genetics, treatment of wet AMD, and on the effects of diet and vitamins. They noted a recent increase in studies in biomarkers including genes.

Conclusion

In terms of publications and citations as a measure of scientific quality, the UK has done well in macular disease research. The UK is one of the leading countries in macular research.

However the distribution of past research topics does not match the priorities set by the SLV-PSP in 2013, and future funding may encourage a different pattern of research.