Introduction

The Knee Society was founded 30 years ago to advance the knowledge of the knee and to promote improved treatment of disorders of the knee. In 1993, The Knee Society established three scientific awards to recognize the best papers presented at the annual meeting from the areas of basic science, surgical technique, and clinical outcomes. Each year, the three award papers and selected nonaward papers presented at the annual meetings of The Knee Society are published, after peer review, in a single Knee Society proceedings issue of Clinical Orthopedics and Related Research ® (CORR®).

Although there is no universally agreed-on method for defining the impact or importance of a research paper, subsequent citations are a commonly used quantitative tool. Prior research has suggested that high citation rates generally correlate to predictably higher methodological quality scores for journal articles, but this association between quality and citations is controversial [5, 15]. Furthermore, quality and impact may not be related. To date, there have not been any quantitative evaluations to determine whether The Knee Society selection process accurately identifies important and potentially influential papers or may be influenced by other factors such as the eminence of the institution or popularity of the authors. In this study, we investigated the impact of the award papers based on subsequent citations in peer-reviewed scientific articles. We sought to use citation analysis to determine whether The Knee Society award selection process consistently identifies potentially highly influential research as measured by subsequent citations.

We therefore asked: (1) Are award papers cited both more than nonaward papers published in the same Knee Society proceedings issue of CORR® and more than all other knee research papers published in all issues of CORR® during any given year? (2) Does the award selection process identify potentially highly influential knee research papers?

Materials and Methods

In this study, all subsequent citations for the award papers and the nonaward papers from the annual meetings of The Knee Society that were published in The Knee Society proceedings issues of CORR® were identified. To allow sufficient time for the papers to be circulated and cited in subsequent works, we allowed a minimum of 5 years from the initial publication date. A power analysis was performed based on a pilot review of award and nonaward articles published in the 2008 Knee Society proceedings issue. Assumptions used in this analysis included: (1) type I error = 0.05; (2) power = 0.8; (3) award to nonaward paper ratio of 1:7 for each year; and (4) the mean citations for award papers is 45. The sample size was evaluated under two scenarios. First, we assumed the award papers are cited five times more than nonaward papers, and SD for the difference in mean citations is 30. In this scenario, 4 years of data would be required. In the second scenario, the SD of the difference in mean citations remained the same but we assumed award papers would only be cited two times more than nonaward papers. In this scenario, 6 years of data would be required. Therefore, to ensure that we had adequate data, we reviewed all the papers published in the proceedings issues from 7 years (2002–2008) and all the knee research papers published in all issues of CORR® during the same time period. All three groups of papers were identified by a senior author (HDC) based on a manual review of the table of contents for each issue of CORR® from the years 2002 through, and including, 2008. In 1 year (2004) four awards were presented; in the other years, three awards were presented each year. In total, 22 Knee Society award papers, 190 Knee Society nonaward papers, and 181 other knee research papers, all published in issues of CORR®, were examined (Table 1). To answer the first research question pertaining to the difference in citation rates between award and nonaward papers, we compared subsequent citation rates for the award papers with the citation rates for the nonaward papers and with the citation rates for all the other knee research papers published in any issue of CORR® during the same year. To answer the second research question regarding the identification of influential knee research papers, we determined how often an award paper was the most cited knee paper in The Knee Society Proceedings issue and in any issue of CORR® during the same year; in addition, we calculated how frequently the award papers were in the top five cited knee papers published in CORR® during any given year. Citations were determined using the citation index in SCOPUS (the largest abstract and citation database of peer-reviewed literature containing 53 million records in 21,915 titles from 5000 publishers). A manual query of the SCOPUS database was performed for each article by title accessed between April 17 and April 30, 2014, through the Internet.

Table 1 Papers reviewed by category by year
Full size table

Statistical Analysis

The mean, SD, median values, and the interquartile range for the number of citations and the 95% confidence interval (CI) for the mean difference in the number of citations (under the assumption of normality) were reported. The Wilcoxon rank-sum test was performed to compare the median number of citation between groups. The percentage of award papers versus nonaward papers that were in the top five cited papers from the Symposium issues was compared by the chi square test or Fisher’s exact test, and the 95% CI for the risk difference was reported. Statistical analysis was performed using SAS 9.3 (SAS Institute, Cary, NC, USA).

Results

Award papers generally garnered more citations than nonaward papers and more than other knee research papers published in nonproceedings issues of CORR® during the same time period. The mean number of citations for an award paper was 86 (SD 95; median 55; 95% CI of the mean, 44–128) versus 33 (SD 30; median 24; 95% CI of the mean, 28–37; p < 0.001) for a nonaward paper. The mean number of citations for award papers was also higher than for all other knee research papers published in nonproceedings issues of CORR®: 86 (SD 95; median 55; 95% CI of the mean, 44–128) versus 30 (SD 31; median 20; 95% CI for the mean, 25–35; p < 0.001).

A total of 12 out of 22 (54.6%) award papers were in the top five cited papers from the respective Knee Society proceedings issue for the specific year versus 24 out of 190 (12.6%) of the nonaward papers (difference in the percentages is 41.9% and the 95% CI for the risk difference is 20.6%–63.3%; p < 0.001). In six of the seven Knee Society proceedings issues reviewed, the most cited article from that year was an award paper (Table 2). Furthermore, in 3 of the 7 years, one of the award papers was the most cited knee paper presented in any issue of CORR® during the year (Table 2).

Table 2 Subsequent citations for papers by category by year
Full size table

Discussion

The Knee Society was founded to expand knowledge of the knee and promote improved treatment of disorders of the knee. In 1993, The Knee Society created three scientific awards to honor the three best research papers submitted to the annual meeting each year: The Mark Coventry Award for the best basic science paper, The Chitranjan Ranawat Award for the best work on a surgical technique, and The John Insall Award for the best work on a clinical subject or outcomes report. The awards are determined by The Knee Society Education Committee. After review of all abstracts submitted for the Annual Meeting of The Knee Society, the education committee invites the authors of the highest ranking abstracts to submit a complete manuscript for consideration of an award. These manuscripts are graded and ranked, and the best in each category is determined. After selection of the award winners, before publication in the proceedings issue of CORR®, the award winners and all other submitted nonaward papers go through the standard peer and editorial review process at CORR®. Although not accepted as the definitive method for evaluating research quality, citation analysis, the evaluation of subsequent citations, has become a frequently used tool in the evaluation of academic productivity and quality. To date, no formal evaluation of The Knee Society selection process for determining the award papers has been undertaken. It is important for the readership of CORR® to understand whether papers receiving these awards are likely to be influential and widely cited in future work or, alternatively are no more likely than other published papers in the journal to achieve widespread recognition. Therefore, in this study we sought to apply citation analysis to examine whether the papers with the highest potential impact were being identified or suggest whether other factors such as author eminence may be biasing award selection.

The primary study limitation is the problem of whether subsequent citations can be used as a proxy measure of an article’s quality. This issue has been previously investigated, yet it remains controversial with some limited support documenting citation analysis as a preferred method of assessing quality of individual papers versus a priori systems [10]. Indeed, many additional factors may bias paper citation rates. These may include eminence of the author or institution [12, 18]; tendency to self-citation by some authors [14]; increased citing of authors published in the same country or language [7, 11]; and research with positive results rather than no difference or negative results [3, 4]. Probably most important among these other factors is the prominence of the journal in which the paper is published. The journal’s prestige, generally measured by its Impact Factor, has been shown to influence subsequent citations [12]. However, contradictory evidence supports that citing of scientific articles is independent of the status of the journals in which they were published [16–19]. Importantly in this study, because all papers were published in the same journal, this potential bias was minimized. Although citations are influenced by a number of factors, increased citations are noted by some as a surrogate marker of increased research quality [4, 9, 10, 13]. However, this relationship between citation rates and quality is controversial; therefore, we should be cautious about drawing any conclusions about whether The Knee Society Award selection process identifies the best quality papers. Rather, increased citations perhaps better reflect the influence of the article in shaping future research and/or clinical practice. The other important limitation of the paper is that the award papers may attract more attention, and consequently receive more citations, precisely because they are award papers that run under an award banner, are prominently displayed on the web site, and are actively promoted by CORR® on social media and elsewhere; their profile alone may increase their citations and potential impact on future orthopaedic research. The third, less significant limitation of the study is the use of SCOPUS as the citation index. Certainly, other citation indices are available and use of this particular tool may have introduced systematic bias. However, it is one of the preeminent citation databases; furthermore, because the goal of this article was not to compare the accuracy of individual citation indices, we think that use of an alternative index would be unlikely to change the major findings. Prior bibliometric studies in orthopaedics have also noted the weakness of using a single database but have not offered a realistic solution [2, 20]. Finally, we did not exclude self-citation during the database queries. Because authors who won the awards may be more published than other authors (and may tend to self-cite more), this may have some effect on the number of citations. However, because the magnitude of the differences between the groups is so large (ie, mean 86 versus 33 citations), exclusion of self-citation would be unlikely, we believe, to change the major findings. Again, recent prior bibliometric studies in orthopaedics either acknowledge this as a limitation of the search strategy used but do not account for this bias or do not state whether self-citations were excluded [1, 2, 6, 8, 20].

Based on the results of this study, we conclude that the selection process for Knee Society scientific awards consistently identified papers that are likely to be highly cited and potentially influential as determined by significantly higher subsequent citations than nonaward papers and compared with knee research papers published in non-Knee Society proceedings issues of CORR® during the same year. Furthermore, there was a significantly greater likelihood of the award papers being in the top five cited papers from The Knee Society proceedings issue for the year; finally, in 6 of 7 years, the most cited paper from the proceedings issue for the year was one of the three award papers. This study also affirms the impact of a Knee Society award because in 3 of 7 years, the most cited knee paper to appear in any CORR® issue during the entire year was an award paper. This supports the assertion that award papers are potentially influential and likely to be highly cited when compared with all knee papers appearing in CORR®.

In conclusion, The Knee Society selection process for the annual scientific awards appears to consistently identify papers that are potentially influential and will become highly cited contributions to orthopaedic knowledge. Therefore, The Knee Society Award Papers should be considered potentially important articles that warrant special attention from the readership of CORR®.