Maintaining relevance in HIV systematic reviews: an evaluation of Cochrane reviews
Research turnover in the HIV field is rapid, and as a result, maintaining high-quality, up-to-date, and relevant systematic reviews is a challenge. One approach is to frequently update published reviews.
We evaluated the methods and relevance of all HIV systematic reviews and protocols published in the Cochrane Library over a 16-year period (2000–2016) to determine the need to update published reviews or complete of reviews in progress.
Of 148 published reviews and protocols, 129 (87%) were identified as not for updating or progression to publication, mostly due to research questions which were either entirely outdated or addressed questions in an outdated manner (N = 89; 60%); this was anticipated for older reviews, but was found also to be the case for recent publications. Some research questions were also inadequately conceptualized, particularly when complex pragmatic trials or behavioral interventions were included.
We suggest that authors clearly characterize interventions and synthesis approaches in their review protocols. In research fields, such as HIV, where questions change frequently, systematic reviews and protocols should be regularly re-evaluated to ensure relevance to current questions. This process of re-evaluation should be incorporated into the methods of living systematic reviews.
Human immunodeficiency virus
Herpes simplex virus 2
Methodological Expectations of Cochrane Intervention Reviews
The HIV field has high research outputs, and the cycle of questions, trials, and guideline modification is rapid. This raises the question of how to keep HIV systematic reviews up to date, relevant, and derived from the highest methodological standards [1, 2]. One approach is to regularly update reviews to incorporate new findings; and indeed, some questions endure and should be updated; and for others however, the review question may have been answered, changed substantially, or become irrelevant, and blindly updating such reviews can result in wasted effort and resources. To provide insight into how review questions change over time, we formally appraised the relevance, methods, and need to update a suite of HIV systematic reviews and protocols published in the Cochrane Library between 2000 and 2016.
Between 1 May 2016 and 30 May 2018, we critically appraised all published Cochrane HIV reviews and protocols based on the Cochrane Updating Classification System [3, 4]. The Updating Classification System acts as a guide to readers to assess whether a review is up to date, needs updating, or might need updating in the future [3, 4]. We modified the tool for assessing protocols. Reviews and protocols were appraised by evaluating: (a) the currency and priority of the review question (through discussion with experts in the field and exploring the literature), (b) the methods, by using an abbreviated version of the MECIR standards , and (c) whether there were new studies to be included in the review, or whether another high-quality review had already been published on the topic outside of Cochrane. Based on the outcome of this assessment, we determined whether reviews should be updated, and whether protocols should continue to full reviews. We categorized them as follows: (1) continue with review update or allow protocol to progress to full review and (2) not for update or protocol progression due to (a) methodological concerns, (b) outdated research question, or (c) review already up to date. At least two people read and appraised each review independently, and the classification was agreed in team meetings chaired by experienced systematic review authors. The tool is available in the Additional file 1.
We identified and appraised 109 Cochrane HIV reviews and 39 HIV protocols (148 articles in total), published on the Cochrane Database of Systematic Reviews between 2000 and 2016. Review and protocol topics included questions about HIV treatment interventions (63%), HIV prevention interventions (32%), health system interventions (2%), and interventions to improve HIV diagnosis (3%). The majority of reviews (N = 98; 66%) were published between 2010 and 2016, with a smaller proportion published between 2000 and 2009 (N = 50; 34%). We concluded that 19 (13%) of all reviews and protocols were eligible for an update or progression to full review, and the remaining 129 (87%) were assessed as “not for updating or progression”.
Thirty-four of 39 protocols were stopped from progressing to full reviews (87%) (Fig. 1). The two main reasons for discontinuation also included outdated research questions. This included protocols where the topic was no longer relevant to policy makers or consumers, or where an important HIV research area was evaluated but the protocol review question did not reflect current practices or thinking. The remaining protocols were discontinued due to methodological limitations, as a result of inadequately conceptualized research questions. Specifically, protocols encompassed a very broad set of treatment strategies that were difficult to combine in one review, or authors did not consider how to meaningfully combine complex HIV interventions. This was common in protocols which included behavioral HIV prevention interventions and pragmatic implementation trials. Thirty protocols were published more than 2 years previously (interquartile range 3–9 years).
This appraisal of Cochrane systematic reviews and protocols demonstrates that topics in the HIV field become outdated quickly, and this was by far the commonest reason for discontinuation of both reviews and protocols, including those from more recent years. In addition, changes in reporting standards over time resulted in outdated methodology. Protocols which progressed slowly became irrelevant before completion, and inadequately conceptualized research questions resulted in disorganized or empty reviews which, at times, did not address issues specifically relevant to HIV populations, or sufficiently characterize interventions and key HIV outcomes.
Evaluating the relevance of systematic reviews prior to updating is essential in all fields but particularly important where there is rapid development of new innovations and care strategies, as seen in HIV research. Where there are delays, protocols should be re-evaluated and adapted to ensure that final reviews answer current questions. With the recent drive to automate and create “living” systematic reviews with frequent updates [5, 6], it is essential that methods of maintaining currency are incorporated into these evidence synthesis approaches as questions may change or become irrelevant [7, 8].
We recommend that authors planning to conduct HIV systematic reviews reflect on how their review can contribute, by thoroughly exploring the current evidence, determining if primary studies exist on the topic of interest, identifying other systematic reviews already conducted, and clearly defining their research question and what their review may add. In addition to presenting the PICO (population, intervention, comparison, and outcomes) elements, authors should fully characterize the interventions of interest, limit the number and types of interventions, and determine how to synthesize results from studies with behavioral or complex interventions. Several available tools can be drawn on and adapted to aid the design of conceptual frameworks and characterization of interventions and complexity: this can help structure review questions to produce meaningful results [7, 8, 9, 10, 11]. Review team effort and Cochrane editorial procedures need to ensure timely completion of reviews to avoid them being outdated before completion.
In summary, HIV research questions and the systematic reviews summarizing the relevant research evidence become outdated quickly, and rapid conduct and publication is critical; clearly defined research questions need to be regularly re-evaluated alongside changes in the field to maintain relevance, particularly in the era of living systematic reviews.
We would like to thank Taryn Young and Dierdre Walsh for facilitating this work.
The work was funded by Cochrane CRG Support Programme in a grant given to the Cochrane Infectious Diseases Group. The work of the Cochrane Infectious Diseases Group is further funded by UK AID through the Research, Evidence and Development Initiative (READ-It, project ID 300342–104). The views expressed do not necessarily reflect the UK government’s official policies.
Availability of data and materials
Data are available on request from Dierdre Walsh from the Cochrane Infectious Diseases Group, Liverpool School of Tropical Medicine, Liverpool, United Kingdom: Deirdre.Walshe@lstmed.ac.uk
All authors contributed to the conduct of the review and protocol appraisal and participated in the drafting and reviewing this manuscript. All authors read and approved the final manuscript.
Ethics approval and consent to participate
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
- 1.Barroso J, Sandelowski M, Voils CI. Research results have expiration dates: ensuring timely systematic reviews. J Eval Clin Pract. 2006; https://doi.org/10.1111/j.1365-2753.2006.00729.x.
- 2.Higgins J, Lasserson T, Chandler J, Tovey D, Churchill R. Standards for the conduct and reporting of new Cochrane Intervention Reviews, reporting of protocols and the planning, conduct and reporting of updates. 2018. Version 1.07. Accessed at: https://community.cochrane.org/mecir-manual.Google Scholar
- 4.Cochrane Editorial Unit. Updating Classification System: guide to applying to Cochrane reviews. 2016. Accessed at: https://community.cochrane.org/editorial-and-publishing-policy-resource/cochrane-review-management/updating-classification-system-cochrane-reviews.
- 6.O’Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Wolfe MS. Moving toward the automation of the systematic review process: a summary of discussions at the second meeting of International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 2018;7(1):3 https://doi.org/10.1186/s13643-017-0667-4.CrossRefGoogle Scholar
- 8.Rohwer A, Pfadenhauer L, Burns J, et al. Series: clinical epidemiology in South Africa. Paper 3: logic models help make sense of complexity in systematic reviews and health technology assessments. J Clin Epidemiol. 2017;83:37–47. https://doi.org/10.1016/j.jclinepi.2016.06.012.CrossRefPubMedGoogle Scholar
- 9.Guise JM, Chang C, Butler M, Viswanathan M, Tugwell P. AHRQ series on complex intervention systematic reviews—paper 1: an introduction to a series of articles that provide guidance and tools for reviews of complex interventions. J Clin Epidemiol. 2017;90:6–10. https://doi.org/10.1016/j.jclinepi.2017.06.011.CrossRefPubMedGoogle Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.