Network meta-analyses should be the highest level of evidence in treatment guidelines
- 3.1k Downloads
The first one is a simple, pragmatic argument: Nowadays, there are so many trials available, that it is simply impossible for a guideline team to read them all and to come up with an objective evaluation. For example, the latest network meta-analysis on antipsychotic drugs for schizophrenia comprised 212 blinded trials , and the last network meta-analysis on antidepressants for major depressive disorder 117 randomised-controlled trials . Nobody can read all these articles and objectively “synthesise” them narratively. We have shown that abstracts from industry-sponsored trials are often biased, thus to read only the abstracts is not sufficient . Actually, the avalanche of evidence is even a problem now for meta-analyses. In 2010, 11 meta-analyses were published per day, the same amount of randomised-controlled trials published three decades ago . There are often several meta-analyses on the same or similar topics. But, their authors do not always come up with the same conclusions, and it is often unclear whether the reason are slightly different research questions, or different interpretation of the results . We have therefore demanded to make a review of the existing systematic reviews mandatory .
The second argument is that science always has to start out from the ideal situation. Imagine 10 identical studies. There is no doubt that the pooled estimate of these 10 studies is better evidence than any of the single studies alone. The simple reason is that a bigger sample size increases the precision of the estimate, meaning that we can be more confident about the result. Consider this thought experiment: there is a trial with 10 participants of which seven responded to treatment and an identical trial with 1000 participants of which 700 responded. In both cases, the response rate is 70 %, but obviously, we would trust the large trial more. The same holds true for network meta-analysis. If all trials are identically designed, and the direct and the indirect evidence are “consistent,” then there is no reason to not use it for complementing the evidence. Therefore, nothing generally speaks against considering network meta-analysis as the highest level of evidence. The problem is rather that often the world is not ideal. For example, it is well known from many medical fields that small trials tend to exaggerate treatment effects . Therefore, the results of meta-analyses based on several trials can be completely reversed by one later published, large randomised-controlled trial [10, 19]. In mental health, for example, it has been shown that the results of meta-analyses only get stable once approximately, 1000 participants have been included in them . In our experience, the results of network meta-analysis can also be distorted by small trials or by differences in other trial characteristics such as different study conditions, differences in patient inclusion criteria etc. But, these potential limitations (which may occur or not occur in a specific case) should not be used to a priori preclude network meta-analysis from the top of the evidence hierarchy. In medicine, we should always start from the theoretically best method, and all methods have limitations. For example, similar problems occur in randomised-controlled trials which are preferred by some guideline producers. Can we really assume that the patients included in them are similar enough that we can average the effects in both groups and compare them? The inclusion criteria usually leave a lot of room for variability leading to large standard deviations in psychiatric studies.
Therefore, in our opinion, systematic reviews based on network meta-analyses should generally be the highest level of evidence in treatment guidelines, but we need to assess them carefully and in certain situations (such as if a meta-analysis is mainly composed of small trials), later published well-designed, large randomised-controlled trials may indeed be preferred .
Nothing comes in complete black or white, but they come in shades of grey in the real world. It is therefore imperative for evidence users to critically appraise each piece of evidence, be it network meta-analysis, pairwise meta-analysis or randomised-controlled trial. One general problem is that publications on the level of evidence often omit the term “systematic review” before meta-analysis—probably only because otherwise the term gets very they long—but a systematic review process must always be implied because without it, any meta-analyses can be useless and should be disregarded. Checklists to assess the quality of systematic reviews such as the AMSTAR instrument exist, but they only check the methodological quality of a systematic review, for example, whether there was a systematic literature search or whether publication bias been investigated . They do not examine the quality and content of the included studies, which should be assessed with the risk of bias tool (bearing in mind the risk of “garbage in garbage out”). It would be laudable if guideline authors could reassess the included studies themselves, but this requires a lot of expert knowledge, it is time consuming, and it opens the doors for selection bias. We would therefore favour the general application of the GRADE approach  which should be ideally already applied by the original systematic review authors, and for which extensions to network meta-analysis has been developed and should be endorsed .
Compliance with ethical standards
Conflict of interest
In the last three years, Stefan Leucht has received honoraria for lectures from EliLilly, Lundbeck (Institute), Pfizer, Janssen, BMS, Johnson and Johnson, Otsuka, Roche, SanofiAventis, ICON, Abbvie, AOP Orphan, Servier; for consulting/advisory boards from Roche, Janssen, Lundbeck, EliLilly, Otsuka, TEVA; for the preparation of educational material and publications from Lundbeck Institute and Roche. EliLilly has provided medication for a clinical trial led by SL as principal investigator. Andrea Cipriani was expert witness for Accord Healthcare for a patent issue about quetiapine extended release. Toshi A Furukawa has received lecture fees from Eli Lilly, Janssen, Meiji, MSD, Otsuka, Pfizer and Tanabe-Mitsubishi, and consultancy fees from Sekisui Chemicals and Takeda Science Foundation. He has received royalties from Igaku-Shoin and Nihon Bunka Kagaku-sha publishers. He has received grant or research support from the Japanese Ministry of Education, Science, and Technology, the Japanese Ministry of Health, Labour and Welfare, the Japan Society for the Promotion of Science, the Japan Foundation for Neuroscience and Mental Health, Mochida and Tanabe-Mitsubishi.
- 2.Cipriani A, Furukawa TA, Salanti G, Geddes JR, Higgins JP, Churchill R, Watanabe N, Nakagawa A, Omori IM, McGuire H, Tansella M, Barbui C (2009) Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis. Lancet 373:746–758CrossRefPubMedGoogle Scholar
- 9.Heres S, Davis J, Maino K, Jetzinger E, Kissling W, Leucht S (2006) Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: an exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. Am J Psychiatry 163:185–194CrossRefPubMedGoogle Scholar
- 12.Leucht S, Cipriani A, Spineli L, Mavridis D, Orey D, Richter F, Samara M, Barbui C, Engel RR, Geddes JR, Kissling W, Stapf MP, Lassig B, Salanti G, Davis JM (2013) Comparative efficacy and tolerability of 15 antipsychotic drugs in schizophrenia: a multiple-treatments meta-analysis. Lancet 382:951–962CrossRefPubMedGoogle Scholar
- 19.Trikalinos TA, Churchill R, Ferri M, Leucht S, Tuunainen A, Wahlbeck K, Ioannidis JPA (2004) Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time. JClinEpidemiol 57:1124–1130Google Scholar