Psychostimulant medications, such as methylphenidate and amphetamine, are commonly prescribed for attention-deficit/hyperactivity disorder (ADHD) and have been a first-line medication for this disorder for over 60 years. However, a recent Cochrane review has raised doubts about the benefit of methylphenidate in the treatment of childhood ADHD (Storebø et al. 2015). Although the authors demonstrated that methylphenidate was associated with improved teacher-rated ADHD symptoms (standardized mean difference [SMD], −0.77 [95% CI −0.90 to −0.64] in 19 parallel group trials and first phase of crossover trials which included a total of 1698 subjects, the authors concluded that the true magnitude and relevance of this effect is uncertain because of the “very low quality” of most of the included studies. They assessed that 96.8% of all 185 trials included in the broader review were at high risk of bias trials according to their interpretation of the evidence. This assumption contrasts with previously published systematic reviews and meta-analyses, including the comprehensive review of the National Institute for Health and Care Excellence (for example, Faraone and Buitelaar 2010; King et al. 2006; National Institute of Health and Clinical Excellence 2008; van der Oord et al. 2008), which all judged study quality more favourably. Interestingly, in the Cochrane review 13 out of the 19 studies were graded as having a high risk of bias due to a vested interest bias. This domain of bias was introduced by the authors on top of and in addition to the standard Cochrane bias domains. It describes a trial as of high risk of bias when the trial was either funded by parties that might have had a conflict of interest (such as a manufacturer of methylphenidate) or where there were potential conflicts of interest among authors, for example, because they had been received remuneration from companies producing or selling ADHD medications.

This Cochrane review led to a fierce discussion and several commentaries that have questioned whether inaccuracies, inconsistencies and misinterpretations of trial data included in the review lead to false results and inadequate conclusions (Banaschewski et al. 2016a, b; Hoekstra and Buitelaar 2016a, b; Romanos et al. 2016a, b; Shaw 2016). Here, we focus on several issues that have not been addressed adequately by the authors of the Cochrane review.

Incorrect selection of studies for estimation of the effect size

The effect size estimate in the Cochrane review was based on 19 of 185 included studies. However, four studies did not compare methylphenidate with placebo but with a co-treatment (i.e. behavioural therapy or clonidine) in both the placebo or non-treatment arm as well as the active methylphenidate arm. This will almost certainly have resulted in a reduction in the reported effect sizes compared to what would be expected with a placebo comparison. Even more surprising, the multimodal treatment of ADHD study was included in the meta-analyses even though it lacked a placebo-arm or non-treatment condition and instead compared medication to treatment as usual (with the majority of those in this arm receiving medication). Finally, one of the included studies was conducted in children aged 3–6 years in whom its use is off-label and in an age range where medication treatments are generally agreed not to be first line and to be associated with lower effect sizes. Accordingly, to enable firm conclusions on the efficacy of methylphenidate in children and adolescents, these five studies should have been excluded from the analysis. The effect size based on teacher reports, after excluding these studies, increases from 0.77 (moderate) to 0.89 (high) (Banaschewski et al. 2016a, b). The authors excluded 56 studies that used a crossover design and provided only endpoint data. The rational for exclusion is that carry-over effects may have inflated the reported effect sizes. Whilst this is possible, including these studies had minimal impact on the effect size which with all 75 eligible studies was 0.91 (Banaschewski et al. 2016a, b).

Judgement of risk of bias that is not justified by “The cochrane collaboration’s risk of bias” tool

It is of utmost importance to critically evaluate the findings of trials in the light of potential bias that may have influenced both study design and conduct or the stated conclusions. This should, however, be supported through empirical testing of whether bias has really distorted the analysis of data, reporting of results and drawing conclusions.

The authors considered seven sources of bias: randomization methods, the concealment of allocation, blinding of participants and personnel, blinding of the outcome assessment, incomplete outcome data, selective reporting, and vested interest bias. Studies were judged as having a high risk of bias if any one source of bias was present, in other words, each type of bias was considered of equal importance. Originally, “The Cochrane Collaboration’s risk of bias” tool was developed on the basis of a combination of empirical and theoretical considerations, leading to a focus on risk of bias rather than study “quality” and a division of assessments into six bias domains (Sterne 2013). Including the funding source as a standard item in the tool is an adaptation that has caused particular controversy. Indeed, there is no evidence that the funding source has really distorted the analysis of data. Previous analysis gave indication, that the principle bias associated with industry funding is associated with the interpretation of the results rather than study design and conduct (Sterne 2013; Lundh et al. 2012). This potential bias of interpretation is, however, not relevant when including study data to calculate effect sizes in meta-analyses.

In the Cochrane review, even those remaining seven studies that were categorized as having overall low risk of bias were still described by the authors as probably biased because in their view frequent adverse events such as loss of appetite or trouble falling asleep may have unmasked the intervention status. This assumption which suggests that the authors entered the review with a prior notion that all studies were probably biased which is an important context for the conduct of the review. There is however no empirical evidence presented that supports bias due to possible de-blinding caused by adverse events. In fact, from a practical and clinical perspective school teachers, whose rating were the basis of the effect sizes, are less likely than parents to detect changes in eating or sleeping patterns in school children and attribute these to adverse drug reactions and therefore less likely than parents to be biased in their assessment of the core symptoms of ADHD.

Conclusion

In summary, incorrect inclusion and exclusion of studies and unjustified judgement of risk of bias has led to results that contradict decades of evidence from randomized clinical trials. Since long-term randomized trials are impossible and unavailable due to ethical limitations, recent analyses from registries provide the most convincing evidence on the long-term benefits of methylphenidate. These studies indicate, for example, that treatment with methylphenidate has beneficial effects in terms of important distal outcomes that go beyond symptom control such as a reduction in comorbid depression, substance-use and dependency, trauma-related visits to the emergency room and mortality (Chang et al. 2014, 2016; Dalsgaard et al. 2015; Man et al. 2015). Furthermore, in the absence of clinically relevant effects of non-pharmacological treatment options on the core symptoms of ADHD (Sonuga-Barke et al. 2013), a dogmatically biased devaluation of the currently most effective treatment is neither helpful for science nor patients.