Unequal distribution of risk factors among the exposed and unexposed groups in observational (non-randomised) studies results in confounding bias. Such bias is almost inherent to the design of observational studies assessing effects of interventions. Most commonly, in daily practice patients with a relatively poor prognosis (i.e. those with an indication) receive a medical intervention. Thus, patient groups differ not only on whether they receive the intervention under study, yet also on other patient characteristics, for example severity of disease [1, 2]. This invariably leads to increased probability of the outcome in those receiving the intervention as compared to patients not receiving the intervention. Such bias, often referred to as “confounding by indication”, typically leads to an underestimation of the intervention effects. In contrast, exposure to interventions like physical exercise and diets are more common in relatively healthy persons (“healthy user bias”) and may lead to overestimation of their effects. Appreciating the role of confounding in observational intervention studies is therefore essential to value the true impact of the interventions under study [3–5]. We conducted a systematic medical literature search to assess trends in attention for confounding bias in observational intervention studies.

We searched the Medline database from 1985 through 2005, to identify observational intervention studies in English on humans (http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?). Therefore we adapted a Health Technology Assessment search query [6]. Inclusion criteria included case-control, cohort, longitudinal, follow-up, cross-sectional, prospective or retrospective studies. We excluded randomised controlled trials, guidelines, reviews, and letters. To ensure that the study indeed pertained to the effect of an intervention, the presence of the term “intervention study”, “treatment outcome”, “therapeutics” or “drug therapy” as medical subject heading was conditional. The main outcome measure was the proportion of observational intervention studies mentioning confounding (confound*) or adjustment/adjusted (adjust*) in any Medline field (including title, abstract, and Medical Subject Heading) among included studies. An additional search included the word “bias”. In a random sample (n = 100) the Medline results were compared to the original article (kappa-statistics). A sub-analysis was planned for seven general medical journals (NEJM, JAMA, Lancet, Ann Intern Med, Ann Rev Med, Arch Intern Med, BMJ).

A total of 119,633 publications were eligible for inclusion and analysed. Of these, 1,260 (1.1%) explicitly mentioned confounding and 7,682 (6.4%) mentioned either confounding or adjustment. Kappa-statistics for the search strategy was 0.68. Fig. 1 illustrates trend in proportions stabilizing over the last years at approximately 1% for mentioning confounding and 9% for mentioning either confounding or adjustment. Among the seven general medical journals the proportions were higher: 5 and 35%, respectively. The additional search including the word “bias” did not alter these findings.

Fig. 1
figure 1

Historical trend lines on prevalence of confounding in peer-reviewed medical publications on observational intervention studies. □ non-randomized intervention studies mentioning confounding among all non-randomized intervention studies, ■ non-randomized intervention studies mentioning confounding (confound*) or adjustment (adjust*) among all non-randomized intervention studies, Δ non-randomized intervention studies mentioning confounding among all non-randomized intervention studies in seven general medical journals, ▴ non-randomized intervention studies mentioning confounding (confound*) or adjustment (adjust*) among all non-randomized intervention studies in seven general medical journals

Full size image

These data show that confounding, although a serious threat to the validity of results of observational intervention studies, does not receive adequate attention in peer-reviewed medical literature. Only 9% of all papers mentioned any of the terms indicating appreciation of confounding. Importantly, despite reports alarming the scientific community, the attention for confounding seems to stabilize at a low level [7].

We assumed a high correlation between terms on confounding in any Medline field and its appreciation in the corresponding publication. It is not inconceivable that such words were not included while authors in fact addressed confounding in their paper. Given the high measurement of agreement between the search strategy and the publication text in a sample of publications it is highly unlikely that the trend is influenced by such misclassification (kappa-statistics = 0.68).

Observational intervention studies are prone to confounding bias, since interventions in such studies are non-randomly allocated, and follow daily practice of medical doctors. Although a threat to the validity of the study, confounding can often effectively be prevented or controlled by application of several design and statistical techniques such as stratification, multivariate regression analysis, and propensity score methods [4]. These methods relate to observed confounders, yet not to unobserved confounders. Confounding that remains after adjustment for observed confounders is known as residual confounding. No methods are known that can control residual confounding. However, its size can be estimated using sensitivity analysis, in which, generally, different potential confounders are simulated to study their consequence on effect of the intervention [8]. When the size of residual confounding is estimated to be small, this is a strong argument in favour of a valid study result.

In our view, issues relating to confounding, either observed or residual, are essential in observational intervention studies and should therefore be addressed in the paper. Until now, the attention for confounding in observational intervention studies is too low. The time has come that guidelines similar to those for the reporting of trials (CONSORT) are developed for observational intervention studies.

Competing interests

None declared.

Funding

This study is part of a personal grant of Dr. E. Hak to study confounding in observational intervention studies by the Netherlands Scientific Organization (VENI no. 916.56.109).