Introduction and methods

In the February 2006 issue of Diabetologia, Martin et al. reported a 51% reduction in the risk of all-cause mortality in patients with type 2 diabetes who perform self-monitoring of blood glucose (SMBG) [1], which seemed to be exciting news for clinical practice. The Retrolective Study: Self-monitoring of Blood Glucose and Outcome in Patients with Type 2 Diabetes (ROSSO) included 3,268 patients aged 45 years and older who were recruited in 192 primary care practices and were initially diagnosed with type 2 diabetes between 1 January 1995 and 31 December 1999. Baseline characteristics, confounders and information on SMBG were collected based on medical records. Patients were assigned to the SMBG group if self-monitoring was documented for at least 1 year during follow-up. Primary study endpoints were morbidity (defined as diabetes-associated microvascular and macrovascular events) and all-cause mortality. The main target variable was the time from the diabetes diagnosis to a fatal or non-fatal event. Cox proportional hazards models were used to estimate adjusted hazard ratios of all-cause mortality in patients with SMBG (n = 1,479) compared with those without SMBG (n = 1,789). During a mean follow-up of 6.5 years, a total of 120 patients died. Mortality was 2.7% in patients with and 4.6% in patients without SMBG, resulting in an adjusted HR of 0.49 (95% CI 0.31–0.78). The authors controlled for a large number of confounders and the estimates also remained stable after using more sophisticated statistical techniques such as matched-pair or propensity score analyses, as recently published by Kolb et al. [2].

Based upon the observational ROSSO study, guidelines of the German Diabetes Association (DDG) recommend the use of SMBG as an integral part of the treatment strategy for patients with type 2 diabetes [3]. Furthermore, recommendations regarding SMBG of the International Diabetes Federation (IDF) and the National Institute for Health and Clinical Excellence (NICE) acknowledge the findings of this study [4, 5]. However, the impressive benefits of SMBG in patients with type 2 diabetes that were shown in the ROSSO study could not be replicated in another observational study [6] and conflict with findings from randomised controlled trials (RCTs). A recent health technology assessment of the National Institute for Health Research reviewed RCTs with a minimum duration of 12 weeks and concluded that SMBG is of limited clinical effectiveness in improving glycaemic control in patients with type 2 diabetes [7]. The German Institute for Quality and Efficiency in Health Care (IQWiG) also found no proof of benefit for SMBG in patients with type 2 diabetes based on randomised trials [8].

Given all these data, the tremendous benefit observed in the ROSSO study is clearly ‘too good to be true’. Most probably, it was caused by a flawed study design that introduced substantial bias from misclassified immortal person-time. In this report, we illustrate this source of bias in the ROSSO study and try to quantify its magnitude by the approximation given by Suissa [9]. We demonstrate that the bias is large enough to completely explain the apparently protective effect of SMBG on all-cause mortality in this study.

Results and discussion

In the ROSSO study, patients were followed up from the date of the first diabetes diagnosis until the end of the study (December 2003) or until the occurrence of a fatal or non-fatal event. Patients were classified as exposed to SMBG for their whole follow-up time if they performed self-monitoring for at least 1 year during the study period. Because of this selection process, the time between cohort entry (diabetes diagnosis) and the date after 1 year of self-monitoring was performed is unavoidably ‘immortal’ for patients with SMBG. In other words, patients had to survive for at least 1 year after the initiation of SMBG to be classified as exposed to this intervention. They were, therefore, artificially ‘protected’ from death until they had received SMBG for 1 year. On the other hand, if they had died before this criterion was met, patients would have been placed in the non-SMBG group. For example, the patient shown in Fig. 1 starts SMBG 2 years after the diagnosis. After performing SMBG for 1 year, this patient would have been classified as exposed (Fig. 1a). He is, by design, protected from death for the first 3 years under study. But if this patient had died after 2.5 years, he would have been considered as unexposed (Fig. 1b). This immortal time bias leads to a substantial underestimation of mortality rates in the SMBG group, as all these patients are, by definition, free of events for at least 1 year.

Fig. 1
figure 1

Illustration of immortal time bias in the ROSSO study. a The patient is classified as ‘exposed’ as he has performed SMBG for at least 1 year. The study design means he is protected from death for the first 3 years of the study. b If the patient had died after 2.5 years, he would have been considered as not exposed to SMBG. Exposure to SMBG is shown by the thick grey line. The ‘×’ indicates death

Full size image

To understand this kind of bias, we have to look at exposed and unexposed person-times instead of exposed and unexposed patients. Because of the fixed-time analysis, where treatment status (SMBG vs non-SMBG group) is assessed during follow-up, all persons in the SMBG group contribute immortal person-time. For the patient shown in Fig. 1a, this is composed of the time between cohort entry and start of SMBG (2 years) and the 1 year for which SMBG had to be performed. These 3 years are misclassified as SMBG person-time, while in fact it was non-SMBG person-time because the patient was not yet exposed to SMBG. According to the authors’ definition, exposure does not start until 1 year of SMBG was performed. A correct analysis would have to classify the (unexposed) immortal person-time as non-SMBG person-time using a Cox proportional hazard model with a time-dependent exposure variable for SMBG status [9].

Is the bias from misclassified immortal time in the ROSSO study large enough to explain the 51% risk reduction in all-cause mortality? By using data given for the incidence calculation of macrovascular events [10] we assume a total follow-up of 9,248 person-years (PY) for patients performing SMBG and 10,560 PY in the non-SMBG group. Based on the published information according to the year after diagnosis in which patients initiated SMBG [1], we estimated the amount of misclassified and immortal person-time in the exposed participants (Table 1). For example, a total of 432 patients (29.2%) started SMBG in the year of the first diagnosis and thus contributed at least 432 PY of immortal person-time to the SMBG group. We estimated the total amount of misclassified immortal person-time in the exposed individuals to be at least 5,082 PY, which corresponds to 55% of the total exposed person-time. Even this is a conservative approximation as we assume that all patients initiated SMBG on the first day of the corresponding year (including patients starting in the seventh or eighth year). By using the number of 120 deaths (41 performed SMBG and 79 did not) and the total person-time as given above (9,248 and 10,560 PY), we estimate a biased relative risk (RRbiased) of 0.59, which is comparable with the unadjusted hazard ratio found in the ROSSO study. A re-classification of the immortal person-time as unexposed leads to an RRunbiased of 1.95:

$$ R{R_{biased}} = \frac{{41/9,248}}{{79/10,560}} = {\mathbf{0}}{\mathbf{.59}} $$
$$ R{R_{unbiased}} = \frac{{41/(9,248 - 5,082)}}{{79/(10,560 + 5,082)}} = {\mathbf{1}}{\mathbf{.95}} $$
Table 1 Estimation of the total misclassified immortal person-time in the SMBG group in the ROSSO study
Full size table

In other words, the strong protective effect of SMBG on all-cause-mortality found in the ROSSO study (i.e. a risk reduction of 51%) is completely explained by ‘guaranteed survival’ due to immortal time bias and is, therefore, introduced by study design. Our corrected estimation even seems to indicate an increased mortality in patients with SMBG. This might reflect a higher morbidity in the SMBG group, for instance because of their poorer metabolic control at baseline and their advanced stage of diabetes as indicated by an approximately tenfold higher proportion of insulin users during follow-up [1, 2]. We have demonstrated the immortal time bias in the ROSSO study for all-cause mortality. However, the analysis for non-fatal events, which shows a reduced risk in the SMBG group (HR 0.68, 95% CI 0.51–0.91) is, of course, also affected by this bias. Our quantification and the correction of the relative risk require several assumptions [9]. An estimation of the ‘true’ influence of immortal time bias on the results would require a re-analysis of the original data using appropriate methods.

Immortal time bias is introduced by the use of a time-fixed analysis, which looks at exposed and unexposed patients instead of exposed and unexposed person-times. This bias cannot be addressed by adjusting for numerous confounders or applying sophisticated statistical techniques such as propensity-score analyses. A substantial number of observational studies have been reported to be subject to immortal time bias [11]. Despite the fact that immortal time bias was already identified in observational studies on survival of patients with heart transplants in the 1970s, studies biased by immortal time seem to have accumulated during the last years. This might be due to the fact that large healthcare or medical-record databases are increasingly used for retrospective observational research. An increased awareness of immortal time bias seems inevitable, particularly if tremendous benefits are shown in observational studies that are not in line with findings of randomised controlled trials [11].