Introduction

The prevalence of type 2 diabetes is increasing worldwide. In developed countries, the increase is mainly due to the ageing of the population and changes in lifestyle (reduced physical activity and an increased prevalence of obesity). The impact of diabetes on national health systems is relevant as direct costs for diabetic individuals are fourfold higher than for those without diabetes of a similar age and sex, the excess being mainly accounted for by hospital admissions for cardiovascular complications [1]. Cardiovascular diseases are the leading causes of morbidity and mortality, with an estimated 10–15 years’ reduction in life expectancy from age 50 years for diabetic compared with non-diabetic individuals [2].

In a global scenario of limited economic resources, healthcare planners need long-term projections of the outcomes and costs of the disease. In this context, prognostic models can assist researchers and help doctors and patients to make decisions by providing estimates of the risk of future outcomes for complications of diabetes [3]. Various computer simulation models have been developed in the field of diabetes research [4]. However, to gain acceptance, it is important that such models can demonstrate reliable predictive performance not just with the patient sample used to develop the model, but with external patient populations who have not been involved in the model’s development and estimation [5]. Published guidelines, such as the American Diabetes Association guidelines for computer modelling of diabetes, strongly encourage external validation exercises to demonstrate the ability of a model to predict outcomes reliably [6].

The United Kingdom Prospective Diabetes Study (UKPDS) Outcomes Model is a widely used example of such a model. It was developed using patient data from the UKPDS trial, and provides estimates of the likely occurrence of major diabetes-related complications in patients with type 2 diabetes over different periods up to a lifetime; all equations pertaining to the model, and details of internal validation exercises, have been published in full [7]. Up until now, it has been used for a range of purposes including extrapolation from clinical trials, prediction of future demands on health services, power calculations during study design, and to provide look-up life-expectancy tables for different combinations of risk factor levels [8–10]. However, validation on different clinical datasets with long follow-up is lacking.

The Casale Monferrato Study is an ongoing population-based study examining the epidemiology of type 2 diabetes over time in a representative Italian population [11]. The completeness of ascertainment of those with diabetes has been estimated to be 80% [12], and this finding, along with the population-based study design and the availability over time of data on both risk factors and outcomes, makes this cohort particularly attractive for performing a validation of a model.

The aim of this study was therefore to assess the performance of the UKPDS Outcomes Model by comparing its predicted cumulative incidences of diabetes-related health outcomes with the observed cumulative incidences in a different clinical setting.

Methods

The UKPDS Outcomes Model

The UKPDS Outcomes Model was developed as a set of linked survival equations estimated using 3,642 patients from the UKPDS, supported by a set of prediction equations for the main risk factors used in these survival equations; an article providing a full description of the model has been published [7]. In summary, the UKPDS Outcomes Model is a probabilistic discrete-time model that uses an integrated set of parametric proportional hazard models to predict the absolute risk of first occurrence of seven major diabetes-related complications—ischaemic heart disease (IHD), fatal and non-fatal myocardial infarction (MI), congestive heart failure (CHF), stroke, blindness, renal failure and amputation, as defined in the electronic supplementary material (ESM) text, and all-cause and diabetes-related mortality.

The patients from whom the prediction equations were estimated were all participants in the UKPDS. A total of 5,102 patients were enrolled from 7,416 patients referred to a UKPDS clinic with newly diagnosed type 2 diabetes (defined as fasting plasma glucose concentrations over 6.0 mmol/l on two separate mornings) and aged 25–65 years. Exclusion occurred mainly because of severe vascular disease, MI or stroke within the year before recruitment, or major systemic illness. Recruitment occurred between 1977 and 1991 at 23 clinical centres in England, Scotland and Northern Ireland. Model estimation was based on the 3,642 patients with complete data for potential confounders. Their detailed characteristics have been described elsewhere [13].

The model’s predictions are based on patients’ characteristics, which are entered as time-varying covariates and are updated at the beginning of each annual cycle. These include age, sex, ethnicity, duration of diabetes, physiological and clinical characteristics (height, weight, total and HDL-cholesterol, HbA1c level and systolic blood pressure [SBP]), lifestyle input variables (smoking status) and the patient’s history of complications. Patients enter the model with a specified set of these characteristics and can experience one or more non-fatal complications or die during any of the annual cycles as the simulation progresses. The updating captures the fact that the occurrence of a non-fatal event will increase the risk of subsequent events: for example, in the model, heart failure significantly elevates the risk of a subsequent MI or stroke. These associations between different types of complications at an individual patient level, sometimes referred to as event-related dependence, are a key feature of the model. The simulation progresses until the patient is predicted to have died.

A validated software version of the UKPDS Outcomes Model is available from the UKPDS website (www.dtu.ox.ac.uk/), using Microsoft Excel to input data and output results.

Patient data

The Casale Monferrato Survey was begun in 1988 with the aim of assessing the prevalence of known diabetes in persons living in the area of Casale Monferrato, northern Italy (93,477 inhabitants). The study protocol was approved by the Ethics Committee of the Local Health Unit of Casale Monferrato (Alessandria). The study design has been described in detail elsewhere [11]. Briefly, persons with a previous diagnosis of diabetes (n = 1,967) were identified from diabetes clinics, general practitioners’ records, hospital discharge records (HDRs), prescriptions and records of sales of reagent strips and syringes. The completeness of ascertainment (80%) was estimated using a capture–recapture method, based upon the linkage between incomplete multiple data sources. In particular, we applied log-linear models to the capture–recapture method, which allowed us to estimate the degree of undercount taking into account both dependence between data sources and the heterogeneity of patients within sources [12]. In 1991–1992, identified patients were invited to a baseline examination to assess the prevalence of cardiovascular risk factors [14, 15]. Out of the initial 1,967 participants, 358 had died, 25 had been lost to follow-up and 45 did not respond to the invitation. The remaining 1,539 participants were visited and interviewed. A flow chart of the patients is shown in Fig. 1.

Fig. 1
figure 1

Flow chart for enrolment of Casale Monferrato patients

Full size image

In 2000, a new survey included all the members of the original cohort who were still alive and living in Casale Monferrato (n = 860), in addition to all persons with a new diagnosis of type 2 diabetes (n = 2,389) identified through the diabetes clinics and from an administrative database—available since 1993—including all those who obtained exemption from payment for glucose-lowering drugs and glucose-monitoring strips (Fig. 1). The two-sample capture–recapture method allowed a completeness of ascertainment of 95% to be achieved [16, 17]. All the participants were white.

In the 1991 survey, patients received both a letter and a telephone invitation to undertake the visit. In the 2000 survey, data were collected during periodic visits either by diabetologists or general practitioners, but no active invitation could be made due to limited resources. Therefore, a higher frequency of missing data was obtained in the 2000 (22.6%) than in the 1991 (6.1%) survey when all risk factors required to run the UKPDS outcome model were considered (SBP, HDL and total cholesterol, HbA1c and smoking status). As these patients were excluded, the current analyses were performed on 1,443 patients from the 1991 survey and 2,514 patients from the 2000 survey. For both the 1991 and 2000 surveys, values for the available risk factors among participants in the missing data group were comparable to those in the complete observations group (data not shown).

As the Casale Monferrato Surveys recruited patients with a previous diagnosis of diabetes, data on risk factors at diagnosis were not available. Data collected at enrolment (the 1988 survey) on weight, height, HbA1c level and smoking status were then used as proxy for these variables at initial diagnosis. Values for SBP, HDL and total cholesterol at diagnosis, which were not collected at enrolment, were assumed in the main analysis to be equal to the first available data, but alternative assumptions were also explored and are reported in a sensitivity analysis. The same approach was used for patients enrolled in 2000, for whom only current risk factors were recorded.

For the 2000 survey participants only, dates of the first occurrence of pre-existing events were also recorded, and diabetes-related complications (MI, IHD, stroke, CHF and amputation) recorded during follow-up were identified from HDRs by means of a record linkage procedure based on the encrypted patient’s tax identification number. The HDR database includes episodes of care—both hospital stays and day care—delivered in the Piedmont region and in other Italian regions for all residents. Only the main diagnosis field (ICD-9, Clinical Modification [ICD-9-CM] codes) was used for identifying the occurrence of complications. Fatal events (MI and stroke) were identified from death certificates. The selection criteria adopted were defined according to the UKPDS trial endpoints [18], except for amputation, where due to some differences between the coding systems, an algorithm already in use locally was applied [19]. The list of ICD-9-CM codes used for defining clinical events is provided in the ESM Appendix. HDRs in the local (Piedmont region) healthcare service are available, complete and reliable only from 1998. As a result, it was not possible to follow up macrovascular complications for the 1991 survey participants.

Vital status was last updated at the end of 2006, and causes of death were ascertained from death certificates. Underlying causes of death were derived and coded by two physicians (ICD-9 codes).

Caution should be applied if the results of the model are extrapolated to populations that differ significantly from that used to develop the UKPDS Outcomes Model. In order to assess comparability, a description of the baseline characteristics in the Casale Monferrato cohorts (the 1991 and 2000 surveys) and in the UKPDS Outcomes Model patients is reported in Table 1. On average the UKPDS patients were slightly younger at diagnosis of diabetes, a higher proportion were male, and higher proportions were past or current smokers and had lower mean values for SBP. Other values were broadly similar.

Table 1 Baseline characteristics of the Casale Monferrato cohort (1991 and 2000 surveys) and the UKPDS trial patients used to estimate the UKPDS Outcomes Model
Full size table

Statistical analysis

The first outcome analysed was all-cause mortality. We predicted the cumulative mortality from the 1991 survey data using the UKPDS Outcomes Model, and compared the predicted with the observed risks over the subsequent 17-year period. Analyses were stratified by sex, age at diagnosis (under 65 vs 65 and older) and duration of diabetes. The analysis was then repeated using data from the 2000 survey participants, over a 7-year period. To better understand the impact of changing risk factors over time on the performance of the model, we also analysed separately individuals who had participated in both surveys, using the 1991 risk factor data updated with the 2000 data at the ninth year of follow-up.

To assess whether the assumption regarding baseline risk factors at diagnosis had much impact on the results, a sensitivity analysis is reported, in which the baseline risk factor values are assumed to be 15% higher or lower than the observed values at first available measure.

For the entire 2000 survey, the predictions of the UKPDS Outcomes Model were compared with observed incidences for the following outcomes: MI, other IHD, stroke, CHF and amputation. A similar analysis could not be performed on the 1991 survey participants due to the lack of complete HDRs before 1998. Data on pre-existing events (MI and stroke) were added to the other risk factors to predict outcomes.

The cumulative incidences for the observed outcomes were determined with the Kaplan–Maier estimator. The simulated and actual cumulative incidences for each of the diabetes-related events and all-cause mortality were compared graphically, reporting 95% CIs around the observed values. Point estimates have been provided at 5, 10 and 15 years for the 1991 survey, and at 5 years for the 2000 survey.

CIs around the predicted cumulative incidence rates were not reported, as these represent uncertainty of variables and are therefore not strictly comparable with the CIs around the observed rates.

Results

Figure 2 shows the simulated and actual cumulative incidences of all-cause mortality up to 17 years from the 1991 survey. Apart from an overestimation of overall all-cause mortality in the first 2 years, UKPDS model predictions are within the 95% CIs around the observed risks up to 12 years. From the 13th year onwards, the model slightly overestimates the cumulative mortality. When the analyses were stratified by sex, the overestimation after 13 years of follow-up was evident only in women.

Fig. 2
figure 2

Observed (95% CI) and predicted cumulative incidence for all-cause mortality over 17 years for the 1991 Casale Monferrato Survey: (a) overall (n = 1443); (b) men (n = 625); (c) women (n = 818); (d) age <65 years (n = 1014); (e) age ≥65 years (n = 429); (f) disease duration <6 years; (g) disease duration ≥6 years. Black lines, predicted; solid grey lines, observed; dashed grey lines, observed upper and lower 95% CIs

Full size image

In people aged less than 65 years, a slight overestimation of mortality is evident after 13 years only. The mortality risk for older individuals at diagnosis (age >65 years) falls slightly below the lower 95% CI of the observed mortality only in years 5–9. Predicted mortality risks were close to observed ones irrespective of the duration of diabetes, except for some overprediction after year 12 in patients with a duration of diabetes of 6 or more years.

Table 2 shows point estimates of predicted mortality at 5, 10 and 15 years. The observed and predicted cumulative risks of dying after 5 and 10 years of observation were identical, at 23% and 47%, respectively. At 15 years, the risk of death was slightly overestimated (an estimate of 67% vs 64% observed, 95% CI 61%, 66%). When analyses were performed in subgroups defined by sex, age of onset and duration of diabetes, differences between the predicted and the observed risks were detectable but very small.

Table 2 Comparison of observed (95% CI) and predicted cumulative incidence for all-cause mortality at 5, 10 and 15 years and for diabetes-related complications at 5 years: Casale Monferrato Surveys in 1991 (n = 1,443) and 2000 (n = 2,514)
Full size table

Updated risk factor predictions based on the patients surveyed in 1991 and also participating in the 2000 survey (n = 813) were very similar to the results reported above and have not been reported separately.

The sensitivity analysis of the baseline risk factors at diagnosis (±15%) had little effect on the predictions, producing lower and upper predicted mortality risks at 5, 10 and 15 years of 22–24%, 45–48% and 66–69%, respectively.

For the 2000 survey, a comparison of cumulative risks for the analysed outcomes during 7 years of follow-up are shown in Fig. 3. Cumulative mortality was slightly overestimated by the UKPDS Outcomes Model over the 7-year comparison period, in line with the results of the 1991 survey for the medium term (from 13 years on). With regard to macrovascular outcomes, MI, CHF and amputations were well predicted, whereas both stroke and IHD were underestimated.

Fig. 3
figure 3

Observed (95% CI) and predicted cumulative incidence for diabetes-related complications over 7 years for the 2000 Casale Monferrato Survey (n = 2514): (a) MI; (b) stroke; (c) CHF; (d) IHD; (e) amputation. Black lines, predicted; solid grey lines, observed; dashed grey lines, observed upper and lower 95% CIs

Full size image

The differences between the observed and predicted point estimates of cumulative incidence at 5 years of macrovascular outcomes ranged between 1% and 2% (Table 2), and were not statistically significant, apart from stroke and IHD, for which predicted incidences were under the lower limit of CI of observed incidences.

Discussion

This study reports the results of an external validation of the UKPDS Outcomes Model against an Italian population-based cohort. The study objective—to assess the performance of the model in predicting the cumulative incidence of diabetes-related health outcomes—is in line with the American Diabetes Association guidelines for computer modelling of diabetes [6], where external validation is strongly encouraged. The model estimates the likely occurrence of major diabetes-related complications over a lifetime, with relevant potential applications to economic evaluations of interventions in type 2 diabetes, to calculating health economic outcomes, and to predicting future demands on health services. Our results showed a good performance of the UKPDS Outcomes Model in predicting death up to a 12-year time period, and the incidence of macrovascular outcomes over a 5-year time span, in an Italian general population with type 2 diabetes. Differences in the predicted and observed risks were quite limited, confirming the applicability of the UKPDS Model to people living in a Mediterranean country.

Whereas the model’s predicted risk of mortality up to 15 years fitted well for men, a slight overestimation was evident after 12 years in women. This effect of sex could be partly due to the difference between the two countries in terms of life expectancy of the general population [20], Italian life expectancy being among the highest in Europe, after Denmark and Malta, with larger differences for women.

The long-term overestimation could also be due to differences between the two populations in mortality risk for diabetes. Prevalence surveys performed in Italy both in Verona [21] and in Casale Monferrato [22] have shown that mortality excess in diabetic vs non-diabetic individuals (35–40%) is lower in Italy compared with the twofold excess reported in other European countries. In contrast, the underestimation of mortality in people aged 65 years and over could be inherent in the UKPDS study design. Indeed, the recruitment criteria of the UKPDS excluded diabetic people aged 65 years and over, as well as those with a previous diagnosis of the disease and those with cardiovascular disease, and this may have reduced the power to estimate mortality in older diabetic individuals. It should be underlined, however, that overestimations in women and in the elderly were quite limited.

The UKPDS Outcomes Model also predicted well the cumulative incidence of macrovascular disease up to 5 years, with very slight differences between the observed and predicted 5-year risks, but with most of the observed risks falling inside or overlapping the CIs of predicted values. The incidence of MI, either fatal or non-fatal, was very well predicted by the model. The ability of the model to accurately predict heterogeneity in the incidence of other outcomes of interest was graphically evident for stroke (Fig. 2), with an underestimation of the model over the whole 7-year period of follow-up of our cohort. As blood pressure is the main risk factor for stroke, baseline differences between the UKPDS and the Casale Monferrato Study might be one contributory explanation for this finding. In both the 1991 and 2000 Casale Monferrato Surveys, mean values of SBP were higher than in the UKPDS cohort, and this might also have affected our estimate of the cumulative incidence of non-fatal IHD, which was underestimated by the UKPDS outcome model. However, as the incidence of MI was well predicted, some methodological problem, possibly in hospital discharge coding, might have been involved in our result.

A small number of previous external validation studies involving the UKPDS Outcomes Model have been reported. Holman et al [23] used baseline characteristics and the reported differences in attained risk factor values in the PROactive trial to predict the main secondary endpoint of the trial: frequency of all-cause mortality plus non-fatal MI plus stroke at 3 years [24]. The trial reported an RR of 0.84, with 95% CIs of 0.72 and 0.98. The UKPDS Outcomes Model predicted an RR of 0.87, well within the CI.

Clarke et al estimated the 4-year coronary event rates likely to occur in the Collaborative Atorvastatin Diabetes Study trial and compared predicted with actual results, as part of a larger comparison of different diabetes models [25]. They reported an acute coronary event rate at 4 years of 5.3 in the control group and 3.6 in the intervention group, close to the reported rates of 5.5 (95% CI 4.3, 6.6) in the control group and 3.6 (95% CI 2.6, 4.5) in the intervention group.

Finally, Song and colleagues compared predicted all-cause mortality using the UKPDS Outcomes Model with participants in the US National Health and Nutrition Examination Survey 1988–1994, and found that the predicted 10-year mortality of 15.7% was similar to the observed mortality of 14.2% in individuals with characteristics comparable with those of the UKPDS cohort [26]. They also reported a similar predicted and observed mortality for those aged over 65, and for those with pre-existing cardiovascular disease, but found that the UKPDS Outcomes Model appeared to substantially overpredict mortality in patients with a duration of diabetes of 6 years or longer.

The strengths of our validation study are the population-based study design of the Casale Monferrato Surveys, the centralised measurement of risk factors and the high degree of completeness of ascertainment of the baseline and follow-up data. Several limitations of our analysis, however, deserve comment. The population-based study design of the Casale Monferrato Study, using multiple independent sources of ascertainment, allowed a high estimated completeness of ascertainment of people with type 2 diabetes. However, the complete set of variables required to run the UKPDS model were missing for 6.1% of patients in the 1991 survey and 22.6% in the 2000 survey. Nevertheless, there was no clear evidence of differences in other characteristics between patients who had been included or excluded, suggesting that a selection bias is unlikely. Moreover, the exclusion of individuals because of missing data should not bias the validation of the UKPDS model performance, so that the results of our study should be unaffected by the frequencies of missing data. For the purposes of this modelling exercise, risk factor values at diagnosis, which were not available, were assumed to be the same as those observed at the first available measure. These risk factor values at diagnosis, however, only affect the equations used to predict future risk factors and do not directly affect the equations for risk of complications. Moreover, the results of the sensitivity analysis demonstrate the limited effect on the results of varying this assumption.

An analysis of diabetes-related complications was not performed for the 1991 survey participants because of the lack of complete and reliable HDRs before 1998 in the Piedmont region; consequently, predictions are only available for the medium term. The divergence between the model’s estimates and the observed risks over the longer time horizon may also be attributable to the lack of updating of risk factors in the Italian cohort, as a result of which the ability of the UKPDS model to adjust for time-dependent risk factors has not been fully used. However, when patients from the 1991 survey were analysed with updating of risk factors using data from the 2000 survey, no significant improvements in the performance of the model were found; that is, once-only updating did not improve the long-term estimates. Finally, the divergence between observed and predicted mortality could be attributable to some specific feature of the way in which mortality is modelled in the UKPDS Outcomes Model; for example, the effects of increasing age and duration of diabetes may not be linear and proportional.

In conclusion, in a cohort of Italian patients with type 2 diabetes, the UKPDS Outcomes Model performed reasonably well in predicting all-cause mortality in the medium term and risk of MI over a shorter period. Most of the divergences between the observed and the predicted risks are explainable at least in part by differences in participants’ characteristics and epidemiological patterns between the two populations. However, the design of the model may also play a part: an updated version of the model is in development incorporating post-trial monitoring data from the UKPDS population, and will in time allow this exercise to be repeated and the results compared. Meanwhile, these external validation results suggest that the UKPDS Outcomes Model is a reliable tool for simulating outcomes in economic evaluations and other studies, but that long-term (>15-year) predictions should be used with caution.