Abstract
Background
Various prognostic models have been derived to predict chronic kidney disease (CKD) development in type 2 diabetes (T2D). However, their generalisability and predictive performance in different populations remain largely unvalidated. This study aimed to externally validate several prognostic models of CKD in a T2D Thai cohort.
Methods
A nationwide survey was linked with hospital databases to create a prospective cohort of patients with diabetes (n = 3416). We undertook a systematic review to identify prognostic models and traditional metrics (i.e., discrimination and calibration) to compare model performance for CKD prediction. We updated prognostic models by including additional clinical parameters to optimise model performance in the Thai setting.
Results
Six relevant previously published models were identified. At baseline, C-statistics ranged from 0.585 (0.565–0.605) to 0.786 (0.765–0.806) for CKD and 0.657 (0.610–0.703) to 0.760 (0.705–0.816) for end-stage renal disease (ESRD). All original CKD models showed fair calibration with Observed/Expected (O/E) ratios ranging from 0.999 (0.975–1.024) to 1.009 (0.929–1.090). Hosmer–Lemeshow tests indicated a good fit for all models. The addition of routine clinical factors (i.e., glucose level and oral diabetes medications) enhanced model prediction by improved C-statistics of Low’s of 0.114 for CKD and Elley’s of 0.025 for ESRD.
Conclusions
All models showed moderate discrimination and fair calibration. Updating models to include routine clinical factors substantially enhanced their accuracy. Low’s (developed in Singapore) and Elley’s model (developed in New Zealand), outperformed the other models evaluated. These models can assist clinicians to improve the risk-stratification of diabetic patients for CKD and/or ESRD in the regions settings are similar to Thailand.
Graphical abstract
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Chronic Kidney Disease (CKD) is a major worldwide health burden and the most common microvascular complication of type 2 diabetes (T2D) [1, 2]. In 2017, more than 840 million individuals developed CKD [3], increasing health care demand, particularly in low to middle-income countries (LMICs) [1]. In the UK and the United States, the prevalence of CKD in T2D was reported to range between 25 and 36%, of which 19% was estimated to be advanced (stages 3–5) [4]. The age-standardised global mortality of CKD due to diabetes has been estimated at 7.6 per 100,000[5].
Early detection and treatment are beneficial in the prevention or delay of CKD progression. Despite improved screening, many CKD patients face delayed diagnosis until an advanced stage due to a lack of overt symptoms. Prognostic models for complications associated with T2D progression that incorporate clinical information systems would facilitate improved treatment allocations, healthcare management, and improve understanding of clinical research strategies [6, 7].
Currently, several prognostic equations [8,9,10,11,12,13,14,15] are available for the prediction of CKD in T2D patients, but their generalisability remains uncertain due to limited external and independent validation, particularly in Asian populations [16, 17]. Indeed, external validation is essential and has become mandatory before implementation in clinical practice [16, 18, 19].
Despite many potential advantages, prognostic models have several shortcomings and frequently reported deficiencies [20]. Multiple models have been developed in different ethnicities [8,9,10,11,12,13,14,15, 21,22,23,24,25,26,27,28,29] but no single model has consistently outperformed all others in Asian populations. For instance, a study based in China performed a limited temporal internal model validation over time on the same data [10]. Most importantly, adaptation of a suitable prognostic model by ethnicity is particularly in an Asian context given that half of the ten countries affected by diabetes worldwide are Asian [4]. Furthermore, recent recommendations have proposed re-evaluation to including race/ethnicity in CKD prediction models [30].
Therefore, this study conducted external validation and improvement of previously published prognostic models of CKD and end stage renal disease (ESRD) in Thai T2D patients.
Methods
We adhered to the TRIPOD guidelines for the development and validation of a clinical prediction score [31, 32]. We focused on external validation of existing models of CKD-ESRD risk predictions in T2D, supplemented with the addition of routine clinical factors to potentially increase the discriminatory power in our local population [18].
We first identified previous prognostic models by performing a systematic review and meta-analysis (SR/MAs), see Figure S1. We selected prognostic models if they: (1) had been internally or externally validated; (2) reported moderate to excellent discrimination of C-statistics, i.e., ≥ 0.70. We identified six studies that met the inclusion criteria for CKD [8,9,10] and ESRD [11,12,13] (Table 1).
Study design and data sources
Data from the Thailand National Health Examination Survey (Thai-NHES) and the standard health databases version 2.4 2019 edition (http://spd.moph.go.th/healthdata/) were used for model validation. The NHES IV and V were population-based cross-sectional surveys conducted in 2009 and 2014, respectively. These surveys captured: health interviews, physical examination, nutrition assessment, and health-related behaviours [33]. Briefly, a multi-stage sampling of adult subjects from the regions, provinces, and districts across the country was used [34, 35].
The standard health databases included medical service records from hospitals, mostly under the direction of the Ministry of Public Health. They comprised a set of tables of all transactions from outpatient and inpatient services for each individual; of 43 files available, only the six that were related to outpatient services (i.e., Person, Diagnosis, Chronic, Drug, Laboratory, and Death) were used for this study.
Settings and participants
A total of 19,671 and 18,564 participants were de-identified from NHES IV-V, respectively; removal of duplicates and missing or invalid citizen identification (CID) resulted in 29,089 participants remaining, see Fig. 1. These were linked with the standard hospital health databases (1999–2019) using an encrypted CID to construct the initial sampling frame, leaving a total of 26,170 participants.
We confirmed T2D status based on self-report, medication use, and/or pathology tests (Fasting Plasma Glucose (FPG) ≥ 126 mg/dL or HbA1c ≥ 6.5%). We excluded type 1 diabetes (T1D) with age at onset less than 30 years with severe insulin treatment. There were 3416 participants with identified T2D, of whom 270 (7.9%) were excluded on the basis that CKD was diagnosed prior to T2D, leaving a total of 3146 participants. Of these, 3014 (10.4%) participated in both NHES IV-V, with 402 newly diagnosed participants identified after the survey, see Fig. 1. These participants were followed up from 1999 to October 31st, 2019.
Outcomes
The primary study outcomes included diabetic nephropathy (CKD stage 3–5) based on the International Classification of Disease, Tenth Edition (ICD-X), which was confirmed by estimated glomerular filtration rate (eGFR) < 60 mL/min/1.73m2 measured within 3 months before and after diagnosis, see Table S1. ESRD (CKD stage 5) was defined as eGFR < 15 mL/min/1.73m2, or dialysis identified by ICD-X code diagnosis. eGFR was based on the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) formula [36].
Established prognostic factors
We focused on prognostic factors identified through our systematic review, including demographics (age, sex, education, income, and area of residence), biomarkers, comorbidities, medication usage, and clinical features; the latter included diabetes duration, body mass index (BMI; kg/m2), waist and hip circumference (cm), systolic/diastolic (SBP/DBP) blood pressure (mmHg), pulse (beat/min), smoking, alcohol consumption, dietary control measures, physical activity, dyslipidaemia, hypertension, and family history of diabetes (FHD, presence of T2D in 1st-degree relatives). Biomarkers included lipid profile (i.e., high-density lipoprotein (HDL), low-density lipoprotein (LDL), triglycerides (TG), total cholesterol (TC) in mg/dL, FPG (mg/dL), haemoglobin (g/dL)) and dipstick proteinuria. Comorbidities included a history of cardiovascular disease (CVD) and stroke. CVD was defined by self-report, clinical diagnosis or receipt of treatment for coronary heart disease. Medications recorded included oral-diabetic, blood pressure or cholesterol-lowering drugs.
We included clinical data associated with diabetic complications (i.e., retinopathy, stroke, and composite CVD’s) based on ICD-X diagnostic codes (Table S1), laboratory follow-up, medication treatment (Table S2), or death certification (based on ICD-X).
Hypertension was defined as SBP ≥ 140 or DBP ≥ 90 mmHg or use of anti-hypertensive medication. Dyslipidaemia was defined as HDL ≤ 40 mg/dL, or LDL, TG and TC levels ≥ 130, ≥ 130, and ≥ 200, mg/dL respectively, according to ATP-III guidelines [37].
All factors were included according to their definitions in the original studies (Table S3–S4).
Statistical analysis
Descriptive statistics for predictor variables were summarised as mean (± standard deviation) or median (interquartile range) for continuous variables or frequency (percentage) for categorical variables. Participant characteristics were compared between groups using Chi-Square or Fisher’s Exact test, where appropriate for categorical variables, and one-way ANOVA or Kruskal Wallis for continuous variables. The predictors which were missing ranged from only 0.1% (n = 3) to 5.8% (n = 199). Therefore, a complete case analysis was applied for the whole analyses.
We evaluated prognostic models originally derived by logistic [8, 10] or Cox regression models [9, 11,12,13] that were identified from our systematic review (PROSPERO: CRD42018105287). Prognostic scores were calculated according to the published regression formulae using the coefficient and intercept or baseline hazard, see Table S4.
External validation was undertaken in accordance with guidelines for the validation and interpretation of risk prediction models [18, 19]. In brief, we evaluated model performance through comparisons between the original published equation and models that included additional adjustment (e.g., intercept, regression coefficients) for other potential predictors, see Appendix [18, 38,39,40].
Briefly, model performance was evaluated as follows [7]. Discrimination was assessed by concordance of C-statistics, area under receiver operator characteristic curves (AUROC) [41], and 95% confidence intervals (CI’s). Calibration, i.e., the closeness between the observed and predicted values, was assessed using the Hosmer–Lemeshow goodness-of-fit test, the observed to expected (O/E) ratios with 95% CI, and calibration plots. We also used global heuristic shrinkage factors and penalised regression to address issues of over-optimism in updated prognostic models [39, 42, 43].
All statistical analyses were conducted using STATA version 16.0. A two-sided p-value less than 0.05 was considered significant.
Results
Characteristics of prognostic models
We identified a total of 6 prognostic studies for CKD-ESRD in T2D patients; see PRISMA flow diagram in Figure S1. Of these, two [8, 10] and four [9, 11,12,13] applied logistic and Cox regressions, respectively (see Table S4).
Five [8,9,10, 12, 13] models were developed in Asia and one in New Zealand [11]. Only two [10, 11] models had been externally validated in either a Chinese or New Zealand population. Five [8,9,10,11,12,13] studies used hospital-based data. The mean age of T2D subjects ranged from 55.4 to 62.9 years with study size ranging between 1582 and 116,509. Five [8,9,10, 12, 13] studies performed internal validation by splitting samples for discovery and validation, and three [9, 12, 13] applied multiple imputation to account for missing data (see Table 1).
T2D was characterized on the basis of a FPG ≥ 7.0 mmol/L in four [9,10,11,12] studies, or medical record review in the remaining two studies [8, 13]. Identification of CKD was mainly based on eGFR and ICD-X codes. The number of prognostic factors included in each model varied between 4 and 11 and included age, sex, SBP, creatinine, and diabetes duration as common predictor variables. These models had fair to good calibration, and discrimination C-statistics ranged between 0.713 [10] and 0.920 [12].
NHES population characteristics
The T2D cohort included 3,416 participants with a median diabetes duration and follow up time of 5.7 (IQR: 2.6–10.1) and 9.9 (IQR: 6.8–12.7) years, respectively, see Table 1. Of these, 1383 and 186 participants developed CKD and ESRD with an incidence (95% CI) of 43.9% (42.2%, 45.7%) and 5.9% (5.1%, 6.8%), respectively; 704 (22.3%) and 495 (14.5%) developed CVD and retinopathy, and 420 (12.3%) died from any cause.
Baseline characteristics of T2D patients are described in Table 2. The mean (SD) age was 56.6 (12.4) years, and 60.2% were female. The mean age at diabetes onset was 60.0 (12.3) years, and 26.5% of patients had a first degree relative with diabetes. Mean BMI was 26.4 (4.7) kg/m2, and the presence of hypertension and dyslipidaemia was 52.3% and 85.2%, respectively.
A total of 1,222 (35.8%), 280 (8.2%), and 1,188 (34.8%) participants were undergoing treatment for diabetes, including oral diabetic medications, insulin, or diet-control, respectively. In general, all prognostic factors including demographics, socioeconomic status, clinical features, biomarkers, treatments, and complications demonstrated significant differences between CKD stages 3–5 (Table 2).
Participant characteristics comparisons
Participants in our study were slightly younger with fewer males (39.8% vs 43.7%–56.2%) compared to the other six CKD-ESRD studies (Table S5). Mean diabetes duration, BMI, serum creatinine, eGFR and SBP-DBP for our cohort fell within the range reported across the various models but the prevalence of dyslipidemia and hypertension was much higher among our participants. Our cohort had lower FPG and HDL-C, but higher lipid levels (i.e., LDL-C, TG and TC). Moreover, the percentages of anti-hypertensive, anti-hyperlipidaemic and oral diabetic medications were lower than for other reported models.
CKD incidence in our study was similar to that reported by Low and colleagues [8] (i.e., 43.9 vs 42.9%), but much higher than that reported in the remaining studies[9, 10], which ranged from 0.7 to 12.3%. The incidence of ESRD in the study by Lin et al. [12] was comparable to our study (5.04% vs 5.90%), but much higher than the other two studies that reported it [11, 13] (0.4% and 2.5%), see Table 1. The coefficients for the associations between prognostic factors and CKD/ESRD in our cohort were estimated and compared to those in the original models, see Table S6. Our coefficients were mostly similar to the model proposed by Low and colleagues [8], but several predictors (i.e., sex, BMI, location, HDL-C, presence of hypertension, and/or dyslipidemia) were not significant compared to the models proposed by Miao et al. [9]. Most predictors in Wu’s model were also significant in our data; however, the effect sizes were lower for SBP, and diabetes duration and the direction of effect was reversed for BMI. Comparison of the corresponding rank odds ratio of predictors included in their respective CKD models identified creatinine (β = 4.653) and retinopathy (β = 1.045) with the strongest effects for females in Miao’s model, whereas SBP (β = 0.902) and diabetes duration (β = 0.891) were highly associated with CKD in Wu’s models, respectively (Table S6).
For modelling ESRD, only three of the 10 predictors were significant in Elley’s [11] equations, including creatinine, diabetes duration and microalbuminuria, whereas in Wan’s [13] models for female participants, insulin use, oral diabetic drug, and SBP were significantly correlated with ESRD in our multivariate analyses (Table S6).
External validation
External validations were performed for models M1 to M6 where applicable (Table S7). Results of CKD-ESRD models are summarised in Table 3. At baseline (M0), all prognostic models showed fair calibration, but discrimination varied from poor to moderate, i.e., 0.585 to 0.707 and 0.671 to 0.760 for CKD and ESRD, respectively (Fig. 2). Sex-specific specific CKD and ESRD models performed better for females. For CKD, Miao’s model for females generated a C-statistic of 0.786 (0.765–0.806) compared to 0.720 (0.691–0.749) for males, see Table 3.
All CKD-ESRD models provided improved C-statistics following additional adjustments of the regression coefficient (M3) and updated models from (M4 – M6), see Figure S2. We updated CKD models by adding biomarkers (i.e., FPG groups < 126 vs ≥ 126 mg/dL) and/or interaction effects with oral diabetic drug use; the greatest improvement was observed in the model by Wu and colleagues with a C-statistic of 0.790 (0.774 – 0.806), see Table 3.
In the baseline validation, most CKD models were well-calibrated in our population with O/E ranging from 0.999 (0.975 –1.024) to 1.009 (0.929 – 1.090). Model calibration remained similar after updating, although Miao’s model for males and females showed a slight overestimation of 1.052 (0.868 – 1.235), and 1.036 (0.917 – 1.156), respectively.
Four ESRD risk scores showed moderate to good calibration for baseline validation, recalibration, and updated models, see Figure S3 and Table 3. Fitting the equations using our validation set of ESRD equations (M5) showed worsening shrinkage, with a penalty of 12.31% and 15.55% for Lin’s and Wan’s male models, respectively.
The Brier score is another measure of prediction accuracy, ranging between 0 and 1, where lower scores indicate better accuracy. The Brier scores for the baseline and updated models are presented in Table 3. In the updated CKD model, the lowest Brier score was observed in Miao’s model for females (0.162), Low’s model (0.168), Miao’s model for males (0.178), and Wu’s model (0.185). Of the four ESRD models, the Brier score for the updated models (M4) was superior and ranged from 0.043 to 0.061.
Table S8 provides a summary of the model improvements implemented following baseline validation. New additional predictor variables (i.e., glucose level and/or interaction with oral diabetic medication) significantly improved the discrimination for the CKD models. The highest improvement was observed in Wu’s models with ∆C-statistic of 0.214 (0.193 – 0.234). Most ESRD models showed minor significant discrimination improvements in the updated models.
Discussion
We externally evaluated, validated, compared and updated six previously published models for predicting CKD/ESRD in a nationwide cohort of Thai participants with T2D, in line with recent framework guidelines [18, 19, 31, 38]. At baseline, most models provided only modest discrimination of T2D patients who developed CKD/ESRD. Two [10, 12] models demonstrated similar performance to their parent models. All models showed good calibration and upon modification, the agreement between observed and expected risk was fair, with only a few models showing slight overestimation.
In this study, the associations observed between prognostic factors and CKD/ESRD risk in Thai participants with T2D differed from previous studies. For instance, either hypertension or dyslipidaemia, LDL-C, and BMI were negatively associated with CKD risk in some models [8,9,10], with only a few predictors (i.e., diabetic duration, creatinine, and oral diabetic medications) significantly correlated with ESRD risk. We suspect that the lack of associations or variation in the direction of effect observed between previously reported predictor outcomes may have resulted from heterogeneity among the predictors and outcomes in our data, and that used previously for the development sets. However, we were unable to include two important biomarker predictor variables for four [8, 11,12,13] models (i.e., UACR and HbA1c) as they were unavailable in our data.
We postulate that the magnitude of the C-statistics and miscalibration observed may be explained by case-mix effects represented by the number of events, predictor effects, and heterogeneity in the population characteristics [19, 44, 45]. Variation of the included predictor variables, and sample size characteristics between derivation and validation settings, are likely responsible for the modest model performance in our population [19, 46].
In general, discrimination and calibration improved in our updated models. Although most models demonstrated lower discrimination in our data compared to their original settings, our updated models showed consistent improvement for all evaluation metrics (i.e., Brier score, shrinkage factor, penalty regression, and C-statistics). Most CKD-ESRD models also showed better reclassification (i.e., ∆C-statistic) for the enhanced models. Despite a lack of existing standards, Pencina et al. proposed that ∆C-statistics greater than 0.01 represents a relevant improvement in model prediction [47, 48]. For our data, all models showed significant improvement on modification, with ∆C-statistics ranging between 0.041 and 0.214 for CKD and 0.025 to 0.089 for ESRD equations.
The Brier score has been proposed as a measure of discrimination and calibration for model validation [49]. In this study, ESRD models performed better compared to those for CKD as determined by Brier scores. Almost every validation and updated model showed improved predictions (as judged by a Benchmark value less than 0.25) [40].
In our updated models, four proved more effective either for the prediction of CKD [8, 9] or ESRD[11, 13] in our population, without the need for recalibration or updated equations. These models consistently exceeded all others in terms of calibration and discrimination, and were more comparable to the derived models. Only Elley’s model [11] provided a web calculator (http://www.nzssd.org.nz/cvd_renal/) to facilitate easier routine clinical practice use.
The strengths of our study include the long-term follow-up of diabetic progression in 26,170 individuals over 20 years, the definition of CKD from multiple data sources, and the evaluation of previously published prognostic models identified from a current SR/MA. This study was based on real world data from a clinical setting that used a broad range of routinely captured potential predictor variables evaluated for prognostic performance of renal outcomes in those with incident diabetes. To our knowledge, this is the first independent validation of CKD-ESRD prognostic models in an Asian population using real world data, beyond the populations from which the models originated. Therefore, our findings should be useful in predicting CKD-ESRD occurrence in other Asian regions where their settings are similar to Thailand.
Our study highlighted that eGFR assessment using creatinine was beneficial to kidney disease surveillance in a Thai population. By avoiding specific race/ethnicity coefficients, our updated models still offered accurate prognostic estimates which could be enhanced further through improved clinical and laboratory standards [30, 50].
Our study has several limitations. Markers of kidney damage, such as albuminuria and cystatin-C were not available in our data and missing data for some predictor variables precluded prognostic risk estimates for some models.
Conclusions
In conclusion, we have provided an independent external validation of prognostic models for the prediction of incident CKD/ESRD in participants with T2D from Thailand. All evaluated prognostic models showed only moderate discriminative performance, but fair calibration at baseline validation. Updated prognostic scores improved predictive performance in most of the evaluation metrics (i.e., discrimination, calibration, and Brier score). An updated prognostic model for clinical use in Asian populations is provided.
Although no model was excellent, prognostic equations not delimited by sex (i.e., Low’s [8] and Elley’s [11]) performed better in our data and may offer clinical utility as a CKD screening tool in primary care for patients with diabetes.
References
George C, Mogueo A, Okpechi I, Echouffo-Tcheugui JB, Kengne AP (2017) Chronic kidney disease in low-income to middle-income countries: the case for increased screening. BMJ Glob Health 2(2):e000256. https://doi.org/10.1136/bmjgh-2016-000256
Hasan M, Sutradhar I, Gupta RD, Sarker M (2018) Prevalence of chronic kidney disease in South Asia: a systematic review. BMC Nephrol 19(1):291. https://doi.org/10.1186/s12882-018-1072-5
Jager KJ, Kovesdy C, Langham R, Rosenberg M, Jha V, Zoccali C (2019) A single number for advocacy and communication-worldwide more than 850 million individuals have kidney diseases. Kidney Int 96(5):1048–1050. https://doi.org/10.1016/j.kint.2019.07.012
International Diabetes Federation (2019) IDF Diabetes Atlas, 9th edn. International Diabetes Federation, Brussels, Belgium
Global Burden of Disease Collaborators (2017) Global, regional, and national age-sex specific mortality for 264 causes of death, 1980–2016: a systematic analysis for the Global burden of disease study 2016. Lancet 390(10100):1151–1210. https://doi.org/10.1016/s0140-6736(17)32152-9
Basu S, Sussman JB, Berkowitz SA, Hayward RA, Bertoni AG, Correa A, Mwasongwe S, Yudkin JS (2018) Validation of risk equations for complications of type 2 diabetes (RECODe) using individual participant data from diverse longitudinal cohorts in the U.S. Diabetes Care 41(3):586–595. https://doi.org/10.2337/dc17-2002
Alba AC, Agoritsas T, Walsh M, Hanna S, Iorio A, Devereaux PJ, McGinn T, Guyatt G (2017) Discrimination and calibration of clinical prediction models: users’ guides to the medical literature. JAMA 318(14):1377–1384. https://doi.org/10.1001/jama.2017.12126
Low S, Lim SC, Zhang X, Zhou S, Yeoh LY, Liu YL, Tavintharan S, Sum CF (2017) Development and validation of a predictive model for chronic kidney disease progression in type 2 diabetes mellitus based on a 13-year study in Singapore. Diabetes Res Clin Pract 123:49–54. https://doi.org/10.1016/j.diabres.2016.11.008
Miao DD, Pan EC, Zhang Q, Sun ZM, Qin Y, Wu M (2017) Development and validation of a model for predicting diabetic nephropathy in Chinese people. Biomed Environ Sci 30(2):106–112. https://doi.org/10.3967/bes2017.014
Wu M, Lu J, Zhang L, Liu F, Chen S, Han Y, Zhao F, Guo K, Bao Y, Chen H, Jia W (2017) A non-laboratory-based risk score for predicting diabetic kidney disease in Chinese patients with type 2 diabetes. Oncotarget 8(60):102550–102558. https://doi.org/10.18632/oncotarget.21684
Elley CR, Robinson T, Moyes SA, Kenealy T, Collins J, Robinson E, Orr-Walker B, Drury PL (2013) Derivation and validation of a renal risk score for people with type 2 diabetes. Diabetes Care 36(10):3113–3120. https://doi.org/10.2337/dc13-0190
Lin CC, Li CI, Liu CS, Lin WY, Lin CH, Yang SY, Li TC (2017) Development and validation of a risk prediction model for end-stage renal disease in patients with type 2 diabetes. Sci Rep 7(1):10177. https://doi.org/10.1038/s41598-017-09243-9
Wan EYF, Fong DYT, Fung CSC, Yu EYT, Chin WY, Chan AKC, Lam CLK (2017) Prediction of new onset of end stage renal disease in Chinese patients with type 2 diabetes mellitus - a population-based retrospective cohort study. BMC Nephrol 18(1):257. https://doi.org/10.1186/s12882-017-0671-x
Hu Y, Shi R, Mo R, Hu F (2020) Nomogram for the prediction of diabetic nephropathy risk among patients with type 2 diabetes mellitus based on a questionnaire and biochemical indicators: a retrospective study. Aging (Albany NY) 12(11):10317–10336. https://doi.org/10.18632/aging.103259
Jiang S, Fang J, Yu T, Liu L, Zou G, Gao H, Zhuo L, Li W (2020) Novel model predicts diabetic nephropathy in type 2 diabetes. Am J Nephrol 51(2):130–138. https://doi.org/10.1159/000505145
Mogueo A, Echouffo-Tcheugui JB, Matsha TE, Erasmus RT, Kengne AP (2015) Validation of two prediction models of undiagnosed chronic kidney disease in mixed-ancestry South Africans. BMC Nephrol 16:94. https://doi.org/10.1186/s12882-015-0093-6
Fraccaro P, van der Veer S, Brown B, Prosperi M, O’Donoghue D, Collins GS, Buchan I, Peek N (2016) An external validation of models to predict the onset of chronic kidney disease using population-based electronic health records from Salford, UK. BMC Med 14:104. https://doi.org/10.1186/s12916-016-0650-2
Moons KG, Kengne AP, Grobbee DE, Royston P, Vergouwe Y, Altman DG, Woodward M (2012) Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98(9):691–698. https://doi.org/10.1136/heartjnl-2011-301247
Debray TP, Vergouwe Y, Koffijberg H, Nieboer D, Steyerberg EW, Moons KG (2015) A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 68(3):279–289. https://doi.org/10.1016/j.jclinepi.2014.06.018
Collins GS, Omar O, Shanyinde M, Yu LM (2013) A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. J Clin Epidemiol 66(3):268–277. https://doi.org/10.1016/j.jclinepi.2012.06.020
Yang XL, So WY, Kong AP, Clarke P, Ho CS, Lam CW, Ng MH, Lyu RR, Yin DD, Chow CC, Cockram CS, Tong PC, Chan JC (2006) End-stage renal disease risk equations for Hong Kong Chinese patients with type 2 diabetes: Hong Kong diabetes registry. Diabetologia 49(10):2299–2308. https://doi.org/10.1007/s00125-006-0376-3
Jardine MJ, Hata J, Woodward M, Perkovic V, Ninomiya T, Arima H, Zoungas S, Cass A, Patel A, Marre M, Mancia G, Mogensen CE, Poulter N, Chalmers J (2012) Prediction of kidney-related outcomes in patients with type 2 diabetes. Am J Kidney Dis 60(5):770–778. https://doi.org/10.1053/j.ajkd.2012.04.025
Wysham CH, Gauthier-Loiselle M, Bailey RA, Manceur AM, Lefebvre P, Greenberg M, Duh MS, Young JB (2020) Development of risk models for major adverse chronic renal outcomes among patients with type 2 diabetes mellitus using insurance claims: a retrospective observational study. Curr Med Res Opin 36(2):219–227. https://doi.org/10.1080/03007995.2019.1682981
Dunkler D, Gao P, Lee SF, Heinze G, Clase CM, Tobe S, Teo KK, Gerstein H, Mann JF, Oberbauer R (2015) Risk prediction for early CKD in type 2 diabetes. Clin J Am Soc Nephrol 10(8):1371–1379. https://doi.org/10.2215/cjn.10321014
Nowak N, Skupien J, Smiles AM, Yamanouchi M, Niewczas MA, Galecki AT, Duffin KL, Breyer MD, Pullen N, Bonventre JV, Krolewski AS (2018) Markers of early progressive renal decline in type 2 diabetes suggest different implications for etiological studies and prognostic tests development. Kidney Int 93(5):1198–1206. https://doi.org/10.1016/j.kint.2017.11.024
Peters KE, Davis WA, Ito J, Winfield K, Stoll T, Bringans SD, Lipscombe RJ, Davis TME (2017) Identification of novel circulating biomarkers predicting rapid decline in renal function in type 2 diabetes: the fremantle diabetes study phase II. Diabetes Care 40(11):1548–1555. https://doi.org/10.2337/dc17-0911
Basu S, Sussman JB, Berkowitz SA, Hayward RA, Yudkin JS (2017) Development and validation of risk equations for complications of type 2 diabetes (RECODe) using individual participant data from randomised trials. Lancet Diabetes Endocrinol 5(10):788–798. https://doi.org/10.1016/s2213-8587(17)30221-8
Keane WF, Zhang Z, Lyle PA, Cooper ME, de Zeeuw D, Grunfeld JP, Lash JP, McGill JB, Mitch WE, Remuzzi G, Shahinfar S, Snapinn SM, Toto R, Brenner BM (2006) Risk scores for predicting outcomes in patients with type 2 diabetes and nephropathy: the RENAAL study. Clin J Am Soc Nephrol 1(4):761–767. https://doi.org/10.2215/cjn.01381005
Liao LN, Li TC, Li CI, Liu CS, Lin WY, Lin CH, Yang CW, Chen CC, Chang CT, Yang YF, Liu YL, Kuo HL, Tsai FJ, Lin CC (2019) Genetic risk score for risk prediction of diabetic nephropathy in Han Chinese type 2 diabetes patients. Sci Rep 9(1):19897. https://doi.org/10.1038/s41598-019-56400-3
Delgado C, Baweja M, Burrows NR, Crews DC, Eneanya ND, Gadegbeku CA, Inker LA, Mendu ML, Miller WG, Moxey-Mims MM, Roberts GV, St Peter WL, Warfield C, Powe NR (2021) Reassessing the inclusion of race in diagnosing kidney diseases: an interim report from the NKF-ASN task force. Am J Kidney Dis. https://doi.org/10.1053/j.ajkd.2021.03.008
Collins GS, Reitsma JB, Altman DG, Moons KG (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. J Clin Epidemiol 68(2):134–143. https://doi.org/10.1016/j.jclinepi.2014.11.010
Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, Vickers AJ, Ransohoff DF, Collins GS (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med 162(1):W1-73. https://doi.org/10.7326/M14-0698
Tantivess S, Yothasamut J, Saengsri W (2019) Utilisation of evidence from Thailand’s national health examination survey in policy development: finding the weakest link. Health Res Policy Syst 17(1):104. https://doi.org/10.1186/s12961-019-0512-4
Aekplakorn W, Chariyalertsak S, Kessomboon P, Assanangkornchai S, Taneepanichskul S, Putwatana P (2018) Prevalence of diabetes and relationship with socioeconomic status in the Thai population: national health examination survey, 2004–2014. J Diabetes Res 2018:1654530. https://doi.org/10.1155/2018/1654530
Nguyen Ngoc H, Kriengsinyos W, Rojroongwasinkul N, Aekplakorn W (2019) Association of adiposity indices with hypertension in middle-aged and elderly Thai population: national health examination survey 2009 (NHES-IV). J Cardiovasc Dev Dis. https://doi.org/10.3390/jcdd6010013
Lee CH, Shih AZL, Woo YC, Fong CHY, Yuen MMA, Chow WS, Lam KSL (2017) Which creatinine-based estimated glomerular filtration rate equation best predicts all-cause mortality in Chinese subjects with type 2 diabetes? Diabetes Res Clin Pract 126:25–29. https://doi.org/10.1016/j.diabres.2017.01.010
Tomeleri CM, Ronque ER, Silva DR, Cardoso Júnior CG, Fernandes RA, Teixeira DC, Barbosa DS, Venturini D, Okino AM, Oliveira JA, Cyrino ES (2015) Prevalence of dyslipidemia in adolescents: comparison between definitions. Rev Port Cardiol 34(2):103–109. https://doi.org/10.1016/j.repc.2014.08.020
Moons KG, Kengne AP, Woodward M, Royston P, Vergouwe Y, Altman DG, Grobbee DE (2012) Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 98(9):683–690. https://doi.org/10.1136/heartjnl-2011-301246
Janssen KJ, Moons KG, Kalkman CJ, Grobbee DE, Vergouwe Y (2008) Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61(1):76–86. https://doi.org/10.1016/j.jclinepi.2007.04.018
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21(1):128–138. https://doi.org/10.1097/EDE.0b013e3181c30fb2
Cook NR (2007) Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 115(7):928–935. https://doi.org/10.1161/circulationaha.106.672402
Steyerberg EW, Borsboom GJ, van Houwelingen HC, Eijkemans MJ, Habbema JD (2004) Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat Med 23(16):2567–2586. https://doi.org/10.1002/sim.1844
Riley RD, Ensor J, Snell KIE, Harrell FE Jr, Martin GP, Reitsma JB, Moons KGM, Collins G, van Smeden M (2020) Calculating the sample size required for developing a clinical prediction model. BMJ 368:m441. https://doi.org/10.1136/bmj.m441
Vergouwe Y, Moons KG, Steyerberg EW (2010) External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 172(8):971–980. https://doi.org/10.1093/aje/kwq223
Candela-Toha A, Elías-Martín E, Abraira V, Tenorio MT, Parise D, de Pablo A, Centella T, Liaño F (2008) Predicting acute renal failure after cardiac surgery: external validation of two new clinical scores. Clin J Am Soc Nephrol 3(5):1260–1265. https://doi.org/10.2215/cjn.00560208
Nieboer D, van der Ploeg T, Steyerberg EW (2016) Assessing discriminative performance at external validation of clinical prediction models. PLoS One 11(2):e0148820. https://doi.org/10.1371/journal.pone.0148820
Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27(2):157–172. https://doi.org/10.1002/sim.2929 (discussion 207-112)
Pencina MJ, D’Agostino RB, Pencina KM, Janssens AC, Greenland P (2012) Interpreting incremental value of markers added to risk prediction models. Am J Epidemiol 176(6):473–481. https://doi.org/10.1093/aje/kws207
Hilden J, Gerds TA (2014) A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med 33(19):3405–3414. https://doi.org/10.1002/sim.5804
Pottel H, Björk J, Courbebaisse M, Couzi L, Ebert N, Eriksen BO, Dalton RN, Dubourg L, Gaillard F, Garrouste C, Grubb A, Jacquemont L, Hansson M, Kamar N, Lamb EJ, Legendre C, Littmann K, Mariat C, Melsom T, Rostaing L, Rule AD, Schaeffner E, Sundin PO, Turner S, Bökenkamp A, Berg U, Åsling-Monemi K, Selistre L, Åkesson A, Larsson A, Nyman U, Delanaye P (2021) Development and validation of a modified full age spectrum creatinine-based equation to estimate glomerular filtration rate : a cross-sectional analysis of pooled data. Ann Intern Med 174(2):183–191. https://doi.org/10.7326/m20-4366
Acknowledgements
This study was a part of a Ph.D. in the Data Science for Healthcare Program for the lead author (Sigit Ari Saputro) under the Department of Clinical Epidemiology and Biostatistics, Faculty of Medicine Ramathibodi Hospital, Mahidol University, Thailand. The authors would like to thank Nattakrit Tongpoonsakdi and Stephen John Pinder for proofreading and conducting a comprehensive English language review.
Funding
This research did not receive any specific grant funding from the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Contributions
SAS, OP, and AT conceptualized the study design and take responsibility for the entire project. OP, AP, WA and AT prepared the data. SAS and AP performed data cleaning, aggregation and analysis. SAS drafted the manuscript with input from all authors. OP, AP, WA and AT edited and revised the manuscript. GM and JA edited and revised the manuscript for significant intellectual issues. All authors have read and approved the final paper.
Corresponding authors
Ethics declarations
Conflict of interest
The authors have no conflicts of interest.
Ethical approval
We explain in a protocol which already approved by the Human Research Ethics Committee of Ramathibodi Hospital (Number: #MURA2019/35).
Informed consent
We use NHES as secondary data which they have individual consent from each participant and we explain in a protocol which already approved by the Human Research Ethics Committee of Ramathibodi Hospital (Number: #MURA2019/35).
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Saputro, S.A., Pattanateepapon, A., Pattanaprateep, O. et al. External validation of prognostic models for chronic kidney disease among type 2 diabetes. J Nephrol 35, 1637–1653 (2022). https://doi.org/10.1007/s40620-021-01220-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40620-021-01220-w