Skip to main content
SpringerLink
Log in

Predictive Accuracy of Prediction Model for Interval-Censored Data

  • Chapter
  • First Online:
Emerging Topics in Modeling Interval-Censored Survival Data

Part of the book series: ICSA Book Series in Statistics ((ICSABSS))

  • 395 Accesses

Abstract

Diverse measures for predictive accuracy have been developed for survival data which require additional investigation to reflect the time dependent outcomes. In this chapter, our purpose is to review several methods to evaluate prediction models and to compare their performance in a context of interval censored data. This chapter provides conceptual and practical explanation of statistical methods for the time-dependent ROC and C-index as the discrimination measure. Also, Brier score is dealt to evaluate overall performance for the prediction model. We aim to provide clarity of each method and identify software tools to carry out analysis in practice. We illustrate the methods using a dementia dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  • Akritas, M. G. (1994), Nearest neighbor estimation of a bivariate distribution under random censoring. Annals of Statistics, 22, 1299–1327.

    Article  MathSciNet  MATH  Google Scholar 

  • Anderson-Bergman, C. (2017). icenReg: Regression models for interval censored data in R. Journal of Statistical Software, 81(12), 1–23.

    Article  Google Scholar 

  • Asano, J., & Hirakawa, A. (2017). Assessing the prediction accuracy of a cure model for censored survival data with long-term survivors: Application to breast cancer data. Journal of Biopharmaceutical statistics, 27, 918–932.

    Article  Google Scholar 

  • Beyene, K. M., Ghouch, A. E., & Oulhaj, A. (2019). On the validity of time-dependent AUC estimation in the presence of cure fraction. Biometrical Journal, 61, 1430–1447.

    Article  MathSciNet  MATH  Google Scholar 

  • Blanche, P., Dartigues, J. F., & Jacqmin-Gadda, H. (2013a). Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biometrical Journal, 55, 687–704.

    Article  MathSciNet  MATH  Google Scholar 

  • Blanche, P., Dartigues J. F, & Jacqmin-Gadda, H. (2013b). Estimating and comparing time-dependent areas under receiver operating characteristic curves for censored event times with competing risks. Statistics in Medicine, 32, 5381–5397.

    Article  MathSciNet  Google Scholar 

  • Brown, B. W., Hollander, M., & Korwar, R. M. (1974). Nonparametric tests of independence for censored data, with applications to heart transplant studies. Reliability and Biometry, 34, 327–354.

    Google Scholar 

  • Chambless, L. E., & Diao, G. (2006). Estimation of time-dependent area under the ROC curve for long-term risk prediction. Statistics in Medicine, 25, 3474–3486.

    Article  MathSciNet  Google Scholar 

  • Díaz-Coto, S., Martínez-Camblor, S., & Corral-Blanco, N. O. (2020), Cumulative/dynamic ROC curve estimation under interval censorship. Journal of Statistical Computation and Simulation, 90, 1570–1590.

    Article  MathSciNet  MATH  Google Scholar 

  • Foucher, Y., Giral, M., Soulillou, J. P., & Daures, J. P. (2010). Time-dependent ROC analysis for a three-class prognostic with application to kidney transplantation. Statistics in Medicine, 29, 3079–3087.

    Article  MathSciNet  Google Scholar 

  • Fu, W., & Simonoff, J. S. (2017). Survival trees for interval censored survival data. Statistics in Medicine, 36(30), 4831–4842.

    Article  MathSciNet  Google Scholar 

  • Gerds, T. A., Kattan, M. W., Schumacher, M., & Yu, C. (2013). Estimating a time-dependent concordance index for survival prediction models with covariate dependent censoring. Statistics in Medicine, 32, 2173–2184.

    Article  MathSciNet  Google Scholar 

  • Gerds, T. A., & Schumacher, M. (2006). Consistent estimation of the expected Brier score in general survival models with right-censored event times. Biometrical Journal, 48, 1029–1040.

    Article  MathSciNet  MATH  Google Scholar 

  • Graf, E., Schmoor, C., Sauerbrei, W., & Schumacher, M. (1999). Assessment and comparison of prognostic classification schemes for survival data. Statistics in Medicine , 18, 2529–2545.

    Article  Google Scholar 

  • Harrell, F. E., Califf, R. M., Pryor, D. B., Lee, K. L., & Rosati, R. A. (1982). Evaluating the yield of medical tests. Journal of the American Medical Association, 247, 2543–2546.

    Article  Google Scholar 

  • Harrell, F. E., Lee, K. L., & Mark, D. B. (1996). Multivariable prognostic models: Issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Statistics in Medicine, 15, 361–387.

    Article  Google Scholar 

  • Heagerty, P. J., Lumley, T., & Pepe, M. S. (2000). Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics, 56, 337–344.

    Article  MATH  Google Scholar 

  • Heagerty, P. J., Saha-Chaudhuri, P., & Saha-Chaudhuri, M. P.(2013). survivalROC:. R package version1.0.5. https://cran.r-project.org/web/packages/survivalROC/survivalROC.pdf

  • Heagerty, P. J., Saha-Chaudhuri, P., & Saha-Chaudhuri, M. P. (2012). risksetROC:. R package version1.0.5. https://cran.r-project.org/web/packages/risksetROC/risksetROC.pdf

  • Heagerty, P. J., & Zheng, Y. (2005). Survival model predictive accuracy and ROC curves. Biometrics, 61, 92–105.

    Article  MathSciNet  MATH  Google Scholar 

  • Hung, H., & Chiang, C. T. (2010). Optimal composite markers for time-dependent receiver operating characteristic curves with censored survival data. Scandinavian Journal of Statistics, 37(4), 664–679.

    Article  MathSciNet  MATH  Google Scholar 

  • Jacqmin-Gadda, H., Blanche, P., Chary, E., Touraine, C., & Dartigues, F. G. (2016). Receiver operating characteristic curve estimation for time-to-event with semicompeting risks and interval censoring. Staticcal Methods in Medical Research, 25(6), 2750–2766.

    Article  MathSciNet  Google Scholar 

  • Kamarudin, A. N., Cox, T., & Kolamunnage-Dona, R. (2017). Time-dependent ROC curve analysis in medical research: Current methods and applications. BMC Medical Research Methodology, 17, 53.

    Article  Google Scholar 

  • Korn, E. L., & Simon, R. (1990). Measures of explained variation for survival data. Statistics in Medicine, 9, 487–503.

    Article  Google Scholar 

  • Letenneur, L., Commenges, D., Dartigues, J. F., & arberger-Gateau, P. (1994). Incidence of dementia and Alzheimer’s disease in elderly community residents of south-western France. International Journal of Epidemiology, 23(6), 1256–1261.

    Google Scholar 

  • Li, J., & Ma, S. (2011). Time-dependent ROC analysis under diverse censoring patterns. Statistics in Medicine, 30, 1266–1277.

    Article  MathSciNet  Google Scholar 

  • Lindsey, J. C., & Ryan, L. M. (1998). Tutorial in Biostatistics methods for interval-censored data. Statistics in Medicine, 17, 219–238.

    Article  Google Scholar 

  • Mogensen, U. B., Ishwaran, H., & Gerds T. A. (2012). Evaluating random forests for survival analysis using prediction error curves. Journal of Statistical Software, 50(11), 1–23.

    Article  Google Scholar 

  • Pepe, M. S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford: Oxford University Press.

    MATH  Google Scholar 

  • Potapov, S., Adler, W., & Schmid, M. (2012). survAUC: Estimators of prediction accuracy for time-to-event data. R package version 1.0-5.

    Google Scholar 

  • Proust-Lima, C., Philipps, V., & Liquet, B. (2017). Estimation of extended mixed models using latent classes and latent processes: The R package. lcmm.

    Google Scholar 

  • Rizopoulos, D. (2011). Dynamic predictions and prospective accuracy in joint models for longitudinal and time-to-event data. Biometrics, 67, 819–829.

    Article  MathSciNet  MATH  Google Scholar 

  • Tsouprou, S. (2015). Measures of Discrimination and Predictive Accuracy for Interval-Censored Data. Master Thesis, Leiden University.

    Google Scholar 

  • Turnbull, B. (1976). The empirical distribution with arbitrarily grouped and censored data. Journal of the Royal Statistical Society, B, 38(3), 290–295.

    MathSciNet  MATH  Google Scholar 

  • Uno, H., Cai, T. X., Tian, L., & Wei, L. J. (2007). Evaluating prediction rules for t-year survivors with censored regression models. Journal of the American Medical Association, 102, 527–537.

    MathSciNet  MATH  Google Scholar 

  • Uno. H., Cai, T., Pencina, M. J., D’Agostino, R. B., & Wei, L. J. (2011). On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data. Statistics in Medicine, 30, 1105–1117.

    Article  MathSciNet  Google Scholar 

  • Wang, Z., & Wang, X. (2020). Evaluating the time-dependent predictive accuracy for event-to-time outcome with a cure fraction. Pharmaceutical Statistics, 19, 955–974.

    Article  Google Scholar 

  • Wolbers, M., Blanche, P., Koller, M. T., Witteman, J. M., & Gerds, T. A. (2014). Concordance for prognostic models with competing risks. Biostatistics, 15, 526–539.

    Article  Google Scholar 

  • Wu, Y., & Cook, R. J. (2022). Assessing the accuracy of predictive models with interval-censored data. Biostatistics, 23, 18–33.

    Article  MathSciNet  Google Scholar 

  • Wu, Y., & Zhang, Y. (2012). Partially monotone tensor spline estimation of the joint distribution function with bivariate current status data. Annals of Statistics, 40, 1609–1636.

    Article  MathSciNet  MATH  Google Scholar 

  • Yao, W., Frydman, H., & Simonoff, J. (2021). An ensemble method for interval-censored time-to-event data. Biostatistics, 22, 198–213.

    Article  MathSciNet  Google Scholar 

  • Wu, Y., Wang, X., Li, J., Jia, B., & Owzar, K. (2020). Predictive accuracy of markers or risk scores for interval censored survival data. Statistics in Medicine, 39, 2437–2446.

    Article  MathSciNet  Google Scholar 

  • Zheng, Y., & Heagerty, P. J. (2007). Prospective accuracy for longitudinal markers. Biometrics, 63(2), 332–341.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This research was supported by Korea National research foundation (NRF-2020R1A2C1A01100755).

Appendix: R code

a. Calculation of AUC

We modified the intDCroc function Díaz-Coto et al. (2020) provided by implementing nonparametric approach and Li and Ma’s approach. A function intCD2roc has four components, (TL, TR, M, k), where k indicates which method is applied. In detail, k = 1 :Li and Ma approach; k = 2: weights based on nonparametric estimator and k = 3 weights based on semiparametric estimator. Function intDC2roc can be downloaded from Springer weblink.

TL left interval time

TR right interval time (TR = TL at packages intcensROC)

M marker value (ex:\(M=X{'}\hat {\beta }\) )

time a time point to obtain the AUC

delta censoring indicator (1:left, 2:interval, 3:right censoring)

LM=intCD2roc(cbind(TL,TR,M),time,1)

IM=intCD2roc(cbind(TL,TR,M),time,2)

DC=intCD2roc(cbind(TL,TR,M),time,3)

res=intcensROC(TL, TR1,M, delta,time,gridNumber = 100)

auc_LM=LM[["auc"]]

auc_IM=IM[["auc"]]

auc_DC=DC[["auc"]]

auc_SM<-intcensAUC(res)

plot(c(0,1),c(0,1), xlab="1-spec", ylab="sens", main="time")

lines(FP_IM[order(TP_IM)],sort(TP_IM), type="l", lty=1)

lines(FP_LM[order(TP_LM)],sort(TP_LM), type="l", lty=3)

lines(FP_SM[order(TP_SM)],sort(TP_SM), type="l", lty=2)

lines(FP_DC[order(TP_DC)],sort(TP_DC), type="l", lty=4)

legend("bottomright", legend=c("IM","SM","LM","CD"), lty=c(1:4)) print(c(auc_IM,auc_SM,auc_LM,auc_DC))

b. Calculation of C-index

A marker is defined as a linear predictor given a model. Function intCindex can be downloaded from Springer weblink. A function intCindex(d, k) requires the following data:

d: dataset including (TL, TR, X)

k: (=1: ph model; 2: po model)

intCindex(d,1)

intCindex(d,2)

c. Calculation of IBS

For calculation of IBS of a imputation point, the packages ipred and pec are implemented. For interval censored data, the function sbrier_IC in a package ICcforest is used. The following variables are defined

obj <- Surv(TL, TR2,type = "interval2")

pred <-icenReg::ic_sp(formula = Surv(TL,TR, type = "interval2") 1,data=d)

pred2<-icenReg::ic_sp(formula = Surv(TL,TR, type = "interval2") X1+X2+X3+X4+X5,data=d)

sbrier_IC(obj, pred, type="IBS")[1]

sbrier_IC(obj, pred2, type="IBS")[1]

TR1<-ifelse(censor==0,99999,TR)

Ctree <- ICtree(Surv(TL,TR,type="interval2") X1+X2+X3+X4+X5, data=d)

Pred5<- predict(Ctree, type="prob")

sbrier_IC(obj,Pred5, type = "IBS")

Author information

Authors and Affiliations

  1. Department of Statistics, Sookmyung Women’s University, Seoul, South Korea

    Yang-Jin Kim

Authors
  1. Yang-Jin Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang-Jin Kim .

Editor information

Editors and Affiliations

  1. Department of Statistics, University of Missouri, Columbia, MO, USA

    Jianguo Sun

  2. College of Health Solutions, Arizona State University, Goodyear, AZ, USA

    Ding-Geng Chen

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Kim, YJ. (2022). Predictive Accuracy of Prediction Model for Interval-Censored Data. In: Sun, J., Chen, DG. (eds) Emerging Topics in Modeling Interval-Censored Survival Data. ICSA Book Series in Statistics. Springer, Cham. https://doi.org/10.1007/978-3-031-12366-5_3

Download citation

Publish with us

Policies and ethics

Search

Navigation