Skip to main content

Prognostic accuracy for predicting ordinal competing risk outcomes using ROC surfaces

Lifetime Data Analysis volume 28, pages 1–22 (2022)Cite this article

Abstract

Many medical conditions are marked by a sequence of events in association with continuous changes in biomarkers. Few works have evaluated the overall accuracy of a biomarker in predicting disease progression. We thus extend the concept of receiver operating characteristic (ROC) surface and the volume under the surface (VUS) from multi-category outcomes to ordinal competing-risk outcomes that are also subject to noninformative censoring. Two VUS estimators are considered. One is based on the definition of the ROC surface and obtained by integrating the estimated ROC surface. The other is an inverse probability weighted U estimator that is built upon the equivalence of the VUS to the concordance probability between the marker and sequential outcomes. Both estimators have nice asymptotic results that can be derived using counting process techniques and U-statistics theory. We illustrate their good practical performances through simulations and applications to two studies of cognition and a transplant dataset.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3

References

  • Andersen PK, Borgan O, Gill RD, Keiding N (1993) Statistical models based on counting processes. Springer, New York

    Book  Google Scholar 

  • Blanche P, Dartigues JF, Jacqmin-Gadda H (2013) Estimating and comparing time-dependent areas under receiver operating characteristic curves for censored event times with competing risks. Stat Med 32(30):5381–5397

    MathSciNet  Article  Google Scholar 

  • Cheng Y, Fine JP, Kosorok MR (2007) Nonparametric analysis of bivariate competing risks data. J Am Stat Assoc 102:1407–1416

    MathSciNet  Article  Google Scholar 

  • Cox DR (1959) The analysis of exponentially distributed life-times with two types of failure. J Roy Stat Soc B 21:411–421

    MathSciNet  MATH  Google Scholar 

  • Dabrowska DM (1989) Kaplan-Meier estimate on the plane: Weak convergence, LIL, and the bootstrap. J Multivar Anal 29:308–325

    MathSciNet  Article  Google Scholar 

  • Dreiseitl S, L O, Binder M, (2000) Comparing three-class diagnostic tests by three-way ROC analysis. Med Decis Making 20(3):323–331

  • Fine JP, Gray RJ (1999) A proportional hazards model for the subdistribution of a competing risk. J Am Stat Assoc 94:496–509

    MathSciNet  Article  Google Scholar 

  • Gooley TA, Leisenring W, Crowley J, Storer BE (1999) Estimation of failure probabilities in the presence of competing risks: new representations of old estimators. Stat Med 18:695–706

    Article  Google Scholar 

  • Heagerty P, Lumley T, Pepe MS (2000) Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics 56:337–344

    Article  Google Scholar 

  • Hung H, Chiang CT (2010) Estimation methods for time-dependent auc models with survival data. Can J Stat 38(1):8–26

    MathSciNet  MATH  Google Scholar 

  • Kalbfleisch JD, Prentice RL (2002) The statistical analysis of failure time data, 2nd edition. Wiley, New York

  • Kaplan EL, Meier P (1958) Nonparametric estimation from incomplete observations. J Am Stat Assoc 53:457–481

    MathSciNet  Article  Google Scholar 

  • Van der Laan MJ, Robins JM (2003) Unified methods for censored longitudinal data and causality. Springer Science & Business Media, Berlin

  • Li J, Fine JP (2008) ROC analysis with multiple classes and multiple tests: methodology and its application in microarray studies. Biostatistics 9(3):566–576

    Article  Google Scholar 

  • Li J, Zhou XH (2009) Nonparametric and semiparametric estimation of the three way receiver operating characteristic surface. J Stat Plann Inference 139(12):4133–4142

    MathSciNet  Article  Google Scholar 

  • Mossman D (1999) Three-way ROCs. Med Decis Making 19(1):78–89

    Article  Google Scholar 

  • Obuchowski NA (2005) Estimating and comparing diagnostic tests’ accuracy when the gold standard is not binary. Acad Radiol 12(9):1198–1204

  • Prentice RL, Kalbfleisch JD, Peterson AV, Flournoy N, Farewell VT, Breslow NE (1978) The analysis of failure time data in the presence of competing risks. Biometrics 12:737–751

    MATH  Google Scholar 

  • Saha P, Heagerty PJ (2010) Time-dependent predictive accuracy in the presence of competing risks. Biometrics 66:999–1011

    MathSciNet  Article  Google Scholar 

  • Scheike TH, Zhang MJ, Gerds TA (2008) Predicting cumulative incidence probability by direct binomial regression. Biometrika 95(1):205–220

    MathSciNet  Article  Google Scholar 

  • Shi H, Cheng Y, Li J (2014) Assessing diagnostic accuracy improvement for survival or competing-risk censored outcomes. Can J Stat 42(1):109–125

    MathSciNet  Article  Google Scholar 

  • Van Der Vaart A (1998) Asymptotic statistics. Cambridge Series in Statistical and Probabilistic Mathematics, Cambridge University Press

  • Wang H, Cheng Y (2014) Piecewise cause-specific association analyses of multivariate untied or tied competing risks data. Int J Biostat 10:197–220

    MathSciNet  Article  Google Scholar 

  • Wolbers M, Blanche P, Koller MT, Witteman JCM, Gerds TA (2014) Concordance for prognostic models with competing risks. Biostatistics 15(3):526–539

    Article  Google Scholar 

  • Wu Y, Chiang C (2013) Optimal receiver operating characteristic manifolds. J Math Psychol 57(5):237–248

    MathSciNet  Article  Google Scholar 

  • Zheng Y, Cai T, Jin Y, Feng Z (2012) Evaluating prognostic accuracy of biomarkers under competing risk. Biometrics 68:388–396

    MathSciNet  Article  Google Scholar 

Download references

Acknowledgements

The authors thank Drs. Mary Ganguli and Joyce Chang for providing the MYHAT data. Cheng was partially supported by the U.S. National Science Foundation (DMS 1916001). This research was supported in part by the University of Pittsburgh Center for Research Computing through the resources provided.

Author information

Authors and Affiliations

  1. Department of Biostatistics, University of Pittsburgh, Pennsylvania, PA, 15260, USA

    Song Zhang & Abdus S. Wahed

  2. Department of Statistics, University of Pittsburgh, Pennsylvania, PA, 15260, USA

    Yang Qu

  3. Departments of Statistics and Biostatistics, University of Pittsburgh, Pennsylvania, PA, 15260, USA

    Yu Cheng

  4. Departments of Neurology, Psychiatry, and Clinical and Translational Sciences, University of Pittsburgh, Pennsylvania, PA, 15260, USA

    Oscar L. Lopez

Authors
  1. Song Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Oscar L. Lopez
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Abdus S. Wahed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yu Cheng.

Ethics declarations

Conflict of interest

The authors declare that they have no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary Materials:

The reader is referred to the on-line Supplementary Materials for detailed proofs, results from additional simulation studies, and some summary statistics for the MYHAT, ADRC and PBC data. (pdf 117KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Qu, Y., Cheng, Y. et al. Prognostic accuracy for predicting ordinal competing risk outcomes using ROC surfaces. Lifetime Data Anal 28, 1–22 (2022). https://doi.org/10.1007/s10985-021-09539-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10985-021-09539-z

Keywords

  • Concordance probability
  • Correct classification probability
  • Discriminative capability
  • Disease progression
  • Inverse probability of censoring weighting