A comparative study of methods for testing the equality of two or more ROC curves

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

The problem of comparing the accuracy of diagnostic tests is usually carried out through the comparison of the corresponding receiver operating characteristic (ROC) curves. This matter has been approached from different perspectives. Usually, ROC curves are compared through their respective areas under the curve, but in cases where there is no uniform dominance between the involved curves other procedures are preferred. Although the asymptotic distributions of the statistics behind these methods are, in general, known, resampling plans are considered. With the purpose of comparing the performance of different approaches, with different ways of calibrating the distribution of the tests, a simulation study is carried out in order to investigate the statistical power and the nominal level of each methodology.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

References

  1. Antoch J, Prchal L, Sarda P (2010) Nonparametric comparison of ROC curves: testing equivalence. In: Nonparametrics and robustness in modern statistical inference and time series analysis: a Festschrift in honor of Professor Jana Jurečková, Institute of Mathematics and Statistics Collection, vol 7. Institute of Mathematics and Statistics, Beachwood, pp 12–24

  2. Bamber D (1975) The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol 12(4):387–415

    MathSciNet  Article  MATH  Google Scholar 

  3. Bandos AI, Rockette HE, Gur D (2005) A permutation test sensitive to differences in areas for comparing ROC curves from a paired design. Stat Med 24(18):2873–2893

    MathSciNet  Article  Google Scholar 

  4. Braun TM, Alonzo TA (2008) A modified sign test for comparing paired roc curves. Biostatistics 9(2):364–372

    Article  MATH  Google Scholar 

  5. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3):837–845

  6. Gonçalves L, Subtil A, Oliveira MR, Bermudez P (2014) Roc curve estimation: an overview. REVSTAT 12(1):1–20

    MathSciNet  MATH  Google Scholar 

  7. Good P (2013) Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer, Berlin

    Google Scholar 

  8. Green D, Swets J (1966) Signal detection theory and psychophysics. Peninsula, Los Altos

    Google Scholar 

  9. Hanley JA, McNeil BJ (1983) A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148(3):839–843

    Article  Google Scholar 

  10. Hsieh F, Turnbull BW (1996) Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat 24(1):25–40

    MathSciNet  Article  MATH  Google Scholar 

  11. Jokiel-Rokita A, Pulit M (2013) Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions. Stat Comput 23(6):703–712

    MathSciNet  Article  MATH  Google Scholar 

  12. Krzanowski WJ, Hand DJ (2009) ROC curves for continuous data. CRC Press, Boca Raton

    Google Scholar 

  13. Lloyd CJ (1998) Using smoothed receiver operating characteristic curves to summarize and compare diagnostic systems. J Am Stat Assoc 93(444):1356–1364

    Article  Google Scholar 

  14. Lloyd CJ, Yong Z (1999) Kernel estimators of the ROC curve are better than empirical. Stat Probab Lett 44(3):221–228

    MathSciNet  Article  MATH  Google Scholar 

  15. Martínez-Camblor P (2007) Comparación de pruebas diagnósticas desde la curva ROC. Rev Colomb Estad 30(2):163–176

    MathSciNet  Google Scholar 

  16. Martínez-Camblor P, Corral N (2012) A general bootstrap algorithm for hypothesis testing. J Stat Plan Inference 142(2):589–600

    MathSciNet  Article  MATH  Google Scholar 

  17. Martínez-Camblor P, Carleos C, Corral N (2011) Powerful nonparametric statistics to compare \(k\) independent roc curves. J Appl Stat 38(7):1317–1332

    MathSciNet  Article  MATH  Google Scholar 

  18. Martínez-Camblor P, Carleos C, Corral N (2013) General nonparametric ROC curve comparison. J Korean Stat Soc 42(1):71–81

    MathSciNet  Article  MATH  Google Scholar 

  19. Pepe MS (2003) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, Cary

    Google Scholar 

  20. Pérez-Fernández S (2017) nsROC: non-standard ROC curve analysis. R package version 1.0. https://CRAN.R-project.org/package=nsROC

  21. R Core Team (2015) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. https://www.R-project.org/

  22. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Mller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77

    Article  Google Scholar 

  23. Sheather SJ, Jones MC (1991) A reliable data-based bandwidth selection method for kernel density estimation. J R Stat Soc Ser B Methodol 53(3):683–690

  24. Venkatraman E (2000) A permutation test to compare receiver operating characteristic curves. Biometrics 56(4):1134–1138

    MathSciNet  Article  MATH  Google Scholar 

  25. Venkatraman E, Begg CB (1996) A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 83(4):835–848

    MathSciNet  Article  MATH  Google Scholar 

  26. Wieand S, Gail MH, James BR, James KL (1989) A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76(3):585–592

    MathSciNet  Article  MATH  Google Scholar 

Download references

Acknowledgements

The authors would like to thank associate editor, the Co-Editor and the reviewers for their constructive comments and suggestions on an earlier version of this manuscript. The research of A. Fanjul-Hevia is supported by the Spanish Ministry of Education, Culture and Sport “Beca de Formación de Profesorado Universitario”; fellowship (FPU14/05316). Both authors acknowledged the support from the Spanish Ministry of Economy and Competitiveness, through grant numbers MTM2013-41383P and MTM2016-76969-P, which includes support from the European Regional Development Fund (ERDF). The research of W. González-Manteiga is also supported by the the IAP network P7/06 StUDyS of the Belgian Government (Belgian Science Policy).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Arís Fanjul-Hevia.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Fanjul-Hevia, A., González-Manteiga, W. A comparative study of methods for testing the equality of two or more ROC curves. Comput Stat 33, 357–377 (2018). https://doi.org/10.1007/s00180-017-0783-6

Download citation

Keywords

  • ROC curves
  • AUC
  • Comparison methods
  • Resampling plans