Computational Statistics

, Volume 33, Issue 1, pp 357–377 | Cite as

A comparative study of methods for testing the equality of two or more ROC curves

  • Arís Fanjul-HeviaEmail author
  • Wenceslao González-Manteiga
Original Paper


The problem of comparing the accuracy of diagnostic tests is usually carried out through the comparison of the corresponding receiver operating characteristic (ROC) curves. This matter has been approached from different perspectives. Usually, ROC curves are compared through their respective areas under the curve, but in cases where there is no uniform dominance between the involved curves other procedures are preferred. Although the asymptotic distributions of the statistics behind these methods are, in general, known, resampling plans are considered. With the purpose of comparing the performance of different approaches, with different ways of calibrating the distribution of the tests, a simulation study is carried out in order to investigate the statistical power and the nominal level of each methodology.


ROC curves AUC Comparison methods Resampling plans 



The authors would like to thank associate editor, the Co-Editor and the reviewers for their constructive comments and suggestions on an earlier version of this manuscript. The research of A. Fanjul-Hevia is supported by the Spanish Ministry of Education, Culture and Sport “Beca de Formación de Profesorado Universitario”; fellowship (FPU14/05316). Both authors acknowledged the support from the Spanish Ministry of Economy and Competitiveness, through grant numbers MTM2013-41383P and MTM2016-76969-P, which includes support from the European Regional Development Fund (ERDF). The research of W. González-Manteiga is also supported by the the IAP network P7/06 StUDyS of the Belgian Government (Belgian Science Policy).


  1. Antoch J, Prchal L, Sarda P (2010) Nonparametric comparison of ROC curves: testing equivalence. In: Nonparametrics and robustness in modern statistical inference and time series analysis: a Festschrift in honor of Professor Jana Jurečková, Institute of Mathematics and Statistics Collection, vol 7. Institute of Mathematics and Statistics, Beachwood, pp 12–24Google Scholar
  2. Bamber D (1975) The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol 12(4):387–415MathSciNetCrossRefzbMATHGoogle Scholar
  3. Bandos AI, Rockette HE, Gur D (2005) A permutation test sensitive to differences in areas for comparing ROC curves from a paired design. Stat Med 24(18):2873–2893MathSciNetCrossRefGoogle Scholar
  4. Braun TM, Alonzo TA (2008) A modified sign test for comparing paired roc curves. Biostatistics 9(2):364–372CrossRefzbMATHGoogle Scholar
  5. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3):837–845Google Scholar
  6. Gonçalves L, Subtil A, Oliveira MR, Bermudez P (2014) Roc curve estimation: an overview. REVSTAT 12(1):1–20MathSciNetzbMATHGoogle Scholar
  7. Good P (2013) Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer, BerlinzbMATHGoogle Scholar
  8. Green D, Swets J (1966) Signal detection theory and psychophysics. Peninsula, Los AltosGoogle Scholar
  9. Hanley JA, McNeil BJ (1983) A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148(3):839–843CrossRefGoogle Scholar
  10. Hsieh F, Turnbull BW (1996) Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat 24(1):25–40MathSciNetCrossRefzbMATHGoogle Scholar
  11. Jokiel-Rokita A, Pulit M (2013) Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions. Stat Comput 23(6):703–712MathSciNetCrossRefzbMATHGoogle Scholar
  12. Krzanowski WJ, Hand DJ (2009) ROC curves for continuous data. CRC Press, Boca RatonCrossRefzbMATHGoogle Scholar
  13. Lloyd CJ (1998) Using smoothed receiver operating characteristic curves to summarize and compare diagnostic systems. J Am Stat Assoc 93(444):1356–1364CrossRefGoogle Scholar
  14. Lloyd CJ, Yong Z (1999) Kernel estimators of the ROC curve are better than empirical. Stat Probab Lett 44(3):221–228MathSciNetCrossRefzbMATHGoogle Scholar
  15. Martínez-Camblor P (2007) Comparación de pruebas diagnósticas desde la curva ROC. Rev Colomb Estad 30(2):163–176MathSciNetGoogle Scholar
  16. Martínez-Camblor P, Corral N (2012) A general bootstrap algorithm for hypothesis testing. J Stat Plan Inference 142(2):589–600MathSciNetCrossRefzbMATHGoogle Scholar
  17. Martínez-Camblor P, Carleos C, Corral N (2011) Powerful nonparametric statistics to compare \(k\) independent roc curves. J Appl Stat 38(7):1317–1332MathSciNetCrossRefzbMATHGoogle Scholar
  18. Martínez-Camblor P, Carleos C, Corral N (2013) General nonparametric ROC curve comparison. J Korean Stat Soc 42(1):71–81MathSciNetCrossRefzbMATHGoogle Scholar
  19. Pepe MS (2003) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, CaryzbMATHGoogle Scholar
  20. Pérez-Fernández S (2017) nsROC: non-standard ROC curve analysis. R package version 1.0.
  21. R Core Team (2015) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna.
  22. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Mller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77CrossRefGoogle Scholar
  23. Sheather SJ, Jones MC (1991) A reliable data-based bandwidth selection method for kernel density estimation. J R Stat Soc Ser B Methodol 53(3):683–690Google Scholar
  24. Venkatraman E (2000) A permutation test to compare receiver operating characteristic curves. Biometrics 56(4):1134–1138MathSciNetCrossRefzbMATHGoogle Scholar
  25. Venkatraman E, Begg CB (1996) A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 83(4):835–848MathSciNetCrossRefzbMATHGoogle Scholar
  26. Wieand S, Gail MH, James BR, James KL (1989) A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76(3):585–592MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2017

Authors and Affiliations

  1. 1.Departamento de Estatística, Análise Matemática e OptimizaciónUniversidade de Santiago de CompostelaA CoruñaSpain

Personalised recommendations