Skip to main content
Log in

A comparative study of methods for testing the equality of two or more ROC curves

  • Original Paper
  • Published:
Computational Statistics Aims and scope Submit manuscript


The problem of comparing the accuracy of diagnostic tests is usually carried out through the comparison of the corresponding receiver operating characteristic (ROC) curves. This matter has been approached from different perspectives. Usually, ROC curves are compared through their respective areas under the curve, but in cases where there is no uniform dominance between the involved curves other procedures are preferred. Although the asymptotic distributions of the statistics behind these methods are, in general, known, resampling plans are considered. With the purpose of comparing the performance of different approaches, with different ways of calibrating the distribution of the tests, a simulation study is carried out in order to investigate the statistical power and the nominal level of each methodology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others


  • Antoch J, Prchal L, Sarda P (2010) Nonparametric comparison of ROC curves: testing equivalence. In: Nonparametrics and robustness in modern statistical inference and time series analysis: a Festschrift in honor of Professor Jana Jurečková, Institute of Mathematics and Statistics Collection, vol 7. Institute of Mathematics and Statistics, Beachwood, pp 12–24

  • Bamber D (1975) The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol 12(4):387–415

    Article  MathSciNet  MATH  Google Scholar 

  • Bandos AI, Rockette HE, Gur D (2005) A permutation test sensitive to differences in areas for comparing ROC curves from a paired design. Stat Med 24(18):2873–2893

    Article  MathSciNet  Google Scholar 

  • Braun TM, Alonzo TA (2008) A modified sign test for comparing paired roc curves. Biostatistics 9(2):364–372

    Article  MATH  Google Scholar 

  • DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3):837–845

  • Gonçalves L, Subtil A, Oliveira MR, Bermudez P (2014) Roc curve estimation: an overview. REVSTAT 12(1):1–20

    MathSciNet  MATH  Google Scholar 

  • Good P (2013) Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer, Berlin

    MATH  Google Scholar 

  • Green D, Swets J (1966) Signal detection theory and psychophysics. Peninsula, Los Altos

    Google Scholar 

  • Hanley JA, McNeil BJ (1983) A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148(3):839–843

    Article  Google Scholar 

  • Hsieh F, Turnbull BW (1996) Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat 24(1):25–40

    Article  MathSciNet  MATH  Google Scholar 

  • Jokiel-Rokita A, Pulit M (2013) Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions. Stat Comput 23(6):703–712

    Article  MathSciNet  MATH  Google Scholar 

  • Krzanowski WJ, Hand DJ (2009) ROC curves for continuous data. CRC Press, Boca Raton

    Book  MATH  Google Scholar 

  • Lloyd CJ (1998) Using smoothed receiver operating characteristic curves to summarize and compare diagnostic systems. J Am Stat Assoc 93(444):1356–1364

    Article  Google Scholar 

  • Lloyd CJ, Yong Z (1999) Kernel estimators of the ROC curve are better than empirical. Stat Probab Lett 44(3):221–228

    Article  MathSciNet  MATH  Google Scholar 

  • Martínez-Camblor P (2007) Comparación de pruebas diagnósticas desde la curva ROC. Rev Colomb Estad 30(2):163–176

    MathSciNet  Google Scholar 

  • Martínez-Camblor P, Corral N (2012) A general bootstrap algorithm for hypothesis testing. J Stat Plan Inference 142(2):589–600

    Article  MathSciNet  MATH  Google Scholar 

  • Martínez-Camblor P, Carleos C, Corral N (2011) Powerful nonparametric statistics to compare \(k\) independent roc curves. J Appl Stat 38(7):1317–1332

    Article  MathSciNet  MATH  Google Scholar 

  • Martínez-Camblor P, Carleos C, Corral N (2013) General nonparametric ROC curve comparison. J Korean Stat Soc 42(1):71–81

    Article  MathSciNet  MATH  Google Scholar 

  • Pepe MS (2003) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, Cary

    MATH  Google Scholar 

  • Pérez-Fernández S (2017) nsROC: non-standard ROC curve analysis. R package version 1.0.

  • R Core Team (2015) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna.

  • Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Mller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77

    Article  Google Scholar 

  • Sheather SJ, Jones MC (1991) A reliable data-based bandwidth selection method for kernel density estimation. J R Stat Soc Ser B Methodol 53(3):683–690

  • Venkatraman E (2000) A permutation test to compare receiver operating characteristic curves. Biometrics 56(4):1134–1138

    Article  MathSciNet  MATH  Google Scholar 

  • Venkatraman E, Begg CB (1996) A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 83(4):835–848

    Article  MathSciNet  MATH  Google Scholar 

  • Wieand S, Gail MH, James BR, James KL (1989) A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76(3):585–592

    Article  MathSciNet  MATH  Google Scholar 

Download references


The authors would like to thank associate editor, the Co-Editor and the reviewers for their constructive comments and suggestions on an earlier version of this manuscript. The research of A. Fanjul-Hevia is supported by the Spanish Ministry of Education, Culture and Sport “Beca de Formación de Profesorado Universitario”; fellowship (FPU14/05316). Both authors acknowledged the support from the Spanish Ministry of Economy and Competitiveness, through grant numbers MTM2013-41383P and MTM2016-76969-P, which includes support from the European Regional Development Fund (ERDF). The research of W. González-Manteiga is also supported by the the IAP network P7/06 StUDyS of the Belgian Government (Belgian Science Policy).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Arís Fanjul-Hevia.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fanjul-Hevia, A., González-Manteiga, W. A comparative study of methods for testing the equality of two or more ROC curves. Comput Stat 33, 357–377 (2018).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: