Recommendations for using the relative operating characteristic (ROC)

Abstract

The relative operating characteristic (ROC) is a widely-used method to measure diagnostic signals including predictions of land changes, species distributions, and ecological niches. The ROC measures the degree to which presence for a Boolean variable is associated with high ranks of an index. The ROC curve plots the rate of true positives versus the rate of false positives obtained from the comparison between the Boolean variable and multiple diagnoses derived from thresholds applied to the index. The area under the ROC curve (AUC) is a summary metric, which is commonly reported and frequently criticized. Our manuscript recommends four improvements in the use and interpretation of the ROC curve and its AUC by: (1) highlighting important threshold points on the ROC curve, (2) interpreting the shape of the ROC curve, (3) defining lower and upper bounds for the AUC, and (4) mapping the density of the presence within each bin of the ROC curve. These recommendations encourage scientists to interpret the rich information that the ROC curve can reveal, in a manner that goes far beyond the potentially misleading AUC. We illustrate the benefit of our recommendations by assessing the prediction of land change in a suburban landscape.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

References

  1. Boscolo D, Metzger JP (2009) Is bird incidence in Atlantic forest fragments influenced by landscape patterns at multiple scales? Landscape Ecol 24(7):907–918

    Article  Google Scholar 

  2. Cortes C, Mohri M (2004) Confidence intervals for the area under the ROC curve. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems. A Bratford Book, Cambridge, pp 305–312

    Google Scholar 

  3. Dlamini WM (2010) A Bayesian belief network analysis of factors influencing wildfire occurrence in Swaziland. Environ Model Softw 25(2):199–208

    Article  Google Scholar 

  4. Eastman JR (2012) Idrisi Andes Guide to GIS and Remote Sensing. Clark Labs, Clark University, Worcester

    Google Scholar 

  5. Eastman JR, Van Fossen M, Solorzano LA (2005) Transition potential modeling for land cover change. In: Maguire DJ, Batty M, Goodchild MF (eds) GIS, spatial analysis, and modeling. ESRI Press, Redlands, CA, pp 357–385

  6. Elith J, Phillips SJ, Hastie T, Dudík M, Chee YE, Yates CJ (2011) A statistical explanation of MaxEnt for ecologists. Divers Distrib 17(1):43–57

    Article  Google Scholar 

  7. Evans JS, Cushman SA (2009) Gradient modeling of conifer species using random forests. Landscape Ecol 24(5):673–683

    Article  Google Scholar 

  8. Fang S, Gertner GZ, Anderson AB (2006) Prediction of multinomial probability of land use change using a bisection decomposition and logistic regression. Landscape Ecol 22(3):419–430

    Article  Google Scholar 

  9. Fawcett T (2006) An introduction to ROC analysis. Pattern Recogn Lett 27(8):861–874

    Article  Google Scholar 

  10. Fielding AH, Bell JF (1997) A review of methods for the assessment of prediction errors in conservation presence/absence models. Environ Conserv 24(01):38–49

    Article  Google Scholar 

  11. Jolliffe IT, Stephenson DB (2003) Forecast verification a practitioner guide in Atmospheric Science. Wiley, Chichester

    Google Scholar 

  12. Lesschen JP, Verburg PH, Staal SJ (2005) Statistical methods for analysing the spatial dimension of changes in land use and farming systems—LUCC Report Series 7. The International Livestock Research Institute, Nairobi, Kenya and Wageningen University, The Netherlands

  13. Lin Y-P, Chu H-J, Wu C-F, Verburg PH (2010) Predictive ability of logistic regression, auto-logistic regression and neural network models in empirical land-use change modeling—a case study. Int J Geogr Inf Sci 25(1):65–87

    Article  Google Scholar 

  14. Lobo JM, Jiménez-Valverde A, Real R (2008) AUC: a misleading measure of the performance of predictive distribution models. Glob Ecol Biogeogr 17(2):145–151

    Article  Google Scholar 

  15. Mas J-F, Soares Filho B, Pontius RG, Farfán Gutiérrez M, Rodrigues H (2013) A suite of tools for ROC analysis of spatial models. ISPRS Int J Geo-Inf 2(3):869–887

    Article  Google Scholar 

  16. MASSGIS (2010) Land use map and topographic information. Executive Office of Environmental Affairs, Boston, MA. http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/. Accessed 10 Nov 2010

  17. Mcskassy SA, Provost F, Rosset S (2005) ROC confidence bands: An Empirical Study. Information Systems Working Papers Series:537-544

  18. Mochizuki S, Murakami T (2011) Change in habitat selection by Japanese macaques (Macaca fuscata) and habitat fragmentation analysis using temporal remotely sensed data in Niigata Prefecture, Japan. Int J Appl Earth Obs Geoinf 13(4):562–571

    Article  Google Scholar 

  19. Muñoz J, Felicísimo ÁM (2004) Comparison of statistical methods commonly used in predictive modelling. J Veg Sci 15(2):285–292

    Article  Google Scholar 

  20. Peterson AT, Papeş M, Soberón J (2008) Rethinking receiver operating characteristic analysis applications in ecological niche modeling. Ecol Model 213(1):63–72

    Article  Google Scholar 

  21. Phillips SJ, Anderson RP, Schapire RE (2006) Maximum entropy modeling of species geographic distributions. Ecol Model 190(3–4):231–259

    Article  Google Scholar 

  22. Pontius RG, Millones M (2011) Death to Kappa: birth of quantity disagreement and allocation disagreement for accuracy assessment. Int J Remote Sens 32(15):4407–4429

    Article  Google Scholar 

  23. Pontius R, Pacheco P (2004) Calibration and validation of a model of forest disturbance in the Western Ghats, India 1920–1990. GeoJournal 61(4):325–334

    Article  Google Scholar 

  24. Pontius RG, Si K (2014) The Total Operating Characteristics to measure diagnostic ability for multiple thresholds. Int J Geogr Inf Sci. doi:10.1080/13658816.2013.862623

  25. Rizkalla CE, Moore JE, Swihart RK (2008) Modeling patch occupancy: relative performance of ecologically scaled landscape indices. Landscape Ecol 24(1):77–88

    Article  Google Scholar 

  26. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Müller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77

    PubMed Central  PubMed  Article  Google Scholar 

  27. Saatchi S, Buermann W, ter Steege H, Mori S, Smith TB (2008) Modeling distribution of Amazonian tree species and diversity using remote sensing measurements. Remote Sens Environ 112(5):2000–2017

    Article  Google Scholar 

  28. Swets JA (2010) Tulips to thresholds. Peninsula Publishing, Los Altos Hills

    Google Scholar 

  29. Taylor RS, Oldland JM, Clarke MF (2008) Edge geometry influences patch-level habitat use by an edge specialist in south-eastern Australia. Landscape Ecol 23(4):377–389

    Article  Google Scholar 

  30. Verburg PH, de Nijs TCM, Ritsema van Eck J, Visser H, de Jong K (2004) A method to analyse neighbourhood characteristics of land use patterns. Comput Environ Urban Syst 28(6):667–690

    Article  Google Scholar 

  31. Westphal MI, Field SA, Tyre AJ, Paton D, Possingham HP (2003) Effects of landscape pattern on bird species distribution in the Mt. Lofty Ranges, South Australia. Landscape Ecol 18(4):413–426

    Article  Google Scholar 

Download references

Acknowledgments

The United States National Science Foundation (NSF) supported this work via three of its programs: LTER via grant OEC-1238212, Coupled Natural Human Systems via grant BCS-0709685, and Research Experiences for Undergraduates via grant 0849985. Any opinions, findings, conclusions, or recommendation expressed in our manuscript are those of the authors and do not necessarily reflect those of the NSF. Massachusetts’ Office of Geographic Information (MassGIS) supplied data for this project. Clark Labs facilitated this work by creating the GIS software Idrisi®. Anonymous reviewers provided constructive feedback that improved our manuscript.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Benoit Parmentier.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 65 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Pontius, R.G., Parmentier, B. Recommendations for using the relative operating characteristic (ROC). Landscape Ecol 29, 367–382 (2014). https://doi.org/10.1007/s10980-013-9984-8

Download citation

Keywords

  • Accuracy
  • AUC
  • Index
  • Land change
  • Map
  • Prediction
  • ROC
  • Threshold
  • Uncertainty