Recommendations for using the relative operating characteristic (ROC)
- 948 Downloads
The relative operating characteristic (ROC) is a widely-used method to measure diagnostic signals including predictions of land changes, species distributions, and ecological niches. The ROC measures the degree to which presence for a Boolean variable is associated with high ranks of an index. The ROC curve plots the rate of true positives versus the rate of false positives obtained from the comparison between the Boolean variable and multiple diagnoses derived from thresholds applied to the index. The area under the ROC curve (AUC) is a summary metric, which is commonly reported and frequently criticized. Our manuscript recommends four improvements in the use and interpretation of the ROC curve and its AUC by: (1) highlighting important threshold points on the ROC curve, (2) interpreting the shape of the ROC curve, (3) defining lower and upper bounds for the AUC, and (4) mapping the density of the presence within each bin of the ROC curve. These recommendations encourage scientists to interpret the rich information that the ROC curve can reveal, in a manner that goes far beyond the potentially misleading AUC. We illustrate the benefit of our recommendations by assessing the prediction of land change in a suburban landscape.
KeywordsAccuracy AUC Index Land change Map Prediction ROC Threshold Uncertainty
The United States National Science Foundation (NSF) supported this work via three of its programs: LTER via grant OEC-1238212, Coupled Natural Human Systems via grant BCS-0709685, and Research Experiences for Undergraduates via grant 0849985. Any opinions, findings, conclusions, or recommendation expressed in our manuscript are those of the authors and do not necessarily reflect those of the NSF. Massachusetts’ Office of Geographic Information (MassGIS) supplied data for this project. Clark Labs facilitated this work by creating the GIS software Idrisi®. Anonymous reviewers provided constructive feedback that improved our manuscript.
- Cortes C, Mohri M (2004) Confidence intervals for the area under the ROC curve. In: Saul LK, Weiss Y, Bottou L (eds) Advances in neural information processing systems. A Bratford Book, Cambridge, pp 305–312Google Scholar
- Eastman JR (2012) Idrisi Andes Guide to GIS and Remote Sensing. Clark Labs, Clark University, WorcesterGoogle Scholar
- Eastman JR, Van Fossen M, Solorzano LA (2005) Transition potential modeling for land cover change. In: Maguire DJ, Batty M, Goodchild MF (eds) GIS, spatial analysis, and modeling. ESRI Press, Redlands, CA, pp 357–385Google Scholar
- Jolliffe IT, Stephenson DB (2003) Forecast verification a practitioner guide in Atmospheric Science. Wiley, ChichesterGoogle Scholar
- Lesschen JP, Verburg PH, Staal SJ (2005) Statistical methods for analysing the spatial dimension of changes in land use and farming systems—LUCC Report Series 7. The International Livestock Research Institute, Nairobi, Kenya and Wageningen University, The NetherlandsGoogle Scholar
- MASSGIS (2010) Land use map and topographic information. Executive Office of Environmental Affairs, Boston, MA. http://www.mass.gov/anf/research-and-tech/it-serv-and-support/application-serv/office-of-geographic-information-massgis/. Accessed 10 Nov 2010
- Mcskassy SA, Provost F, Rosset S (2005) ROC confidence bands: An Empirical Study. Information Systems Working Papers Series:537-544Google Scholar
- Pontius RG, Si K (2014) The Total Operating Characteristics to measure diagnostic ability for multiple thresholds. Int J Geogr Inf Sci. doi: 10.1080/13658816.2013.862623
- Swets JA (2010) Tulips to thresholds. Peninsula Publishing, Los Altos HillsGoogle Scholar