ROC analysis in medical imaging: a tutorial review of the literature
 Charles E. Metz
 … show all 1 hide
Abstract
Receiver operating characteristic (ROC) analysis measures the “diagnostic accuracy” of a medical imaging system, which represents the second level of diagnostic efficacy in the hierarchical model described by Fryback and Thornbury (Med Decis Making 11:88–94, 1991). After describing the historical origins of ROC analysis, this paper reviews the importance of sampling cases appropriately, designing an observer study to avoid bias, and collecting data on a useful scale. A variety of methods for fitting ROC curves to observer data and testing the statistical significance of apparent differences are then reported. Finally, generalized forms of ROC analysis that require lesion localization or allow more than two states of truth are surveyed briefly.
 Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making. 1991;11:88–94
 Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8:283–98
 Wald A. Statistical decision functions. New York: Wiley; 1950
 Egan JP. Signal detection theory and ROC analysis. New York: Academic; 1975
 Van Meter D, Middleton D. Modern statistical approaches to reception in communication theory. IRE Trans. 1954;PGIT4:119–41
 Peterson WW, Birdsall TG, Fox WC. The theory of signal detectability. IRE Trans. 1954;PGIT4:171–212
 Tanner WP Jr, Swets JA. A decisionmaking theory of visual detection. Psych Rev. 1954;61:401–9
 Swets JA, Birdsall TG. Human use of information III: decisionmaking in signal detection and recognition situations involving multiple alternatives. IEEE Trans Inf Theory. 1956;IT2:138–65
 Swets JA, Tanner WP Jr, Birdsall TG. Decision processes in perception. Psych Rev. 1961;68:301–40
 Swets JA, (editor). Signal detection and recognition by human observers: contemporary readings. New York: Wiley; 1964
 Green DM, Swets JA. Signal detection theory and psychophysics. New York: Wiley; 1966. [Reprinted with corrections and an updated topical bibliography by Krieger (Huntington, NY, 1974) and by Peninsula Publishing (Los Altos, CA, 1988)]
 Swets JA. The relative operating characteristic in psychology. Science. 1973;182:990–1000
 Swets JA, Green DM. Applications of signal detection theory. In: Pick HA, Liebowitz HL, Singer A, et al. editors. Psychology: from research to practice. New York: Plenum; 1978. p. 311–331
 Swets JA. Effectiveness of information retrieval methods. Am Doc. 1969;20:72–89
 Griner PF, Mayewski RJ, Mushlin AI, Greenland P. Selection and interpretation of diagnostic tests and procedures: principles and applications. Ann Intern Med. 1981;94:553–92
 Swets JA. Assessment of NDT systems (Parts I and II). Mater Eval. 1983;41:1294–303
 Robertson EA, Zweig MH, Van Steirtghem AC. Evaluating the clinical efficacy of laboratory tests. Am J Clin Path. 1983;79:78–86
 Zweig MH, Campbell G. Receiveroperating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39:561–77. (Erratum published in Clin Chem. 1993;39:1589.)
 Swets JA. Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Mahwah: Lawrence Erlbaum Associates; 1996
 Lusted LB. Personal communication in conversations with CE Metz. circa 1975
 Lusted LB. Logical analysis in roentgen diagnosis. Radiology. 1960;74:178–93
 Lusted LB. Introduction to medical decision making. Springfield: Thomas; 1968
 Lusted LB. Decisionmaking studies in patient management. New Engl J Med. 1971;284:416–24 CrossRef
 Lusted LB. Signal detectability and medical decisionmaking. Science. 1971;171:1217–9
 Lusted LB. Observer error, signal detectability, medical decision making. In: Jacquez JA, editor. Computer diagnosis and diagnostic methods. Springfield: Thomas; 1972. p. 29–44
 Lusted LB. Receiver operating chararcteristic analysis and its significance in interpretation of radiologic images. In: Potchen E, editor. Current concepts in radiology. St Louis: Mosby; 1975. p. 117–130
 Lusted LB. General problems in medical decision making, with comments on ROC analysis. Semin Nucl Med. 1978;8:299–306
 Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of signal detection theory. Radiology. 1972;105:199–200
 Goodenough DJ, Rossmann K, Lusted LB. Factors affecting the detectability of a simulated radiographic signal. Invest Radiol. 1973;8:339–44
 Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of receiver operating characteristic (ROC) analysis. Radiology. 1974;110:89–95
 Swets JA. Signal detection in medical diagnosis. In: Jacquez JA, editor. Computer diagnosis and diagnostic methods. Springfield: Thomas; 1972. p. 8–28
 Morgan RH, Donner MW, Gayler BW, et al. Decision processes and observer error in the diagnosis of pneumoconiosis by chest roentgenography. Am J Roentgenol. 1973;117:757–64
 Kundel HL, Revesz G. The evaluation of radiograghic techniques by observer tests: problems, pitfalls and procedures. Invest Radiol. 1974;9:166–73
 Metz CE, Starr SJ, Lusted LB, Rossmann K. Progress in evaluation of human observer visual detection performance using the ROC curve approach. In: Raynaud C, ToddPokropek AE, editors. Information processing in scintigraphy. Orsay, France: Commissariat à l’Energie Atomique, Département de Biologie, Service Hospitalier Frédéric Joliot; 1975. p. 420–439
 Metz CE, Starr SJ, Lusted LB. Quantitative evaluation of visual detection performance in medicine: ROC analysis and determination of diagnostic benefit. In: Hay GA, editor. Medical images: formation, perception and measurement. London: Wiley; 1977. p. 220–240
 Andrus WS, Bird KT. Radiology and the receiver operating characteristic (ROC) curve. Chest. 1975;67:378–9
 McNeil BJ, Keeler E, Adelstein SJ. Primer on certain elements of medical decision making. New Engl J Med. 1975;293:211–5 CrossRef
 Turner DA. An intuitive approach to receiver operating chararcteristic curve analysis. J Nucl Med. 1978;19:213–20
 Swets JA. ROC analysis applied to the evaluation of medical imaging techniques. Invest Radiol. 1979;14:109–21
 Swets JA, Pickett RM. Evaluation of diagnostic systems: methods from signal detection theory. New York: Academic; 1982
 Metz CE. ROC methodology in radiologic imaging. Invest Radiol. 1986;21:720–33
 Swets JA. Measuring the accuracy of diagnostic systems. Science. 1988;240:1285–93
 Hanley JA. Receiver operating characteristic (ROC) methodology: the state of the art. CRC Crit Rev Diagn Imaging. 1989;29:307–35
 Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24:234–45
 Metz CE, Wagner RF, Doi K, Brown DG, Nishikawa RN, Myers KJ. Toward consensus on quantitative assessment of medical imaging systems. Med Phys. 1995;22:1057–61
 Allisy A, (editor). Medical imaging—the assessment of image quality. ICRU report #54. Bethesda: International Commission for Radiation Units and Measurements, Inc.; 1996
 Metz CE. Evaluation of CAD. In: Doi K, MacMahon H, Giger ML, Hoffmann KR, editors. Computeraided diagnosis in medical imaging. Amsterdam: Elsevier; 1999. p. 543. (Excerpta Medica International Congress Series, vol. 1182)
 Metz CE. Fundamental ROC analysis. In: Beutel J, Kundel H, Van Metter R, editors. Handbook of medical imaging, vol. 1: physics and psychophysics. Bellingham: SPIE Press; 2000. p. 751
 Wagner RF, Beiden SV, Campbell G, Metz CE, Sacks WM. Assessment of medical imaging and computerassist systems: lessons from recent experience. Acad Radiol. 2002;8:1264–77
 Metz CE. Receiver operating characteristic (ROC) analysis: a tool for quantitative evaluation of observer performance and imaging systems. JACR. 2006;3:413–22
 Wagner RF, Metz CE, Campbell G. Assessment of medial imaging systems and computer aids: a tutorial review. Acad Radiol. 2007;14:723–48
 Krupinski EA, Jiang Y. Evaluation of medical imaging systems. Med Phys. 2008 (in press)
 Gur D. Objectively measuring and comparing performance levels of diagnostic imaging systems and practices (editorial). Acad Radiol. 2007;14:641–2
 Gur D, Rockette HE, Good W, Slasky BS, Cooperstein LA, Straub WH, et al. Effect of observer instruction on ROC study of chest images. Invest Radiol. 1990;25:230–4 CrossRef
 Kobayashi T, Xu XW, MacMahon H, Metz CE, Doi K. Effect of a computeraided diagnosis scheme on radiologists’ performance in detection of lung nodules on chest radiographs. Radiology. 1996;199:843–8
 Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New Engl J Med. 1978;299:926–30 CrossRef
 Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39:207–15
 Revesz G, Kundel HL, Bonitatibus M. The effect of verification on the assessment of imaging techniques. Invest Radiol. 1983;18:194–8
 Gray R, Begg CB, Greenes RA. Construction of receiver operating characteristic curves when disease verification is subject to selection bias. Med Decis Making. 1984;4:151–64
 Begg CB, McNeil BJ. Assessment of radiologic tests: control of bias and other design considerations. Radiology. 1988;167:565–9
 Gur D, Rockette HE, Armfield DR, et al. Prevalence effect in a laboratory environment. Radiology. 2003;228:10–4
 Gur D, Bandos AI, Fuhrman CR, Klym AH, King JL, Rockette HE. The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol. 2007;14:49–53
 Rockette HE, Gur D, Metz CE. The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. Invest Radiol. 1992;27:169–72
 King JL, Britton CA, Gur D, Rockette HE, Davis PL. On the validity of continuous and discrete confidence rating scales in receiver operating characteristic studies. Invest Radiol. 1993;28:962–3
 Walsh SJ. Limitations to the robustness of binormal ROC curves: effects of model misspecification and location of decision thresholds on bias, precision, size and power. Stat Med. 1997;16:669–79
 Wagner RF, Beiden SV, Metz CE. Continuous vs. categorical data for ROC analysis: some quantitative considerations. Acad Radiol. 2001;8:328–34
 Hadjiiski L, Chan HP, Sahiner B, Helvie MA, Roubidoux MA. Quasicontinuous and discrete confidence rating scales for observer performance studies: effects on ROC analysis. Acad Radiol. 2007;14:38–48
 Swets JA. Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychol Bull. 1986;99:181–98
 Hanley JA. The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis Making. 1988;8:197–203
 Hanley JA. The use of the “binormal” model for parametric ROC analysis of quantitative diagnostic tests. Stat Med. 1996;15:1575–85
 Dorfman DD, Alf E. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals—rating method data. J Math Psych. 1969;6:487–96
 Grey DR, Morgan BJT. Some aspects of ROC curvefitting: normal and logistic models. J Math Psych. 1972;9:128–39
 Metz CE, Herman BA, Shen JH. Maximumlikelihood estimation of ROC curves from continuouslydistributed data. Stat Med. 1998 17:1033–53
 Dorfman DD, Berbaum KS. Degeneracy and discrete receiver operating characteristic rating data. Acad Radiol. 1995;2:907–15
 Dorfman DD, Berbaum KS, Metz CE, Lenth RV, Hanley JA, Dagga HA. Proper ROC analysis: the bigamma model. Acad Radiol. 1997;4:138–49
 Pan X, Metz CE. The “proper” binormal model: parametric ROC curve estimation with degenerate data. Acad Radiol. 1997;4:380–9
 Metz CE, Pan X. “Proper” binormal ROC curves: theory and maximumlikelihood estimation. J Math Psych. 1999;43:1–33
 Pesce LL, Metz CE. Reliable and computationally efficient maximumlikelihood estimation of “proper” binormal ROC curves. Acad Radiol. 2007;14:814–29
 Tosteson A, Begg C. A general regression methodology for ROC curve estimation. Med Decis Making. 1988;8:204–15
 Toledano AY, Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Med. 1996;15:1807–26
 Hellmich M, Abrams KR, Jones DR, Lambert PC. A Bayesian approach to a general regression model for ROC curves. Med Decis Making. 1998;18:436–43
 Pepe MS. The statistical evaluation of medical tests for classification and prediction. New York: Oxford University Press; 2004
 Metz CE. Quantification of failure to demonstrate statistical significance: the usefulness of confidence intervals. Invest Radiol. 1993;28:59–63
 Metz CE, Kronman HB. Statistical significance tests for binormal ROC curves. J Math Psych. 1980;22:218–43
 Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29–36
 McClish DK. Analyzing a portion of the ROC curve. Med Decis Making. 1989;9:190–5
 Jiang Y, Metz CE, Nishikawa RM. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 1996;201:745–50
 Halpern EJ, Alpert M, Krieger AM, Metz CE, Maidment AD. Comparisons of ROC curves on the basis of optimal operating points. Acad Radiol. 1996;3:245–53
 Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148:839–43
 DeLong ER, DeLong DM, ClarkePearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–45
 Wieand S, Gail MH, James BR, James KL. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika. 1989;76:585–92
 Thompson ML, Zucchini W. On the statistical analysis of ROC curves. Stat Med. 1989;8:1277–90
 Zhou XH, Gatsonis CA. A simple method for comparing correlated ROC curves using incomplete data. Stat Med. 1996;15:1687–93
 Metz CE, Wang PL, Kronman HB. A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Deconinck F, editor. Information processing in medical imaging. The Hague: Nijhoff; 1984. p. 432–445
 Metz CE. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert D, Myers R, editors. Multiple regression analysis: applications in the health sciences. New York: American Institute of Physics; 1986. p. 365–384
 Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC curve estimates obtained from partiallypaired datasets. Med Decis Making. 1998;18:110–21
 HajianTilaki KO, Hanley JA, Joseph L, Collet JP. A comparison of parametric and nonparametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Making. 1997;17:94–102
 Hsieh F, Turnbull BW. Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat. 1996;24:25–40
 Roe CA, Metz CE. Variancecomponent modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol. 1997;4:587–600
 Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992;27:723–31
 Dorfman DD, Metz CE. Multireader multicase ROC analysis: comments on Begg’s commentary. Acad Radiol. 1995;2 Suppl 1:S76
 Dorfman DD, Berbaum KS, Lenth RV. Multireader, multicase receiver operating characteristic methodology: a bootstrap analysis. Acad Radiol. 1995;2:626–33
 Roe CA, Metz CE. The Dorfman–Berbaum–Metz method for statistical analysis of multireader, multimodality ROC data: validation by computer simulation. Acad Radiol. 1997;4:298–303
 Dorfman DD, Berbaum KS, Lenth RV, Chen YF, Donaghy BA. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol. 1998;5:591–602
 Hillis SL, Berbaum KS. Power estimation for the Dorfman–Berbaum–Metz method. Acad Radiol. 2004;11:1260–73
 Hillis SL, Berbaum KS. Monte Carlo validation of the Dorfman–Berbaum–Metz method using normalized pseudovalues and less databased model simplification. Acad Radiol. 2005;12:1534–42
 Obuchowski NA, Rockette HE. Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: an ANOVA approach with dependent observations. Commun Stat Simul Comput. 1995;24:285–308
 Obuchowski NA. Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol. 1995;2:522–9
 Obuchowski NA. Sample size tables for receiver operating characteristic studies. Am J Roentgenol. 2000;175:603–8
 Toledano AY, Gatsonis C. GEEs for ordinal categorical data: arbitrary patterns of missing responses and missingness in a key covariate. Biometrics. 1999;22:488–96
 Beiden SV, Wagner RF, Campbell G. Componentsofvariance models and multiplebootstrap experiments: an alternative method for randomeffects, receiver operating characteristic analysis. Acad Radiol. 2000;7:341–9
 Beiden SV, Wagner RF, Campbell G, Chan HP. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 2001;8:616–22
 Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y. Componentsofvariance models for randomeffects ROC analysis: the case of unequal variance structures across modalities. Acad Radiol. 2001;8:605–15
 Beiden SV, Wagner RF, Campbell G, Chan HP. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 2001;8:616–22
 Obuchowski NA, Beiden SV, Berbaum KS, Hillis SL, Ishwaran H, Song HH, et al. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol. 2004;11:980–95
 Hillis SL, Obuchowski NA, Schartz KM, Berbaum KS. A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data. Stat Med. 2005;24:1579–607
 Hillis, SL: Sample size estimates for DBM MRMC based on analysis of published data. http://perception.radiology.uiowa.edu/SampleSize/tabid/182/Default.aspx
 Dorfman DD. RSCORE II. In: Swets JA, Pickett RM, editors. Evaluation of diagnostic systems: methods from signal detection theory. New York: Academic; 1982. p. 208–232
 University of Chicago Receiver Operating Characteristic program software downloads. http://xray.bsd.uchicago.edu/krl/KRL_ROC/software_index6.htm
 University of Iowa Receiver Operating Characteristic program software downloads. http://perception.radiology.uiowa.edu/
 Cleveland Clinic Receiver Operating Characteristic program software downloads. http://www.bio.ri.ccf.org/html/obumrm.html
 Obuchowski, NA: Research activities: ROC analysis. http://www.bio.ri.ccf.org/html/rocanalysis.html
 Starr SJ, Metz CE, Lusted LB, Goodenough DJ. Visual detection and localization of radiographic images. Radiology. 1975;116:533–8
 Starr SJ, Metz CE, Lusted LB. Comments on generalization of receiver operating characteristic analysis to detection and localization tasks (Letter to the Editor). Phys Med Biol. 1977;22:376–9
 Swensson RG. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys. 1996;23:1709–25
 Egan JP, Greenberg GZ, Schulman AI. Operating characteristics, signal detection, and the method of free response. J Acoust Soc Am. 1961;33:993–1007
 International Atomic Energy Agency. IAEA coordinated research programme on the intercomparison of computerassisted scintigraphic techniques: third progress report. In: Medical radionuclide imaging, vol. 1. Vienna: IAEA; 1977. p. 585–615
 Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free response approach to the measurement and characterization of radiographic observer performance. Proc SPIE. 1977;127:124–35
 Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free response approach to the measurement and characterization of radiographic observer performance. J Appl Photogr Eng. 1978;4:166–72
 Chakraborty DP. Maximum likelihood analysis of freeresponse receiver operating characteristic (FROC) data. Med Phys. 1989;16:561–8
 Chakraborty DP, Winter LHL. Freeresponse methodology: alternate analysis and a new observerperformance experiment. Radiology. 1990;33:873–81
 Obuchowski NA, Lieber ML, Powell KA. Data analysis for detection and localization of multiple abnormalities with application to mammography. Acad Radiol. 2000;7:516–25
 Chakraborty DP. Statistical power in observer performance studies: a comparison of the ROC and freeresponse methods in tasks involving localization. Acad Radiol. 2002;9:147–56
 Edwards DC, Kupinski MA, Metz CE, Nishikawa RM. Maximumlikelihood fitting of FROC curves under an initialdetectionandcandidateanalysis model. Med Phys. 2002;29:2861–70
 Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys. 2004;31:2313–30
 Chakraborty DP. A search model and figure of merit for observer data acquired according to the freeresponse paradigm. Phys Med Biol. 2006;51:3449–62
 Chakraborty DP. Analysis of location specific observer performance data: validated extensions of the jackknife freeresponse (JAFROC) method. Acad Radiol. 2006;13:1187–93
 Chakraborty D, Yoon HJ, MelloThoms C. Spatial localization accuracy of radiologists in freeresponse studies: inferring perceptual FROC curves from markrating data. Acad Radiol. 2007;14:4–18
 Edwards DC, Metz CE, Kupinski MA. Ideal observers and optimal ROC hypersurfaces in Nclass classification. IEEE Trans Med Imaging. 2004;23:891–5
 Edwards DC, Metz CE, Nishikawa RM. The hypervolume under the ROC hypersurface of “nearguessing” and “nearperfect” observers in Nclass classification tasks. IEEE Trans Med Imaging. 2005;24:293–9
 Edwards DC, Lan L, Metz CE, Giger ML, Nishikawa RM. Estimating threeclass ideal observer decision variables for computerized detection and classification of mammographic mass lesions. Med Phys. 2004;31:81–90
 Edwards DC, Metz CE. Review of several proposed threeclass classification decision rules and their relation to the ideal observer decision rule. Proc SPIE. 2005;5749:128–37
 Edwards DC, Metz CE. Restrictions on the threeclass ideal observer’s decision boundary lines. IEEE Trans Med Imaging. 2005;24:1566–73
 Edwards DC, Metz CE. Analysis of proposed threeclass classification decision rules in terms of the ideal observer decision rule. J Math Psych. 2006;50:478–87
 He X, Metz CE, Tsui BMW, Links JM, Frey EC. Threeclass ROC analysis—I: a decision theoretic approach under the ideal observer framework. IEEE Trans Med Imaging. 2006;25:571–81
 He X, Fry EC. An optimal threeclass linear observer derived from decision theory. IEEE Trans Med Imaging. 2007;26:77–83
 Chan HP, Sahiner B, Hadjiiski LM, Petrick N, Zhou C. Design of threeclass classifiers in computeraided diagnosis: Monte Carlo simulation study. Proc SPIE. 2003;5032:567–78
 Sahiner B, Chan HP, Hadjiiski LM. Performance analysis of 3class classifiers: properties of the 3D ROC surface and the normalized volume under the surface. Proc SPIE. 2006;6146:87–93
 Title
 ROC analysis in medical imaging: a tutorial review of the literature
 Journal

Radiological Physics and Technology
Volume 1, Issue 1 , pp 212
 Cover Date
 20080101
 DOI
 10.1007/s1219400700021
 Print ISSN
 18650333
 Online ISSN
 18650341
 Publisher
 SpringerVerlag
 Additional Links
 Topics
 Keywords

 Receiver operating characteristic analysis
 ROC analysis
 Image evaluation
 Diagnostic accuracy
 Diagnostic efficacy
 Observer performance
 Industry Sectors
 Authors

 Charles E. Metz ^{(1)}
 Author Affiliations

 1. Radiology and Medical Physics, The University of Chicago, Chicago, IL, USA