Skip to main content
SpringerLink
Log in

ROC analysis in medical imaging: a tutorial review of the literature

  • Published:
Radiological Physics and Technology Aims and scope Submit manuscript

Abstract

Receiver operating characteristic (ROC) analysis measures the “diagnostic accuracy” of a medical imaging system, which represents the second level of diagnostic efficacy in the hierarchical model described by Fryback and Thornbury (Med Decis Making 11:88–94, 1991). After describing the historical origins of ROC analysis, this paper reviews the importance of sampling cases appropriately, designing an observer study to avoid bias, and collecting data on a useful scale. A variety of methods for fitting ROC curves to observer data and testing the statistical significance of apparent differences are then reported. Finally, generalized forms of ROC analysis that require lesion localization or allow more than two states of truth are surveyed briefly.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

References

  1. Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making. 1991;11:88–94

    PubMed  CAS  Google Scholar 

  2. Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8:283–98

    PubMed  CAS  Google Scholar 

  3. Wald A. Statistical decision functions. New York: Wiley; 1950

    Google Scholar 

  4. Egan JP. Signal detection theory and ROC analysis. New York: Academic; 1975

    Google Scholar 

  5. Van Meter D, Middleton D. Modern statistical approaches to reception in communication theory. IRE Trans. 1954;PGIT-4:119–41

    Google Scholar 

  6. Peterson WW, Birdsall TG, Fox WC. The theory of signal detectability. IRE Trans. 1954;PGIT-4:171–212

    Google Scholar 

  7. Tanner WP Jr, Swets JA. A decision-making theory of visual detection. Psych Rev. 1954;61:401–9

    Google Scholar 

  8. Swets JA, Birdsall TG. Human use of information III: decision-making in signal detection and recognition situations involving multiple alternatives. IEEE Trans Inf Theory. 1956;IT-2:138–65

    Google Scholar 

  9. Swets JA, Tanner WP Jr, Birdsall TG. Decision processes in perception. Psych Rev. 1961;68:301–40

    CAS  Google Scholar 

  10. Swets JA, (editor). Signal detection and recognition by human observers: contemporary readings. New York: Wiley; 1964

    Google Scholar 

  11. Green DM, Swets JA. Signal detection theory and psychophysics. New York: Wiley; 1966. [Reprinted with corrections and an updated topical bibliography by Krieger (Huntington, NY, 1974) and by Peninsula Publishing (Los Altos, CA, 1988)]

    Google Scholar 

  12. Swets JA. The relative operating characteristic in psychology. Science. 1973;182:990–1000

    PubMed  CAS  Google Scholar 

  13. Swets JA, Green DM. Applications of signal detection theory. In: Pick HA, Liebowitz HL, Singer A, et al. editors. Psychology: from research to practice. New York: Plenum; 1978. p. 311–331

  14. Swets JA. Effectiveness of information retrieval methods. Am Doc. 1969;20:72–89

    Google Scholar 

  15. Griner PF, Mayewski RJ, Mushlin AI, Greenland P. Selection and interpretation of diagnostic tests and procedures: principles and applications. Ann Intern Med. 1981;94:553–92

    Google Scholar 

  16. Swets JA. Assessment of NDT systems (Parts I and II). Mater Eval. 1983;41:1294–303

    Google Scholar 

  17. Robertson EA, Zweig MH, Van Steirtghem AC. Evaluating the clinical efficacy of laboratory tests. Am J Clin Path. 1983;79:78–86

    PubMed  CAS  Google Scholar 

  18. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39:561–77. (Erratum published in Clin Chem. 1993;39:1589.)

    PubMed  CAS  Google Scholar 

  19. Swets JA. Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Mahwah: Lawrence Erlbaum Associates; 1996

    Google Scholar 

  20. Lusted LB. Personal communication in conversations with CE Metz. circa 1975

  21. Lusted LB. Logical analysis in roentgen diagnosis. Radiology. 1960;74:178–93

    PubMed  CAS  Google Scholar 

  22. Lusted LB. Introduction to medical decision making. Springfield: Thomas; 1968

    Google Scholar 

  23. Lusted LB. Decision-making studies in patient management. New Engl J Med. 1971;284:416–24

    Article  PubMed  CAS  Google Scholar 

  24. Lusted LB. Signal detectability and medical decision-making. Science. 1971;171:1217–9

    PubMed  CAS  Google Scholar 

  25. Lusted LB. Observer error, signal detectability, medical decision making. In: Jacquez JA, editor. Computer diagnosis and diagnostic methods. Springfield: Thomas; 1972. p. 29–44

    Google Scholar 

  26. Lusted LB. Receiver operating chararcteristic analysis and its significance in interpretation of radiologic images. In: Potchen E, editor. Current concepts in radiology. St Louis: Mosby; 1975. p. 117–130

    Google Scholar 

  27. Lusted LB. General problems in medical decision making, with comments on ROC analysis. Semin Nucl Med. 1978;8:299–306

    PubMed  CAS  Google Scholar 

  28. Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of signal detection theory. Radiology. 1972;105:199–200

    PubMed  CAS  Google Scholar 

  29. Goodenough DJ, Rossmann K, Lusted LB. Factors affecting the detectability of a simulated radiographic signal. Invest Radiol. 1973;8:339–44

    PubMed  CAS  Google Scholar 

  30. Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of receiver operating characteristic (ROC) analysis. Radiology. 1974;110:89–95

    PubMed  CAS  Google Scholar 

  31. Swets JA. Signal detection in medical diagnosis. In: Jacquez JA, editor. Computer diagnosis and diagnostic methods. Springfield: Thomas; 1972. p. 8–28

    Google Scholar 

  32. Morgan RH, Donner MW, Gayler BW, et al. Decision processes and observer error in the diagnosis of pneumoconiosis by chest roentgenography. Am J Roentgenol. 1973;117:757–64

    CAS  Google Scholar 

  33. Kundel HL, Revesz G. The evaluation of radiograghic techniques by observer tests: problems, pitfalls and procedures. Invest Radiol. 1974;9:166–73

    PubMed  CAS  Google Scholar 

  34. Metz CE, Starr SJ, Lusted LB, Rossmann K. Progress in evaluation of human observer visual detection performance using the ROC curve approach. In: Raynaud C, Todd-Pokropek AE, editors. Information processing in scintigraphy. Orsay, France: Commissariat à l’Energie Atomique, Département de Biologie, Service Hospitalier Frédéric Joliot; 1975. p. 420–439

    Google Scholar 

  35. Metz CE, Starr SJ, Lusted LB. Quantitative evaluation of visual detection performance in medicine: ROC analysis and determination of diagnostic benefit. In: Hay GA, editor. Medical images: formation, perception and measurement. London: Wiley; 1977. p. 220–240

    Google Scholar 

  36. Andrus WS, Bird KT. Radiology and the receiver operating characteristic (ROC) curve. Chest. 1975;67:378–9

    PubMed  CAS  Google Scholar 

  37. McNeil BJ, Keeler E, Adelstein SJ. Primer on certain elements of medical decision making. New Engl J Med. 1975;293:211–5

    Article  PubMed  CAS  Google Scholar 

  38. Turner DA. An intuitive approach to receiver operating chararcteristic curve analysis. J Nucl Med. 1978;19:213–20

    PubMed  CAS  Google Scholar 

  39. Swets JA. ROC analysis applied to the evaluation of medical imaging techniques. Invest Radiol. 1979;14:109–21

    PubMed  CAS  Google Scholar 

  40. Swets JA, Pickett RM. Evaluation of diagnostic systems: methods from signal detection theory. New York: Academic; 1982

    Google Scholar 

  41. Metz CE. ROC methodology in radiologic imaging. Invest Radiol. 1986;21:720–33

    PubMed  CAS  Google Scholar 

  42. Swets JA. Measuring the accuracy of diagnostic systems. Science. 1988;240:1285–93

    PubMed  CAS  Google Scholar 

  43. Hanley JA. Receiver operating characteristic (ROC) methodology: the state of the art. CRC Crit Rev Diagn Imaging. 1989;29:307–35

    CAS  Google Scholar 

  44. Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24:234–45

    PubMed  CAS  Google Scholar 

  45. Metz CE, Wagner RF, Doi K, Brown DG, Nishikawa RN, Myers KJ. Toward consensus on quantitative assessment of medical imaging systems. Med Phys. 1995;22:1057–61

    PubMed  CAS  Google Scholar 

  46. Allisy A, (editor). Medical imaging—the assessment of image quality. ICRU report #54. Bethesda: International Commission for Radiation Units and Measurements, Inc.; 1996

  47. Metz CE. Evaluation of CAD. In: Doi K, MacMahon H, Giger ML, Hoffmann KR, editors. Computer-aided diagnosis in medical imaging. Amsterdam: Elsevier; 1999. p. 543. (Excerpta Medica International Congress Series, vol. 1182)

    Google Scholar 

  48. Metz CE. Fundamental ROC analysis. In: Beutel J, Kundel H, Van Metter R, editors. Handbook of medical imaging, vol. 1: physics and psychophysics. Bellingham: SPIE Press; 2000. p. 751

    Google Scholar 

  49. Wagner RF, Beiden SV, Campbell G, Metz CE, Sacks WM. Assessment of medical imaging and computer-assist systems: lessons from recent experience. Acad Radiol. 2002;8:1264–77

    Google Scholar 

  50. Metz CE. Receiver operating characteristic (ROC) analysis: a tool for quantitative evaluation of observer performance and imaging systems. JACR. 2006;3:413–22

    PubMed  Google Scholar 

  51. Wagner RF, Metz CE, Campbell G. Assessment of medial imaging systems and computer aids: a tutorial review. Acad Radiol. 2007;14:723–48

    PubMed  Google Scholar 

  52. Krupinski EA, Jiang Y. Evaluation of medical imaging systems. Med Phys. 2008 (in press)

  53. Gur D. Objectively measuring and comparing performance levels of diagnostic imaging systems and practices (editorial). Acad Radiol. 2007;14:641–2

    PubMed  Google Scholar 

  54. Gur D, Rockette HE, Good W, Slasky BS, Cooperstein LA, Straub WH, et al. Effect of observer instruction on ROC study of chest images. Invest Radiol. 1990;25:230–4

    Article  PubMed  CAS  Google Scholar 

  55. Kobayashi T, Xu X-W, MacMahon H, Metz CE, Doi K. Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on chest radiographs. Radiology. 1996;199:843–8

    PubMed  CAS  Google Scholar 

  56. Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New Engl J Med. 1978;299:926–30

    Article  PubMed  CAS  Google Scholar 

  57. Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39:207–15

    PubMed  CAS  Google Scholar 

  58. Revesz G, Kundel HL, Bonitatibus M. The effect of verification on the assessment of imaging techniques. Invest Radiol. 1983;18:194–8

    PubMed  CAS  Google Scholar 

  59. Gray R, Begg CB, Greenes RA. Construction of receiver operating characteristic curves when disease verification is subject to selection bias. Med Decis Making. 1984;4:151–64

    PubMed  CAS  Google Scholar 

  60. Begg CB, McNeil BJ. Assessment of radiologic tests: control of bias and other design considerations. Radiology. 1988;167:565–9

    PubMed  CAS  Google Scholar 

  61. Gur D, Rockette HE, Armfield DR, et al. Prevalence effect in a laboratory environment. Radiology. 2003;228:10–4

    PubMed  Google Scholar 

  62. Gur D, Bandos AI, Fuhrman CR, Klym AH, King JL, Rockette HE. The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol. 2007;14:49–53

    PubMed  Google Scholar 

  63. Rockette HE, Gur D, Metz CE. The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. Invest Radiol. 1992;27:169–72

    PubMed  CAS  Google Scholar 

  64. King JL, Britton CA, Gur D, Rockette HE, Davis PL. On the validity of continuous and discrete confidence rating scales in receiver operating characteristic studies. Invest Radiol. 1993;28:962–3

    PubMed  CAS  Google Scholar 

  65. Walsh SJ. Limitations to the robustness of binormal ROC curves: effects of model misspecification and location of decision thresholds on bias, precision, size and power. Stat Med. 1997;16:669–79

    PubMed  CAS  Google Scholar 

  66. Wagner RF, Beiden SV, Metz CE. Continuous vs. categorical data for ROC analysis: some quantitative considerations. Acad Radiol. 2001;8:328–34

    PubMed  CAS  Google Scholar 

  67. Hadjiiski L, Chan H-P, Sahiner B, Helvie MA, Roubidoux MA. Quasi-continuous and discrete confidence rating scales for observer performance studies: effects on ROC analysis. Acad Radiol. 2007;14:38–48

    PubMed  Google Scholar 

  68. Swets JA. Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychol Bull. 1986;99:181–98

    PubMed  CAS  Google Scholar 

  69. Hanley JA. The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis Making. 1988;8:197–203

    PubMed  CAS  Google Scholar 

  70. Hanley JA. The use of the “binormal” model for parametric ROC analysis of quantitative diagnostic tests. Stat Med. 1996;15:1575–85

    PubMed  CAS  Google Scholar 

  71. Dorfman DD, Alf E. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals—rating method data. J Math Psych. 1969;6:487–96

    Google Scholar 

  72. Grey DR, Morgan BJT. Some aspects of ROC curve-fitting: normal and logistic models. J Math Psych. 1972;9:128–39

    Google Scholar 

  73. Metz CE, Herman BA, Shen J-H. Maximum-likelihood estimation of ROC curves from continuously-distributed data. Stat Med. 1998 17:1033–53

    PubMed  CAS  Google Scholar 

  74. Dorfman DD, Berbaum KS. Degeneracy and discrete receiver operating characteristic rating data. Acad Radiol. 1995;2:907–15

    PubMed  CAS  Google Scholar 

  75. Dorfman DD, Berbaum KS, Metz CE, Lenth RV, Hanley JA, Dagga HA. Proper ROC analysis: the bigamma model. Acad Radiol. 1997;4:138–49

    PubMed  CAS  Google Scholar 

  76. Pan X, Metz CE. The “proper” binormal model: parametric ROC curve estimation with degenerate data. Acad Radiol. 1997;4:380–9

    PubMed  CAS  Google Scholar 

  77. Metz CE, Pan X. “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psych. 1999;43:1–33

    Google Scholar 

  78. Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Acad Radiol. 2007;14:814–29

    PubMed  Google Scholar 

  79. Tosteson A, Begg C. A general regression methodology for ROC curve estimation. Med Decis Making. 1988;8:204–15

    PubMed  CAS  Google Scholar 

  80. Toledano AY, Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Med. 1996;15:1807–26

    PubMed  CAS  Google Scholar 

  81. Hellmich M, Abrams KR, Jones DR, Lambert PC. A Bayesian approach to a general regression model for ROC curves. Med Decis Making. 1998;18:436–43

    PubMed  CAS  Google Scholar 

  82. Pepe MS. The statistical evaluation of medical tests for classification and prediction. New York: Oxford University Press; 2004

    Google Scholar 

  83. Metz CE. Quantification of failure to demonstrate statistical significance: the usefulness of confidence intervals. Invest Radiol. 1993;28:59–63

    PubMed  CAS  Google Scholar 

  84. Metz CE, Kronman HB. Statistical significance tests for binormal ROC curves. J Math Psych. 1980;22:218–43

    Google Scholar 

  85. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29–36

    PubMed  CAS  Google Scholar 

  86. McClish DK. Analyzing a portion of the ROC curve. Med Decis Making. 1989;9:190–5

    PubMed  CAS  Google Scholar 

  87. Jiang Y, Metz CE, Nishikawa RM. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 1996;201:745–50

    PubMed  CAS  Google Scholar 

  88. Halpern EJ, Alpert M, Krieger AM, Metz CE, Maidment AD. Comparisons of ROC curves on the basis of optimal operating points. Acad Radiol. 1996;3:245–53

    PubMed  CAS  Google Scholar 

  89. Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148:839–43

    PubMed  CAS  Google Scholar 

  90. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–45

    PubMed  CAS  Google Scholar 

  91. Wieand S, Gail MH, James BR, James KL. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika. 1989;76:585–92

    Google Scholar 

  92. Thompson ML, Zucchini W. On the statistical analysis of ROC curves. Stat Med. 1989;8:1277–90

    PubMed  CAS  Google Scholar 

  93. Zhou XH, Gatsonis CA. A simple method for comparing correlated ROC curves using incomplete data. Stat Med. 1996;15:1687–93

    PubMed  CAS  Google Scholar 

  94. Metz CE, Wang P-L, Kronman HB. A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Deconinck F, editor. Information processing in medical imaging. The Hague: Nijhoff; 1984. p. 432–445

    Google Scholar 

  95. Metz CE. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert D, Myers R, editors. Multiple regression analysis: applications in the health sciences. New York: American Institute of Physics; 1986. p. 365–384

    Google Scholar 

  96. Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC curve estimates obtained from partially-paired datasets. Med Decis Making. 1998;18:110–21

    PubMed  CAS  Google Scholar 

  97. Hajian-Tilaki KO, Hanley JA, Joseph L, Collet J-P. A comparison of parametric and nonparametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Making. 1997;17:94–102

    PubMed  CAS  Google Scholar 

  98. Hsieh F, Turnbull BW. Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat. 1996;24:25–40

    Google Scholar 

  99. Roe CA, Metz CE. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol. 1997;4:587–600

    PubMed  CAS  Google Scholar 

  100. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992;27:723–31

    PubMed  CAS  Google Scholar 

  101. Dorfman DD, Metz CE. Multi-reader multi-case ROC analysis: comments on Begg’s commentary. Acad Radiol. 1995;2 Suppl 1:S76

    Google Scholar 

  102. Dorfman DD, Berbaum KS, Lenth RV. Multireader, multicase receiver operating characteristic methodology: a bootstrap analysis. Acad Radiol. 1995;2:626–33

    Google Scholar 

  103. Roe CA, Metz CE. The Dorfman–Berbaum–Metz method for statistical analysis of multi-reader, multi-modality ROC data: validation by computer simulation. Acad Radiol. 1997;4:298–303

    PubMed  CAS  Google Scholar 

  104. Dorfman DD, Berbaum KS, Lenth RV, Chen Y-F, Donaghy BA. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol. 1998;5:591–602

    PubMed  CAS  Google Scholar 

  105. Hillis SL, Berbaum KS. Power estimation for the Dorfman–Berbaum–Metz method. Acad Radiol. 2004;11:1260–73

    PubMed  Google Scholar 

  106. Hillis SL, Berbaum KS. Monte Carlo validation of the Dorfman–Berbaum–Metz method using normalized pseudovalues and less data-based model simplification. Acad Radiol. 2005;12:1534–42

    PubMed  Google Scholar 

  107. Obuchowski NA, Rockette HE. Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: an ANOVA approach with dependent observations. Commun Stat Simul Comput. 1995;24:285–308

    Google Scholar 

  108. Obuchowski NA. Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol. 1995;2:522–9

    Google Scholar 

  109. Obuchowski NA. Sample size tables for receiver operating characteristic studies. Am J Roentgenol. 2000;175:603–8

    CAS  Google Scholar 

  110. Toledano AY, Gatsonis C. GEEs for ordinal categorical data: arbitrary patterns of missing responses and missingness in a key covariate. Biometrics. 1999;22:488–96

    Google Scholar 

  111. Beiden SV, Wagner RF, Campbell G. Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol. 2000;7:341–9

    PubMed  CAS  Google Scholar 

  112. Beiden SV, Wagner RF, Campbell G, Chan HP. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 2001;8:616–22

    PubMed  CAS  Google Scholar 

  113. Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y. Components-of-variance models for random-effects ROC analysis: the case of unequal variance structures across modalities. Acad Radiol. 2001;8:605–15

    PubMed  CAS  Google Scholar 

  114. Beiden SV, Wagner RF, Campbell G, Chan H-P. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 2001;8:616–22

    PubMed  CAS  Google Scholar 

  115. Obuchowski NA, Beiden SV, Berbaum KS, Hillis SL, Ishwaran H, Song HH, et al. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol. 2004;11:980–95

    PubMed  Google Scholar 

  116. Hillis SL, Obuchowski NA, Schartz KM, Berbaum KS. A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data. Stat Med. 2005;24:1579–607

    PubMed  Google Scholar 

  117. Hillis, SL: Sample size estimates for DBM MRMC based on analysis of published data. http://perception.radiology.uiowa.edu/SampleSize/tabid/182/Default.aspx

  118. Dorfman DD. RSCORE II. In: Swets JA, Pickett RM, editors. Evaluation of diagnostic systems: methods from signal detection theory. New York: Academic; 1982. p. 208–232

    Google Scholar 

  119. University of Chicago Receiver Operating Characteristic program software downloads. http://xray.bsd.uchicago.edu/krl/KRL_ROC/software_index6.htm

  120. University of Iowa Receiver Operating Characteristic program software downloads. http://perception.radiology.uiowa.edu/

  121. Cleveland Clinic Receiver Operating Characteristic program software downloads. http://www.bio.ri.ccf.org/html/obumrm.html

  122. Obuchowski, NA: Research activities: ROC analysis. http://www.bio.ri.ccf.org/html/rocanalysis.html

  123. Starr SJ, Metz CE, Lusted LB, Goodenough DJ. Visual detection and localization of radiographic images. Radiology. 1975;116:533–8

    PubMed  CAS  Google Scholar 

  124. Starr SJ, Metz CE, Lusted LB. Comments on generalization of receiver operating characteristic analysis to detection and localization tasks (Letter to the Editor). Phys Med Biol. 1977;22:376–9

    PubMed  CAS  Google Scholar 

  125. Swensson RG. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys. 1996;23:1709–25

    PubMed  CAS  Google Scholar 

  126. Egan JP, Greenberg GZ, Schulman AI. Operating characteristics, signal detection, and the method of free response. J Acoust Soc Am. 1961;33:993–1007

    Google Scholar 

  127. International Atomic Energy Agency. IAEA co-ordinated research programme on the intercomparison of computer-assisted scintigraphic techniques: third progress report. In: Medical radionuclide imaging, vol. 1. Vienna: IAEA; 1977. p. 585–615

  128. Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free response approach to the measurement and characterization of radiographic observer performance. Proc SPIE. 1977;127:124–35

    Google Scholar 

  129. Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free response approach to the measurement and characterization of radiographic observer performance. J Appl Photogr Eng. 1978;4:166–72

    Google Scholar 

  130. Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys. 1989;16:561–8

    PubMed  CAS  Google Scholar 

  131. Chakraborty DP, Winter LHL. Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology. 1990;33:873–81

    Google Scholar 

  132. Obuchowski NA, Lieber ML, Powell KA. Data analysis for detection and localization of multiple abnormalities with application to mammography. Acad Radiol. 2000;7:516–25

    PubMed  CAS  Google Scholar 

  133. Chakraborty DP. Statistical power in observer performance studies: a comparison of the ROC and free-response methods in tasks involving localization. Acad Radiol. 2002;9:147–56

    PubMed  Google Scholar 

  134. Edwards DC, Kupinski MA, Metz CE, Nishikawa RM. Maximum-likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys. 2002;29:2861–70

    PubMed  Google Scholar 

  135. Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys. 2004;31:2313–30

    PubMed  Google Scholar 

  136. Chakraborty DP. A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol. 2006;51:3449–62

    PubMed  CAS  Google Scholar 

  137. Chakraborty DP. Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol. 2006;13:1187–93

    PubMed  Google Scholar 

  138. Chakraborty D, Yoon H-J, Mello-Thoms C. Spatial localization accuracy of radiologists in free-response studies: inferring perceptual FROC curves from mark-rating data. Acad Radiol. 2007;14:4–18

    PubMed  Google Scholar 

  139. Edwards DC, Metz CE, Kupinski MA. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans Med Imaging. 2004;23:891–5

    PubMed  Google Scholar 

  140. Edwards DC, Metz CE, Nishikawa RM. The hypervolume under the ROC hypersurface of “near-guessing” and “near-perfect” observers in N-class classification tasks. IEEE Trans Med Imaging. 2005;24:293–9

    PubMed  Google Scholar 

  141. Edwards DC, Lan L, Metz CE, Giger ML, Nishikawa RM. Estimating three-class ideal observer decision variables for computerized detection and classification of mammographic mass lesions. Med Phys. 2004;31:81–90

    PubMed  Google Scholar 

  142. Edwards DC, Metz CE. Review of several proposed three-class classification decision rules and their relation to the ideal observer decision rule. Proc SPIE. 2005;5749:128–37

    Google Scholar 

  143. Edwards DC, Metz CE. Restrictions on the three-class ideal observer’s decision boundary lines. IEEE Trans Med Imaging. 2005;24:1566–73

    PubMed  Google Scholar 

  144. Edwards DC, Metz CE. Analysis of proposed three-class classification decision rules in terms of the ideal observer decision rule. J Math Psych. 2006;50:478–87

    Google Scholar 

  145. He X, Metz CE, Tsui BMW, Links JM, Frey EC. Three-class ROC analysis—I: a decision theoretic approach under the ideal observer framework. IEEE Trans Med Imaging. 2006;25:571–81

    PubMed  Google Scholar 

  146. He X, Fry EC. An optimal three-class linear observer derived from decision theory. IEEE Trans Med Imaging. 2007;26:77–83

    PubMed  Google Scholar 

  147. Chan H-P, Sahiner B, Hadjiiski LM, Petrick N, Zhou C. Design of three-class classifiers in computer-aided diagnosis: Monte Carlo simulation study. Proc SPIE. 2003;5032:567–78

    Google Scholar 

  148. Sahiner B, Chan H-P, Hadjiiski LM. Performance analysis of 3-class classifiers: properties of the 3D ROC surface and the normalized volume under the surface. Proc SPIE. 2006;6146:87–93

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Radiology and Medical Physics, The University of Chicago, Chicago, IL, USA

    Charles E. Metz

Authors
  1. Charles E. Metz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Charles E. Metz.

About this article

Cite this article

Metz, C.E. ROC analysis in medical imaging: a tutorial review of the literature. Radiol Phys Technol 1, 2–12 (2008). https://doi.org/10.1007/s12194-007-0002-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12194-007-0002-1

Keywords

Search

Navigation