Radiological Physics and Technology

, Volume 1, Issue 1, pp 2–12 | Cite as

ROC analysis in medical imaging: a tutorial review of the literature

Article
First Online:
Received:
Accepted:

Abstract

Receiver operating characteristic (ROC) analysis measures the “diagnostic accuracy” of a medical imaging system, which represents the second level of diagnostic efficacy in the hierarchical model described by Fryback and Thornbury (Med Decis Making 11:88–94, 1991). After describing the historical origins of ROC analysis, this paper reviews the importance of sampling cases appropriately, designing an observer study to avoid bias, and collecting data on a useful scale. A variety of methods for fitting ROC curves to observer data and testing the statistical significance of apparent differences are then reported. Finally, generalized forms of ROC analysis that require lesion localization or allow more than two states of truth are surveyed briefly.

Keywords

Receiver operating characteristic analysis ROC analysis Image evaluation Diagnostic accuracy Diagnostic efficacy Observer performance 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Med Decis Making. 1991;11:88–94PubMedGoogle Scholar
  2. 2.
    Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8:283–98PubMedGoogle Scholar
  3. 3.
    Wald A. Statistical decision functions. New York: Wiley; 1950Google Scholar
  4. 4.
    Egan JP. Signal detection theory and ROC analysis. New York: Academic; 1975Google Scholar
  5. 5.
    Van Meter D, Middleton D. Modern statistical approaches to reception in communication theory. IRE Trans. 1954;PGIT-4:119–41Google Scholar
  6. 6.
    Peterson WW, Birdsall TG, Fox WC. The theory of signal detectability. IRE Trans. 1954;PGIT-4:171–212Google Scholar
  7. 7.
    Tanner WP Jr, Swets JA. A decision-making theory of visual detection. Psych Rev. 1954;61:401–9Google Scholar
  8. 8.
    Swets JA, Birdsall TG. Human use of information III: decision-making in signal detection and recognition situations involving multiple alternatives. IEEE Trans Inf Theory. 1956;IT-2:138–65Google Scholar
  9. 9.
    Swets JA, Tanner WP Jr, Birdsall TG. Decision processes in perception. Psych Rev. 1961;68:301–40Google Scholar
  10. 10.
    Swets JA, (editor). Signal detection and recognition by human observers: contemporary readings. New York: Wiley; 1964Google Scholar
  11. 11.
    Green DM, Swets JA. Signal detection theory and psychophysics. New York: Wiley; 1966. [Reprinted with corrections and an updated topical bibliography by Krieger (Huntington, NY, 1974) and by Peninsula Publishing (Los Altos, CA, 1988)]Google Scholar
  12. 12.
    Swets JA. The relative operating characteristic in psychology. Science. 1973;182:990–1000PubMedGoogle Scholar
  13. 13.
    Swets JA, Green DM. Applications of signal detection theory. In: Pick HA, Liebowitz HL, Singer A, et al. editors. Psychology: from research to practice. New York: Plenum; 1978. p. 311–331Google Scholar
  14. 14.
    Swets JA. Effectiveness of information retrieval methods. Am Doc. 1969;20:72–89Google Scholar
  15. 15.
    Griner PF, Mayewski RJ, Mushlin AI, Greenland P. Selection and interpretation of diagnostic tests and procedures: principles and applications. Ann Intern Med. 1981;94:553–92Google Scholar
  16. 16.
    Swets JA. Assessment of NDT systems (Parts I and II). Mater Eval. 1983;41:1294–303Google Scholar
  17. 17.
    Robertson EA, Zweig MH, Van Steirtghem AC. Evaluating the clinical efficacy of laboratory tests. Am J Clin Path. 1983;79:78–86PubMedGoogle Scholar
  18. 18.
    Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39:561–77. (Erratum published in Clin Chem. 1993;39:1589.)PubMedGoogle Scholar
  19. 19.
    Swets JA. Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Mahwah: Lawrence Erlbaum Associates; 1996Google Scholar
  20. 20.
    Lusted LB. Personal communication in conversations with CE Metz. circa 1975Google Scholar
  21. 21.
    Lusted LB. Logical analysis in roentgen diagnosis. Radiology. 1960;74:178–93PubMedGoogle Scholar
  22. 22.
    Lusted LB. Introduction to medical decision making. Springfield: Thomas; 1968Google Scholar
  23. 23.
    Lusted LB. Decision-making studies in patient management. New Engl J Med. 1971;284:416–24PubMedCrossRefGoogle Scholar
  24. 24.
    Lusted LB. Signal detectability and medical decision-making. Science. 1971;171:1217–9PubMedGoogle Scholar
  25. 25.
    Lusted LB. Observer error, signal detectability, medical decision making. In: Jacquez JA, editor. Computer diagnosis and diagnostic methods. Springfield: Thomas; 1972. p. 29–44Google Scholar
  26. 26.
    Lusted LB. Receiver operating chararcteristic analysis and its significance in interpretation of radiologic images. In: Potchen E, editor. Current concepts in radiology. St Louis: Mosby; 1975. p. 117–130Google Scholar
  27. 27.
    Lusted LB. General problems in medical decision making, with comments on ROC analysis. Semin Nucl Med. 1978;8:299–306PubMedGoogle Scholar
  28. 28.
    Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of signal detection theory. Radiology. 1972;105:199–200PubMedGoogle Scholar
  29. 29.
    Goodenough DJ, Rossmann K, Lusted LB. Factors affecting the detectability of a simulated radiographic signal. Invest Radiol. 1973;8:339–44PubMedGoogle Scholar
  30. 30.
    Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of receiver operating characteristic (ROC) analysis. Radiology. 1974;110:89–95PubMedGoogle Scholar
  31. 31.
    Swets JA. Signal detection in medical diagnosis. In: Jacquez JA, editor. Computer diagnosis and diagnostic methods. Springfield: Thomas; 1972. p. 8–28Google Scholar
  32. 32.
    Morgan RH, Donner MW, Gayler BW, et al. Decision processes and observer error in the diagnosis of pneumoconiosis by chest roentgenography. Am J Roentgenol. 1973;117:757–64Google Scholar
  33. 33.
    Kundel HL, Revesz G. The evaluation of radiograghic techniques by observer tests: problems, pitfalls and procedures. Invest Radiol. 1974;9:166–73PubMedGoogle Scholar
  34. 34.
    Metz CE, Starr SJ, Lusted LB, Rossmann K. Progress in evaluation of human observer visual detection performance using the ROC curve approach. In: Raynaud C, Todd-Pokropek AE, editors. Information processing in scintigraphy. Orsay, France: Commissariat à l’Energie Atomique, Département de Biologie, Service Hospitalier Frédéric Joliot; 1975. p. 420–439Google Scholar
  35. 35.
    Metz CE, Starr SJ, Lusted LB. Quantitative evaluation of visual detection performance in medicine: ROC analysis and determination of diagnostic benefit. In: Hay GA, editor. Medical images: formation, perception and measurement. London: Wiley; 1977. p. 220–240Google Scholar
  36. 36.
    Andrus WS, Bird KT. Radiology and the receiver operating characteristic (ROC) curve. Chest. 1975;67:378–9PubMedGoogle Scholar
  37. 37.
    McNeil BJ, Keeler E, Adelstein SJ. Primer on certain elements of medical decision making. New Engl J Med. 1975;293:211–5PubMedCrossRefGoogle Scholar
  38. 38.
    Turner DA. An intuitive approach to receiver operating chararcteristic curve analysis. J Nucl Med. 1978;19:213–20PubMedGoogle Scholar
  39. 39.
    Swets JA. ROC analysis applied to the evaluation of medical imaging techniques. Invest Radiol. 1979;14:109–21PubMedGoogle Scholar
  40. 40.
    Swets JA, Pickett RM. Evaluation of diagnostic systems: methods from signal detection theory. New York: Academic; 1982Google Scholar
  41. 41.
    Metz CE. ROC methodology in radiologic imaging. Invest Radiol. 1986;21:720–33PubMedGoogle Scholar
  42. 42.
    Swets JA. Measuring the accuracy of diagnostic systems. Science. 1988;240:1285–93PubMedGoogle Scholar
  43. 43.
    Hanley JA. Receiver operating characteristic (ROC) methodology: the state of the art. CRC Crit Rev Diagn Imaging. 1989;29:307–35Google Scholar
  44. 44.
    Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24:234–45PubMedGoogle Scholar
  45. 45.
    Metz CE, Wagner RF, Doi K, Brown DG, Nishikawa RN, Myers KJ. Toward consensus on quantitative assessment of medical imaging systems. Med Phys. 1995;22:1057–61PubMedGoogle Scholar
  46. 46.
    Allisy A, (editor). Medical imaging—the assessment of image quality. ICRU report #54. Bethesda: International Commission for Radiation Units and Measurements, Inc.; 1996Google Scholar
  47. 47.
    Metz CE. Evaluation of CAD. In: Doi K, MacMahon H, Giger ML, Hoffmann KR, editors. Computer-aided diagnosis in medical imaging. Amsterdam: Elsevier; 1999. p. 543. (Excerpta Medica International Congress Series, vol. 1182)Google Scholar
  48. 48.
    Metz CE. Fundamental ROC analysis. In: Beutel J, Kundel H, Van Metter R, editors. Handbook of medical imaging, vol. 1: physics and psychophysics. Bellingham: SPIE Press; 2000. p. 751Google Scholar
  49. 49.
    Wagner RF, Beiden SV, Campbell G, Metz CE, Sacks WM. Assessment of medical imaging and computer-assist systems: lessons from recent experience. Acad Radiol. 2002;8:1264–77Google Scholar
  50. 50.
    Metz CE. Receiver operating characteristic (ROC) analysis: a tool for quantitative evaluation of observer performance and imaging systems. JACR. 2006;3:413–22PubMedGoogle Scholar
  51. 51.
    Wagner RF, Metz CE, Campbell G. Assessment of medial imaging systems and computer aids: a tutorial review. Acad Radiol. 2007;14:723–48PubMedGoogle Scholar
  52. 52.
    Krupinski EA, Jiang Y. Evaluation of medical imaging systems. Med Phys. 2008 (in press)Google Scholar
  53. 53.
    Gur D. Objectively measuring and comparing performance levels of diagnostic imaging systems and practices (editorial). Acad Radiol. 2007;14:641–2PubMedGoogle Scholar
  54. 54.
    Gur D, Rockette HE, Good W, Slasky BS, Cooperstein LA, Straub WH, et al. Effect of observer instruction on ROC study of chest images. Invest Radiol. 1990;25:230–4PubMedCrossRefGoogle Scholar
  55. 55.
    Kobayashi T, Xu X-W, MacMahon H, Metz CE, Doi K. Effect of a computer-aided diagnosis scheme on radiologists’ performance in detection of lung nodules on chest radiographs. Radiology. 1996;199:843–8PubMedGoogle Scholar
  56. 56.
    Ransohoff DF, Feinstein AR. Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New Engl J Med. 1978;299:926–30PubMedCrossRefGoogle Scholar
  57. 57.
    Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39:207–15PubMedGoogle Scholar
  58. 58.
    Revesz G, Kundel HL, Bonitatibus M. The effect of verification on the assessment of imaging techniques. Invest Radiol. 1983;18:194–8PubMedGoogle Scholar
  59. 59.
    Gray R, Begg CB, Greenes RA. Construction of receiver operating characteristic curves when disease verification is subject to selection bias. Med Decis Making. 1984;4:151–64PubMedGoogle Scholar
  60. 60.
    Begg CB, McNeil BJ. Assessment of radiologic tests: control of bias and other design considerations. Radiology. 1988;167:565–9PubMedGoogle Scholar
  61. 61.
    Gur D, Rockette HE, Armfield DR, et al. Prevalence effect in a laboratory environment. Radiology. 2003;228:10–4PubMedGoogle Scholar
  62. 62.
    Gur D, Bandos AI, Fuhrman CR, Klym AH, King JL, Rockette HE. The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol. 2007;14:49–53PubMedGoogle Scholar
  63. 63.
    Rockette HE, Gur D, Metz CE. The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. Invest Radiol. 1992;27:169–72PubMedGoogle Scholar
  64. 64.
    King JL, Britton CA, Gur D, Rockette HE, Davis PL. On the validity of continuous and discrete confidence rating scales in receiver operating characteristic studies. Invest Radiol. 1993;28:962–3PubMedGoogle Scholar
  65. 65.
    Walsh SJ. Limitations to the robustness of binormal ROC curves: effects of model misspecification and location of decision thresholds on bias, precision, size and power. Stat Med. 1997;16:669–79PubMedGoogle Scholar
  66. 66.
    Wagner RF, Beiden SV, Metz CE. Continuous vs. categorical data for ROC analysis: some quantitative considerations. Acad Radiol. 2001;8:328–34PubMedGoogle Scholar
  67. 67.
    Hadjiiski L, Chan H-P, Sahiner B, Helvie MA, Roubidoux MA. Quasi-continuous and discrete confidence rating scales for observer performance studies: effects on ROC analysis. Acad Radiol. 2007;14:38–48PubMedGoogle Scholar
  68. 68.
    Swets JA. Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychol Bull. 1986;99:181–98PubMedGoogle Scholar
  69. 69.
    Hanley JA. The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis Making. 1988;8:197–203PubMedGoogle Scholar
  70. 70.
    Hanley JA. The use of the “binormal” model for parametric ROC analysis of quantitative diagnostic tests. Stat Med. 1996;15:1575–85PubMedGoogle Scholar
  71. 71.
    Dorfman DD, Alf E. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals—rating method data. J Math Psych. 1969;6:487–96Google Scholar
  72. 72.
    Grey DR, Morgan BJT. Some aspects of ROC curve-fitting: normal and logistic models. J Math Psych. 1972;9:128–39Google Scholar
  73. 73.
    Metz CE, Herman BA, Shen J-H. Maximum-likelihood estimation of ROC curves from continuously-distributed data. Stat Med. 1998 17:1033–53PubMedGoogle Scholar
  74. 74.
    Dorfman DD, Berbaum KS. Degeneracy and discrete receiver operating characteristic rating data. Acad Radiol. 1995;2:907–15PubMedGoogle Scholar
  75. 75.
    Dorfman DD, Berbaum KS, Metz CE, Lenth RV, Hanley JA, Dagga HA. Proper ROC analysis: the bigamma model. Acad Radiol. 1997;4:138–49PubMedGoogle Scholar
  76. 76.
    Pan X, Metz CE. The “proper” binormal model: parametric ROC curve estimation with degenerate data. Acad Radiol. 1997;4:380–9PubMedGoogle Scholar
  77. 77.
    Metz CE, Pan X. “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psych. 1999;43:1–33Google Scholar
  78. 78.
    Pesce LL, Metz CE. Reliable and computationally efficient maximum-likelihood estimation of “proper” binormal ROC curves. Acad Radiol. 2007;14:814–29PubMedGoogle Scholar
  79. 79.
    Tosteson A, Begg C. A general regression methodology for ROC curve estimation. Med Decis Making. 1988;8:204–15PubMedGoogle Scholar
  80. 80.
    Toledano AY, Gatsonis C. Ordinal regression methodology for ROC curves derived from correlated data. Stat Med. 1996;15:1807–26PubMedGoogle Scholar
  81. 81.
    Hellmich M, Abrams KR, Jones DR, Lambert PC. A Bayesian approach to a general regression model for ROC curves. Med Decis Making. 1998;18:436–43PubMedGoogle Scholar
  82. 82.
    Pepe MS. The statistical evaluation of medical tests for classification and prediction. New York: Oxford University Press; 2004Google Scholar
  83. 83.
    Metz CE. Quantification of failure to demonstrate statistical significance: the usefulness of confidence intervals. Invest Radiol. 1993;28:59–63PubMedGoogle Scholar
  84. 84.
    Metz CE, Kronman HB. Statistical significance tests for binormal ROC curves. J Math Psych. 1980;22:218–43Google Scholar
  85. 85.
    Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29–36PubMedGoogle Scholar
  86. 86.
    McClish DK. Analyzing a portion of the ROC curve. Med Decis Making. 1989;9:190–5PubMedGoogle Scholar
  87. 87.
    Jiang Y, Metz CE, Nishikawa RM. A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 1996;201:745–50PubMedGoogle Scholar
  88. 88.
    Halpern EJ, Alpert M, Krieger AM, Metz CE, Maidment AD. Comparisons of ROC curves on the basis of optimal operating points. Acad Radiol. 1996;3:245–53PubMedGoogle Scholar
  89. 89.
    Hanley JA, McNeil BJ. A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 1983;148:839–43PubMedGoogle Scholar
  90. 90.
    DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837–45PubMedGoogle Scholar
  91. 91.
    Wieand S, Gail MH, James BR, James KL. A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika. 1989;76:585–92Google Scholar
  92. 92.
    Thompson ML, Zucchini W. On the statistical analysis of ROC curves. Stat Med. 1989;8:1277–90PubMedGoogle Scholar
  93. 93.
    Zhou XH, Gatsonis CA. A simple method for comparing correlated ROC curves using incomplete data. Stat Med. 1996;15:1687–93PubMedGoogle Scholar
  94. 94.
    Metz CE, Wang P-L, Kronman HB. A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Deconinck F, editor. Information processing in medical imaging. The Hague: Nijhoff; 1984. p. 432–445Google Scholar
  95. 95.
    Metz CE. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert D, Myers R, editors. Multiple regression analysis: applications in the health sciences. New York: American Institute of Physics; 1986. p. 365–384Google Scholar
  96. 96.
    Metz CE, Herman BA, Roe CA. Statistical comparison of two ROC curve estimates obtained from partially-paired datasets. Med Decis Making. 1998;18:110–21PubMedGoogle Scholar
  97. 97.
    Hajian-Tilaki KO, Hanley JA, Joseph L, Collet J-P. A comparison of parametric and nonparametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Making. 1997;17:94–102PubMedGoogle Scholar
  98. 98.
    Hsieh F, Turnbull BW. Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat. 1996;24:25–40Google Scholar
  99. 99.
    Roe CA, Metz CE. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol. 1997;4:587–600PubMedGoogle Scholar
  100. 100.
    Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992;27:723–31PubMedGoogle Scholar
  101. 101.
    Dorfman DD, Metz CE. Multi-reader multi-case ROC analysis: comments on Begg’s commentary. Acad Radiol. 1995;2 Suppl 1:S76Google Scholar
  102. 102.
    Dorfman DD, Berbaum KS, Lenth RV. Multireader, multicase receiver operating characteristic methodology: a bootstrap analysis. Acad Radiol. 1995;2:626–33Google Scholar
  103. 103.
    Roe CA, Metz CE. The Dorfman–Berbaum–Metz method for statistical analysis of multi-reader, multi-modality ROC data: validation by computer simulation. Acad Radiol. 1997;4:298–303PubMedGoogle Scholar
  104. 104.
    Dorfman DD, Berbaum KS, Lenth RV, Chen Y-F, Donaghy BA. Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol. 1998;5:591–602PubMedGoogle Scholar
  105. 105.
    Hillis SL, Berbaum KS. Power estimation for the Dorfman–Berbaum–Metz method. Acad Radiol. 2004;11:1260–73PubMedGoogle Scholar
  106. 106.
    Hillis SL, Berbaum KS. Monte Carlo validation of the Dorfman–Berbaum–Metz method using normalized pseudovalues and less data-based model simplification. Acad Radiol. 2005;12:1534–42PubMedGoogle Scholar
  107. 107.
    Obuchowski NA, Rockette HE. Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: an ANOVA approach with dependent observations. Commun Stat Simul Comput. 1995;24:285–308Google Scholar
  108. 108.
    Obuchowski NA. Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol. 1995;2:522–9Google Scholar
  109. 109.
    Obuchowski NA. Sample size tables for receiver operating characteristic studies. Am J Roentgenol. 2000;175:603–8Google Scholar
  110. 110.
    Toledano AY, Gatsonis C. GEEs for ordinal categorical data: arbitrary patterns of missing responses and missingness in a key covariate. Biometrics. 1999;22:488–96Google Scholar
  111. 111.
    Beiden SV, Wagner RF, Campbell G. Components-of-variance models and multiple-bootstrap experiments: an alternative method for random-effects, receiver operating characteristic analysis. Acad Radiol. 2000;7:341–9PubMedGoogle Scholar
  112. 112.
    Beiden SV, Wagner RF, Campbell G, Chan HP. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 2001;8:616–22PubMedGoogle Scholar
  113. 113.
    Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y. Components-of-variance models for random-effects ROC analysis: the case of unequal variance structures across modalities. Acad Radiol. 2001;8:605–15PubMedGoogle Scholar
  114. 114.
    Beiden SV, Wagner RF, Campbell G, Chan H-P. Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 2001;8:616–22PubMedGoogle Scholar
  115. 115.
    Obuchowski NA, Beiden SV, Berbaum KS, Hillis SL, Ishwaran H, Song HH, et al. Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol. 2004;11:980–95PubMedGoogle Scholar
  116. 116.
    Hillis SL, Obuchowski NA, Schartz KM, Berbaum KS. A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data. Stat Med. 2005;24:1579–607PubMedGoogle Scholar
  117. 117.
    Hillis, SL: Sample size estimates for DBM MRMC based on analysis of published data. http://perception.radiology.uiowa.edu/SampleSize/tabid/182/Default.aspx
  118. 118.
    Dorfman DD. RSCORE II. In: Swets JA, Pickett RM, editors. Evaluation of diagnostic systems: methods from signal detection theory. New York: Academic; 1982. p. 208–232Google Scholar
  119. 119.
    University of Chicago Receiver Operating Characteristic program software downloads. http://xray.bsd.uchicago.edu/krl/KRL_ROC/software_index6.htm
  120. 120.
    University of Iowa Receiver Operating Characteristic program software downloads. http://perception.radiology.uiowa.edu/
  121. 121.
    Cleveland Clinic Receiver Operating Characteristic program software downloads. http://www.bio.ri.ccf.org/html/obumrm.html
  122. 122.
    Obuchowski, NA: Research activities: ROC analysis. http://www.bio.ri.ccf.org/html/rocanalysis.html
  123. 123.
    Starr SJ, Metz CE, Lusted LB, Goodenough DJ. Visual detection and localization of radiographic images. Radiology. 1975;116:533–8PubMedGoogle Scholar
  124. 124.
    Starr SJ, Metz CE, Lusted LB. Comments on generalization of receiver operating characteristic analysis to detection and localization tasks (Letter to the Editor). Phys Med Biol. 1977;22:376–9PubMedGoogle Scholar
  125. 125.
    Swensson RG. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys. 1996;23:1709–25PubMedGoogle Scholar
  126. 126.
    Egan JP, Greenberg GZ, Schulman AI. Operating characteristics, signal detection, and the method of free response. J Acoust Soc Am. 1961;33:993–1007Google Scholar
  127. 127.
    International Atomic Energy Agency. IAEA co-ordinated research programme on the intercomparison of computer-assisted scintigraphic techniques: third progress report. In: Medical radionuclide imaging, vol. 1. Vienna: IAEA; 1977. p. 585–615Google Scholar
  128. 128.
    Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free response approach to the measurement and characterization of radiographic observer performance. Proc SPIE. 1977;127:124–35Google Scholar
  129. 129.
    Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free response approach to the measurement and characterization of radiographic observer performance. J Appl Photogr Eng. 1978;4:166–72Google Scholar
  130. 130.
    Chakraborty DP. Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys. 1989;16:561–8PubMedGoogle Scholar
  131. 131.
    Chakraborty DP, Winter LHL. Free-response methodology: alternate analysis and a new observer-performance experiment. Radiology. 1990;33:873–81Google Scholar
  132. 132.
    Obuchowski NA, Lieber ML, Powell KA. Data analysis for detection and localization of multiple abnormalities with application to mammography. Acad Radiol. 2000;7:516–25PubMedGoogle Scholar
  133. 133.
    Chakraborty DP. Statistical power in observer performance studies: a comparison of the ROC and free-response methods in tasks involving localization. Acad Radiol. 2002;9:147–56PubMedGoogle Scholar
  134. 134.
    Edwards DC, Kupinski MA, Metz CE, Nishikawa RM. Maximum-likelihood fitting of FROC curves under an initial-detection-and-candidate-analysis model. Med Phys. 2002;29:2861–70PubMedGoogle Scholar
  135. 135.
    Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys. 2004;31:2313–30PubMedGoogle Scholar
  136. 136.
    Chakraborty DP. A search model and figure of merit for observer data acquired according to the free-response paradigm. Phys Med Biol. 2006;51:3449–62PubMedGoogle Scholar
  137. 137.
    Chakraborty DP. Analysis of location specific observer performance data: validated extensions of the jackknife free-response (JAFROC) method. Acad Radiol. 2006;13:1187–93PubMedGoogle Scholar
  138. 138.
    Chakraborty D, Yoon H-J, Mello-Thoms C. Spatial localization accuracy of radiologists in free-response studies: inferring perceptual FROC curves from mark-rating data. Acad Radiol. 2007;14:4–18PubMedGoogle Scholar
  139. 139.
    Edwards DC, Metz CE, Kupinski MA. Ideal observers and optimal ROC hypersurfaces in N-class classification. IEEE Trans Med Imaging. 2004;23:891–5PubMedGoogle Scholar
  140. 140.
    Edwards DC, Metz CE, Nishikawa RM. The hypervolume under the ROC hypersurface of “near-guessing” and “near-perfect” observers in N-class classification tasks. IEEE Trans Med Imaging. 2005;24:293–9PubMedGoogle Scholar
  141. 141.
    Edwards DC, Lan L, Metz CE, Giger ML, Nishikawa RM. Estimating three-class ideal observer decision variables for computerized detection and classification of mammographic mass lesions. Med Phys. 2004;31:81–90PubMedGoogle Scholar
  142. 142.
    Edwards DC, Metz CE. Review of several proposed three-class classification decision rules and their relation to the ideal observer decision rule. Proc SPIE. 2005;5749:128–37Google Scholar
  143. 143.
    Edwards DC, Metz CE. Restrictions on the three-class ideal observer’s decision boundary lines. IEEE Trans Med Imaging. 2005;24:1566–73PubMedGoogle Scholar
  144. 144.
    Edwards DC, Metz CE. Analysis of proposed three-class classification decision rules in terms of the ideal observer decision rule. J Math Psych. 2006;50:478–87Google Scholar
  145. 145.
    He X, Metz CE, Tsui BMW, Links JM, Frey EC. Three-class ROC analysis—I: a decision theoretic approach under the ideal observer framework. IEEE Trans Med Imaging. 2006;25:571–81PubMedGoogle Scholar
  146. 146.
    He X, Fry EC. An optimal three-class linear observer derived from decision theory. IEEE Trans Med Imaging. 2007;26:77–83PubMedGoogle Scholar
  147. 147.
    Chan H-P, Sahiner B, Hadjiiski LM, Petrick N, Zhou C. Design of three-class classifiers in computer-aided diagnosis: Monte Carlo simulation study. Proc SPIE. 2003;5032:567–78Google Scholar
  148. 148.
    Sahiner B, Chan H-P, Hadjiiski LM. Performance analysis of 3-class classifiers: properties of the 3D ROC surface and the normalized volume under the surface. Proc SPIE. 2006;6146:87–93Google Scholar

Copyright information

© Japanese Society of Radiological Technology and Japan Society of Medical Physics 2007

Authors and Affiliations

  1. 1.Radiology and Medical PhysicsThe University of ChicagoChicagoUSA

Personalised recommendations