ROC analysis in medical imaging: a tutorial review of the literature
 Charles E. Metz
 … show all 1 hide
Abstract
Receiver operating characteristic (ROC) analysis measures the “diagnostic accuracy” of a medical imaging system, which represents the second level of diagnostic efficacy in the hierarchical model described by Fryback and Thornbury (Med Decis Making 11:88–94, 1991). After describing the historical origins of ROC analysis, this paper reviews the importance of sampling cases appropriately, designing an observer study to avoid bias, and collecting data on a useful scale. A variety of methods for fitting ROC curves to observer data and testing the statistical significance of apparent differences are then reported. Finally, generalized forms of ROC analysis that require lesion localization or allow more than two states of truth are surveyed briefly.
 Fryback, DG, Thornbury, JR. (1991) The efficacy of diagnostic imaging. Med Decis Making. 11: pp. 8894
 Metz, CE. (1978) Basic principles of ROC analysis. Semin Nucl Med. 8: pp. 28398
 Wald, A. (1950) Statistical decision functions. Wiley, New York
 Egan, JP. (1975) Signal detection theory and ROC analysis. Academic, New York
 Meter, D, Middleton, D. (1954) Modern statistical approaches to reception in communication theory. IRE Trans. PGIT4: pp. 11941
 Peterson, WW, Birdsall, TG, Fox, WC. (1954) The theory of signal detectability. IRE Trans. PGIT4: pp. 171212
 Tanner, WP, Swets, JA. (1954) A decisionmaking theory of visual detection. Psych Rev. 61: pp. 4019
 Swets, JA, Birdsall, TG. (1956) Human use of information III: decisionmaking in signal detection and recognition situations involving multiple alternatives. IEEE Trans Inf Theory. IT2: pp. 13865
 Swets, JA, Tanner, WP, Birdsall, TG. (1961) Decision processes in perception. Psych Rev. 68: pp. 30140
 Swets, JA eds. (1964) Signal detection and recognition by human observers: contemporary readings. Wiley, New York
 Green, DM, Swets, JA. (1966.) Signal detection theory and psychophysics. Wiley, New York
 Swets, JA. (1973) The relative operating characteristic in psychology. Science. 182: pp. 9901000
 Swets JA, Green DM. Applications of signal detection theory. In: Pick HA, Liebowitz HL, Singer A, et al. editors. Psychology: from research to practice. New York: Plenum; 1978. p. 311–331
 Swets, JA. (1969) Effectiveness of information retrieval methods. Am Doc. 20: pp. 7289
 Griner, PF, Mayewski, RJ, Mushlin, AI, Greenland, P. (1981) Selection and interpretation of diagnostic tests and procedures: principles and applications. Ann Intern Med. 94: pp. 55392
 Swets, JA. (1983) Assessment of NDT systems (Parts I and II). Mater Eval. 41: pp. 1294303
 Robertson, EA, Zweig, MH, Steirtghem, AC. (1983) Evaluating the clinical efficacy of laboratory tests. Am J Clin Path. 79: pp. 7886
 Zweig, MH, Campbell, G. (1993) Receiveroperating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 39: pp. 56177
 Swets, JA. (1996) Signal detection theory and ROC analysis in psychology and diagnostics: collected papers. Lawrence Erlbaum Associates, Mahwah
 Lusted LB. Personal communication in conversations with CE Metz. circa 1975
 Lusted, LB. (1960) Logical analysis in roentgen diagnosis. Radiology. 74: pp. 17893
 Lusted, LB. (1968) Introduction to medical decision making. Thomas, Springfield
 Lusted, LB. (1971) Decisionmaking studies in patient management. New Engl J Med. 284: pp. 41624 CrossRef
 Lusted, LB. (1971) Signal detectability and medical decisionmaking. Science. 171: pp. 12179
 Lusted, LB. Observer error, signal detectability, medical decision making. In: Jacquez, JA eds. (1972.) Computer diagnosis and diagnostic methods. Thomas, Springfield, pp. 2944
 Lusted, LB. Receiver operating chararcteristic analysis and its significance in interpretation of radiologic images. In: Potchen, E eds. (1975.) Current concepts in radiology. Mosby, St Louis, pp. 117130
 Lusted, LB. (1978) General problems in medical decision making, with comments on ROC analysis. Semin Nucl Med. 8: pp. 299306
 Goodenough, DJ, Rossmann, K, Lusted, LB. (1972) Radiographic applications of signal detection theory. Radiology. 105: pp. 199200
 Goodenough, DJ, Rossmann, K, Lusted, LB. (1973) Factors affecting the detectability of a simulated radiographic signal. Invest Radiol. 8: pp. 33944
 Goodenough, DJ, Rossmann, K, Lusted, LB. (1974) Radiographic applications of receiver operating characteristic (ROC) analysis. Radiology. 110: pp. 8995
 Swets, JA. Signal detection in medical diagnosis. In: Jacquez, JA eds. (1972.) Computer diagnosis and diagnostic methods. Thomas, Springfield, pp. 828
 Morgan, RH, Donner, MW, Gayler, BW (1973) Decision processes and observer error in the diagnosis of pneumoconiosis by chest roentgenography. Am J Roentgenol. 117: pp. 75764
 Kundel, HL, Revesz, G. (1974) The evaluation of radiograghic techniques by observer tests: problems, pitfalls and procedures. Invest Radiol. 9: pp. 16673
 Metz, CE, Starr, SJ, Lusted, LB, Rossmann, K. Progress in evaluation of human observer visual detection performance using the ROC curve approach. In: Raynaud, C, ToddPokropek, AE eds. (1975.) Information processing in scintigraphy. Commissariat à l’Energie Atomique, Département de Biologie, Service Hospitalier Frédéric Joliot, Orsay, France, pp. 420439
 Metz, CE, Starr, SJ, Lusted, LB. Quantitative evaluation of visual detection performance in medicine: ROC analysis and determination of diagnostic benefit. In: Hay, GA eds. (1977.) Medical images: formation, perception and measurement. Wiley, London, pp. 220240
 Andrus, WS, Bird, KT. (1975) Radiology and the receiver operating characteristic (ROC) curve. Chest. 67: pp. 3789
 McNeil, BJ, Keeler, E, Adelstein, SJ. (1975) Primer on certain elements of medical decision making. New Engl J Med. 293: pp. 2115 CrossRef
 Turner, DA. (1978) An intuitive approach to receiver operating chararcteristic curve analysis. J Nucl Med. 19: pp. 21320
 Swets, JA. (1979) ROC analysis applied to the evaluation of medical imaging techniques. Invest Radiol. 14: pp. 10921
 Swets, JA, Pickett, RM. (1982) Evaluation of diagnostic systems: methods from signal detection theory. Academic, New York
 Metz, CE. (1986) ROC methodology in radiologic imaging. Invest Radiol. 21: pp. 72033
 Swets, JA. (1988) Measuring the accuracy of diagnostic systems. Science. 240: pp. 128593
 Hanley, JA. (1989) Receiver operating characteristic (ROC) methodology: the state of the art. CRC Crit Rev Diagn Imaging. 29: pp. 30735
 Metz, CE. (1989) Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 24: pp. 23445
 Metz, CE, Wagner, RF, Doi, K, Brown, DG, Nishikawa, RN, Myers, KJ. (1995) Toward consensus on quantitative assessment of medical imaging systems. Med Phys. 22: pp. 105761
 Allisy A, (editor). Medical imaging—the assessment of image quality. ICRU report #54. Bethesda: International Commission for Radiation Units and Measurements, Inc.; 1996
 Metz, CE. Evaluation of CAD. In: Doi, K, MacMahon, H, Giger, ML, Hoffmann, KR eds. (1999.) Computeraided diagnosis in medical imaging. Elsevier, Amsterdampp. 543
 Metz, CE. Fundamental ROC analysis. In: Beutel, J, Kundel, H, Metter, R eds. (2000.) Handbook of medical imaging, vol. 1: physics and psychophysics. SPIE Press, Bellinghampp. 751
 Wagner, RF, Beiden, SV, Campbell, G, Metz, CE, Sacks, WM. (2002) Assessment of medical imaging and computerassist systems: lessons from recent experience. Acad Radiol. 8: pp. 126477
 Metz, CE. (2006) Receiver operating characteristic (ROC) analysis: a tool for quantitative evaluation of observer performance and imaging systems. JACR. 3: pp. 41322
 Wagner, RF, Metz, CE, Campbell, G. (2007) Assessment of medial imaging systems and computer aids: a tutorial review. Acad Radiol. 14: pp. 72348
 Krupinski EA, Jiang Y. Evaluation of medical imaging systems. Med Phys. 2008 (in press)
 Gur, D. (2007) Objectively measuring and comparing performance levels of diagnostic imaging systems and practices (editorial). Acad Radiol. 14: pp. 6412
 Gur, D, Rockette, HE, Good, W, Slasky, BS, Cooperstein, LA, Straub, WH (1990) Effect of observer instruction on ROC study of chest images. Invest Radiol. 25: pp. 2304 CrossRef
 Kobayashi, T, Xu, XW, MacMahon, H, Metz, CE, Doi, K. (1996) Effect of a computeraided diagnosis scheme on radiologists’ performance in detection of lung nodules on chest radiographs. Radiology. 199: pp. 8438
 Ransohoff, DF, Feinstein, AR. (1978) Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. New Engl J Med. 299: pp. 92630 CrossRef
 Begg, CB, Greenes, RA. (1983) Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 39: pp. 20715
 Revesz, G, Kundel, HL, Bonitatibus, M. (1983) The effect of verification on the assessment of imaging techniques. Invest Radiol. 18: pp. 1948
 Gray, R, Begg, CB, Greenes, RA. (1984) Construction of receiver operating characteristic curves when disease verification is subject to selection bias. Med Decis Making. 4: pp. 15164
 Begg, CB, McNeil, BJ. (1988) Assessment of radiologic tests: control of bias and other design considerations. Radiology. 167: pp. 5659
 Gur, D, Rockette, HE, Armfield, DR (2003) Prevalence effect in a laboratory environment. Radiology. 228: pp. 104
 Gur, D, Bandos, AI, Fuhrman, CR, Klym, AH, King, JL, Rockette, HE. (2007) The prevalence effect in a laboratory environment: changing the confidence ratings. Acad Radiol. 14: pp. 4953
 Rockette, HE, Gur, D, Metz, CE. (1992) The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. Invest Radiol. 27: pp. 16972
 King, JL, Britton, CA, Gur, D, Rockette, HE, Davis, PL. (1993) On the validity of continuous and discrete confidence rating scales in receiver operating characteristic studies. Invest Radiol. 28: pp. 9623
 Walsh, SJ. (1997) Limitations to the robustness of binormal ROC curves: effects of model misspecification and location of decision thresholds on bias, precision, size and power. Stat Med. 16: pp. 66979
 Wagner, RF, Beiden, SV, Metz, CE. (2001) Continuous vs. categorical data for ROC analysis: some quantitative considerations. Acad Radiol. 8: pp. 32834
 Hadjiiski, L, Chan, HP, Sahiner, B, Helvie, MA, Roubidoux, MA. (2007) Quasicontinuous and discrete confidence rating scales for observer performance studies: effects on ROC analysis. Acad Radiol. 14: pp. 3848
 Swets, JA. (1986) Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurement of performance. Psychol Bull. 99: pp. 18198
 Hanley, JA. (1988) The robustness of the “binormal” assumptions used in fitting ROC curves. Med Decis Making. 8: pp. 197203
 Hanley, JA. (1996) The use of the “binormal” model for parametric ROC analysis of quantitative diagnostic tests. Stat Med. 15: pp. 157585
 Dorfman, DD, Alf, E. (1969) Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals—rating method data. J Math Psych. 6: pp. 48796
 Grey, DR, Morgan, BJT. (1972) Some aspects of ROC curvefitting: normal and logistic models. J Math Psych. 9: pp. 12839
 Metz, CE, Herman, BA, Shen, JH. (1998) Maximumlikelihood estimation of ROC curves from continuouslydistributed data. Stat Med. 17: pp. 103353
 Dorfman, DD, Berbaum, KS. (1995) Degeneracy and discrete receiver operating characteristic rating data. Acad Radiol. 2: pp. 90715
 Dorfman, DD, Berbaum, KS, Metz, CE, Lenth, RV, Hanley, JA, Dagga, HA. (1997) Proper ROC analysis: the bigamma model. Acad Radiol. 4: pp. 13849
 Pan, X, Metz, CE. (1997) The “proper” binormal model: parametric ROC curve estimation with degenerate data. Acad Radiol. 4: pp. 3809
 Metz, CE, Pan, X. (1999) “Proper” binormal ROC curves: theory and maximumlikelihood estimation. J Math Psych. 43: pp. 133
 Pesce, LL, Metz, CE. (2007) Reliable and computationally efficient maximumlikelihood estimation of “proper” binormal ROC curves. Acad Radiol. 14: pp. 81429
 Tosteson, A, Begg, C. (1988) A general regression methodology for ROC curve estimation. Med Decis Making. 8: pp. 20415
 Toledano, AY, Gatsonis, C. (1996) Ordinal regression methodology for ROC curves derived from correlated data. Stat Med. 15: pp. 180726
 Hellmich, M, Abrams, KR, Jones, DR, Lambert, PC. (1998) A Bayesian approach to a general regression model for ROC curves. Med Decis Making. 18: pp. 43643
 Pepe, MS. (2004) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, New York
 Metz, CE. (1993) Quantification of failure to demonstrate statistical significance: the usefulness of confidence intervals. Invest Radiol. 28: pp. 5963
 Metz, CE, Kronman, HB. (1980) Statistical significance tests for binormal ROC curves. J Math Psych. 22: pp. 21843
 Hanley, JA, McNeil, BJ. (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 143: pp. 2936
 McClish, DK. (1989) Analyzing a portion of the ROC curve. Med Decis Making. 9: pp. 1905
 Jiang, Y, Metz, CE, Nishikawa, RM. (1996) A receiver operating characteristic partial area index for highly sensitive diagnostic tests. Radiology. 201: pp. 74550
 Halpern, EJ, Alpert, M, Krieger, AM, Metz, CE, Maidment, AD. (1996) Comparisons of ROC curves on the basis of optimal operating points. Acad Radiol. 3: pp. 24553
 Hanley, JA, McNeil, BJ. (1983) A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology. 148: pp. 83943
 DeLong, ER, DeLong, DM, ClarkePearson, DL. (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 44: pp. 83745
 Wieand, S, Gail, MH, James, BR, James, KL. (1989) A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika. 76: pp. 58592
 Thompson, ML, Zucchini, W. (1989) On the statistical analysis of ROC curves. Stat Med. 8: pp. 127790
 Zhou, XH, Gatsonis, CA. (1996) A simple method for comparing correlated ROC curves using incomplete data. Stat Med. 15: pp. 168793
 Metz, CE, Wang, PL, Kronman, HB. A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Deconinck, F eds. (1984.) Information processing in medical imaging. Nijhoff, The Hague, pp. 432445
 Metz, CE. Statistical analysis of ROC data in evaluating diagnostic performance. In: Herbert, D, Myers, R eds. (1986.) Multiple regression analysis: applications in the health sciences. American Institute of Physics, New York, pp. 365384
 Metz, CE, Herman, BA, Roe, CA. (1998) Statistical comparison of two ROC curve estimates obtained from partiallypaired datasets. Med Decis Making. 18: pp. 11021
 HajianTilaki, KO, Hanley, JA, Joseph, L, Collet, JP. (1997) A comparison of parametric and nonparametric approaches to ROC analysis of quantitative diagnostic tests. Med Decis Making. 17: pp. 94102
 Hsieh, F, Turnbull, BW. (1996) Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat. 24: pp. 2540
 Roe, CA, Metz, CE. (1997) Variancecomponent modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol. 4: pp. 587600
 Dorfman, DD, Berbaum, KS, Metz, CE. (1992) Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol. 27: pp. 72331
 Dorfman, DD, Metz, CE. (1995) Multireader multicase ROC analysis: comments on Begg’s commentary. Acad Radiol. 2: pp. S76
 Dorfman, DD, Berbaum, KS, Lenth, RV. (1995) Multireader, multicase receiver operating characteristic methodology: a bootstrap analysis. Acad Radiol. 2: pp. 62633
 Roe, CA, Metz, CE. (1997) The Dorfman–Berbaum–Metz method for statistical analysis of multireader, multimodality ROC data: validation by computer simulation. Acad Radiol. 4: pp. 298303
 Dorfman, DD, Berbaum, KS, Lenth, RV, Chen, YF, Donaghy, BA. (1998) Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design. Acad Radiol. 5: pp. 591602
 Hillis, SL, Berbaum, KS. (2004) Power estimation for the Dorfman–Berbaum–Metz method. Acad Radiol. 11: pp. 126073
 Hillis, SL, Berbaum, KS. (2005) Monte Carlo validation of the Dorfman–Berbaum–Metz method using normalized pseudovalues and less databased model simplification. Acad Radiol. 12: pp. 153442
 Obuchowski, NA, Rockette, HE. (1995) Hypothesis testing of the diagnostic accuracy for multiple diagnostic tests: an ANOVA approach with dependent observations. Commun Stat Simul Comput. 24: pp. 285308
 Obuchowski, NA. (1995) Multireader, multimodality receiver operating characteristic curve studies: hypothesis testing and sample size estimation using an analysis of variance approach with dependent observations. Acad Radiol. 2: pp. 5229
 Obuchowski, NA. (2000) Sample size tables for receiver operating characteristic studies. Am J Roentgenol. 175: pp. 6038
 Toledano, AY, Gatsonis, C. (1999) GEEs for ordinal categorical data: arbitrary patterns of missing responses and missingness in a key covariate. Biometrics. 22: pp. 48896
 Beiden, SV, Wagner, RF, Campbell, G. (2000) Componentsofvariance models and multiplebootstrap experiments: an alternative method for randomeffects, receiver operating characteristic analysis. Acad Radiol. 7: pp. 3419
 Beiden, SV, Wagner, RF, Campbell, G, Chan, HP. (2001) Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 8: pp. 61622
 Beiden, SV, Wagner, RF, Campbell, G, Metz, CE, Jiang, Y. (2001) Componentsofvariance models for randomeffects ROC analysis: the case of unequal variance structures across modalities. Acad Radiol. 8: pp. 60515
 Beiden, SV, Wagner, RF, Campbell, G, Chan, HP. (2001) Analysis of uncertainties in estimates of components of variance in multivariate ROC analysis. Acad Radiol. 8: pp. 61622
 Obuchowski, NA, Beiden, SV, Berbaum, KS, Hillis, SL, Ishwaran, H, Song, HH (2004) Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods. Acad Radiol. 11: pp. 98095
 Hillis, SL, Obuchowski, NA, Schartz, KM, Berbaum, KS. (2005) A comparison of the Dorfman–Berbaum–Metz and Obuchowski–Rockette methods for receiver operating characteristic (ROC) data. Stat Med. 24: pp. 1579607
 Hillis, SL: Sample size estimates for DBM MRMC based on analysis of published data. http://perception.radiology.uiowa.edu/SampleSize/tabid/182/Default.aspx
 Dorfman, DD. RSCORE II. In: Swets, JA, Pickett, RM eds. (1982.) Evaluation of diagnostic systems: methods from signal detection theory. Academic, New York, pp. 208232
 University of Chicago Receiver Operating Characteristic program software downloads. http://xray.bsd.uchicago.edu/krl/KRL_ROC/software_index6.htm
 University of Iowa Receiver Operating Characteristic program software downloads. http://perception.radiology.uiowa.edu/
 Cleveland Clinic Receiver Operating Characteristic program software downloads. http://www.bio.ri.ccf.org/html/obumrm.html
 Obuchowski, NA: Research activities: ROC analysis. http://www.bio.ri.ccf.org/html/rocanalysis.html
 Starr, SJ, Metz, CE, Lusted, LB, Goodenough, DJ. (1975) Visual detection and localization of radiographic images. Radiology. 116: pp. 5338
 Starr, SJ, Metz, CE, Lusted, LB. (1977) Comments on generalization of receiver operating characteristic analysis to detection and localization tasks (Letter to the Editor). Phys Med Biol. 22: pp. 3769
 Swensson, RG. (1996) Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys. 23: pp. 170925
 Egan, JP, Greenberg, GZ, Schulman, AI. (1961) Operating characteristics, signal detection, and the method of free response. J Acoust Soc Am. 33: pp. 9931007
 International Atomic Energy Agency. IAEA coordinated research programme on the intercomparison of computerassisted scintigraphic techniques: third progress report. In: Medical radionuclide imaging, vol. 1. Vienna: IAEA; 1977. p. 585–615
 Bunch, PC, Hamilton, JF, Sanderson, GK, Simmons, AH. (1977) A free response approach to the measurement and characterization of radiographic observer performance. Proc SPIE. 127: pp. 12435
 Bunch, PC, Hamilton, JF, Sanderson, GK, Simmons, AH. (1978) A free response approach to the measurement and characterization of radiographic observer performance. J Appl Photogr Eng. 4: pp. 16672
 Chakraborty, DP. (1989) Maximum likelihood analysis of freeresponse receiver operating characteristic (FROC) data. Med Phys. 16: pp. 5618
 Chakraborty, DP, Winter, LHL. (1990) Freeresponse methodology: alternate analysis and a new observerperformance experiment. Radiology. 33: pp. 87381
 Obuchowski, NA, Lieber, ML, Powell, KA. (2000) Data analysis for detection and localization of multiple abnormalities with application to mammography. Acad Radiol. 7: pp. 51625
 Chakraborty, DP. (2002) Statistical power in observer performance studies: a comparison of the ROC and freeresponse methods in tasks involving localization. Acad Radiol. 9: pp. 14756
 Edwards, DC, Kupinski, MA, Metz, CE, Nishikawa, RM. (2002) Maximumlikelihood fitting of FROC curves under an initialdetectionandcandidateanalysis model. Med Phys. 29: pp. 286170
 Chakraborty, DP, Berbaum, KS. (2004) Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys. 31: pp. 231330
 Chakraborty, DP. (2006) A search model and figure of merit for observer data acquired according to the freeresponse paradigm. Phys Med Biol. 51: pp. 344962
 Chakraborty, DP. (2006) Analysis of location specific observer performance data: validated extensions of the jackknife freeresponse (JAFROC) method. Acad Radiol. 13: pp. 118793
 Chakraborty, D, Yoon, HJ, MelloThoms, C. (2007) Spatial localization accuracy of radiologists in freeresponse studies: inferring perceptual FROC curves from markrating data. Acad Radiol. 14: pp. 418
 Edwards, DC, Metz, CE, Kupinski, MA. (2004) Ideal observers and optimal ROC hypersurfaces in Nclass classification. IEEE Trans Med Imaging. 23: pp. 8915
 Edwards, DC, Metz, CE, Nishikawa, RM. (2005) The hypervolume under the ROC hypersurface of “nearguessing” and “nearperfect” observers in Nclass classification tasks. IEEE Trans Med Imaging. 24: pp. 2939
 Edwards, DC, Lan, L, Metz, CE, Giger, ML, Nishikawa, RM. (2004) Estimating threeclass ideal observer decision variables for computerized detection and classification of mammographic mass lesions. Med Phys. 31: pp. 8190
 Edwards, DC, Metz, CE. (2005) Review of several proposed threeclass classification decision rules and their relation to the ideal observer decision rule. Proc SPIE. 5749: pp. 12837
 Edwards, DC, Metz, CE. (2005) Restrictions on the threeclass ideal observer’s decision boundary lines. IEEE Trans Med Imaging. 24: pp. 156673
 Edwards, DC, Metz, CE. (2006) Analysis of proposed threeclass classification decision rules in terms of the ideal observer decision rule. J Math Psych. 50: pp. 47887
 He, X, Metz, CE, Tsui, BMW, Links, JM, Frey, EC. (2006) Threeclass ROC analysis. IEEE Trans Med Imaging. 25: pp. 57181
 He, X, Fry, EC. (2007) An optimal threeclass linear observer derived from decision theory. IEEE Trans Med Imaging. 26: pp. 7783
 Chan, HP, Sahiner, B, Hadjiiski, LM, Petrick, N, Zhou, C. (2003) Design of threeclass classifiers in computeraided diagnosis: Monte Carlo simulation study. Proc SPIE. 5032: pp. 56778
 Sahiner, B, Chan, HP, Hadjiiski, LM. (2006) Performance analysis of 3class classifiers: properties of the 3D ROC surface and the normalized volume under the surface. Proc SPIE. 6146: pp. 8793
 Title
 ROC analysis in medical imaging: a tutorial review of the literature
 Journal

Radiological Physics and Technology
Volume 1, Issue 1 , pp 212
 Cover Date
 20080101
 DOI
 10.1007/s1219400700021
 Print ISSN
 18650333
 Online ISSN
 18650341
 Publisher
 SpringerVerlag
 Additional Links
 Topics
 Keywords

 Receiver operating characteristic analysis
 ROC analysis
 Image evaluation
 Diagnostic accuracy
 Diagnostic efficacy
 Observer performance
 Industry Sectors
 Authors

 Charles E. Metz ^{(1)}
 Author Affiliations

 1. Radiology and Medical Physics, The University of Chicago, Chicago, IL, USA