Evaluation of Medical Images

  • Charles E. Metz
Conference paper
Part of the NATO ASI Series book series (volume 98)


From a practical point of view, image quality in medicine must be defined in terms of the decisons that physicians can make by reading the images. Image-based decisions concerning the actual state of an object or patient are evaluated most meaningfully by ROC analysis, which has been used to quantify detection performance in sensory psychophysics since the early 1960s and more recently has been applied to evaluate diagnostic systems. After surveying briefly some considerations that motivate the use of ROC analysis in medical image evauation, this manuscript addresses various practical issues of experimental design and data analysis, including the selection of cases and observers, the need for standards of truth, reading-order effects, data collection, curve fitting, statistical testing, and possible generalizations of conventional ROC methodology.


Receiver Operating Characteristic Receiver Operating Characteristic Curve Receiver Operating Characteristic Analysis False Positive Fraction True Positive Fraction 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Begg, C.B., and McNeil, B.J. (1988). Assessment of radiologic tests: control of bias and other design considerations, Radiology 167, pp. 565–569.PubMedGoogle Scholar
  2. Bunch, P.C., Hamilton, J.F., Sanderson, G.K., and Simmons, A.H. (1977). A free response approach to the measurement and characterization of radiographic observer performance. Proc. SPIE 127, pp. 124–135.Google Scholar
  3. Chakraborty, D.P. (1988). Maximum likelihood analysis of free response receiver operating characteristic (FROC) data. Personal communication.Google Scholar
  4. Dorfman, D.D., and Alf, E. (1969). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals — rating method data, J. Math. Psychol. 6, pp. 487–496.CrossRefGoogle Scholar
  5. Egan, J.P. (1975). Signal detection theory and ROC analysis, Academic Press, New York.Google Scholar
  6. Grey, D.R., and Morgan, B.J.T. (1972). Some aspects of ROC curve-fitting: normal and logistic models, J. Math. Psychol. 9, pp. 128–139.CrossRefGoogle Scholar
  7. Getty, D.J., Pickett, R.M., D’Orsi, C.J., and Swets, J.A. (1988). Enhanced interpretation of diagnostic images, Invest. Radiol. 23, pp. 240–252.PubMedCrossRefGoogle Scholar
  8. Goodenough, D.J., and Metz, C.E. (1977). Implications of a “noisy” observer to data processing techniques. In: Information processing in medical imaging, C. Raynaud and A.E. Todd-Pokropek (eds.), CEA, Orsay, pp. 400–419.Google Scholar
  9. Gray, R., Begg, C.B., and Greenes, R.A. (1984). Construction of receiver operating characteristic curves when disease verification is subject to selection bias, Med. Decis. Making 4, pp. 151–164.PubMedCrossRefGoogle Scholar
  10. Green, D.M., and Swets, J.A. (1966). Signal detection theory and psychophysics, Wiley, New York. Reprint with corrections: (1974). Krieger, Huntington NY.Google Scholar
  11. Hanley, J.A., and McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology 143, pp. 29–36.PubMedGoogle Scholar
  12. Hanley, J.A., and McNeil, B.J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases, Radiology 148, pp. 839–843.PubMedGoogle Scholar
  13. International Atomic Energy Agency (1977). IAEA co-ordinated research programme on the intercomparison of computer-assisted scinigraphic techniques. In: Medical radionuclide imaging, IAEA, Vienna, 1, pp. 585–615.Google Scholar
  14. Kendall, M., and Stewart, A. (1976). The advanced thoery of statistics, Hafner, New York, 3, pp. 182–243.Google Scholar
  15. Lusted, L.B. (1978). General problems in medical decision making, with comments on ROC analysis, Seminars Nucl. Med. 8, pp. 299–306.CrossRefGoogle Scholar
  16. Loo, L.-N., Doi, K., Ishida, M., Metz, C.E., Chan, H.-P., Higashida, Y., and Kodera, Y. (1983). An empirical investigation of variability in contrast-detail diagram measurements, Proc. SPIE 419, pp. 68–76.Google Scholar
  17. MacMillan, N.A., and Kaplan, H.L. (1985). Detection theory analysis of group data: estimating sensitivity from average hit and false alarm rates, Psychol. Bull. 98, pp. 185–199.PubMedCrossRefGoogle Scholar
  18. McNeil, B.J., Keeler, E., and Adelstein, S.J. (1975). Primer on certain elements of medical decision making, N. Engl. J. Med. 293, pp. 211–215.PubMedCrossRefGoogle Scholar
  19. McNeil, B.J., and Hanley, J.A. (1984). Statistical approaches to the analysis of receiver operating characteristic (ROC) curves, Med. Decis. Making 4, pp. 137–150.PubMedCrossRefGoogle Scholar
  20. Metz, C.E. (1978). Basic principles of ROC analysis, Seminars Nucl. Med. 8, pp. 283–298.CrossRefGoogle Scholar
  21. Metz, C.E. (1979). Applications of ROC analysis in diagnostic image evaluation. In: The physics of medical imaging: recording system measurements and techniques, A. Haus (ed.), Am. Inst. Phys., New York, pp. 546–572.Google Scholar
  22. Metz, C.E. (1986a). ROC methodology in radiologic imaging, Invest. Radiol. 21, pp. 720–733.PubMedCrossRefGoogle Scholar
  23. Metz, C.E. (1986b). Statistical analysis of ROC data in evaluating diagnostic performance. In: Multiple regression analysis: applications in the health sciences, D. Herbert and R. Myers (eds.), Am. Inst. Phys., New York, pp. 365–384.Google Scholar
  24. Metz, C.E. (1988). Some practical issues of experimental design and data analysis in radiological ROC syudies, Invest. Radiol. (in press).Google Scholar
  25. Metz, C.E., and Kronman, H.B. (1980). Statistical significance tests for binormal ROC curves, J. Math. Psychol. 22, pp. 218–243.CrossRefGoogle Scholar
  26. Metz, C.E., Kronman, H.B., Wang, P.-L., and Shen, J.-H. (1985). ROCFIT: a modified maximum likelihood algorithm for estimating a binormal ROC curve from confidence-rating data, Dept. Radiology, Univ. Chicago, Chicago. Mini-and micro-computer versions of this and other software for ROC analysis are available from the author at no cost.Google Scholar
  27. Metz, C.E., Starr, S.J., Lusted, L.B., and Rossmann, K. (1975). Progress in evaluation of human observer visual detection performance using the ROC curve approach. In: Information processing in medical imaging, C. Raynaud and A.E. Todd-Pokropek (eds.), CEA, Orsay, pp. 420–436.Google Scholar
  28. Metz, C.E., Wang, P.-L., and Kronman, H.B. (1984). A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Information processing in medical imaging, F. Deconinck (ed.), Nijhoff, The Hague, pp. 432–445.CrossRefGoogle Scholar
  29. Ransohoff, D.F., and Feinstein, A.R., (1978). Problems of spectrum and bias in evaluating the efficacy of diagnostic tests, N. Engl. J. Med. 299, pp. 926–930.PubMedCrossRefGoogle Scholar
  30. Starr, S.J., Metz, C.E., Lusted, L.B., and Goodenough, D.J. (1975). Visual detection and localization of radiographic images, Radiology 116, pp. 533–538.PubMedGoogle Scholar
  31. Swets, J.A. (1979). ROC analysis applied to the evaluation of medical imaging techniques, Invest. Radiol. 14, pp. 109–121.PubMedCrossRefGoogle Scholar
  32. Swets, J.A. (1986a). Indices of discrimination or diagnostic accuracy: their ROCs and implied models, Psychol. Bull. 99, pp. 100–117.PubMedCrossRefGoogle Scholar
  33. Swets, J.A. (1986b). Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurementof performance, Psychol. Bull. 99, pp. 181–198.PubMedCrossRefGoogle Scholar
  34. Swets, J.A., and Pickett, R.M. (1982). Evaluation of diagnostic systems: methods from signal detection theory, Academic Press, New York.Google Scholar
  35. Weinstein, M.C., and Feinberg, H.V. (1980). Clinical decision analysis, Saunders, Philadelphia.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1992

Authors and Affiliations

  • Charles E. Metz
    • 1
  1. 1.Department of RadiologyThe University of ChicagoUSA

Personalised recommendations