## Abstract

From a practical point of view, image quality in medicine must be defined in terms of the decisons that physicians can make by reading the images. Image-based decisions concerning the actual state of an object or patient are evaluated most meaningfully by ROC analysis, which has been used to quantify detection performance in sensory psychophysics since the early 1960s and more recently has been applied to evaluate diagnostic systems. After surveying briefly some considerations that motivate the use of ROC analysis in medical image evauation, this manuscript addresses various practical issues of experimental design and data analysis, including the selection of cases and observers, the need for standards of truth, reading-order effects, data collection, curve fitting, statistical testing, and possible generalizations of conventional ROC methodology.

## Keywords

Receiver Operating Characteristic Receiver Operating Characteristic Curve Receiver Operating Characteristic Analysis False Positive Fraction True Positive Fraction## Preview

Unable to display preview. Download preview PDF.

## References

- Begg, C.B., and McNeil, B.J. (1988). Assessment of radiologic tests: control of bias and other design considerations, Radiology 167, pp. 565–569.PubMedGoogle Scholar
- Bunch, P.C., Hamilton, J.F., Sanderson, G.K., and Simmons, A.H. (1977). A free response approach to the measurement and characterization of radiographic observer performance. Proc. SPIE 127, pp. 124–135.Google Scholar
- Chakraborty, D.P. (1988). Maximum likelihood analysis of free response receiver operating characteristic (FROC) data. Personal communication.Google Scholar
- Dorfman, D.D., and Alf, E. (1969). Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals — rating method data, J. Math. Psychol. 6, pp. 487–496.CrossRefGoogle Scholar
- Egan, J.P. (1975). Signal detection theory and ROC analysis, Academic Press, New York.Google Scholar
- Grey, D.R., and Morgan, B.J.T. (1972). Some aspects of ROC curve-fitting: normal and logistic models, J. Math. Psychol. 9, pp. 128–139.CrossRefGoogle Scholar
- Getty, D.J., Pickett, R.M., D’Orsi, C.J., and Swets, J.A. (1988). Enhanced interpretation of diagnostic images, Invest. Radiol. 23, pp. 240–252.PubMedCrossRefGoogle Scholar
- Goodenough, D.J., and Metz, C.E. (1977). Implications of a “noisy” observer to data processing techniques. In: Information processing in medical imaging, C. Raynaud and A.E. Todd-Pokropek (eds.), CEA, Orsay, pp. 400–419.Google Scholar
- Gray, R., Begg, C.B., and Greenes, R.A. (1984). Construction of receiver operating characteristic curves when disease verification is subject to selection bias, Med. Decis. Making 4, pp. 151–164.PubMedCrossRefGoogle Scholar
- Green, D.M., and Swets, J.A. (1966). Signal detection theory and psychophysics, Wiley, New York. Reprint with corrections: (1974). Krieger, Huntington NY.Google Scholar
- Hanley, J.A., and McNeil, B.J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology 143, pp. 29–36.PubMedGoogle Scholar
- Hanley, J.A., and McNeil, B.J. (1983). A method of comparing the areas under receiver operating characteristic curves derived from the same cases, Radiology 148, pp. 839–843.PubMedGoogle Scholar
- International Atomic Energy Agency (1977). IAEA co-ordinated research programme on the intercomparison of computer-assisted scinigraphic techniques. In: Medical radionuclide imaging, IAEA, Vienna, 1, pp. 585–615.Google Scholar
- Kendall, M., and Stewart, A. (1976). The advanced thoery of statistics, Hafner, New York, 3, pp. 182–243.Google Scholar
- Lusted, L.B. (1978). General problems in medical decision making, with comments on ROC analysis, Seminars Nucl. Med. 8, pp. 299–306.CrossRefGoogle Scholar
- Loo, L.-N., Doi, K., Ishida, M., Metz, C.E., Chan, H.-P., Higashida, Y., and Kodera, Y. (1983). An empirical investigation of variability in contrast-detail diagram measurements, Proc. SPIE 419, pp. 68–76.Google Scholar
- MacMillan, N.A., and Kaplan, H.L. (1985). Detection theory analysis of group data: estimating sensitivity from average hit and false alarm rates, Psychol. Bull. 98, pp. 185–199.PubMedCrossRefGoogle Scholar
- McNeil, B.J., Keeler, E., and Adelstein, S.J. (1975). Primer on certain elements of medical decision making, N. Engl. J. Med. 293, pp. 211–215.PubMedCrossRefGoogle Scholar
- McNeil, B.J., and Hanley, J.A. (1984). Statistical approaches to the analysis of receiver operating characteristic (ROC) curves, Med. Decis. Making 4, pp. 137–150.PubMedCrossRefGoogle Scholar
- Metz, C.E. (1978). Basic principles of ROC analysis, Seminars Nucl. Med. 8, pp. 283–298.CrossRefGoogle Scholar
- Metz, C.E. (1979). Applications of ROC analysis in diagnostic image evaluation. In: The physics of medical imaging: recording system measurements and techniques, A. Haus (ed.), Am. Inst. Phys., New York, pp. 546–572.Google Scholar
- Metz, C.E. (1986a). ROC methodology in radiologic imaging, Invest. Radiol. 21, pp. 720–733.PubMedCrossRefGoogle Scholar
- Metz, C.E. (1986b). Statistical analysis of ROC data in evaluating diagnostic performance. In: Multiple regression analysis: applications in the health sciences, D. Herbert and R. Myers (eds.), Am. Inst. Phys., New York, pp. 365–384.Google Scholar
- Metz, C.E. (1988). Some practical issues of experimental design and data analysis in radiological ROC syudies, Invest. Radiol. (in press).Google Scholar
- Metz, C.E., and Kronman, H.B. (1980). Statistical significance tests for binormal ROC curves, J. Math. Psychol. 22, pp. 218–243.CrossRefGoogle Scholar
- Metz, C.E., Kronman, H.B., Wang, P.-L., and Shen, J.-H. (1985). ROCFIT: a modified maximum likelihood algorithm for estimating a binormal ROC curve from confidence-rating data, Dept. Radiology, Univ. Chicago, Chicago. Mini-and micro-computer versions of this and other software for ROC analysis are available from the author at no cost.Google Scholar
- Metz, C.E., Starr, S.J., Lusted, L.B., and Rossmann, K. (1975). Progress in evaluation of human observer visual detection performance using the ROC curve approach. In: Information processing in medical imaging, C. Raynaud and A.E. Todd-Pokropek (eds.), CEA, Orsay, pp. 420–436.Google Scholar
- Metz, C.E., Wang, P.-L., and Kronman, H.B. (1984). A new approach for testing the significance of differences between ROC curves measured from correlated data. In: Information processing in medical imaging, F. Deconinck (ed.), Nijhoff, The Hague, pp. 432–445.CrossRefGoogle Scholar
- Ransohoff, D.F., and Feinstein, A.R., (1978). Problems of spectrum and bias in evaluating the efficacy of diagnostic tests, N. Engl. J. Med. 299, pp. 926–930.PubMedCrossRefGoogle Scholar
- Starr, S.J., Metz, C.E., Lusted, L.B., and Goodenough, D.J. (1975). Visual detection and localization of radiographic images, Radiology 116, pp. 533–538.PubMedGoogle Scholar
- Swets, J.A. (1979). ROC analysis applied to the evaluation of medical imaging techniques, Invest. Radiol. 14, pp. 109–121.PubMedCrossRefGoogle Scholar
- Swets, J.A. (1986a). Indices of discrimination or diagnostic accuracy: their ROCs and implied models, Psychol. Bull. 99, pp. 100–117.PubMedCrossRefGoogle Scholar
- Swets, J.A. (1986b). Form of empirical ROCs in discrimination and diagnostic tasks: implications for theory and measurementof performance, Psychol. Bull. 99, pp. 181–198.PubMedCrossRefGoogle Scholar
- Swets, J.A., and Pickett, R.M. (1982). Evaluation of diagnostic systems: methods from signal detection theory, Academic Press, New York.Google Scholar
- Weinstein, M.C., and Feinberg, H.V. (1980). Clinical decision analysis, Saunders, Philadelphia.Google Scholar