Skip to main content
SpringerLink
Log in

Verification of modified receiver-operating characteristic software using simulated rating data

  • Published:
Radiological Physics and Technology Aims and scope Submit manuscript

Abstract

ROCKIT, which is a receiver-operating characteristic (ROC) curve-fitting software package, was developed by Metz et al. In the early 1990s, it is a very frequently used ROC software throughout the world. In addition to ROCKIT, DBM-MRMC software was developed for multi-reader multi-case analysis of the difference in average area under ROC curves (AUCs). Because this old software cannot run on a PC with Windows 7 or a more recent operating system, we developed new software that employs the same basic algorithms with minor modifications. In this study, we verified our modified software and tested the differences between the index of diagnostic accuracies using simulated rating data. In our simulation model, all data were generated using target AUCs and a binormal parameter b. In ROC curve fitting with simulated rating data, we varied four factors: the total number of case samples, the ratio of positive-to-negative cases, a binormal parameter b, and the preset AUC. To investigate the differences between the statistical test results obtained from our software and the existing software, we generated simulated rating data sets with three levels of case difficulty and three degrees of difference in AUCs obtained from two modalities. As a result of the simulation, the AUCs estimated by the new and existing software were highly correlated (R > 0.98), and there were high agreements (85% or more) in the statistical test results. In conclusion, we believe that our modified software is as capable as the existing software.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. Imaging Section Website in JSRT: http://imgcom.jsrt.or.jp/rocGroup/.

References

  1. Green DM, Swets JA. Signal detection theory and psychophysics. New York: Wiley; 1966 (reprinted with updated topical bibliographies by Kreiger, New York, 1974).

    Google Scholar 

  2. Metz CE, Herman BA, Shen J-H. Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data. Stat Med. 1998;17:1033–53.

    Article  CAS  Google Scholar 

  3. Lusted LB. Logical analysis in Roentgen diagnosis. Radiology. 1960;74:178–93.

    Article  CAS  Google Scholar 

  4. Lusted LB. Introduction to medical decision making. Springfield: Charles C Thomas; 1968.

    Google Scholar 

  5. Swets JA. Measuring the accuracy of diagnostic systems. Science. 1988;240:1285–93.

    Article  CAS  Google Scholar 

  6. Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of receiver operating characteristic (ROC) curves. Radiology. 1974;110:89–95.

    Article  CAS  Google Scholar 

  7. Metz CE. ROC methodology in radiologic imaging. Invest Radiol. 1986;21:720–33.

    Article  CAS  Google Scholar 

  8. Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8:283–98.

    Article  CAS  Google Scholar 

  9. Obuchowski NA. Receiver operating characteristic curves and their use in radiology. Radiology. 2003;229:3–8.

    Article  Google Scholar 

  10. ICRU Report 79. Receiver operating characteristic analysis in medical imaging, vol. 8, No.1. Oxford: Oxford University Press; 2008 (J. of the ICRU).

    Google Scholar 

  11. Metz CE. ROC analysis in medical imaging: a tutorial review of the literature. Radiol Phys Technol. 2008;1:2–12.

    Article  Google Scholar 

  12. Shiraishi J, Pesce L, Metz CE, Doi K. Experimental design and data analysis in receiver operating characteristic studies: lessons learned from reports in Radiology from 1997 to 2006. Radiology. 2009;253:822–30.

    Article  Google Scholar 

  13. Dorfman DD, Alf E. Maximum likelihood estimation of parameters of signal detection theory and determination of confidence intervals—rating method data. J Math Psychol. 1969;6:487–96.

    Article  Google Scholar 

  14. Metz CE, Pan X. “Proper” binormal ROC curves: theory and maximum-likelihood estimation. J Math Psychol. 1999;43(1):1–33.

    Article  CAS  Google Scholar 

  15. Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis: generalization to the population of readers and patients with the jackknife method. Invest Radiol. 1992;27:723–31.

    Article  CAS  Google Scholar 

  16. Metz CE, Roe CA. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol. 1997;4(4):298–303.

    Article  Google Scholar 

  17. Shiraishi J, Fukuoka D, Hara T, Abe H. Basic concepts and development of an all-purpose computer interface for ROC/FROC observer study. Radiol Phys Technol. 2013;6(1):35–41.

    Article  Google Scholar 

  18. Waldrop MM. More than Moore. Nature. 2016;530:144–7.

    Article  CAS  Google Scholar 

  19. Metz CE. Receiver operating characteristic analysis: a tool for the quantitative evaluation of observer performance and imaging systems. J Am Coll Radiol. 2006;3:413–22.

    Article  Google Scholar 

  20. Dorfman DD, Berbaum KS, Metz CE, Lenth RV, Hanley JA, Dagga HA. Proper receiver operating characteristic analysis: the Bigamma model. Acad Radiol. 1996;4:138–49.

    Article  Google Scholar 

  21. Roe CA, Metz CE. Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: validation with computer simulation. Acad Radiol. 1997;4:298–303.

    Article  CAS  Google Scholar 

  22. Roe CA, Metz CE. Variance-component modeling in the analysis of receiver operating characteristic index estimates. Acad Radiol. 1997;4:587–600.

    Article  CAS  Google Scholar 

  23. Pan X, Metz CE. The “Proper” binormal model: parametric receiver operating characteristic curve estimation with degenerate data. Acad Radiol. 1997;4:380–9.

    Article  CAS  Google Scholar 

  24. Wagner RF, Beiden SV, Metz CE. Continuous versus categorical data for ROC analysis: some quantitative considerations. Acad Radiol. 2001;8(4):328–34.

    Article  CAS  Google Scholar 

  25. Pesce LL, Horsch K, Drukker K, Metz CE. Semiparametric estimation of the relationship between ROC operating points and the test-result scale: application to the proper binormal model. Acad Radiol. 2011;18:1537–48.

    Article  Google Scholar 

  26. Hillis SL, Berbaum KS. Monte Carlo validation of the Dorfman-Berbaum-Metz method using normalized pseudo values and less data-based model simplification. Acad Radiol. 2005;12:1534–41.

    Article  Google Scholar 

  27. Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol. 2008;15:647–61.

    Article  Google Scholar 

  28. Shiraishi J, Katsuragawa S, Ikezoe J, Matsumoto T, Kobayashi T, Komatsu K, Matsui M, Fujita H, Kodera Y, Doi K. Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules. AJR. 2000;174:71–4.

    Article  CAS  Google Scholar 

Download references

Acknowledgements

We gratefully acknowledge the support of a Japanese Society of Radiological Technology (JSRT) research grant (2016 and 2017). This work was also partially supported by JSPS KAKENHI Grant number 15K09898.

Author information

Authors and Affiliations

  1. Faculty of Life Sciences, Kumamoto University, 4-24-1 Kuhonji, Kumamoto, 862-0976, Japan

    Junji Shiraishi

  2. Faculty of Education, Gifu University, 1-1 Yanagido, Gifu, 501-1193, Japan

    Daisuke Fukuoka

  3. School of Health Sciences, Kumamoto University, 4-24-1 Kuhonji, Kumamoto, 862-0976, Japan

    Reimi Iha & Haruka Inada

  4. College of Medical, Pharmaceutical and Health Sciences, Kanazawa University, 5-11-80 Kodatsuno, Kanazawa, Ishikawa, 920-0942, Japan

    Rie Tanaka

  5. Faculty of Engineering, Gifu University, 1-1 Yanagido, Gifu, 501-1193, Japan

    Takeshi Hara

Authors
  1. Junji Shiraishi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daisuke Fukuoka
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Reimi Iha
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haruka Inada
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rie Tanaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Takeshi Hara
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junji Shiraishi.

Ethics declarations

Ethical approval

This article does not contain any studies with human participants performed, and thus, we have no informed consent from any individuals. In addition, this article does not contain any studies with animals performed.

Conflict of interest

The authors declare that they have no conflict of interest about this article.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shiraishi, J., Fukuoka, D., Iha, R. et al. Verification of modified receiver-operating characteristic software using simulated rating data. Radiol Phys Technol 11, 406–414 (2018). https://doi.org/10.1007/s12194-018-0479-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12194-018-0479-9

Keywords

Search

Navigation