Abstract
In this study, we initially investigated various aspects of requirements for a computer interface employed in receiver operating characteristic (ROC) and free-response ROC (FROC) observer studies which involve digital images and ratings obtained by observers (radiologists). Secondly, by taking into account these aspects, an all-purpose computer interface utilized for these observer performance studies was developed. Basically, the observer studies can be classified into three paradigms, such as one rating for one case without an identification of a signal location, one rating for one case with an identification of a signal location, and multiple ratings for one case with identification of signal locations. For these paradigms, display modes on the computer interface can be used for single/multiple views of a static image, continuous viewing with cascade images (i.e., CT, MRI), and dynamic viewing of movies (i.e., DSA, ultrasound). Various functions on these display modes, which include windowing (contrast/level), magnifications, and annotations, are needed to be selected by an experimenter corresponding to the purpose of the research. In addition, the rules of judgment for distinguishing between true positives and false positives are an important factor for estimating diagnostic accuracy in an observer study. We developed a computer interface which runs on a Windows operating system by taking into account all aspects required for various observer studies. This computer interface requires experimenters to have sufficient knowledge about ROC/FROC observer studies, but allows its use for any purpose of the observer studies. This computer interface will be distributed publicly in the near future.
Notes
Available at http://dicom.offis.de/dcmtk.php.en.
References
Goodenough DJ, Rossmann K, Lusted LB. Radiographic applications of receiver operating characteristic (ROC) curves. Radiology. 1974;110:89–95.
Metz CE. ROC methodology in radiologic imaging. Invest Radiol. 1986;21:720–33.
ICRU Report 79. Receiver operating characteristic analysis in medical imaging. Oxford University Press, Oxford; J ICRU 2008;8(1).
Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29–36.
Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8:283–98.
Metz CE, Herman BA, Shen J-H. Maximum-likelihood estimation of ROC curves from continuously-distributed data. Stat Med. 1998;17:1033–53.
Pesce LL, Metz CE. On the convexity of ROC curves estimated from radiological test results. Acad. Radiol. 2010;256(1):29–31.
Shiraishi J, Pesce L, Metz CE, Doi K. Experimental design and data analysis in receiver operating characteristic studies: lessons learned from reports in Radiology from 1997 to 2006. Radiology. 2009;253:822–30.
Swensson RG. Unified measurement of observer performance in detecting and localizing target objects on images. Med Phys. 1996;23:1709–25.
Chakraborty DP, Breatnach EA, Yester MV, Soto B, Barnes GT, Fraser RG. Digital and conventional chest imaging: a modified ROC study of observer performance using simulated nodules. Radiology. 1986;158:35–9.
Chakraborty DP, Winter LH. Free-response methodology: alternative analysis and a new observer-performance experiment. Radiology. 1990;174:873–81.
Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys. 2004;31:2313–30.
Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24:234–45.
Metz ROC Software web site in Department of Radiology, The University of Chicago. http://xray.bsd.uchicago.edu/krl/roc_soft6.htm. (Accessed 14 Mar 2012)
Dr. Dev’s Chakraborty’s FROC web site http://www.devchakraborty.com/index.php. (Accessed 14 Mar 2012)
Tanaka R, Shiraishi J, Takamori M, Watari C. ROC analysis for evaluating the detectability of image unsharpness due to the patient’s movement: phantom study comparing preview and diagnostic LCDs. J Radiol. Technol. 2011;67(7):772–8.
Acknowledgments
We gratefully acknowledge support from the Japanese Society of Radiological Technology (JSRT) Research Grant (2020, and 2011). This work is also supported in part by Scientific Research Grant (C), 22611014, 2010.
Author information
Authors and Affiliations
Corresponding author
About this article
Cite this article
Shiraishi, J., Fukuoka, D., Hara, T. et al. Basic concepts and development of an all-purpose computer interface for ROC/FROC observer study. Radiol Phys Technol 6, 35–41 (2013). https://doi.org/10.1007/s12194-012-0166-1
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12194-012-0166-1