Skip to main content
SpringerLink
Log in

Evaluating the Usefulness of Binary Classifier Based on Enhanced ROC Analysis

  • Published:
Cybernetics and Systems Analysis Aims and scope

Abstract

The definition of the usefulness of the binary classifier from the point of view of reducing the a priori risk of false classification is formulated. Sufficient conditions are proposed to guarantee the utility of a diagnostic test according to this definition. The obtained conditions improved the traditional ROC analysis by limiting the corresponding region of the ROC curve. The line limiting the region of the guaranteed useful test is shown to coincide with the known iso-performance line corresponding to the a priori risk level. The feasible limits of the ratio of losses from target misses and false alarms were determined, according to which a test with appropriate operational characteristics remains useful for screening a disease with a known prevalence. Based on the obtained results, the authors substantiated the efficiency of the new method of the analysis and interpretation of electrocardiograms, which is based on determining the original diagnostic feature in the phase space and enables detecting persons with a high risk of coronary heart disease in the early stages of the disease.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. D. Maxim, R. Niebo, and M. J. Utel, “Screening tests: A review with examples,” Inhal. Toxicol., Vol. 26, Iss. 13, 811–828 (2014). http://doi.org/https://doi.org/10.3109/08958378.2014.955932.

  2. O. Zhukovska, “Decision-making model on potential borrower lending for independent experts group,” in: Proc. IEEE 3rd Intern. Conf. on System Analysis & Intelligent Computing (SAIC) (Kyiv, Ukraine, Oct. 4–7, 2022), IEEE (2022), pp. 118–121. http://doi.org/https://doi.org/10.1109/SAIC57818.2022.9923015.

  3. C. Dendek and J. Mańdziuk, “Improving performance of a binary classifier by training set selection,” in: V. Kurková, R. Neruda, and J. Koutník (eds.), Artificial Neural Networks — ICANN 2008. Lecture Notes in Computer Science, Vol. 5163, Springer, Berlin–Heidelberg (2008), pp. 128–135. https://doi.org/10.1007/978-3-540-87536-9_14.

  4. C. E. Metz, “Fundamental ROC analysis,” in: R. L. Van Metter, J. Beutel, and H. L. Kundel (eds.), Handbook of Medical Imaging, Vol. 1, Physics and Psychophysics, Ch. 15, SPIE Press, Bellingham (2000), pp. 751–769. https://doi.org/10.1117/3.832716.ch15.

  5. T. Fawcett, “Using rule sets to maximize ROC performance,” in: Proc. IEEE Intern. Conf. on Data Mining (ICDM-2001), IEEE Computer Society (2001), pp. 131–138. https://doi.org/10.1109/ICDM.2001.989510.

  6. P. Flach and S. Wu, “Repairing concavities in ROC curves,” in: Proc. 2003 UK Workshop on Computational Intelligence, University of Bristol (2003), pp. 38–44.

  7. P. Sonego, A. Kocsor, and S. Pongor, “ROC analysis: Applications to the classification of biological sequences and 3D structures,” Brief. Bioinform., Vol. 9, Iss. 3, 198–209 (2008). https://doi.org/10.1093/bib/bbm064.

  8. K. Feng, H. Hong, K. Nang, and J. Wang, “Decision making with machine learning and ROC curves,” arXiv:1905.02810v1 [stat.ME] 5 May (2019). https://doi.org/10.48550/arXiv.1905.02810.

  9. J. Davis and M. Goadrich, “The relationship between Precision-Recall and ROC curves” in: Proc. 23rd Intern. Conf. on Machine Learning (ICML’06), Association for Computing Machinery, New York (2006), pp. 233–240. https://doi.org/10.1145/1143844.1143874.

  10. W. B. van den Hout, “The area under an ROC curve with limited information,” Med. Decis. Making, Vol. 23, Iss. 2, 160–166 (2003). https://doi.org/10.1177/0272989X03251246.

  11. M. S. Pepe, G. Longton, and H. Janes, “Estimation and comparison of receiver operating characteristic curves,” Stata J., Vol. 9, No. 1, 1–16 (2009). https://doi.org/https://doi.org/10.1177/1536867X0900900101.

    Article  Google Scholar 

  12. T. A. Alonzo and M. S. Pepe, “Distribution-free ROC analysis using binary regression techniques,” Biostatistics, Vol. 3, Iss. 3, 421–432 (2002). https://doi.org/10.1093/biostatistics/3.3.421.

  13. F. Provost and T. Fawcett, “Robust classification for imprecise environments,” Machine Learning, Vol. 42, No. 3, 203–231 (2001). https://doi.org/https://doi.org/10.1023/A:1007601015854.

    Article  MATH  Google Scholar 

  14. M. Majnik and Z. Bosnić, “ROC analysis of classifiers in machine learning: A survey,” Intell. Data Anal., Vol. 17, No. 3, 531–558 (2013). https://doi.org/https://doi.org/10.3233/IDA-130592.

    Article  Google Scholar 

  15. N. Hu, “Using receiver operating characteristic (ROC) analysis to evaluate information-based decision-making,” in: M. Khosrow-Pour, D.B.A. (ed.), Advanced Methodologies and Technologies in Business Operations and Management, IGI Global, Hershey, PA (2019), pp. 764–776. https://doi.org/10.4018/978-1-5225-7362-3.ch057.

  16. D. J. Hand and R. J. Till, “A simple generalization of the area under the ROC curve to multiple class classification problems,” Machine Learning, Vol. 45, No. 2, 171–186 (2001). https://doi.org/https://doi.org/10.1023/A:1010920819831.

    Article  MATH  Google Scholar 

  17. C. Ferri, J. Hernández-Orallo, and M. A. Salido, “Volume under the ROC surface for multi-class problems,” in: N. LavračD. Gamberger, H. Blockeel, and L. Todorovski (eds.), Machine Learning: ECML 2003. ECML 2003; Lecture Notes in Computer Science, Vol. 2837, Springer, Berlin–Heidelberg (2003), pp. 108–120. https://doi.org/10.1007/978-3-540-39857-8_12.

  18. D. C. Edwards, C. E. Metz, and M. A. Kupinski, “Ideal observers and optimal ROC hypersurfaces in N-class classification,” IEEE Trans. Med. Imaging, Vol. 23, No. 7, 891–895 (2004). https://doi.org/https://doi.org/10.1109/TMI.2004.828358.

    Article  Google Scholar 

  19. D. C. Edwards, C. E. Metz, and R. M. Nishikawa, “The hypervolume under the ROC hypersurface of “Near-Guessing” and “Near-Perfect” observers in N-class classification tasks,” IEEE Trans. Med. Imaging, Vol. 24, No. 3, 293–299 (2005). https://doi.org/https://doi.org/10.1109/tmi.2004.841227.

    Article  Google Scholar 

  20. X. He and E. C. Fry, “An optimal three-class linear observer derived from decision theory,” IEEE Trans. Med. Imaging, Vol. 26, No. 1, 77–83 (2007). https://doi.org/https://doi.org/10.1109/TMI.2006.885335.

    Article  Google Scholar 

  21. B. Sahiner, H.-P. Chan, and L. M. Hadjiiski, “Performance analysis of 3-class classifiers: Properties of the 3D ROC surface and the normalized volume under the surface,” IEEE Trans. Med. Imaging, Vol. 27, No. 2, 215–227 (2008). https://doi.org/https://doi.org/10.1109/TMI.2007.905822.

    Article  Google Scholar 

  22. T. Fawcett, “ROC graphs with instance-varying costs,” Pattern Recognit. Lett., Vol. 27, Iss. 8, 882–891 (2006). https://doi.org/10.1016/j.patrec.2005.10.012.

  23. R. Meekins, S. Adams, P. A. Beling, K. Farinholt, N. Hipwell, A. Chaudhry, S. Polter, and Q. Dong, “Cost-sensitive classifier selection when there is additional cost information,” Proc. Mach. Learn. Res., Vol. 88, 17–30 (2018).

    Google Scholar 

  24. R. C. Holte and C. Drummond, “Cost-sensitive classifier evaluation using cost curves,” in: T. Washio, E. Suzuki, K. M. Ting, and A. Inokuchi (eds.), Advances in Knowledge Discovery and Data Mining, PAKDD 2008; Lecture Notes in Computer Science, Vol. 5012, Springer, Berlin–Heidelberg (2008), pp. 26–29. https://doi.org/10.1007/978-3-540-68125-0_4.

  25. L. S. Fainzilberg, “Plausible but groundless premises when constructing diagnostic models,” J. Autom. Inform. Sci., Vol. 52, Iss. 5, 38–50 (2020). https://doi.org/10.1615/JAutomatInfScien.v52.i5.40.

  26. L. S. Fainzilberg, “Conditions of utility of diagnostic tests from the point of view of the statistical theory of decision making,” J. Autom. Inform. Sci., Vol. 35, Iss. 4, 63–73 (2003). https://doi.org/10.1615/JAutomatInfScien.v35.i4.30.

  27. L. S. Fainzilberg, “New opportunities of phasegraphy in medical practice,” Sci. Innov., Vol. 1, Iss. 3, 37–50 (2017). https://doi.org/10.15407/scine13.03.037.

  28. L. S. Fainzilberg, “New approaches to the analysis and interpretation of the shape of cyclic signals,” Cybern. Syst. Analysis, Vol. 56, No. 4, 665–674 (2020). https://doi.org/10.1007/s10559-020-00283-0.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute, Kyiv, Ukraine

    O. A. Zhukovska

  2. International Scientific and Training Center of Information Technologies and Systems, National Academy of Sciences of Ukraine and Ministry of Education and Science of Ukraine, Kyiv, Ukraine

    L. S. Fainzilberg

  3. National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute, Kyiv, Ukraine

    L. S. Fainzilberg

Authors
  1. O. A. Zhukovska
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. L. S. Fainzilberg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to O. A. Zhukovska.

Additional information

Translated from Kibernetyka ta Systemnyi Analiz, No. 3, May–June, 2023, pp. 95–105

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhukovska, O.A., Fainzilberg, L.S. Evaluating the Usefulness of Binary Classifier Based on Enhanced ROC Analysis. Cybern Syst Anal 59, 439–448 (2023). https://doi.org/10.1007/s10559-023-00578-y

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10559-023-00578-y

Keywords

Search

Navigation