Calculation of a Composite DET Curve

  • Andy Adler
  • Michael E. Schuckers
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3546)

Abstract

The verification performance of biometric systems is normally evaluated using the receiver operating characteristic (ROC) or detection error trade-off (DET) curve. We propose two new ideas for statistical evaluation of biometric systems based on these data. The first is a new way to normalize match score distributions. A normalized match score, \(\hat{t}\), is calculated as a function of the angle from a representation of (FMR, FNMR) values in polar coordinates from some center. This has the advantage that it does not produce counterintuitive results for systems with unusual DET performance. Secondly, building on this normalization we develop a methodology to calculate an average DET curve. Each biometric system is represented in terms of \(\hat{t}\) to allow genuine and impostor distributions to be combined, and an average DET is then calulated from these new distributions. We then show that this method is equivalent to direct averaging of DET data along each angle from the center. This procedure is then applied to data from a study of human matchers of facial images.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Adler, A., Maclean, J.: Performance comparison of human and automatic face recognition. In: Biometrics Consortium Conference 2004, Washington, DC, USA, Septmeber 20-22 (2004)Google Scholar
  2. 2.
    Bradley, A.P.: The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition 7, 1145–1159 (1997)CrossRefMathSciNetGoogle Scholar
  3. 3.
    Burton, A.M., Miller, P., Bruce, V., Hancock, P.J.B., Henderson, Z.: Human and automatic face recognition: a comparison across image formats. Vision Research 41, 3185–3195 (2001)CrossRefGoogle Scholar
  4. 4.
    Drummond, C., Holte, R.C.: What ROC Curves Can’t Do (and Cost Curves Can). In: Proc. 1st Workshop ROC Analysis in AI:ROCAI, pp. 19–26 (2004)Google Scholar
  5. 5.
    Fawcett, T.: ROC graphs: Notes and practical considerations for data mining researchers, Technical Report HPL-2003-4. HP Labs (2003)Google Scholar
  6. 6.
    Green, D.M., Swets, J.A.: Signal Detection Theory and Psychophysics. JohnWiley & Sons, New York (1966)Google Scholar
  7. 7.
    Golfarelli, M., Maio, D., Maltoni, D.: On the Error-Reject Trade-Off in Biometric Verification Systems. IEEE Trans. Pattern Anal. Machine Intel. 19, 786–796 (1997)CrossRefGoogle Scholar
  8. 8.
    Hancock, P.J.B., Bruce, V., Burton, M.A.: A comparison of two computerbased face identification systems with human perceptions of faces. Vision Research 38, 2277–2288 (1998)CrossRefGoogle Scholar
  9. 9.
    Hanley, J.A., McNeil, B.J.: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143, 29–36 (1982)Google Scholar
  10. 10.
    Hernández-Orallo, J., Ferri, C., Lachiche, N., Flach, P.A. (eds.): ROC Analysis in Artificial Intelligence, 1st Int. Workshop, ROCAI-2004, Valencia, Spain (2004)Google Scholar
  11. 11.
    Jain, A.K., Nandakumar, K., Ross, A.: Score Normalization in Multimodal Biometric Systems. Pattern Recognition (2005) (in press)Google Scholar
  12. 12.
    Karduan, J., Karduan, O.: Comparative diagnostic performance of three radiological procedures for the detection of lumbar disk herniation. Methods Inform. Med. 29, 12–22 (1990)Google Scholar
  13. 13.
    Macskassy, S., Provost, F.: Confidence Bands for ROC Curves: Methods and an Empirical Study. In: Proc. 1st Workshop ROC Analysis in AI:ROCAI, pp. 61–70 (2004)Google Scholar
  14. 14.
    NIST: Face Recognition Vendor test (2002), http://frvt.org/frvt2002
  15. 15.
    NIST: Fingerprint Vendor Technology Evaluation, FpVTE (2003), http://fpvte.nist.gov/
  16. 16.
    NIST: NIST Special Database 18: Mugshot Identification Database (MID), http://www.nist.gov/srd/nistsd18.htm
  17. 17.
    Provost, F.J., Fawcett, T., Kohavi, R.: The case against accuracy estimation for comparing induction algorithms. In: Proc. 15th Int. Conf. Machine Learning, pp. 445–453 (1998)Google Scholar
  18. 18.
    Rukhin, A., Grother, P., Phillips, P.J., Newton, E.: Dependence characteristics of face recognition algorithms. In: Proc. Int. Conf Pattern Recog., vol. 16, pp. 36–39 (2002)Google Scholar
  19. 19.
    Zhou, X.-H., McClish, D.K., Obuchowski, N.A.: Statistical Methods in Diagnostic Medicine. John W. Wiley & Sons, Chichester (2002)MATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Andy Adler
    • 1
  • Michael E. Schuckers
    • 2
    • 3
  1. 1.School of Information Technology and EngineeringUniversity of OttawaCanada
  2. 2.Mathematics, Computer Science and Statistics DepartmentSt. Lawrence UniversityCantonUSA
  3. 3.Center for Identification Technology Research (CITeR)West Virginia UniversityMorgantownUSA

Personalised recommendations