Advertisement

Localization Recall Precision (LRP): A New Performance Metric for Object Detection

  • Kemal Oksuz
  • Baris Can Cam
  • Emre Akbas
  • Sinan Kalkan
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11211)

Abstract

Average precision (AP), the area under the recall-precision (RP) curve, is the standard performance measure for object detection. Despite its wide acceptance, it has a number of shortcomings, the most important of which are (i) the inability to distinguish very different RP curves, and (ii) the lack of directly measuring bounding box localization accuracy. In this paper, we propose “Localization Recall Precision (LRP) Error”, a new metric specifically designed for object detection. LRP Error is composed of three components related to localization, false negative (FN) rate and false positive (FP) rate. Based on LRP, we introduce the “Optimal LRP” (oLRP), the minimum achievable LRP error representing the best achievable configuration of the detector in terms of recall-precision and the tightness of the boxes. In contrast to AP, which considers precisions over the entire recall domain, oLRP determines the “best” confidence score threshold for a class, which balances the trade-off between localization and recall-precision. In our experiments, we show that oLRP provides richer and more discriminative information than AP. We also demonstrate that the best confidence score thresholds vary significantly among classes and detectors. Moreover, we present LRP results of a simple online video object detector and show that the class-specific optimized thresholds increase the accuracy against the common approach of using a general threshold for all classes. Our experiments demonstrate that LRP is more competent than AP in capturing the performance of detectors. Our source code for PASCAL VOC AND MSCOCO datasets are provided at https://github.com/cancam/LRP.

Keywords

Average precision Object detection Performance metric Optimal threshold Recall-precision 

Notes

Acknowledgements

This work was supported by the Scientific and Technological Research Council of Turkey (TÜBİTAK) through project called “Object Detection in Videos with Deep Neural Networks” (project no 117E054). We also gratefully acknowledge (i) the support of NVIDIA Corporation with the donation of the Tesla K40 GPU and (ii) the computational resources kindly provided by Roketsan Missiles Inc. used for this research. Kemal Oksuz is supported by the TÜBİTAK 2211-A National Scholarship Programme for Ph.D. students.

Supplementary material

474212_1_En_31_MOESM1_ESM.pdf (205 kb)
Supplementary material 1 (pdf 204 KB)

References

  1. 1.
    Bernardin, K., Stiefelhagen, R.: Evaluating multiple object tracking performance: the clear mot metrics. EURASIP J. Image Video Process. 2008(1), 246309 (2008).  https://doi.org/10.1155/2008/246309CrossRefGoogle Scholar
  2. 2.
    Bourgeois, F., Lassalle, J.C.: An extension of the munkres algorithm for the assignment problem to rectangular matrices. Commun. ACM 14(12), 802–804 (1971)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Breitenstein, M.D., Reichlin, F., Leibe, B., Koller-Meier, E., Gool, L.V.: Robust tracking-by-detection using a detector confidence particle filter. In: IEEE International Conference on Computer Vision ICCV (2009)Google Scholar
  4. 4.
    Breitenstein, M.D., Reichlin, F., Leibe, B., Koller-Meier, E., Van Gool, L.: Online multiperson tracking-by-detection from a single, uncalibrated camera. IEEE Trans. Pattern Anal. Mach. Intell. 33(9), 1820–1833 (2011)CrossRefGoogle Scholar
  5. 5.
    Brzezinski, D., Stefanowski, J., Susmaga, R., Szczech, I.: Visual-based analysis of classification measures with applications to imbalanced data. arXiv: 1704.07122 (2017)
  6. 6.
    Dai, J., Li, Y., He, K., Sun, J.: R-FCN: object detection via region-based fully convolutional networks. In: Advances in Neural Information Processing Systems, NIPS (2016)Google Scholar
  7. 7.
    Dembczynski, K.J., Waegeman, W., Cheng, W., Hüllermeier, E.: An exact algorithm for F-measure maximization. In: Advances in Neural Information Processing, NIPS, pp. 1404–1412 (2011)Google Scholar
  8. 8.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)CrossRefGoogle Scholar
  9. 9.
    Feichtenhofer, C., Pinz, A., Zisserman, A.: Detect to track and track to detect. In: IEEE International Conference on Computer Vision, ICCV (2017)Google Scholar
  10. 10.
    Ferrari, P.: A keras port of single shot multibox detector. https://github.com/pierluigiferrari/ssd_keras. Accessed 13 Mar 2018
  11. 11.
    Girshick, R., Radosavovic, I., Gkioxari, G., Dollár, P., He, K.: Detectron. https://github.com/facebookresearch/detectron. Accessed 13 Mar 2018
  12. 12.
    Kang, K., et al.: T-CNN: tubelets with convolutional neural networks for object detection from videos. IEEE Trans. Circuits Syst. Video Technol. PP(99), 1 (2017)Google Scholar
  13. 13.
    Lin, T., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal loss for dense object detection. In: IEEE International Conference on Computer Vision, ICCV (2017)Google Scholar
  14. 14.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10602-1_48CrossRefGoogle Scholar
  15. 15.
    Lipton, Z.C., Elkan, C., Naryanaswamy, B.: Optimal thresholding of classifiers to maximize F1 measure. In: Calders, T., Esposito, F., Hüllermeier, E., Meo, R. (eds.) ECML PKDD 2014. LNCS (LNAI), vol. 8725, pp. 225–239. Springer, Heidelberg (2014).  https://doi.org/10.1007/978-3-662-44851-9_15CrossRefGoogle Scholar
  16. 16.
    Liu, W., et al.: SSD: single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46448-0_2CrossRefGoogle Scholar
  17. 17.
    Lu, Y., Lu, C., Tang, C.: Online video object detection using association LSTM. In: IEEE International Conference on Computer Vision, ICCV (2017)Google Scholar
  18. 18.
    Musicant, D.R., Kumar, V., Ozgur, A.: Optimizing F-measure with support vector machines. In: The Florida Artifical Intelligence Research Society Conference, FLAIRS Conference (2003)Google Scholar
  19. 19.
    Oksuz, K., Cemgil, A.T.: Multitarget tracking performance metric: deficiency aware subpattern assignment. IET Radar Sonar Navig. 12(3), 373–381 (2018)CrossRefGoogle Scholar
  20. 20.
    Powers, D.M.W.: What the F-measure doesn’t measure: features, flaws, fallacies and fixes. arXiv: 1503.06410 (2015)
  21. 21.
    Puthiya Parambath, S., Usunier, N., Grandvalet, Y.: Optimizing F-measures by cost-sensitive classification. In: Advances in Neural Information Processing, NIPS (2014)Google Scholar
  22. 22.
    Quevedo, J.R., Luaces, O., Bahamonde, A.: Multilabel classifiers with a probabilistic thresholding strategy. Pattern Recogn. 45(2), 876–883 (2012)zbMATHGoogle Scholar
  23. 23.
    Rahmathullah, A.S., Garcia-Fernandez, A.F., Svensson, L.: Generalized optimal sub-pattern assignment metric. In: IEEE International Conference on Information Fusion, FUSION (2017)Google Scholar
  24. 24.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, NIPS (2015)Google Scholar
  25. 25.
    Rijsbergen, C.J.V.: Information Retrieval. 2nd edn. Butterworth-Heinemann (1979)Google Scholar
  26. 26.
    Ristic, B., Vo, B.N., Clark, D.: Performance evaluation of multi-target tracking using the OSPA metric. In: IEEE International Conference on Information Fusion, FUSION (2010)Google Scholar
  27. 27.
    Ristic, B., Vo, B.N., Clark, D., Vo, B.T.: A metric for performance evaluation of multi-target tracking algorithms. IEEE Trans. Signal Process. 59(7), 3452–3457 (2011)MathSciNetCrossRefGoogle Scholar
  28. 28.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Schuhmacher, D., Vo, B.T., Vo, B.N.: A consistent metric for performance evaluation of multi-object filters. IEEE Trans. Signal Process. 56(8), 3447–3457 (2008)MathSciNetCrossRefGoogle Scholar
  30. 30.
    Shi, X., Yang, F., Tong, F., Lian, H.: A comprehensive performance metric for evaluation of multi-target tracking algorithms. In: International Conference on Information Management, ICIM (2017)Google Scholar
  31. 31.
    Shu, G., Dehghan, A., Shah, M.: Improving an object detector and extracting regions using superpixels. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2013)Google Scholar
  32. 32.
    Stalder, S., Grabner, H., Van Gool, L.: Cascaded confidence filtering for improved tracking-by-detection. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 369–382. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15549-9_27CrossRefGoogle Scholar
  33. 33.
    Suzuki, J., McDermott, E., Isozaki, H.: Training conditional random fields with multivariate evaluation measures. In: International Conference on Computational Linguistics and the Annual Meeting of the Association for Computational Linguistics, ACL-44 (2006)Google Scholar
  34. 34.
    Tripathi, S., Lipton, Z.C., Belongie, S.J., Nguyen, T.Q.: Context matters: refining object detection in video with recurrent neural networks. In: British Machine Vision Conference, BMVC (2016)Google Scholar
  35. 35.
    Vu, T., Evans, R.: A new performance metric for multiple target tracking based on optimal subpattern assignment. In: IEEE International Conference on Information Fusion, FUSION (2014)Google Scholar
  36. 36.
    Zhu, X., Dai, J., Yuan, L., Wei, Y.: Towards high performance video object detection. arXiv: 1711.11577 (2017)
  37. 37.
    Zou, X., Wen, J.: Detection of object security in crowed environment. In: IEEE International Conference on Communication Problem-Solving, ICCP (2015)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.Department of Computer EngineeringMiddle East Technical UniversityAnkaraTurkey

Personalised recommendations