Skip to main content

Object Detection as Probabilistic Set Prediction

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Abstract

Accurate uncertainty estimates are essential for deploying deep object detectors in safety-critical systems. The development and evaluation of probabilistic object detectors have been hindered by shortcomings in existing performance measures, which tend to involve arbitrary thresholds or limit the detector’s choice of distributions. In this work, we propose to view object detection as a set prediction task where detectors predict the distribution over the set of objects. Using the negative log-likelihood for random finite sets, we present a proper scoring rule for evaluating and training probabilistic object detectors. The proposed method can be applied to existing probabilistic detectors, is free from thresholds, and enables fair comparison between architectures. Three different types of detectors are evaluated on the COCO dataset. Our results indicate that the training of existing detectors is optimized toward non-probabilistic metrics. We hope to encourage the development of new object detectors that can accurately estimate their own uncertainty.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    Hyperparameters are used as is unless stated otherwise.

References

  1. Abdar, M., et al.: A review of uncertainty quantification in deep learning: techniques, applications and challenges. Inf. Fusion 76, 243–297 (2021)

    Article  Google Scholar 

  2. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  3. Choi, J., Chun, D., Kim, H., Lee, H.J.: Gaussian YOLOv3: an accurate and fast object detector using localization uncertainty for autonomous driving. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 502–511 (2019)

    Google Scholar 

  4. Dawid, A.P., Musio, M.: Theory and applications of proper scoring rules. Metron 72(2), 169–183 (2014). https://doi.org/10.1007/s40300-014-0039-y

    Article  MathSciNet  MATH  Google Scholar 

  5. Feng, D., Harakeh, A., Waslander, S.L., Dietmayer, K.: A review and comparative study on probabilistic object detection in autonomous driving. IEEE Trans. Intell. Transp. Syst. 23, 9961–9980 (2021)

    Article  Google Scholar 

  6. Feng, D., Wei, X., Rosenbaum, L., Maki, A., Dietmayer, K.: Deep active learning for efficient training of a LiDAR 3D object detector. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 667–674. IEEE (2019)

    Google Scholar 

  7. García-Fernández, Á.F., Williams, J.L., Granström, K., Svensson, L.: Poisson multi-Bernoulli mixture filter: direct derivation and implementation. IEEE Trans. Aerosp. Electron. Syst. 54(4), 1883–1901 (2018)

    Article  Google Scholar 

  8. Gawlikowski, J., et al.: A survey of uncertainty in deep neural networks. arXiv preprint arXiv:2107.03342 (2021)

  9. Gneiting, T., Balabdaoui, F., Raftery, A.E.: Probabilistic forecasts, calibration and sharpness. J. Roy. Stat. Soc.: Ser. B (Stat. Methodol.) 69(2), 243–268 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  10. Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102(477), 359–378 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  11. Hall, D., et al.: Probabilistic object detection: definition and evaluation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1031–1040 (2020)

    Google Scholar 

  12. Harakeh, A., Smart, M., Waslander, S.L.: BayesOD: a Bayesian approach for uncertainty estimation in deep object detectors. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 87–93. IEEE (2020)

    Google Scholar 

  13. Harakeh, A., Waslander, S.L.: Estimating and evaluating regression predictive uncertainty in deep object detectors. In: International Conference on Learning Representations (2021)

    Google Scholar 

  14. He, Y., Wang, J.: Deep mixture density network for probabilistic object detection. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10550–10555. IEEE (2020)

    Google Scholar 

  15. Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  16. Lee, Y., Hwang, J.W., Kim, H.I., Yun, K., Kwon, Y.: Localization uncertainty estimation for anchor-free object detection. arXiv preprint arXiv:2006.15607 (2020)

  17. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)

    Google Scholar 

  18. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  19. Lyu, Z., Gutierrez, N., Rajguru, A., Beksi, W.J.: Probabilistic object detection via deep ensembles. In: Bartoli, A., Fusiello, A. (eds.) ECCV 2020. LNCS, vol. 12540, pp. 67–75. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-65414-6_7

    Chapter  Google Scholar 

  20. Mahler, R.P.: Advances in Statistical Multisource-Multitarget Information Fusion. Artech House, Norwood (2014)

    MATH  Google Scholar 

  21. Miller, D., Nicholson, L., Dayoub, F., Sünderhauf, N.: Dropout sampling for robust object detection in open-set conditions. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3243–3249. IEEE (2018)

    Google Scholar 

  22. Motro, M., Ghosh, J.: Scaling data association for hypothesis-oriented MHT. In: 2019 22th International Conference on Information Fusion (FUSION), pp. 1–8. IEEE (2019)

    Google Scholar 

  23. Murty, K.G.: An algorithm for ranking all the assignments in order of increasing cost. Oper. Res. 16(3), 682–687 (1968)

    Article  MATH  Google Scholar 

  24. Pinto, J., Xia, Y., Svensson, L., Wymeersch, H.: An uncertainty-aware performance measure for multi-object tracking. IEEE Signal Process. Lett. 28, 1689–1693 (2021)

    Article  Google Scholar 

  25. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

  26. Skafte, N., Jørgensen, M., Hauberg, S.R.: Reliable training and estimation of variance networks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  27. Sun, Z., Cao, S., Yang, Y., Kitani, K.M.: Rethinking transformer-based set prediction for object detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3611–3620 (2021)

    Google Scholar 

  28. Valdenegro-Toro, M.: I find your lack of uncertainty in computer vision disturbing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1263–1272 (2021)

    Google Scholar 

  29. Vo, B., Vo, B., Hoang, H.G.: An efficient implementation of the generalized labeled multi-Bernoulli filter. IEEE Trans. Signal Process. 65(8), 1975–1987 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  30. Vo, B.T., Vo, B.N.: Labeled random finite sets and multi-object conjugate priors. IEEE Trans. Signal Process. 61(13), 3460–3475 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  31. Williams, J.L.: Hybrid poisson and multi-Bernoulli filters. In: 2012 15th International Conference on Information Fusion, pp. 1103–1110. IEEE (2012)

    Google Scholar 

  32. Williams, J.L.: Marginal multi-Bernoulli filters: RFS derivation of MHT, JIPDA, and association-based MeMBer. IEEE Trans. Aerosp. Electron. Syst. 51(3), 1664–1687 (2015)

    Article  Google Scholar 

  33. Wirges, S., Reith-Braun, M., Lauer, M., Stiller, C.: Capturing object detection uncertainty in multi-layer grid maps. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 1520–1526. IEEE (2019)

    Google Scholar 

  34. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019). https://github.com/facebookresearch/detectron2

  35. Zhang, Y., Hare, J., Prugel-Bennett, A.: Deep set prediction networks. In: Advances in Neural Information Processing Systems, vol. 32 (2019)

    Google Scholar 

  36. Zhou, X., Koltun, V., Krähenbühl, P.: Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461 (2021)

  37. Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: deformable transformers for end-to-end object detection. In: International Conference on Learning Representations (2021)

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Computational resources were provided by the Swedish National Infrastructure for Computing at C3SE and NSC, partially funded by the Swedish Research Council, grant agreement no. 2018-05973.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Georg Hess .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 15023 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hess, G., Petersson, C., Svensson, L. (2022). Object Detection as Probabilistic Set Prediction. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13670. Springer, Cham. https://doi.org/10.1007/978-3-031-20080-9_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20080-9_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20079-3

  • Online ISBN: 978-3-031-20080-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics