Abstract
Reliable spatial uncertainty evaluation of object detection models is of special interest and has been subject of recent work. In this work, we review the existing definitions for uncertainty calibration of probabilistic regression tasks. We inspect the calibration properties of common detection networks and extend state-of-the-art recalibration methods. Our methods use a Gaussian process (GP) recalibration scheme that yields parametric distributions as output (e.g. Gaussian or Cauchy). The usage of GP recalibration allows for a local (conditional) uncertainty calibration by capturing dependencies between neighboring samples. The use of parametric distributions such as Gaussian allows for a simplified adaption of calibration in subsequent processes, e.g., for Kalman filtering in the scope of object tracking.
In addition, we use the GP recalibration scheme to perform covariance estimation which allows for post-hoc introduction of local correlations between the output quantities, e.g., position, width, or height in object detection. To measure the joint calibration of multivariate and possibly correlated data, we introduce the quantile calibration error which is based on the Mahalanobis distance between the predicted distribution and the ground truth to determine whether the ground truth is within a predicted quantile.
Our experiments show that common detection models overestimate the spatial uncertainty in comparison to the observed error. We show that the simple Isotonic Regression recalibration method is sufficient to achieve a good uncertainty quantification in terms of calibrated quantiles. In contrast, if normal distributions are required for subsequent processes, our GP-Normal recalibration method yields the best results. Finally, we show that our covariance estimation method is able to achieve best calibration results for joint multivariate calibration. All code is open source and available at https://github.com/EFS-OpenSource/calibration-framework.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Banerjee, K., Notz, D., Windelen, J., Gavarraju, S., He, M.: Online camera lidar fusion and object detection on hybrid data for autonomous driving. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1632–1638. IEEE (2018)
Bar-Shalom, Y., Li, X.R., Kirubarajan, T.: Estimation with Applications to Tracking and Navigation: Theory Algorithms and Software. Wiley, New York (2004)
Bhatt, D., Mani, K., Bansal, D., Murthy, K., Lee, H., Paull, L.: \( f \)-Cal: calibrated aleatoric uncertainty estimation from neural networks for robot perception. arXiv preprint arXiv:2109.13913 (2021)
Chung, Y., Neiswanger, W., Char, I., Schneider, J.: Beyond pinball loss: quantile methods for calibrated uncertainty quantification. In: Advances in Neural Information Processing Systems, vol. 34 (2021)
Cui, P., Hu, W., Zhu, J.: Calibrated reliable regression using maximum mean discrepancy. In: Advances in Neural Information Processing Systems, vol. 33 (2020)
Fasiolo, M., Wood, S.N., Zaffran, M., Nedellec, R., Goude, Y.: Fast calibrated additive quantile regression. J. Am. Stat. Assoc. 116, 1402–1412 (2020). https://doi.org/10.1080/01621459.2020.1725521
Feng, D., Harakeh, A., Waslander, S.L., Dietmayer, K.: A review and comparative study on probabilistic object detection in autonomous driving. IEEE Trans. Intell. Transp. Syst. (2021)
Feng, D., Rosenbaum, L., Glaeser, C., Timm, F., Dietmayer, K.: Can we trust you? On calibration of a probabilistic object detector for autonomous driving. arXiv preprint (2019)
Gardner, J.R., Pleiss, G., Bindel, D., Weinberger, K.Q., Wilson, A.G.: GPyTorch: blackbox matrix-matrix Gaussian process inference with GPU acceleration. In: Advances in Neural Information Processing Systems (2018)
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 70, pp. 1321–1330. PMLR, August 2017
Hall, D., et al.: Probabilistic object detection: definition and evaluation. In: The IEEE Winter Conference on Applications of Computer Vision, pp. 1031–1040 (2020)
Harakeh, A., Smart, M., Waslander, S.L.: BayesOD: a Bayesian approach for uncertainty estimation in deep object detectors. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 87–93. IEEE (2020)
He, Y., Zhu, C., Wang, J., Savvides, M., Zhang, X.: Bounding box regression with uncertainty for accurate object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2888–2897 (2019)
Hensman, J., Matthews, A., Ghahramani, Z.: Scalable variational gaussian process classification. In: Artificial Intelligence and Statistics, pp. 351–360. PMLR (2015)
Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems (NIPS), pp. 5574–5584 (2017)
Kuleshov, V., Fenner, N., Ermon, S.: Accurate uncertainties for deep learning using calibrated regression. In: International Conference on Machine Learning (ICML), pp. 2801–2809 (2018)
Kull, M., Silva Filho, T., Flach, P.: Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In: Artificial Intelligence and Statistics, pp. 623–631 (2017)
Kumar, A., Liang, P.S., Ma, T.: Verified uncertainty calibration. In: Advances in Neural Information Processing Systems, vol. 32, pp. 3792–3803. Curran Associates, Inc. (2019). http://papers.nips.cc/paper/8635-verified-uncertainty-calibration.pdf
Küppers, F., Kronenberger, J., Shantia, A., Haselhoff, A.: Multivariate confidence calibration for object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 326–327 (2020)
Laves, M.H., Ihler, S., Fast, J.F., Kahrs, L.A., Ortmaier, T.: Well-calibrated regression uncertainty in medical imaging with deep learning. In: Medical Imaging with Deep Learning, pp. 393–412. PMLR (2020)
Levi, D., Gispan, L., Giladi, N., Fetaya, E.: Evaluating and calibrating uncertainty prediction in regression tasks. arXiv preprint abs/1905.11659 (2019)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988 (2017)
Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Naeini, M.P., Cooper, G.F., Hauskrecht, M.: Binary classifier calibration using a Bayesian non-parametric approach. In: Proceedings of the 2015 SIAM International Conference on Data Mining, pp. 208–216 (2015). https://doi.org/10.1137/1.9781611974010.24
Naeini, M., Cooper, G., Hauskrecht, M.: Obtaining well calibrated probabilities using Bayesian binning. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence, pp. 2901–2907 (2015)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 91–99 (2015)
Schwaiger, F., Henne, M., Küppers, F., Schmoeller Roza, F., Roscher, K., Haselhoff, A.: From black-box to white-box: examining confidence calibration under different conditions. In: Proceedings of the Workshop on Artificial Intelligence Safety 2021 (SafeAI 2021) Co-located with the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI 2021) (2021)
SenGupta, A.: Tests for standardized generalized variances of multivariate normal populations of possibly different dimensions. J. Multivariate Anal. 23(2), 209–219 (1987)
SenGupta, A.: Generalized variance. Encycl. Stat. Sci. 6053 (2004)
Snelson, E., Ghahramani, Z.: Sparse Gaussian processes using pseudo-inputs. Adv. Neural Inf. Process. Syst. 18, 1257 (2006)
Song, H., Diethe, T., Kull, M., Flach, P.: Distribution calibration for regression. In: Proceedings of the 36th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 97, pp. 5897–5906. PMLR, Long Beach, California, USA, 9–15 June 2019
Song, L., Zhang, X., Smola, A., Gretton, A., Schölkopf, B.: Tailoring density estimation via reproducing Kernel moment matching. In: Proceedings of the 25th International Conference on Machine Learning, pp. 992–999 (2008)
Steinwart, I., Christmann, A.: Estimating conditional quantiles with the help of the pinball loss. Bernoulli 17(1), 211–225 (2011)
Van Der Heijden, F., Duin, R.P., De Ridder, D., Tax, D.M.: Classification, Parameter Estimation and State Estimation: An Engineering Approach Using MATLAB. Wiley, New York (2005)
Yang, Z., Li, J., Li, H.: Real-time pedestrian and vehicle detection for autonomous driving. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 179–184. IEEE (2018)
Yu, F., et al.: BDD100K: a diverse driving video database with scalable annotation tooling. CoRR (2018)
Zadrozny, B., Elkan, C.: Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the Eighth International Conference on Knowledge Discovery and Data Mining, 23–26 July 2002, Edmonton, Alberta, Canada, pp. 694–699 (2002). https://doi.org/10.1145/775047.775151
Acknowledgement
The authors gratefully acknowledge support of this work by Elektronische Fahrwerksysteme GmbH, Gaimersheim, Germany. The research leading to the results presented above are funded by the German Federal Ministry for Economic Affairs and Energy within the project “KI Absicherung - Safe AI for automated driving”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Küppers, F., Schneider, J., Haselhoff, A. (2023). Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection. In: Karlinsky, L., Michaeli, T., Nishino, K. (eds) Computer Vision – ECCV 2022 Workshops. ECCV 2022. Lecture Notes in Computer Science, vol 13805. Springer, Cham. https://doi.org/10.1007/978-3-031-25072-9_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-25072-9_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-25071-2
Online ISBN: 978-3-031-25072-9
eBook Packages: Computer ScienceComputer Science (R0)