Advertisement

6D Camera Relocalization in Ambiguous Scenes via Continuous Multimodal Inference

Conference paper
  • 517 Downloads
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12363)

Abstract

We present a multimodal camera relocalization framework that captures ambiguities and uncertainties with continuous mixture models defined on the manifold of camera poses. In highly ambiguous environments, which can easily arise due to symmetries and repetitive structures in the scene, computing one plausible solution (what most state-of-the-art methods currently regress) may not be sufficient. Instead we predict multiple camera pose hypotheses as well as the respective uncertainty for each prediction. Towards this aim, we use Bingham distributions, to model the orientation of the camera pose, and a multivariate Gaussian to model the position, with an end-to-end deep neural network. By incorporating a Winner-Takes-All training scheme, we finally obtain a mixture model that is well suited for explaining ambiguities in the scene, yet does not suffer from mode collapse, a common problem with mixture density networks. We introduce a new dataset specifically designed to foster camera localization research in ambiguous environments and exhaustively evaluate our method on synthetic as well as real data on both ambiguous scenes and on non-ambiguous benchmark datasets. We plan to release our code and dataset under multimodal3dvision.github.io.

Notes

Acknowledgements

This project is supported by Bavaria California Technology Center (BaCaTeC), Stanford-Ford Alliance, NSF grant IIS-1763268, Vannevar Bush Faculty Fellowship, Samsung GRO program, the Stanford SAIL Toyota Research, and the PRIME programme of the German Academic Exchange Service (DAAD) with funds from the German Federal Ministry of Education and Research (BMBF).

Supplementary material

504473_1_En_9_MOESM1_ESM.pdf (4.6 mb)
Supplementary material 1 (pdf 4717 KB)

References

  1. 1.
    Arun Srivatsan, R., Xu, M., Zevallos, N., Choset, H.: Probabilistic pose estimation using a Bingham distribution-based linear filter. Int. J. Robot. Res. 37(13–14), 1610–1631 (2018)CrossRefGoogle Scholar
  2. 2.
    Barfoot, T.D., Furgale, P.T.: Associating uncertainty with three-dimensional poses for use in estimation problems. IEEE Trans. Robot. 30(3), 679–693 (2014)CrossRefGoogle Scholar
  3. 3.
    Bingham, C.: An antipodally symmetric distribution on the sphere. Ann. Stat. 1201–1225 (1974)Google Scholar
  4. 4.
    Birdal, T., Arbel, M., Şimşekli, U., Guibas, L.: Synchronizing probability measures on rotations via optimal transport. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020)Google Scholar
  5. 5.
    Birdal, T., Bala, E., Eren, T., Ilic, S.: Online inspection of 3D parts via a locally overlapping camera network. In: 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE (2016)Google Scholar
  6. 6.
    Birdal, T., Simsekli, U.: Probabilistic permutation synchronization using the Riemannian structure of the Birkhoff polytope. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 11105–11116 (2019)Google Scholar
  7. 7.
    Birdal, T., Simsekli, U., Eken, M.O., Ilic, S.: Bayesian pose graph optimization via Bingham distributions and tempered geodesic MCMC. In: Advances in Neural Information Processing Systems, pp. 308–319 (2018)Google Scholar
  8. 8.
    Bishop, C.M.: Mixture density networks (1994)Google Scholar
  9. 9.
    Bourmaud, G., Mégret, R., Arnaudon, M., Giremus, A.: Continuous-discrete extended Kalman filter on matrix lie groups using concentrated Gaussian distributions. Jo. Math. Imaging Vis. 51(1), 209–228 (2015)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Brachmann, E., et al.: DSAC-differentiable RANSAC for camera localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  11. 11.
    Brachmann, E., Michel, F., Krull, A., Ying Yang, M., Gumhold, S., et al.: Uncertainty-driven 6D pose estimation of objects and scenes from a single RGB image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3364–3372 (2016)Google Scholar
  12. 12.
    Brachmann, E., Rother, C.: Learning less is more-6D camera localization via 3D surface regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4654–4662 (2018)Google Scholar
  13. 13.
    Brahmbhatt, S., Gu, J., Kim, K., Hays, J., Kautz, J.: Geometry-aware learning of maps for camera localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2616–2625 (2018)Google Scholar
  14. 14.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefGoogle Scholar
  15. 15.
    Bui, M., Albarqouni, S., Ilic, S., Navab, N.: Scene coordinate and correspondence learning for image-based localization. In: British Machine Vision Conference (BMVC) (2018)Google Scholar
  16. 16.
    Busam, B., Birdal, T., Navab, N.: Camera pose filtering with local regression geodesics on the Riemannian manifold of dual quaternions. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2436–2445 (2017)Google Scholar
  17. 17.
    Cadena, C., et al.: Past, present, and future of simultaneous localization and mapping: toward the robust-perception age. IEEE Trans. Robot. 32(6), 1309–1332 (2016)CrossRefGoogle Scholar
  18. 18.
    Clark, R., Wang, S., Markham, A., Trigoni, N., Wen, H.: VidLoc: a deep spatio-temporal model for 6-DoF video-clip relocalization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  19. 19.
    Corona, E., Kundu, K., Fidler, S.: Pose estimation for objects with rotational symmetry. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7215–7222. IEEE (2018)Google Scholar
  20. 20.
    Cui, H., et al.: Multimodal trajectory predictions for autonomous driving using deep convolutional networks. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 2090–2096. IEEE (2019)Google Scholar
  21. 21.
    Deng, H., Birdal, T., Ilic, S.: 3D local features for direct pairwise registration. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019Google Scholar
  22. 22.
    Durrant-Whyte, H., Bailey, T.: Simultaneous localization and mapping: part I. IEEE Robot. Autom. Mag. 13(2), 99–110 (2006)CrossRefGoogle Scholar
  23. 23.
    Falorsi, L., de Haan, P., Davidson, T.R., Forré, P.: Reparameterizing distributions on lie groups. arXiv preprint arXiv:1903.02958 (2019)
  24. 24.
    Feng, W., Tian, F.P., Zhang, Q., Sun, J.: 6D dynamic camera relocalization from single reference image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4049–4057 (2016)Google Scholar
  25. 25.
    Firman, M., Campbell, N.D., Agapito, L., Brostow, G.J.: DiverseNet: when one right answer is not enough. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5598–5607 (2018)Google Scholar
  26. 26.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  27. 27.
    Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: International Conference on Machine Learning, pp. 1050–1059 (2016)Google Scholar
  28. 28.
    Gilitschenski, I., Sahoo, R., Schwarting, W., Amini, A., Karaman, S., Rus, D.: Deep orientation uncertainty learning based on a Bingham loss. In: International Conference on Learning Representations (2020)Google Scholar
  29. 29.
    Glover, J., Kaelbling, L.P.: Tracking the spin on a ping pong ball with the quaternion Bingham filter. In: 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 4133–4140, May 2014Google Scholar
  30. 30.
    Glover, J., Bradski, G., Rusu, R.B.: Monte Carlo pose estimation with quaternion kernels and the Bingham distribution. In: Robotics Science System (2012)Google Scholar
  31. 31.
    Glover, J.M.: The quaternion Bingham distribution, 3D object detection, and dynamic manipulation. Ph.D. thesis, Massachusetts Institute of Technology (2014)Google Scholar
  32. 32.
    Grassia, F.S.: Practical parameterization of rotations using the exponential map. J. Graph. Tools 3(3), 29–48 (1998)CrossRefGoogle Scholar
  33. 33.
    Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 1321–1330. JMLR. org (2017)Google Scholar
  34. 34.
    Guzman-Rivera, A., Batra, D., Kohli, P.: Multiple choice learning: learning to produce multiple structured outputs. In: Advances in Neural Information Processing Systems, pp. 1799–1807 (2012)Google Scholar
  35. 35.
    Haarbach, A., Birdal, T., Ilic, S.: Survey of higher order rigid body motion interpolation methods for keyframe animation and continuous-time trajectory estimation. In: 2018 Sixth International Conference on 3D Vision (3DV), pp. 381–389. IEEE (2018).  https://doi.org/10.1109/3DV.2018.00051
  36. 36.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  37. 37.
    Herz, C.S.: Bessel functions of matrix argument. Ann. Math. 61(3), 474–523 (1955). http://www.jstor.org/stable/1969810MathSciNetCrossRefGoogle Scholar
  38. 38.
    Hinterstoisser, S., et al.: Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes. In: Lee, K.M., Matsushita, Y., Rehg, J.M., Hu, Z. (eds.) ACCV 2012. LNCS, vol. 7724, pp. 548–562. Springer, Heidelberg (2013).  https://doi.org/10.1007/978-3-642-37331-2_42CrossRefGoogle Scholar
  39. 39.
    Horaud, R., Conio, B., Leboulleux, O., Lacolle, B.: An analytic solution for the perspective 4-point problem. In: Proceedings CVPR 1989: IEEE Computer Society Conference on Computer Vision and Pattern Recognition. IEEE (1989)Google Scholar
  40. 40.
    Kendall, A., Cipolla, R.: Modelling uncertainty in deep learning for camera relocalization. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 4762–4769. IEEE (2016)Google Scholar
  41. 41.
    Kendall, A., Cipolla, R., et al.: Geometric loss functions for camera pose regression with deep learning. In: Proceedings of CVPR, vol. 3, p. 8 (2017)Google Scholar
  42. 42.
    Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems (2017)Google Scholar
  43. 43.
    Kendall, A., Grimes, M., Cipolla, R.: PoseNet: a convolutional network for real-time 6-DoF camera relocalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2938–2946 (2015)Google Scholar
  44. 44.
    Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  45. 45.
    Kume, A., Wood, A.T.: Saddlepoint approximations for the bingham and fisher-bingham normalising constants. Biometrika 92(2), 465–476 (2005)MathSciNetCrossRefGoogle Scholar
  46. 46.
    Kurz, G., Gilitschenski, I., Julier, S., Hanebeck, U.D.: Recursive estimation of orientation based on the Bingham distribution. In: 2013 16th International Conference on Information Fusion (FUSION), pp. 1487–1494. IEEE (2013)Google Scholar
  47. 47.
    Kurz, G., et al.: Directional statistics and filtering using libdirectional. arXiv preprint arXiv:1712.09718 (2017)
  48. 48.
    Labbé, M., Michaud, F.: Rtab-map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. J. Field Robot. 36(2), 416–446 (2019)CrossRefGoogle Scholar
  49. 49.
    Makansi, O., Ilg, E., Cicek, O., Brox, T.: Overcoming limitations of mixture density networks: a sampling and fitting framework for multimodal future prediction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7144–7153 (2019)Google Scholar
  50. 50.
    Manhardt, F., et al.: Explaining the ambiguity of object detection and 6D pose from visual data. In: International Conference of Computer Vision. IEEE/CVF (2019)Google Scholar
  51. 51.
    Mardia, K.V., Jupp, P.E.: Directional Statistics. Wiley, Hoboken (2009)zbMATHGoogle Scholar
  52. 52.
    Massiceti, D., Krull, A., Brachmann, E., Rother, C., Torr, P.H.: Random forests versus neural networks–what’s best for camera localization? In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2017)Google Scholar
  53. 53.
    Morawiec, A., Field, D.: Rodrigues parameterization for orientation and misorientation distributions. Philos. Mag. A 73(4), 1113–1130 (1996)CrossRefGoogle Scholar
  54. 54.
    Murray, R.M.: A Mathematical Introduction to Robotic Manipulation. CRC Press, Boca Raton (1994)zbMATHGoogle Scholar
  55. 55.
    Paszke, A., et al.: Automatic differentiation in PyTorch. In: NIPS Autodiff Workshop (2017)Google Scholar
  56. 56.
    Peretroukhin, V., Wagstaff, B., Giamou, M., Kelly, J.: Probabilistic regression of rotations using quaternion averaging and a deep multi-headed network. arXiv preprint arXiv:1904.03182 (2019)
  57. 57.
    Piasco, N., Sidibé, D., Demonceaux, C., Gouet-Brunet, V.: A survey on visual-based localization: on the benefit of heterogeneous data. Pattern Recogn. 74, 90–109 (2018)CrossRefGoogle Scholar
  58. 58.
    Pitteri, G., Ramamonjisoa, M., Ilic, S., Lepetit, V.: On object symmetries and 6D pose estimation from images. In: 3D Vision (3DV). IEEE (2019)Google Scholar
  59. 59.
    Prokudin, S., Gehler, P., Nowozin, S.: Deep directional statistics: pose estimation with uncertainty quantification. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 534–551 (2018)Google Scholar
  60. 60.
    Qi, C.R., Litany, O., He, K., Guibas, L.J.: Deep hough voting for 3D object detection in point clouds. In: The IEEE International Conference on Computer Vision (ICCV), October 2019Google Scholar
  61. 61.
    Riedel, S., Marton, Z.C., Kriegel, S.: Multi-view orientation estimation using Bingham mixture models. In: 2016 IEEE International Conference on Automation, Quality and Testing, Robotics (AQTR), pp. 1–6. IEEE (2016)Google Scholar
  62. 62.
    Rubner, Y., Tomasi, C., Guibas, L.J.: The earth mover’s distance as a metric for image retrieval. Int. J. Comput. Vis. 40(2), 99–121 (2000)CrossRefGoogle Scholar
  63. 63.
    Rupprecht, C., et al.: Learning in an uncertain world: representing ambiguity through multiple hypotheses. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3591–3600 (2017)Google Scholar
  64. 64.
    Salas-Moreno, R.F., Newcombe, R.A., Strasdat, H., Kelly, P.H., Davison, A.J.: SLAM++: simultaneous localisation and mapping at the level of objects. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1352–1359 (2013)Google Scholar
  65. 65.
    Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., Chen, X.: Improved techniques for training GANs. In: Advances in Neural Information Processing Systems, pp. 2234–2242 (2016)Google Scholar
  66. 66.
    Sattler, T., Havlena, M., Radenovic, F., Schindler, K., Pollefeys, M.: Hyperpoints and fine vocabularies for large-scale location recognition. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2102–2110 (2015)Google Scholar
  67. 67.
    Sattler, T., Zhou, Q., Pollefeys, M., Leal-Taixe, L.: Understanding the limitations of CNN-based absolute camera pose regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3302–3312 (2019)Google Scholar
  68. 68.
    Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  69. 69.
    Shotton, J., Glocker, B., Zach, C., Izadi, S., Criminisi, A., Fitzgibbon, A.: Scene coordinate regression forests for camera relocalization in RGB-D images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2930–2937 (2013)Google Scholar
  70. 70.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  71. 71.
    Suvrit, S., Ley, C., Verdebout, T.: Directional statistics in machine learning: a brief review. In: Applied Directional Statistics. Chapman and Hall/CRC (2018)Google Scholar
  72. 72.
    Ullman, S.: The interpretation of structure from motion. Proc. Roy. Soc. London. Ser. B. Biol. Sci. 203(1153), 405–426 (1979)Google Scholar
  73. 73.
    Valentin, J., Nießner, M., Shotton, J., Fitzgibbon, A., Izadi, S., Torr, P.H.: Exploiting uncertainty in regression forests for accurate camera relocalization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4400–4408 (2015)Google Scholar
  74. 74.
    Yamaji, A.: Genetic algorithm for fitting a mixed bingham distribution to 3D orientations: a tool for the statistical and paleostress analyses of fracture orientations. Island Arc 25(1), 72–83 (2016)CrossRefGoogle Scholar
  75. 75.
    Zakharov, S., Shugurov, I., Ilic, S.: DPOD: 6D pose object detector and refiner. In: The IEEE International Conference on Computer Vision (ICCV) (2019)Google Scholar
  76. 76.
    Zeisl, B., Sattler, T., Pollefeys, M.: Camera pose voting for large-scale image-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2704–2712 (2015)Google Scholar
  77. 77.
    Zolfaghari, M., Çiçek, Ö., Ali, S.M., Mahdisoltani, F., Zhang, C., Brox, T.: Learning representations for predicting future activities. arXiv:1905.03578 (2019)

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Technical University of MunichMunichGermany
  2. 2.Stanford UniversityStanfordUSA
  3. 3.Siemens AGMunichGermany
  4. 4.ETH ZurichZurichSwitzerland
  5. 5.Johns Hopkins UniversityBaltimoreUSA

Personalised recommendations