Advertisement

International Journal of Computer Vision

, Volume 127, Issue 5, pp 415–436 | Cite as

Robust and Optimal Registration of Image Sets and Structured Scenes via Sum-of-Squares Polynomials

  • Danda Pani PaudelEmail author
  • Adlane Habed
  • Cédric Demonceaux
  • Pascal Vasseur
Article
  • 227 Downloads

Abstract

This paper addresses the problem of registering a known structured 3D scene, typically a 3D scan, and its metric Structure-from-Motion (SfM) counterpart. The proposed registration method relies on a prior plane segmentation of the 3D scan. Alignment is carried out by solving either the point-to-plane assignment problem, should the SfM reconstruction be sparse, or the plane-to-plane one in case of dense SfM. A Polynomial Sum-of-Squares optimization theory framework is employed for identifying point-to-plane and plane-to-plane mismatches, i.e. outliers, with certainty. An inlier set maximization approach within a Branch-and-Bound search scheme is adopted to iteratively build potential inlier sets and converge to the solution satisfied by the largest number of assignments. Plane visibility conditions and vague camera locations may be incorporated for better efficiency without sacrificing optimality. The registration problem is solved in two cases: (i) putative correspondences (with possibly overwhelmingly many outliers) are provided as input and (ii) no initial correspondences are available. Our approach yields outstanding results in terms of robustness and optimality.

Keywords

2D–3D registration Structure-from-Motion Polynomial Sum-of-Squares optimization 

Notes

Acknowledgements

This research has been funded by the International Project NRF-ANR DrAACaR: ANR-11-ISO3-0003, the Regional Council of Bourgogne and European Regional Development Fund.

References

  1. Bartoli, A., & Castellani, U. (2012). 3D shape registration. In 3D imaging, analysis, and applications, Springer (pp. 221–264).Google Scholar
  2. Bazin, J., Li, H., Kweon, I. S., Demonceaux, C., Vasseur, P., & Ikeuchi, K. (2013). A branch-and-bound approach to correspondence and grouping problems. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 35, 1565–1576.CrossRefGoogle Scholar
  3. Borrmann, D., Elseberg, J., Lingemann, K., & Nüchter, A. (2011). The 3D hough transform for plane detection in point clouds: A review and a new accumulator design. 3D. Research, 32(1–32), 13.Google Scholar
  4. Boyd, S., & Vandenberghe, L. (2004). Convex optimization. New York, NY: Cambridge University Press.CrossRefzbMATHGoogle Scholar
  5. Breuel, T. M. (2003). Implementation techniques for geometric branch-and-bound matching methods. Computer Vision and Image Understanding, 90(3), 258–294.CrossRefzbMATHGoogle Scholar
  6. Chandraker, M., Agarwal, S., Kahl, F., Nister, D., & Kriegman, D. (2007). Autocalibration via rank-constrained estimation of the absolute quadric. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–8).Google Scholar
  7. Chesi, G., Garulli, A., Vicino, A., & Cipolla, R. (2002). Estimating the fundamental matrix via constrained least-squares: A convex approach. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 24, 397–401.CrossRefGoogle Scholar
  8. Choi, M. D., Lam, T. Y., & Reznick, B. (1995). Sums of squares of real polynomials. Proceedings of Symposia in Pure Mathematics, 2(58), 103–126.MathSciNetzbMATHGoogle Scholar
  9. Christy, S., & Horaud, R. (1999). Iterative pose computation from line correspondences. Computer Vision and Image Understanding (CVIU), 73, 137–144.CrossRefzbMATHGoogle Scholar
  10. Corsini, M., Dellepiane, M., Ganovelli, F., Gherardi, R., Fusiello, A., & Scopigno, R. (2013). Fully automatic registration of image sets on approximate geometry. International Journal of Computer Vision (IJCV), 102, 91–111.CrossRefGoogle Scholar
  11. Du, S., Zheng, N., Ying, S., You, Q., & Wu, Y. (2007). An extension of the ICP algorithm considering scale factor. In IEEE international conference on image processing (ICIP) (pp. V–193).Google Scholar
  12. Enqvist, O., Josephson, K., & Kahl, F. (2009). Optimal correspondences from pairwise constraints. In 2009 IEEE 12th international conference on computer vision (pp. 1295–1302). IEEE.Google Scholar
  13. Enqvist, O., & Kahl, F. (2008). Robust optimal pose estimation. Computer Vision-ECCV, 2008, 141–153.Google Scholar
  14. Ferraz, L., Binefa, X., & Moreno-Noguer, F. (2014). Very fast solution to the pnp problem with algebraic outlier rejection. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 501–508).Google Scholar
  15. Finsler, P. (1936/37). Uber das vorkommen definiter und semidefiniter formen in scharen quadratischer formen. Commentarii Mathematici Helvetici, 9, 188–192.Google Scholar
  16. Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communication of the ACM, 24, 381–395.MathSciNetCrossRefGoogle Scholar
  17. Fitzgibbon, A. W. (2001). Robust registration of 2D and 3D point sets. In The British machine vision conference (BMVC) (pp. 662–670).Google Scholar
  18. Fraundorfer, F., & Scaramuzza, D. (2012). Visual odometry: Part II: Matching, robustness, optimization, and applications. IEEE Robotics & Automation Magazine, 19(2), 78–90.CrossRefGoogle Scholar
  19. Habed, A., Al Ismaeil, K., & Fofi, D. (2012). A new set of quartic trivariate polynomial equations for stratified camera self-calibration under zero-skew and constant parameters assumptions. In European conference on computer vision (ECCV) (pp. 710–723).Google Scholar
  20. Hartley, R. I., & Zisserman, A. (2004). Multiple view geometry in computer vision (2nd ed.). Cambridge: Cambridge University Press.CrossRefzbMATHGoogle Scholar
  21. Hilbert, D. (1888). Uber die darstellung definiter formen als summe von formen quadraten. Mathematische Annalen, 32, 342–350.MathSciNetCrossRefzbMATHGoogle Scholar
  22. Hough Transform Plane Detector. (2015). Howpublished. https://github.com/daviddoria/vtkhoughplanes/
  23. Jensen, R., Dahl, A., Vogiatzis, G., Tola, E., & Aanæs, H. (2014). Large scale multi-view stereopsis evaluation. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 406–413).Google Scholar
  24. Jurie, F. (1999). Solution of the simultaneous pose and correspondence problem using gaussian error model. Computer Vision and Image Understanding, 73(3), 357–373.CrossRefzbMATHGoogle Scholar
  25. Kahl, F., & Henrion, D. (2007). Globally optimal estimates for geometric reconstruction problems. International Journal of Computer Vision, 74(1), 3–15.CrossRefGoogle Scholar
  26. Lasserre, J. B. (2000). Global optimization with polynomials and the problem of moments. SIAM Journal on Optimization, 11, 796–817.MathSciNetCrossRefzbMATHGoogle Scholar
  27. Li, H. (2009). Consensus set maximization with guaranteed global optimality for robust geometry estimation. In IEEE international conference on computer vision (ICCV) (pp. 1074–1080).Google Scholar
  28. Liu, L., & Stamos, I. (2005). Automatic 3D to 2D registration for the photorealistic rendering of urban scenes. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 137–143).Google Scholar
  29. Mastin, A., Kepner, J., & Fisher III, J. W. (2009). Automatic registration of LIDAR and optical images of urban scenes. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2639–2646).Google Scholar
  30. Moulon, P., Monasse, P., & Marlet, R. (2013). Adaptive structure from motion with a contrario model estimation. In Asian conference on computer vision (ACCV) (pp. 257–270).Google Scholar
  31. Olsson, C., Kahl, F., & Oskarsson, M. (2006). The registration problem revisited: Optimal solutions from points, lines and planes. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1206–1213).Google Scholar
  32. Parrilo, P. A. (2000). Structured semidefinite programs and semialgebraic geometry methods in robustness and optimization. Technical report, California Institute of TechnologyGoogle Scholar
  33. Paudel, D. P., Demonceaux, C., Habed, A., & Vasseur, P. (2014). Localization of 2D cameras in a known environment using direct 2D–3D registration. In International conference on pattern recognition (ICPR) (pp. 1–6).Google Scholar
  34. Plotz, T., & Roth, S. (2015). Registering images to untextured geometry using average shading gradients. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 2030–2038).Google Scholar
  35. Powers, V., & Wörmann, T. (1998). An algorithm for sums of squares of real polynomials. Journal of Pure and Applied Algebra, 127(1), 99–104.MathSciNetCrossRefzbMATHGoogle Scholar
  36. Putinar, M. (1993). Positive polynomials on compact semi-algebraic sets. Indiana University Mathematics Journal, 42, 969–984.MathSciNetCrossRefzbMATHGoogle Scholar
  37. Ramalingam, S., & Taguchi, Y. (2013). A theory of minimal 3D point to 3D plane registration and its generalization. International Journal of Computer Vision, 102(1–3), 73–90.MathSciNetCrossRefzbMATHGoogle Scholar
  38. Rusinkiewicz, S., & Levoy, M. (2001). Efficient variants of the ICP algorithm. In 3-D digital imaging and modeling (3DIM) (pp. 145–152).Google Scholar
  39. Schindler, G., Krishnamurthy, P., Lublinerman, R., Liu, Y., & Dellaert, F. (2008). Detecting and matching repeated patterns for automatic geo-tagging in urban environments. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–7).Google Scholar
  40. Schweighofer, G., & Pinz, A. (2008). Globally optimal o(n) solution to the pnp problem for general camera models. In The British machine vision conference (BMVC) (pp. 1–10).Google Scholar
  41. Segal, A. V., Haehnel, D., & Thrun, S. (2009). Generalized-ICP. In Robotics: Science and systems (RSS).Google Scholar
  42. Strecha, C., von Hansen, W., Van Gool, L., Fua, P., & Thoennessen, U. (2008). On benchmarking camera calibration and multi-view stereo for high resolution imagery. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 1–8).Google Scholar
  43. Tamaazousti, M., Gay-Bellile, V., Collette, S. N., Bourgeois, S., & Dhome, M. (2011). Nonlinear refinement of structure from motion reconstruction by taking advantage of a partial knowledge of the environment. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 3073–3080).Google Scholar
  44. Taneja, A., Ballan, L., & Pollefeys, M. (2013). City-scale change detection in cadastral 3D models using images. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 113–120).Google Scholar
  45. Verschelde, J. (1999). Algorithm 795: Phcpack: A general-purpose solver for polynomial systems by homotopy continuation. ACM Transactions on Mathematical Software (TOMS), 25(2), 251–276.CrossRefzbMATHGoogle Scholar
  46. Viola, P., & Wells, M, I. I. I. (1997). Alignment by maximization of mutual information. International Journal of Computer Vision (IJCV), 24, 137–154.CrossRefGoogle Scholar
  47. Wagner, S. (2009). Archimedean quadratic modules: A decision problem for real multivariate polynomials. Ph.D. thesis, Universität Konstanz.Google Scholar
  48. Yang, J., Li, H., & Jia, Y. (2013). Go-icp: Solving 3D registration efficiently and globally optimally. In IEEE international conference on computer vision (ICCV) (pp. 1457–1464).Google Scholar
  49. Yang, J., Li, H., & Jia, Y. (2014). Optimal essential matrix estimation via inlier-set maximization. In European conference on computer vision (ECCV) (pp. 111–126).Google Scholar
  50. Zhang, X., Agam, G., & Chen, X. (2014). Alignment of 3d building models with satellite images using extended chamfer matching. In IEEE conference on computer vision and pattern recognition workshops (CVPRW) (pp. 746–753).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Computer Vision LabETH ZurichZurichSwitzerland
  2. 2.ICube Laboratory, CNRSUniversity of StrasbourgStrasbourgFrance
  3. 3.Le2i Laboratory, CNRSUniversity of Bourgogne Franche-ComtéDijonFrance
  4. 4.Laboratoire d’Informatique, de Traitement de l’Information et des SystèmesNormandie Univ, UNIROUEN, UNIHAVRE, INSA Rouen, LITISRouenFrance

Personalised recommendations