Globally Optimal Object Pose Estimation in Point Clouds with Mixed-Integer Programming

  • Gregory IzattEmail author
  • Hongkai Dai
  • Russ Tedrake
Conference paper
Part of the Springer Proceedings in Advanced Robotics book series (SPAR, volume 10)


Motivated by the limitations of local object trackers, we present a formulation of the underlying point-cloud object pose estimation problem as a mixed-integer convex program, which we efficiently solve to optimality with an off-the-shelf branch and bound solver. We show that reasoning about object pose estimation in this way allows natural extension to point-to-mesh correspondence, multiple simultaneous object pose estimation, and outlier rejection without losing the ability to obtain a globally optimal solution. We probe the extent to which rich problem-specific formulations typically tackled with unreliable nonlinear optimization can be rigorously treated in a global optimization framework to overcome the limitations of other global pose estimation methods.



This material is based upon work supported by NSF Contract IIS-1427050, a National Science Foundation Graduate Research Fellowship under Grant No. 1122374, as well as support from ABB and Draper Laboratory.


  1. 1.
    Akgül, C.B., Sankur, B., Yemez, Y., Schmitt, F.: 3d model retrieval using probability density-based shape descriptors. IEEE Trans. Pattern Anal. Mach. Intell. 31(6), 1117–1133 (2009)CrossRefGoogle Scholar
  2. 2.
    Arie-Nachimson, M., Kovalsky, S.Z., Kemelmacher-Shlizerman, I., Singer, A., Basri, R.: Global motion estimation from point matches. In 2012 second international conference on 3D imaging, modeling, processing, visualization and transmission (3DIMPVT), pp. 81–88. IEEE (2012)Google Scholar
  3. 3.
    Besl, P.J., McKay, N.D.: Method for registration of 3-d shapes. In: Robotics-DL tentative, pp. 586–606. International Society for Optics and Photonics (1992)Google Scholar
  4. 4.
    Chaudhury, K.N., Khoo, Y., Singer, A.: Global registration of multiple point clouds using semidefinite programming. SIAM J. Optim. 25(1), 468–501 (2015)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Chen, Y., Medioni, G.: Object modelling by registration of multiple range images. Image Vis. Comput. 10(3), 145–155 (1992)CrossRefGoogle Scholar
  6. 6.
    Dai, H., Izatt, G., Tedrake, R.: Global inverse kinematics via mixed-integer convex optimization (2017)Google Scholar
  7. 7.
    Drost, B., Ulrich, M., Navab, N., Ilic, S.: Model globally, match locally: efficient and robust 3d object recognition. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp. 998–1005. IEEE (2010)Google Scholar
  8. 8.
    Dunning, I., Huchette, J., Lubin, M.: Jump: a modeling language for mathematical optimization. SIAM Rev. 59(2), 295–320 (2017)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Eggert, D.W., Lorusso, A., Fisher, R.B.: Estimating 3-d rigid body transformations: a comparison of four major algorithms. Mach. Vis. Appl. 9(5–6), 272–290 (1997)CrossRefGoogle Scholar
  10. 10.
    Enqvist, O., Josephson, K., Kahl, F.: Optimal correspondences from pairwise constraints. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1295–1302. IEEE (2009)Google Scholar
  11. 11.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Gelfand, N., Mitra, N.J., Guibas, L.J., Pottmann, H.: Robust global registration. In: Symposium on Geometry Processing, vol. 2, p. 5 (2005)Google Scholar
  13. 13.
    G.O. Inc. Gurobi optimizer reference manual (2016)Google Scholar
  14. 14.
    Hartley, R.I., Kahl, F.: Global optimization through rotation space search. Int. J. Comput. Vis. 82(1), 64–79 (2009)CrossRefGoogle Scholar
  15. 15.
    Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., Navab, N.: Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In: Asian Conference on Computer Vision, pp. 548–562. Springer (2012)Google Scholar
  16. 16.
    Izatt, G., Mirano, G., Adelson, E., Tedrake, R.: Tracking objects with point clouds from vision and touch. In: 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE (2017)Google Scholar
  17. 17.
    Johnson, A.E., Hebert, M.: Using spin images for efficient object recognition in cluttered 3d scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 433–449 (1999)CrossRefGoogle Scholar
  18. 18.
    Klingensmith, M., Koval, M.C., Srinivasa, S.S., Pollard, N.S., Kaess, M.: The manifold particle filter for state estimation on high-dimensional implicit manifolds (2016). arXiv:1604.07224
  19. 19.
    Li, H., Hartley, R.: The 3d-3d registration problem revisited. In: IEEE 11th International Conference on Computer Vision, 2007. ICCV 2007, pp. 1–8. IEEE (2007)Google Scholar
  20. 20.
    Li, S., Lyu, S., Trinkle, J.: State estimation for dynamic systems with intermittent contact. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 3709–3715. IEEE (2015)Google Scholar
  21. 21.
    Marion, P., Florence, P.R., Manuelli, L., Tedrake, R.: A pipeline for generating ground truth labels for real rgbd data of cluttered scenes. Under Review (2017)Google Scholar
  22. 22.
    Maron, H., Dym, N., Kezurer, I., Kovalsky, S., Lipman, Y.: Point registration via efficient convex relaxation. ACM Trans. Gr. (TOG) 35(4), 73 (2016)Google Scholar
  23. 23.
    McCormick, G.P.: Computability of global solutions to factorable nonconvex programs: part i-convex underestimating problems. Math. Program. 10(1), 147–175 (1976)CrossRefGoogle Scholar
  24. 24.
    Mellado, N., Aiger, D., Mitra, N.J.: Super 4pcs fast global pointcloud registration via smart indexing. In: Computer Graphics Forum, vol. 33, pp. 205–215. Wiley Online Library (2014)Google Scholar
  25. 25.
    Narayanan, V., Likhachev, M.: Deliberative object pose estimation in clutter. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3125–3130. IEEE (2017)Google Scholar
  26. 26.
    Nemhauser, G.L., Wolsey, L.A., Integer programming and combinatorial optimization. Wiley, Chichester. Nemhauser, G.L., Savelsbergh, M.W.P., Sigismondi, G.S.: Constraint classification for mixed integer programming formulations. COAL Bull. 20(8–12) (1988) (1992)Google Scholar
  27. 27.
    Olsson, C., Kahl, F., Oskarsson, M.: Branch-and-bound methods for euclidean registration problems. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 783–794 (2009)CrossRefGoogle Scholar
  28. 28.
    Papazov, C., Burschka, D.: An efficient ransac for 3d object recognition in noisy and occluded scenes. Comput. Vis.-ACCV 2010, 135–148 (2011)Google Scholar
  29. 29.
    Rosen, D.M., Carlone, L., Bandeira, A.S., Leonard, J.J.: Se-sync: a certifiably correct algorithm for synchronization over the special Euclidean group (2016)Google Scholar
  30. 30.
    Schmidt, T., Hertkorn, K., Newcombe, R., Marton, Z., Suppa, M., Fox, D.: Depth-based tracking with physical constraints for robot manipulation. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 119–126. IEEE (2015)Google Scholar
  31. 31.
    Schmidt, T., Newcombe, R., Fox, D.: Dart: dense articulated real-time tracking with consumer depth cameras. Auton. Robots 39(3), 239–258 (2015)CrossRefGoogle Scholar
  32. 32.
    Schulman, J., Lee, A., Ho, J., Abbeel, P.: Tracking deformable objects with point clouds. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), pp. 1130–1137. IEEE (2013)Google Scholar
  33. 33.
    Tedrake, R., The Drake Development Team: Drake: A planning, control, and analysis toolbox for nonlinear dynamical systems (2016)Google Scholar
  34. 34.
    Tombari, F., Salti, S., Di Stefano, L.: Unique signatures of histograms for local surface description. In: European Conference on Computer Vision, pp. 356–369. Springer (2010)Google Scholar
  35. 35.
    Wohlhart, P., Lepetit, V.: Learning descriptors for object recognition and 3d pose estimation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)Google Scholar
  36. 36.
    Wong, J.M., Kee, V., Le, T., Wagner, S., Mariottini, G.-L., Schneider, A., Hamilton, L., Chipalkatty, R., Hebert, M., Johnson, D. et al.: Segicp: integrated deep semantic segmentation and pose estimation (2017). arXiv:1703.01661
  37. 37.
    Yang, J., Li, H., Campbell, D., Jia, Y.: Go-icp: a globally optimal solution to 3d icp point-set registration. IEEE Trans. Pattern Anal. Mach. Intell. 38(11), 2241–2254 (2016)CrossRefGoogle Scholar
  38. 38.
    Zhou, Q.-Y., Park, J., Koltun, V.: Fast global registration. In: European Conference on Computer Vision, pp. 766–782. Springer (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Computer Science and Artificial Intelligence LabMITCambridgeUSA
  2. 2.Toyota Research InstituteAnn ArborUSA

Personalised recommendations