Skip to main content
SpringerLink
Log in

Photometric Bundle Adjustment for Vision-Based SLAM

  • Conference paper
  • First Online:
Computer Vision – ACCV 2016 (ACCV 2016)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10114))

Included in the following conference series:

Abstract

We propose a novel algorithm for the joint refinement of structure and motion parameters from image data directly without relying on fixed and known correspondences. In contrast to traditional bundle adjustment (BA) where the optimal parameters are determined by minimizing the reprojection error using tracked features, the proposed algorithm relies on maximizing the photometric consistency and estimates the correspondences implicitly. Since the proposed algorithm does not require correspondences, its application is not limited to corner-like structure; any pixel with nonvanishing gradient could be used in the estimation process. Furthermore, we demonstrate the feasibility of refining the motion and structure parameters simultaneously using the photometric error in unconstrained scenes and without requiring restrictive assumptions such as planarity. The proposed algorithm is evaluated on range of challenging outdoor datasets, and it is shown to improve upon the accuracy of the state-of-the-art VSLAM methods obtained using the minimization of the reprojection error using traditional BA as well as loop closure.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    While this work was under review, Engel et al. proposed a similar photometric (direct) formulation for VSLAM [70].

References

  1. Sun, D., Roth, S., Black, M.: Secrets of optical flow estimation and their principles. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2432–2439 (2010)

    Google Scholar 

  2. Vedula, S., Baker, S., Rander, P., Collins, R., Kanade, T.: Three-dimensional scene flow. IEEE Trans. Pattern Anal. Mach. Intell. 27, 475–480 (2005)

    Article  Google Scholar 

  3. Seitz, S., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 519–528 (2006)

    Google Scholar 

  4. Furukawa, Y., Hernndez, C.: Multi-view stereo: a tutorial. Found. Trends Comput. Graph. Vis. 9, 1–148 (2015)

    Article  Google Scholar 

  5. Engel, J., Schöps, T., Cremers, D.: LSD-SLAM: large-scale direct monocular SLAM. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8690, pp. 834–849. Springer, Heidelberg (2014). doi:10.1007/978-3-319-10605-2_54

    Google Scholar 

  6. Kerl, C., Sturm, J., Cremers, D.: Robust odometry estimation for RGB-D cameras. In: International Conference on Robotics and Automation (ICRA) (2013)

    Google Scholar 

  7. Steinbrucker, F., Sturm, J., Cremers, D.: Real-time visual odometry from dense RGB-D images. In: IEEE International Conference on Computer Vision, ICCV Workshops (2011)

    Google Scholar 

  8. Meilland, M., Comport, A.: On unifying key-frame and voxel-based dense visual SLAM at large scales. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3677–3683 (2013)

    Google Scholar 

  9. Newcombe, R., Lovegrove, S., Davison, A.: DTAM: dense tracking and mapping in real-time. In: IEEE International Conference on Computer Vision (ICCV), pp. 2320–2327 (2011)

    Google Scholar 

  10. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment — a modern synthesis. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) IWVA 1999. LNCS, vol. 1883, pp. 298–372. Springer, Heidelberg (2000). doi:10.1007/3-540-44480-7_21

    Chapter  Google Scholar 

  11. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision, 2nd edn. Cambridge University Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  12. Torr, P.H.S., Zisserman, A.: Feature based methods for structure and motion estimation. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) IWVA 1999. LNCS, vol. 1883, pp. 278–294. Springer, Heidelberg (2000). doi:10.1007/3-540-44480-7_19

    Chapter  Google Scholar 

  13. Kanazawa, Y., Kanatani, K.: Do we really have to consider covariance matrices for image features? In: Proceedings of the Eighth IEEE International Conference on Computer Vision, 2001. ICCV 2001, vol. 2, pp. 301–306. IEEE (2001)

    Google Scholar 

  14. Brooks, M.J., Chojnacki, W., Gawley, D., Van Den Hengel, A.: What value covariance information in estimating vision parameters? In: Proceedings of the Eighth IEEE International Conference on Computer Vision, 2001. ICCV 2001, vol. 1, pp. 302–308. IEEE (2001)

    Google Scholar 

  15. Furukawa, Y., Ponce, J.: Accurate camera calibration from multi-view stereo and bundle adjustment. In: IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008, pp. 1–8. IEEE (2008)

    Google Scholar 

  16. Deriche, R., Giraudon, G.: Accurate corner detection: an analytical study. In: Proceedings of the Third International Conference on Computer Vision, pp. 66–70. IEEE (1990)

    Google Scholar 

  17. Shimizu, M., Okutomi, M.: Precise sub-pixel estimation on area-based matching. In: ICCV, pp. 90–97 (2001)

    Google Scholar 

  18. Mur-Artal, R., Montiel, J.M.M., Tardós, J.D.: ORB-SLAM: a versatile and accurate monocular SLAM system. CoRR abs/1502.00956 (2015)

    Google Scholar 

  19. Milford, M., Wyeth, G.: SeqSLAM: visual route-based navigation for sunny summer days and stormy winter nights. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 1643–1649 (2012)

    Google Scholar 

  20. Reid, I.: Towards semantic visual SLAM. In: 13th International Conference on Control Automation Robotics Vision (ICARCV), p. 1 (2014)

    Google Scholar 

  21. Salas-Moreno, R., Newcombe, R., Strasdat, H., Kelly, P., Davison, A.: SLAM++: simultaneous localisation and mapping at the level of objects. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1352–1359 (2013)

    Google Scholar 

  22. Murray, R.M., Li, Z., Sastry, S.S., Sastry, S.S.: A Mathematical Introduction to Robotic Manipulation. CRC Press, Boca Raton (1994)

    MATH  Google Scholar 

  23. Ma, Y., Soatto, S., Kosecka, J., Sastry, S.S.: An Invitation to 3-D Vision: From Images to Geometric Models. Springer, New York (2003)

    MATH  Google Scholar 

  24. Hartley, R., Trumpf, J., Dai, Y., Li, H.: Rotation averaging. Int. J. Comput. Vis. 103, 267–305 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  25. Civera, J., Davison, A.J., Montiel, J.M.: Inverse depth parametrization for monocular SLAM. IEEE Trans. Robot. 24, 932–945 (2008)

    Article  Google Scholar 

  26. Zhao, L., Huang, S., Sun, Y., Yan, L., Dissanayake, G.: ParallaxBA: bundle adjustment using parallax angle feature parametrization. Int. J. Robot. Res. 34, 493–516 (2015)

    Article  Google Scholar 

  27. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision (DARPA). In: Proceedings of the 1981 DARPA Image Understanding Workshop, pp. 121–130 (1981)

    Google Scholar 

  28. Horn, B.K., Schunck, B.G.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)

    Article  Google Scholar 

  29. Baker, S., Matthews, I.: Lucas-Kanade 20 years on: a unifying framework. Int. J. Comput. Vis. 56, 221–255 (2004)

    Article  Google Scholar 

  30. Engel, J., Stueckler, J., Cremers, D.: Large-scale direct SLAM with stereo cameras. In: International Conference on Intelligent Robots and Systems (IROS) (2015)

    Google Scholar 

  31. Nocedal, J., Wright, S.J.: Numerical Optimization, 2nd edn. Springer, New York (2006)

    MATH  Google Scholar 

  32. Harris, C., Stephens, M.: A combined corner and edge detector. In: Alvey Vision Conference, Manchester, vol. 15, p. 50 (1988)

    Google Scholar 

  33. Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). doi:10.1007/11744023_34

    Chapter  Google Scholar 

  34. Dellaert, F., Seitz, S.M., Thorpe, C.E., Thrun, S.: Structure from motion without correspondence. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 557–564. IEEE (2000)

    Google Scholar 

  35. Meilland, M., Comport, A., Rives, P.: A spherical robot-centered representation for urban navigation. In: IROS (2010)

    Google Scholar 

  36. Nister, D., Naroditsky, O., Bergen, J.: Visual odometry. In: Computer Vision and Pattern Recognition (CVPR) (2004)

    Google Scholar 

  37. Irani, M., Anandan, P., Cohen, M.: Direct recovery of planar-parallax from multiple frames. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) IWVA 1999. LNCS, vol. 1883, pp. 85–99. Springer, Heidelberg (2000). doi:10.1007/3-540-44480-7_6

    Chapter  Google Scholar 

  38. Stein, G., Shashua, A.: Model-based brightness constraints: on direct estimation of structure and motion. IEEE Trans. Pattern Anal. Mach. Intell. 22, 992–1015 (2000)

    Article  Google Scholar 

  39. Agouris, P., Schenk, T.: Automated aerotriangulation using multiple image multipoint matching. Photogramm. Eng. Remote Sens. 62, 703–710 (1996)

    Google Scholar 

  40. Agarwal, S., Mierle, K., et al.: Ceres solver (2016). http://ceres-solver.org

  41. Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. J. Appl. Maths. 2, 164–168 (1944)

    Article  MathSciNet  MATH  Google Scholar 

  42. Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11, 431–441 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  43. Snderhauf, N., Konolige, K., Lacroix, S., Protzel, P.: Visual odometry using sparse bundle adjustment on an autonomous outdoor vehicle. In: Levi, P., Schanz, M., Lafrenz, R., Avrutin, V. (eds.) Autonome Mobile Systems 2005. Informatik aktuell, pp. 157–163. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  44. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)

    Google Scholar 

  45. Blanco, J.L., Moreno, F.A., González-Jiménez, J.: The málaga urban dataset: high-rate stereo and lidars in a realistic urban scenario. Int. J. Robot. Res. 33, 207–214 (2014)

    Article  Google Scholar 

  46. Tardif, J.P., George, M., Laverne, M., Kelly, A., Stentz, A.: A new approach to vision-aided inertial navigation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4161–4168. IEEE (2010)

    Google Scholar 

  47. Badino, H., Yamamoto, A., Kanade, T.: Visual odometry by multi-frame feature integration. In: IEEE International Conference on Computer Vision Workshops (ICCVW), pp. 222–229 (2013)

    Google Scholar 

  48. Lindeberg, T.: Scale-space Theory in Computer Vision. Springer, New York (1994)

    Book  MATH  Google Scholar 

  49. Alismail, H., Browning, B., Lucey, S.: Direct visual odometry using bit-planes. CoRR abs/1604.00990 (2016)

    Google Scholar 

  50. Delaunoy, A., Pollefeys, M.: Photometric bundle adjustment for dense multi-view 3D modeling. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1486–1493. IEEE (2014)

    Google Scholar 

  51. Hirschmuller, H.: Accurate and efficient stereo processing by semi-global matching and mutual information. In: Computer Vision and Pattern Recognition (2005)

    Google Scholar 

  52. Agarwal, S., Snavely, N., Seitz, S.M., Szeliski, R.: Bundle adjustment in the large. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 29–42. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15552-9_3

    Chapter  Google Scholar 

  53. Konolige, K., Garage, W.: Sparse sparse bundle adjustment. In: BMVC, pp. 1–11 (2010)

    Google Scholar 

  54. Jeong, Y., Nister, D., Steedly, D., Szeliski, R., Kweon, I.S.: Pushing the envelope of modern methods for bundle adjustment. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1605–1617 (2012)

    Article  Google Scholar 

  55. Engels, C., Stewnius, H., Nister, D.: Bundle adjustment rules. In: Photogrammetric Computer Vision (2006)

    Google Scholar 

  56. Wu, C., Agarwal, S., Curless, B., Seitz, S.M.: Multicore bundle adjustment. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3057–3064. IEEE (2011)

    Google Scholar 

  57. Ni, K., Steedly, D., Dellaert, F.: Out-of-core bundle adjustment for large-scale 3D reconstruction. In: IEEE 11th International Conference on Computer Vision, pp. 1–8 (2007)

    Google Scholar 

  58. Konolige, K., Agrawal, M.: FrameSLAM: from bundle adjustment to real-time visual mapping. IEEE Trans. Robot. 24, 1066–1077 (2008)

    Article  Google Scholar 

  59. Kaess, M., Ila, V., Roberts, R., Dellaert, F.: The Bayes tree: an algorithmic foundation for probabilistic robot mapping. In: Hsu, D., Isler, V., Latombe, J.C., Lin, M. (eds.) Algorithmic Foundations of Robotics IX. Springer Tracts in Advanced Robotics, vol. 68, pp. 157–173. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  60. Kaess, M., Ranganathan, A., Dellaert, F.: iSAM: incremental smoothing and mapping. IEEE Trans. Robot. (TRO) 24, 1365–1378 (2008)

    Article  Google Scholar 

  61. Kahl, F., Agarwal, S., Chandraker, M.K., Kriegman, D., Belongie, S.: Practical global optimization for multiview geometry. Int. J. Comput. Vis. 79, 271–284 (2008)

    Article  Google Scholar 

  62. Hartley, R., Kahl, F., Olsson, C., Seo, Y.: Verifying global minima for \(L_2\) minimization problems in multiple view geometry. Int. J. Comput. Vis. 101, 288–304 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  63. Aftab, K., Hartley, R.: LQ-bundle adjustment. In: IEEE International Conference on Image Processing (ICIP), pp. 1275–1279 (2015)

    Google Scholar 

  64. Irani, M., Anandan, P.: About direct methods. In: Triggs, B., Zisserman, A., Szeliski, R. (eds.) IWVA 1999. LNCS, vol. 1883, pp. 267–277. Springer, Heidelberg (2000). doi:10.1007/3-540-44480-7_18

    Chapter  Google Scholar 

  65. Horn, B.K.P., Weldon, E.J.: Direct methods for recovering motion (1988)

    Google Scholar 

  66. Oliensis, J.: Direct multi-frame structure from motion for hand-held cameras. In: Proceedings of the 15th International Conference on Pattern Recognition, vol. 1, pp. 889–895 (2000)

    Google Scholar 

  67. Mandelbaum, R., Salgian, G., Sawhney, H.: Correlation-based estimation of ego-motion and structure from motion and stereo. In: The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, pp. 544–550 (1999)

    Google Scholar 

  68. Silveira, G., Malis, E., Rives, P.: An efficient direct approach to visual SLAM. IEEE Trans. Robot. 24(5), 969–979 (2008). doi:10.1109/TRO.2008.2004829

    Article  Google Scholar 

  69. Lovegrove, S., Davison, A.J.: Real-time spherical mosaicing using whole image alignment. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6313, pp. 73–86. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15558-1_6

    Chapter  Google Scholar 

  70. Engel, J., Koltun, V., Cremers, D.: Direct sparse odometry. ArXiv e-prints (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The Robotics Institute, Carnegie Mellon University, Pittsburgh, USA

    Hatem Alismail, Brett Browning & Simon Lucey

Authors
  1. Hatem Alismail
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brett Browning
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Simon Lucey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hatem Alismail .

Editor information

Editors and Affiliations

  1. National Tsing Hua University, Hsinchu, Taiwan

    Shang-Hong Lai

  2. Graz University of Technology, Graz, Austria

    Vincent Lepetit

  3. Drexel University, Philadelphia, Pennsylvania, USA

    Ko Nishino

  4. The University of Tokyo, Tokyo, Japan

    Yoichi Sato

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Alismail, H., Browning, B., Lucey, S. (2017). Photometric Bundle Adjustment for Vision-Based SLAM. In: Lai, SH., Lepetit, V., Nishino, K., Sato, Y. (eds) Computer Vision – ACCV 2016. ACCV 2016. Lecture Notes in Computer Science(), vol 10114. Springer, Cham. https://doi.org/10.1007/978-3-319-54190-7_20

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-54190-7_20

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-54189-1

  • Online ISBN: 978-3-319-54190-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation