Skip to main content
SpringerLink
Log in

Joint optimization based on direct sparse stereo visual-inertial odometry

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

This paper proposes a novel fusion of an inertial measurement unit (IMU) and stereo camera method based on direct sparse odometry (DSO) and stereo DSO. It jointly optimizes all model parameters within a sliding window, including the inverse depth of all selected pixels and the internal or external camera parameters of all keyframes. The vision part uses a photometric error function that optimizes 3D geometry and camera pose in a combined energy functional. The proposed algorithm uses image blocks to extract neighboring image features and directly forms measurement residuals in the image intensity space. A fixed-baseline stereo camera solves scale drift. IMU information is accumulated between several frames using manifold pre-integration and is inserted into the optimization as additional constraints between keyframes. The scale and gravity inserted are incorporated into the stereo visual inertial odometry model and are optimized together with other variables such as poses. The experimental results show that the tracking accuracy and robustness of the proposed method are superior to those of the state-of-the-art fused IMU method. In addition, compared with previous semi-dense direct methods, the proposed method displays a higher reconstruction density and scene recovery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  • Bowman, S. L., Atanasov, N., Daniilidis, K., & Pappas G. J. (2017). Probabilistic data association for semantic SLAM. In IEEE international conference on robotics and automation (pp. 1722–1729).

  • Comport, A. I., Malis, E., & Rives, P. (2007). Accurate quadrifocal tracking for robust 3D visual odometry. In IEEE international conference on robotics and automation (pp. 40–45).

  • Engel, J., Koltun, V., & Cremers, D. (2016). Direct sparse odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, PP(99), 1–1.

    Google Scholar 

  • Engel, J., Schps, T., & Cremers, D. (2014). LSD-SLAM: Large-scale direct monocular SLAM. Berlin: Springer.

    Google Scholar 

  • Engel, J., Stckler, J., & Cremers, D. (2015). Large-scale direct SLAM with stereo cameras. In IEEE/RSJ international conference on intelligent robots and systems (pp. 1935–1942).

  • Engel, J., Sturm, J., & Cremers, D. (2013). Semi-dense visual odometry for a monocular camera. In IEEE international conference on computer vision (pp. 1449–1456).

  • Engel, J., Sturm, J., & Cremers, D. (2012). Camera-based navigation of a low-cost quadrocopter. In IEEE/RSJ international conference on intelligent robots and systems.

  • Engel, J., Usenko, V., Cremers, D. (2016). A photometrically calibrated benchmark for monocular visual odometry. arXiv:1607.02555 [cs.CV].

  • Forster, C., Carlone, L., Dellaert, F., & Scaramuzza, D. (2015). On-manifold preintegration for real-time visual-inertial odometry. IEEE Transactions on Robotics, 33(1), 1–21.

    Article  Google Scholar 

  • Forster, C., Carlone, L., Dellaert, F., & Scaramuzza, D. (2017). On-manifold preintegration for real-time visual-inertial odometry. IEEE Transactions on Robotics, 33(1), 1–21.

    Article  Google Scholar 

  • Forster, C., Pizzoli, M.,& Scaramuzza, D. (2014). SVO: Fast semi-direct monocular visual odometry. In IEEE international conference on robotics and automation (pp. 15–22).

  • Kerl, C., Sturm, J., & Cremers, D. (2013). Robust odometry estimation for RGB-D cameras. In IEEE international conference on robotics and automation (pp. 3748–3754).

  • Klein, G., & Murray, D. (2008). Parallel tracking and mapping for small AR workspaces. In IEEE and ACM international symposium on mixed and augmented reality (pp. 1–10).

  • Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R., & Furgale, P. (2014). Keyframe-based visual-inertial odometry using nonlinear optimization. International Journal of Robotics Research, 34(3), 314–334.

    Article  Google Scholar 

  • Mahmoud, N., Cirauqui, I., Hostettler, A., Doignon, C., Soler, L., Marescaux, J., Montiel, M. M. (2016). ORBSLAM-based endoscope tracking and 3D reconstruction. arXiv:1608.08149 [cs.CV]

  • Meier, L., Tanskanen, P., Fraundorfer, F., & Pollefeys, M. (2012). The Pixhawk open-source computer vision framework for mavs. ISPRS - International Archives of the Photogrammetry, XXXVIII–1/C22, 13–18.

    Google Scholar 

  • Mur-Artal, R., Montiel, J. M. M., & Tardos, J. D. (2017). ORB-SLAM: A versatile and accurate monocular slam system. IEEE Transactions on Robotics, 31(5), 1147–1163.

    Article  Google Scholar 

  • Mur-Artal, R., & Tardos, J. D. (2017). ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics, 33(5), 1255–1262.

    Article  Google Scholar 

  • Mur-Artal, R., & Tards, J. D. (2016). Visual-inertial monocular SLAM with map reuse. IEEE Robotics and Automation Letters, 2(2), 796–803.

    Article  Google Scholar 

  • Newcombe, R. A., Lovegrove, S. J., & Davison, A. J. (2010). DTAM: Dense tracking and mapping in real-time. In International conference on computer vision (pp. 2320–2327).

  • Qin, T., Li, P., & Shen, S. (2017). VINS-Mono: A robust and versatile monocular visual-inertial state estimator. IEEE Transactions on Robotics, PP(99), 1–17.

    Google Scholar 

  • Stumberg, L. V., Usenko, V., & Cremers, D. (2018). Direct sparse visual-inertial odometry using dynamic marginalization. arXiv:1804.05625 [cs.CV].

  • Urtasun, R., Lenz, P., & Geiger, A. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. In IEEE conference on computer vision and pattern recognition (pp. 3354–3361).

  • Usenko, V., Engel, J., Stckler, J., & Cremers, D. (2016). Direct visual-inertial odometry with stereo cameras. In IEEE international conference on robotics and automation (pp. 1885–1892).

  • Wang, R., Schworer, M., & Cremers, D. (2017). Stereo DSO: Large-scale direct sparse visual odometry with stereo cameras (pp. 3923–3931). arXiv:1708.07878 [cs.CV].

Download references

Acknowledgements

The work is supported by the national Natural Science Foundation of China (Project Nos. 61673125, 61773333), China Scholarship Council (CSC, Project No. 201908130016).

Author information

Authors and Affiliations

  1. Key Lab of Industrial Computer Control Engineering of Hebei Province, Yanshan University, Qinhuangdao, China

    Shuhuan Wen & Yanfang Zhao

  2. Department of Computing Science, University of Alberta, Edmonton, AB, T6G 2E8, Canada

    Shuhuan Wen & Hong Zhang

  3. Department of Informatics, King’s College London, 30 Aldwych, London, WC2B 4BG, UK

    Hak Keung Lam

  4. Institute for Medical Science and Technology (IMSaT), University of Dundee, Dundee, UK

    Luigi Manfredi

Authors
  1. Shuhuan Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanfang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hak Keung Lam
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Luigi Manfredi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wen, S., Zhao, Y., Zhang, H. et al. Joint optimization based on direct sparse stereo visual-inertial odometry. Auton Robot 44, 791–809 (2020). https://doi.org/10.1007/s10514-019-09897-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-019-09897-6

Keywords

Search

Navigation