Skip to main content
SpringerLink
Log in

Multimodal Feature Association-based Stereo Visual SLAM Method

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Much work has been done to improve visual SLAM systems by integrating point, line, and plane features into the bundle adjustment model, however little attention has been paid to explore the associations between these spatial features that could be used to achieve better performance. This study proposes a multimodal feature association-based stereo SLAM method. Firstly, a method to extract point and line features from stereo images and estimate plane features through the line features is proposed. Then, the corresponding mathematic models for representing the association relations between different features are given. After that, the association relations are integrated into the back-end optimization model to improve system robustness and accuracy by adjusting reprojection errors with confidence weights. Finally, comparison tests are performed to evaluate the performance of the proposed method with the mainstream stereo SLAM systems using the EuRoC and KITTI datasets. The test results show that our approach not only generates semantic maps with higher fidelity but also provides a better positioning capability, with the positioning accuracy of the algorithm on the EuRoC and KITTI datasets improved by an average of 14.78% and 20.09%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

We used pure open datasets EuRoC and KITTI to test our algorithm.

Code Availability

Code generated or used during the study is available from the corresponding author by request.

References

  1. Qiao, C., Bai, T., Xiang, Z., Qian, Q., Bi, Y.: Superline: A Robust Line Segment Feature for Visual SLAM. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp:5664–5670 (2021). https://doi.org/10.1109/IROS51168.2021.9636435

  2. Zhang, F., Rui, T., Yang, C., Shi, J.: LAP-SLAM: A Line-Assisted Point-Based Monocular VSLAM. Electronics (2021). https://doi.org/10.3390/ELECTRONICS8020243

    Article  Google Scholar 

  3. Tourani, A., et al.: Visual SLAM: What Are the Current Trends and What to Expect?. Sensors (22), 9297 (2022). https://doi.org/10.3390/s22239297

  4. Li, Y., et al.: RGB-D SLAM with Structural Regularities. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp: 11581–11587 (2020). https://doi.org/10.1109/ICRA48506.2021.9561560

  5. Gomez-Ojeda, R., Moreno, F.A., Zuñiga-Noël, D., Scaramuzza, D., Gonzalez-Jimenez, J.: PL-SLAM: A Stereo SLAM System Through the Combination of Points and Line Segments. IEEE Trans. Rob. 35, 734–746 (2019). https://doi.org/10.1109/TRO.2019.2899783

    Article  Google Scholar 

  6. Shu, F., Wang, J., Pagani, A., Stricker, D.: Structure PLP-SLAM: Efficient Sparse Mapping and Localization using Point, Line and Plane for Monocular, RGB-D and Stereo Cameras. arXiv preprint arXiv:2207.06058 (2022). https://doi.org/10.48550/arXiv.2207.06058

  7. Rong, H., Gao, Y., Guan, L., Ramirez-Serrano, A., Xu, X., Zhu, Y.: Point-Line Visual Stereo SLAM Using EDlines and PL-BoW. Remote Sensing 13(18), 3591 (2021). https://doi.org/10.3390/rs13183591

    Article  Google Scholar 

  8. Li, G., Zeng, Y., Huang, H., et al.: A multi-feature fusion slam system attaching semantic invariant to points and lines. Sensors. 21(4), 1196 (2021). https://doi.org/10.3390/s21041196

    Article  Google Scholar 

  9. Yunus, R., Li, Y., Tombari, F.: Manhattanslam: Robust planar tracking and mapping leveraging mixture of manhattan frames. In 2021 IEEE International Conference on Robotics and Automation (ICRA). pp. 6687–6693) (2021). https://doi.org/10.1109/ICRA48506.2021.9562030

  10. Zhang, X., Wang, W., Qi, X., Liao, Z., Wei, R.: Point-plane slam using supposed planes for indoor environments. Sensors 19(17), 3795 (2019). https://doi.org/10.3390/s19173795

    Article  Google Scholar 

  11. Yu, Z., Liang, Q., Zhang, S., & Chen, X. A Graph-based Visual SLAM System Fusing Multiple Feature. In 2021 6th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM). pp: 777–784 (2021). https://doi.org/10.1109/ICARM52023.2021.9536072

  12. Zhang, X., Wang, W., Qi, X., Liao, Z.: Stereo plane slam based on intersecting lines. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) pp. 6566–6572 (2021). https://doi.org/10.1109/IROS51168.2021.9635961

  13. Zhou, F., Zhang, L., Deng, C., Fan, X.: Improved Point-Line Feature Based Visual SLAM Method for Complex Environments. Sensors (Basel, Switzerland) 21(13), 4604 (2021). https://doi.org/10.3390/s21134604

    Article  Google Scholar 

  14. Wei, C., Tang, Y., Yang, L., Huang, Z.: SLC-VIO: a stereo visual-inertial odometry based on structural lines and points belonging to lines. Robotica, pp. 1–21 (2022). https://doi.org/10.1017/S0263574721001958

  15. Zhang, X., Li, S., Liu, Q., Liu, P., Wang, Q.: A Visual SLAM Approach to Deeply Explore the Spatial Constraints in Indoor Environment Based on the Manhattan Hypothesis. J Sensors (2023). https://doi.org/10.1155/2023/4152171

    Article  Google Scholar 

  16. Sun, Q., Yuan, J., Zhang, X., Duan, F.: Plane-Edge-SLAM: Seamless fusion of planes and edges for SLAM in indoor environments. IEEE Trans. Autom. Sci. Eng. 18(4), 2061–2075 (2020). https://doi.org/10.1109/tase.2020.3032831

    Article  Google Scholar 

  17. Wang, K., Ma, S. , Ren, F., Lu, J.: SBAS: Salient Bundle Adjustment for Visual SLAM, IEEE Transactions on Instrumentation and Measurement 70, pp. 1–9 (2021). https://doi.org/10.1109/tim.2021.3105243

  18. Shi, L., Zheng, F., Shi, Y.: Multi-constraint slam optimisation algorithm for indoor scenes. Int. J. Robot. Autom. (2023). https://doi.org/10.2316/J.2023.206-0826

    Article  Google Scholar 

  19. Theodorou, C., Velisavljevic, V., Dyo, V., Nonyelu, F.: Visual SLAM algorithms and their application for AR, mapping, localization and wayfinding. Array 15, 100222 (2022). https://doi.org/10.1016/j.array.2022.100222

    Article  Google Scholar 

  20. Von Gioi, R.G., Jakubowicz, J., Morel, J.M., Randall, G.: LSD: A Fast Line Segment Detector with a False Detection Control. IEEE Trans. Pattern Anal. Mach. Intell. 32(4), 722–732 (2010). https://doi.org/10.1109/TPAMI.2008.300

    Article  Google Scholar 

  21. Zhang, L., Koch, R.: An efficient and robust line segment matching approach based on LBD descriptor and pairwise geometric consistency. J. Vis. Commun. Image Represent. 24(7), 794–805 (2013). https://doi.org/10.1016/j.jvcir.2013.05.006

    Article  Google Scholar 

  22. Grisetti, G., Kümmerle, R., Strasdat, H., et al.: g2o: A general framework for (hyper) graph optimization, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 9–13 (2011). https://doi.org/10.1109/ICRA.2011.5979949

  23. Mur-Artal, R., Tardos, J.D.: ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras. IEEE Trans. Robot. 33(5), 1255–1262 (2017). https://doi.org/10.1109/TRO.2017.2705103

    Article  Google Scholar 

  24. Qian, K., Zhao, W., Li, K., Yu, H.: Visual SLAM With BoPLW Pairs Using Egocentric Stereo Camera for Wearable-Assisted Substation Inspection. IEEE Sens. J. 20(3), 1630–1641 (2020). https://doi.org/10.1109/JSEN.2019.2947275

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to acknowledge the support of the National Natural Science Foundation of China—Grant 62073078.

Author information

Authors and Affiliations

  1. School of Instrument Science and Engineering, Southeast University, Nanjing, 210096, China

    Shangzhe Li, Yafei Liu, Huiqing Wang & Xiaoguo Zhang

Authors
  1. Shangzhe Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yafei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Huiqing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaoguo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Author Contributions The overall study supervised by Xiaoguo Zhang; Methodology, software, preparing the original draft and the results were analyzed and validated by Shangzhe Li, Yafei Liu, and Xiaoguo Zhang; Review and editing by Xiaoguo Zhang and Huiqing Wang. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Xiaoguo Zhang.

Ethics declarations

Ethics approval

This study does not involve human or animal subjects. No ethical approval is required

Conflict of Interests

All the authors of this paper have no conflicts of interest, financial or otherwise.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, S., Liu, Y., Wang, H. et al. Multimodal Feature Association-based Stereo Visual SLAM Method. J Intell Robot Syst 109, 37 (2023). https://doi.org/10.1007/s10846-023-01976-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-023-01976-2

Keywords

Search

Navigation