Skip to main content
SpringerLink
Log in

A Fusion Method for 2D LiDAR and RGB-D Camera Depth Image Without Calibration

  • Conference paper
  • First Online:
Cognitive Computation and Systems (ICCCS 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1732))

Included in the following conference series:

  • 420 Accesses

Abstract

Two-dimensional (2D) LiDAR and RGB-D camera are two widely used sensors in various tasks of robot navigation. In spite of calibration, there are still quite a few noises such as hollows and speckle burrs in both of them caused by possibly external complex environments. This paper provides a data fusion method for 2D LiDAR and RGB-D depth images. The proposed method utilizes the phenomenon that the data provided by 2D LiDAR and RGB-D are significantly different in format but tightly relative in their depth information. With time alignment and correlation analysis, we find that the lines of 2D LiDAR are able to register to the RGB-D images in height, and conversely, the corresponding lines of RGB-D depth in height could be registered to the range of 2D LiDAR curves in width automatically, even there is no calibration of them. In experiments, we evaluate the proposed method on the Robot@Home dataset: a widely recognized in-doors open robot navigation database. The results show that the proposed method contributes to de-noise of the original data both for 2D LiDAR and RGB-D depth image simultaneously. The proposed method is also validated on the realistic navigation environments, and it could be extended to the application of more precise 2D map construction for robot navigation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ruiz-Sarmiento, J.R., Galindo, C., GonzáLez-JiméNez, J.: Robot@Home, a robotic dataset for semantic mapping of home environments. Int. J. Robot. Res. 36(2), 131–141 (2017)

    Google Scholar 

  2. Kolhatkar, C., Wagle, K.: Review of SLAM algorithms for indoor mobile robot with LIDAR and RGB-D camera technology. In: Favorskaya, M.N., Mekhilef, S., Pandey, R.K., Singh, N. (eds.) Innovations in Electrical and Electronic Engineering. Lecture Notes in Electrical Engineering, vol. 661, pp. 397–409. Springer, Singapore (2021). https://doi.org/10.1007/978-981-15-4692-1_30

    Chapter  Google Scholar 

  3. Tian, B., Shim, V.A., Yuan, M., Srinivasan, C., Tang, H., Li, H.: RGB-D based cognitive map building and navigation. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Tokyo, Japan, 3–7 November 2013 (2013)

    Google Scholar 

  4. Wellhausen, L., Ranftl, R., Hutter, M.: Safe robot navigation via multi-modal anomaly detection. IEEE Robot. Autom. Lett. 5(2), 1326–1333 (2020)

    Article  Google Scholar 

  5. Fu, Y., Yan, Q., Yang, L., Liao, J., Xiao, C.: Texture mapping for 3D reconstruction with RGB-D sensor. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  6. Kuan, Y.W., Ee, N.O., Wei, L.S.: Comparative study of intel R200, kinect v2, and Primesense RGB-D sensors performance outdoors. IEEE Sensors J. 19(19), 8741–8750 (2019)

    Google Scholar 

  7. Shin, Y.-S., Park, Y.S., Kim, A.: Direct visual SLAM using sparse depth for camera-LiDAR system. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 21–25 May 2018 (2018)

    Google Scholar 

  8. Winterhalter, W., Fleckenstein, F., Steder, B., Spinello, L., Burgard, W.: Accurate indoor localization for RGB-D smartphones and tablets given 2D floor plans. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, Germany, 28 Sept–2 Oct 2015 (2015)

    Google Scholar 

  9. Song, H., Choi, W., Kim, H.: Robust vision-based relative-localization approach using an RGB-depth camera and lidar sensor fusion. IEEE Trans. Indust. Electron. 63(6), 3725–3736 (2016)

    Google Scholar 

  10. Hess, W., Kohler, D., Rapp, H., Andor, D.: Real-time loop closure in 2D LIDAR SLAM. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), Stockholm, Sweden, 16–21 May 2016 (2016)

    Google Scholar 

  11. Himstedt, M., Frost, J., Hellbach, S., Böhme, H.-J., Maehle, E.: Large scale place recognition in 2D LIDAR scans using geometrical landmark relations.  In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2014), Chicago, IL, USA, 14–18 September 2014 (2014)

    Google Scholar 

  12. Zlot, R., Bosse, M.: Place recognition using keypoint similarities in 2D lidar maps. Springer Tracts Adv. Robot. 54(2009), 363–372 (2009)

    Article  Google Scholar 

  13. Wang, D.Z., Posner, I., Newman, P.: Model-free detection and tracking of dynamic objects with 2D lidar. Int. J. Robot. Res. 34(7), 1039–1063 (2015)

    Google Scholar 

  14. Arras, K.O., Mozos, O.M., Burgard, W.: Using boosted features for the detection of people in 2D range data. In: IEEE International Conference on Robotics and Automation (ICRA 2007), Roma, Italy, 10–14 April 2007 (2007)

    Google Scholar 

  15. Holz, D., Holzer, S., Rusu, R.B., Behnke, S.: Real-time plane segmentation using RGB-D cameras.  In: Röfer, T., Mayer, N.M., Savage, J., Saranli, U. (eds.) RoboCup 2011: Robot Soccer World Cup XV. RoboCup 2011. LNCS, vol. 7416, pp. 306–317. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-32060-6_26

    Google Scholar 

  16. Wang, Y., Wei, X., Shen, H., Dinge, L., Wan, J.: Robust fusion for RGB-D tracking using CNN features. Appl. Soft Comput. 92, 106302 (2020)

    Google Scholar 

  17. Zhong, J., Li, M., Liao, X., Qin, J.: A Real-time infrared stereo matching algorithm for RGB-D cameras’ indoor 3D perception. Int. J. Geo-Inf. 9(8), 472 2020

    Google Scholar 

  18. Zollhöfer, M., et al.: State of the art reportstate of the art on 3D reconstruction with RGB-D cameras. In: Eurographics 2018: Annual Conference of the European Association for Computer Graphics (2018)

    Google Scholar 

  19. Ylimaki, M., Heikkil, J., Kannala, J.: Accurate 3-D Reconstruction with RGB-D Cameras using depth map fusion and pose refinement. In: 2018 24th International Conference on Pattern Recognition (ICPR) Beijing, China (2018)

    Google Scholar 

  20. Chong, Z.J., Qin, B., Bandyopadhyay, T., Ang, M.H., Frazzoli, E., Rus, D.: Synthetic 2D LIDAR for precise vehicle localization in 3D urban environment. In: 2013 IEEE International Conference on Robotics and Automation (ICRA), Karlsruhe, Germany (2013)

    Google Scholar 

  21. Myronenko, A., Song, X.: Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2262–2275 (2010)

    Article  Google Scholar 

  22. Lian, W.: Rotation-invariant nonrigid point set matching in cluttered scenes. IEEE Trans. Image Process. 21(5), 2786–2797 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  23. Ma, J., Zhao, J., Yuille, A.L.: Non-rigid point set registration by preserving global and local structures. IEEE Trans. Image Process. 25(1), 53–64 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  24. MA, F., RC, B.: Random Sample Consensus (RANSAC): a Paradigm for model fitting with applications to image analysis and automated cartography. In: Readings in Computer Vision, pp. 726–740 (1987)

    Google Scholar 

  25. Malis, S.B.: Homography-based 2D visual tracking and servoing. Int. J. Robot. Res. 26(7), 661–676 (2007)

    Article  Google Scholar 

  26. Robot@Home (http://mapir.isa.uma.es/mapirwebsite/index.php/mapir-downloads/203-robot-at-home-dataset) (31 Oct 2017)

Download references

Acknowledgements

This work is supported by the National Key Research & Development Program of China (No. 2018AAA0102902), the Guangxi Key Research and Development Program (AB21220038), the National Natural Science Foundation of China (NSFC) (No.61873269), Beijing Natural Science Foundation + J210012, Hebei Natural Science Foundation (F2021205014), Key Laboratory Foundation (KGJ6142210210311), the Beijing Natural Science Foundation (No: L192005), Key Laboratory Fund (KGJ6142210210311), the CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-20202-027A), the Guangxi Key Research and Development Program (AB18221011,AB21075004, AD18281002, AD19110137), the Natural Science Foundation of Guangxi of China (No: 2020GXNSFAA297061, 2019GXNSFDA185006, 2019GXNSFDA185007), Guangxi Key Laboratory of Intelligent Processing of Computer Images and Graphics (No GIIP201702) and Guangxi Key Laboratory of Trusted Software (NO kx201621,kx201715).

Author information

Authors and Affiliations

  1. School of Computer, Beijing Information Science and Technology University, Beijing, China

    Xia Hou & Hong Shi

  2. Institute of Automation, Chinese Academy of Science, Beijing, China

    Yuanhao Qu & Minghao Yang

Authors
  1. Xia Hou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuanhao Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Minghao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong Shi .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Fuchun Sun

  2. Tsinghua University, Beijing, China

    Jianmin Li

  3. Tsinghua University, Beijing, China

    Huaping Liu

  4. Beihang University, Beijing, China

    Zhongyi Chu

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hou, X., Shi, H., Qu, Y., Yang, M. (2023). A Fusion Method for 2D LiDAR and RGB-D Camera Depth Image Without Calibration. In: Sun, F., Li, J., Liu, H., Chu, Z. (eds) Cognitive Computation and Systems. ICCCS 2022. Communications in Computer and Information Science, vol 1732. Springer, Singapore. https://doi.org/10.1007/978-981-99-2789-0_8

Download citation

  • DOI: https://doi.org/10.1007/978-981-99-2789-0_8

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-99-2788-3

  • Online ISBN: 978-981-99-2789-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation