Abstract
The problem of visual odometry (VO) and localization in extreme illumination conditions is widely concerned. In this paper, we propose a novel SLAM algorithm namely DVT-SLAM (Deep-learning based Visible-Thermal SLAM). It focuses on the fusion of thermal infrared image and visible image which have complementary advantages in characteristics. With the contrastive learning and the measurement of mutual information between multi-modal images, the first part of DVT-SLAM is the DVT-GAN network to fuse visible-thermal images and generate pseudo visible images at night. Given the generated images, visual odometry is applied for pose estimation base. Extensive evaluations are performed on the Brno Urban Dataset, a multi-modal dataset containing different time and weather conditions in diverse scenarios. Series of experiments show that DVT-SLAM is a robustness and suitability solution for single visible camera failures, which can reduce positioning error by half and achieve superior SLAM performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mouats, T., Aouf, N., Sappa, A.D., et al.: Multispectral stereo odometry. IEEE Trans. Intell. Transp. Syst. 16(3), 1210–1224 (2014)
Beauvisage, A., Aouf, N., Courtois, H.: Multi-spectral visual odometry for unmanned air vehicles. In: 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 001994–001999 IEEE (2016)
Poujol, J., Aguilera, C.A., Danos, E., et al.: A visible-thermal fusion based monocular visual odometry. In: Reis, L., Moreira, A., Lima, P., Montano, L., Muñoz-Martinez, V. (eds.) Robot 2015: Second Iberian Robotics Conference. Advances in Intelligent Systems and Computing, vol. 417. Springer, Cham pp. 517–528 (2016)
Beauvisage, A., Ahiska, K., Aouf, N.: Multimodal tracking framework for visual odometry in challenging illumination conditions. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 11133–11139. IEEE (2020)
Choi, Y., et al.: Thermal image enhancement using convolutional neural network. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2016)
Sun, Y., Zuo, W., Yun, P., et al.: FuseSeg: semantic segmentation of urban scenes based on RGB and thermal data fusion. IEEE Trans. Autom. Sci. Eng. (2020)
Mur-Artal, R., Tardós, J.D.: Orb-slam2: an open-source slam system for monocular, stereo, and RGB-D cameras. IEEE Trans. Rob. 33(5), 1255–1262 (2017)
Campos, C., Elvira, R., RodrÃguez, J.J.G., et al.: ORB-SLAM3: An accurate open-source library for visual, visual-inertial and multi-map SLAM. arXiv preprint arXiv:2007.11898, (2020)
Krotosky, S.J., Trivedi, M.M.: Mutual information based registration of multimodal stereo videos for person tracking. Comput. Vis. Image Underst. 106(2–3), 270–287 (2007)
Zhu, J.Y., Park, T., Isola, P., et al.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Park, T., Efros, A.A., Zhang, R., Zhu, J.: Contrastive learning for unpaired image-to-image translation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX, pp. 319–345. Springer, Cham (2020)
Li, C., Liu, H., Chen, C., Pu, Y., Chen, L., Henao, R., Carin, L.: Alice: Towards understanding adversarial learning for joint distribution matching. In: Advances in Neural Information Processing Systems (2017)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In: NIPSs, pp. 6626–6637 (2017)
Ligocki, A., Jelinek, A., Zalud, L.: Brno urban dataset-the new data for self-driving agents and mapping tasks. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3284–3290. IEEE (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, R. et al. (2021). DVT-SLAM: Deep-Learning Based Visible and Thermal Fusion SLAM. In: Yang, C., Xie, J. (eds) China Satellite Navigation Conference (CSNC 2021) Proceedings. Lecture Notes in Electrical Engineering, vol 773. Springer, Singapore. https://doi.org/10.1007/978-981-16-3142-9_37
Download citation
DOI: https://doi.org/10.1007/978-981-16-3142-9_37
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-3141-2
Online ISBN: 978-981-16-3142-9
eBook Packages: EngineeringEngineering (R0)