Abstract
The real-world deployment of fully autonomous mobile robots depends on a robust simultaneous localization and mapping (SLAM) system, capable of handling dynamic environments, where objects are moving in front of the robot, and changing environments, where objects are moved or replaced after the robot has already mapped the scene. This paper proposes Changing-SLAM, a method for robust Visual SLAM in both dynamic and changing environments. This is achieved by using a Bayesian filter combined with a long-term data association algorithm. Also, it employs an efficient algorithm for dynamic keypoints filtering based on object detection that correctly identifies features inside the bounding box that are not dynamic, preventing a depletion of features that could cause lost tracks. Furthermore, a new dataset was developed with RGB-D data specially designed for the evaluation of changing environments on an object level, called PUC-USP dataset. Six sequences were created using a mobile robot, an RGB-D camera and a motion capture system. The sequences were designed to capture different scenarios that could lead to a tracking failure or map corruption. Changing-SLAM does not assume a given camera pose or a known map, being also able to operate in real time. The proposed method was evaluated using benchmark datasets and compared with other state-of-the-art methods, proving to be highly accurate.
Article PDF
Similar content being viewed by others
Availability of data and materials
The datasets used are publicly available in: \(\bullet \) PUC-USP Dataset \(\bullet \)TUM RGB-D Dataset: https://vision.in.tum.de/data/datasets/rgbd-dataset\(\bullet \)Bonn RGB-D Dynamic Dataset: http://www.ipb.uni-bonn.de/data/rgbd-dynamic-dataset/
References
Klein, G., Murray, D.: Parallel tracking and mapping for small AR workspaces. In: Proceedings of the 2007 6th IEEE and ACM international symposium on mixed and augmented reality, pp. 1–10 (2007). IEEE Computer Society
Mur-Artal, R., Montiel, J.M.M., Tardós, J.D.: ORB-SLAM: A Versatile and Accurate Monocular SLAM System. IEEE Trans Robot 31(5), 1147–1163 (2015)
Bescos, B., Fácil, J.M., Civera, J., Neira, J.: DynaSLAM: Tracking, mapping, and inpainting in dynamic scenes. IEEE Robot Autom Lett 3(4), 4076–4083 (2018)
Yu, C., Liu, Z., Liu, X.-J., Xie, F., Yang, Y., Wei, Q., Fei, Q.: DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 1168–1174 (2018). https://doi.org/10.1109/IROS.2018.8593691
Yuan, X., Chen, S.: SaD-SLAM: A Visual SLAM Based on Semantic and Depth Information. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4930–4935 (2020). https://doi.org/10.1109/IROS45743.2020.9341180
Vincent, J., Labbé, M., Lauzon, J., Grondin, F., Comtois-Rivet, P., Michaud, F.: Dynamic Object Tracking and Masking for Visual SLAM. 2020 IEEE/RSJ Int Conf Intell Robot Syst (IROS), 4974–4979 (2020)
Olson, E., Agarwal, P.: Inference on networks of mixtures for robust robot mapping. Int J Robot Res 32(7), 826–840 (2013). https://doi.org/10.1177/0278364913479413
Agarwal, P., Tipaldi, G.D., Spinello, L., Stachniss, C., Burgard, W.: Robust map optimization using dynamic covariance scaling. In: 2013 IEEE international conference on robotics and automation, pp. 62–69 (2013)
Sünderhauf, N., Protzel, P.: Switchable constraints for robust pose graph SLAM. In: 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 1879–1884 (2012). IEEE
Lee, D., Myung, H.: Solution to the SLAM problem in low dynamic environments using a pose graph and an RGB-D sensor. Sensors 14(7), 12467–12496 (2014)
Gomez, C., Hernandez, A., Derner, E.: Object-Based Pose Graph for Dynamic Indoor Environments. IEEE Robot Autom Lett 5(4), 5401–5408 (2020)
Derner, E., Gomez, C., Hernandez, A.C., Barber, R., Babuska, R.: Change detection using weighted features for image-based localization. Robot Auton Syst 135, 103676 (2021)
Li, D., Shi, X., Long, Q., Liu, S., Yang, W., Wang, F., Wei, Q., Qiao, F.: DXSLAM: A Robust and Efficient Visual SLAM System with Deep Features. 2020 IEEE/RSJ Int Conf Intell Robot Syst (IROS), 4958–4965 (2020)
Zhao, M., Guo, X., Song, L., Qin, B., Shi, X., Lee, G., Sun, G.: A General Framework for Lifelong Localization and Mapping in Changing Environment. In: 2021 IEEE/RSJ International conference on intelligent robots and systems (2021). IEEE
Campos, C., Elvira, R., Rodríguez, J.J.G., Montiel, J.M.M., Tardøs, J.D.: ORB-SLAM3: An Accurate Open-Source Library for Visual, Visual-Inertial, and Multimap SLAM. IEEE Trans Robot 37(6), 1874–1890 (2021). https://doi.org/10.1109/TRO.2021.3075644
Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: An efficient alternative to SIFT or SURF. In: 2011 International conference on computer vision, pp. 2564–2571 (2011)
Zhong, F., Wang, S., Zhang, Z., Chen, C., Wang, Y.: Detect-SLAM: Making Object Detection and SLAM Mutually Beneficial. In: 2018 IEEE winter conference on applications of computer vision (WACV), pp. 1001–1010 (2018). https://doi.org/10.1109/WACV.2018.00115
Yang, S., Fan, G., Bai, L., Zhao, C., Li, D.: SGC-VSLAM: A Semantic and Geometric Constraints VSLAM for Dynamic Indoor Environments. Sensors 20(8), (2020)
Xiao, L., Wang, J., Qiu, X., Rong, Z., Zou, X.: Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment. Robot Auton Syst 117, 1–16 (2019). https://doi.org/10.1016/j.robot.2019.03.012
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C., Berg, A.: SSD: single shot multibox detector. In: Proceedings of the european conference on computer vision, pp. 21–37 (2016)
Rother, C., Kolmogorov, V., Blake, A.: GrabCut: Interactive Foreground Extraction Using Iterated Graph Cuts. ACM Trans. Graph. 23(3), 309–314 (2004). https://doi.org/10.1145/1015706.1015720
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE international conference on computer vision, pp. 2961–2969 (2017)
Li, A., Wang, J., Xu, M., Chen, Z.: DP-SLAM: A visual SLAM with moving probability towards dynamic environments. Inf Sci 556, 128–142 (2021). https://doi.org/10.1016/j.ins.2020.12.019
Ji, T., Wang, C., Xie, L.: Towards Real-time Semantic RGB-D SLAM in Dynamic Environments. In: 2021 IEEE international conference on robotics and automation (ICRA), pp. 11175–11181 (2021). https://doi.org/10.1109/ICRA48506.2021.9561743
Soares, J.C.V., Gattass, M., Meggiolaro, M.A.: Crowd-SLAM: Visual SLAM Towards Crowded Environments using Object Detection. J Intell Robot Syst 102(2), 50 (2021). https://doi.org/10.1007/s10846-021-01414-1
Mur-Artal, R., Tardśs, J.D.: ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras. IEEE Trans Robot 33(5), 1255–1262 (2017)
Bochkovskiy, A., Wang, C., Liao, H.M.: YOLOv4: Optimal Speed and Accuracy of Object Detection. CoRR (2020) arXiv:2004.10934
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In: Conference on computer vision and pattern recognition (CVPR) (2012)
Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of RGB-D SLAM systems. In: 2012 IEEE/RSJ International conference on intelligent robots and systems, pp. 573–580 (2012)
Palazzolo, E., Behley, J., Lottes, P., Giguère, P., Stachniss, C.: ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals. In: 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 7855–7862 (2019). https://doi.org/10.1109/IROS40897.2019.8967590
Shi, X., Li, D., Zhao, P., Tian, Q., Tian, Y., Long, Q., Zhu, C., Song, J., Qiao, F., Song, L., Guo, Y., Wang, Z., Zhang, Y., Qin, B., Yang, W., Wang, F., Chan, R.H.M., She, Q.: Are We Ready for Service Robots? The OpenLORIS-Scene Datasets for Lifelong SLAM. In: 2020 International conference on robotics and automation (ICRA), pp. 3139–3145 (2020)
Rosen, D.M., Mason, J., Leonard, J.J.: Towards lifelong feature-based mapping in semi-static environments. 2016 IEEE Int Conf Robo Autom (ICRA), 1063–1070 (2016)
Hashemifar, Z., Dantu, K.: Practical Persistence Reasoning in Visual SLAM. In: 2020 IEEE international conference on robotics and automation (ICRA), pp. 7307–7313 (2020). https://doi.org/10.1109/ICRA40945.2020.9196913
Lázaro, M.T., Capobianco, R., Grisetti, G.: Efficient long-term mapping in dynamic environments. In: 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 153–160 (2018). https://doi.org/10.1109/IROS.2018.8594310
Schmid, L.M., Delmerico, J.A., Schönberger, J.L., Nieto, J.I., Pollefeys, M., Siegwart, R., Cadena, C.: Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency. CoRR (2021) arXiv:2109.10165
Walcott-Bryant, A., Kaess, M., Johannsson, H., Leonard, J.J.: Dynamic pose graph SLAM: Long-term mapping in low dynamic environments. In: 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 1871–1878 (2012). IEEE
Bore, N., Ekekrantz, J., Jensfelt, P., Folkesson, J.: Detection and tracking of general movable objects in large three-dimensional maps. IEEE Trans Robot 35(1), 231–247 (2019). https://doi.org/10.1109/TRO.2018.2876111
Xie, H., Deng, T., Wang, J., Chen, W.: Robust incremental long-term visual topological localization in changing environments. IEEE Trans Instrum Meas 72, 1–14 (2023). https://doi.org/10.1109/TIM.2022.3220270
Adkins, A., Chen, T., Biswas, J.: Probabilistic object maps for longterm robot localization. In: 2022 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 931–938 (2022). https://doi.org/10.1109/IROS47612.2022.9981316
Elvira, R., Tardós, J.D., Montiel, J.M.M.: ORBSLAM-Atlas: a robust and accurate multi-map system. 2019 IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), 6253–6259 (2019)
Lin, T., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European Conference on Computer Vision, pp. 740–755 (2014). Springer
Saho, K.: Kalman filter for moving object tracking: Performance analysis and filter design. In: de Oliveira Serra, G.L. (ed.) Kalman Filters. IntechOpen, Rijeka (2017). Chap. 12. https://doi.org/10.5772/intechopen.71731
EarthSense. https://www.earthsense.co/home. Accessed: 2022-04-22
Redmon, J., Farhadi, A.: YOLOv3: An incremental improvement (2018). arXiv:1804.02767
Scona, R., Jaimez, M., Petillot, Y.R., Fallon, M., Cremers, D.: StaticFusion: Background Reconstruction for Dense RGB-D SLAM in Dynamic Environments. In: 2018 IEEE international conference on robotics and automation (ICRA), pp. 3849–3856 (2018). https://doi.org/10.1109/ICRA.2018.8460681
Liu, H., Liu, G., Tian, G., Xin, S., Ji, Z.: Visual SLAM Based on Dynamic Object Removal. In: 2019 IEEE international conference on robotics and biomimetics (ROBIO), pp. 596–601 (2019). https://doi.org/10.1109/ROBIO49542.2019.8961397
Cui, L., Ma, C.: SOF-SLAM: A Semantic Visual SLAM for Dynamic Environments. IEEE Access 7, 166528–166539 (2019)
Sun, Y., Liu, M., Meng, M.: Improving RGB-D SLAM in dynamic environments: A motion removal approach. Robot Auton Syst 89 (2016). https://doi.org/10.1016/j.robot.2016.11.012
Sun, Y., Liu, M., Meng, M.Q.-H.: Motion removal for reliable RGB-D SLAM in dynamic environments. Rob Auton Syst 108, 115–128 (2018). https://doi.org/10.1016/j.robot.2018.07.002
Chen, Z., Liu, L.: Navigable space construction from sparse noisy pointclouds. IEEE Robot. Autom. Lett. 6(3), 4720–4727 (2021). https://doi.org/10.1109/LRA.2021.3068939
Funding
Open access funding provided by Istituto Italiano di Tecnologia within the CRUI-CARE Agreement. This research was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001, and in part by FAPESP, process number 2021/05336-3.
Author information
Authors and Affiliations
Contributions
Conceptualization and methodology, J.S, V.M and G.A; software, J.S, V.M and G.A; validation, J.S, V.M and G.A.; experiment J.S, V.M, G.A, G.C and M.B; writing – original draft preparation, J.S, V.M and G.A; writing – review and editing, J.S, V.M, G.A, M.B, M.G and M.M; supervision, M.B, M.G and M.M; All authors have read and agreed to the published version of the manuscript.
Corresponding author
Ethics declarations
Competing Interests
The authors declare that they have no competing interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file 1 (mp4 33846 KB)
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Virgolino Soares, J., Medeiros, V.S., Abati, G.F. et al. Visual Localization and Mapping in Dynamic and Changing Environments. J Intell Robot Syst 109, 95 (2023). https://doi.org/10.1007/s10846-023-02019-6
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-023-02019-6