Abstract
Mobile robot localization is an important task in navigation and can be challenging, especially in non-static environments as the scene naturally involves movable objects and appearance changes. In this paper, we address the problem of estimating the robot’s pose in non-static environments containing movable objects. We understand as non-static environments, dynamic environments in which objects might be moved or changed their appearance. We propose a probabilistic localization approach that combines metric and semantic information and takes into account both, static and movable objects. We perform a pixel-wise association of depth and semantic data from an RGB-D sensor with a semantically-augmented truncated signed distance field (TSDF) in order to estimate the robot’s pose. The combination of metric and semantic information increases the robustness w.r.t. movable objects and object appearance changes. The experiments conducted in a real indoor environment and a publicly-available dataset suggest that our approach successfully estimates robot pose in non-static environments and they show an improvement compared to robot localization based only on metric or semantic information and compared to a feature-based method.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Han, S., Xi, Z.: Dynamic Scene Semantics SLAM Based on Semantic Segmentation. IEEE Access 8, 43563–43570 (2020)
Palazzolo, E., Behley, J., Lottes, P., Giguère, P., Stachniss, C.: ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals. arXiv preprint (2019)
Li, S., Lee, D.: RGB-D SLAM in dynamic environments using static point weighting. IEEE Robot. Autom. Lett. (RA-L) 2(4), 2263–2270 (2017)
Bavle, H., Manthe, S., de la Puente, P., Rodriguez-Ramos, A., Sampedro, C., Campoy, P.: Stereo visual odometry and semantics based localization of aerial robots in indoor environments. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp. 1018–1023 (2018)
Zhang, W., Liu, G., Tian, G.: A coarse to fine indoor visual localization method using environmental semantic information. IEEE Access 7, 21963–21970 (2019)
Cramariuc, A., Tschopp, F., Alatur, N., Benz, S., Falck, T., Brühlmeier, M., Hahn, B., Nieto, J., Siegwart, R.: SemSegMap–3D segment-based semantic localization. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp. 1183–1190 (2021)
Hughes, N., Chang, Y., Carlone, L.: Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization. Robotics: Science and Systems (RSS) (2022)
Yu, J., Shen, S.: SemanticLoop: loop closure with 3D semantic graph matching. IEEE Robot. Autom. Lett. (RA-L) 8(2), 568–575 (2022)
Kim, J.J., Urschler, M., Riddle, P.J., Wicker, J.S.: Closing the Loop: Graph Networks to Unify Semantic Objects and Visual Features for Multi-object Scenes. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp. 4352–4358 (2022)
Xiao, L., Wang, J., Qiu, X., Rong, Z., Zou, X.: Dynamic-SLAM: Semantic monocular visual localization and mapping based on deep learning in dynamic environment. J Robot. Autonomous Syst. (RAS) 117, 1–16 (2019)
Xu, B., Li, W., Tzoumanikas, D., Bloesch, M., Davison, A., Leutenegger, S.: MID-Fusion: Octree-based object-level multi-instance dynamic SLAM. In: Proceedings of the IEEE International Conference on Robotics & Automation (ICRA), IEEE, pp. 5231–5237 (2019)
Yang, S., Scherer, S.: CubeSLAM: Monocular 3D object SLAM. IEEE Trans. Robot. 35(4), 925–938 (2019)
Singh, G., Wu, M., Lam, S.-K., et al.: Hierarchical Loop Closure Detection for Long-term Visual SLAM with Semantic-Geometric Descriptors. In: IEEE International Intelligent Transportation Systems Conference (ITSC), IEEE, pp. 2909–2916 (2021)
Cummins, M., Newman, P.: FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance. Intl. J Robot. Res. (IJRR) 27(6), 647–665 (2008)
Dayoub, F., Duckett, T.: An adaptive appearance-based map for long-term topological localization of mobile robots. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp. 3364–3369 (2008)
Derner, E., Gomez, C., Hernandez, A.C., Barber, R., Babuška, R.: Towards life-long autonomy of mobile robots through feature-based change detection. In: Proceedings of the European Conference on Mobile Robotics (ECMR), IEEE, pp. 1–6 (2019)
Patel, N., Khorrami, F., Krishnamurthy, P., Tzes, A.: Tightly Coupled Semantic RGB-D Inertial Odometry for Accurate Long-Term Localization and Mapping. In: Proceedings of the International Conference on Advanced Robotics (ICAR), IEEE, pp. 523–528 (2019)
Stachniss, C., Burgard, W.: Mobile Robot Mapping and Localization in Non-Static Environments. In: Proceedings of the National Conference on Artificial Intelligence (AAAI), Pittsburgh, PA, USA, pp. 1324–1329 (2005)
Vysotska, O., Stachniss, C.: Lazy Data Association For Image Sequences Matching Under Substantial Appearance Changes. IEEE Robot. Autom. Lett. (RA-L) 1(1), 213–220 (2016)
Vysotska, O., Stachniss, C.: Effective Visual Place Recognition Using Multi-Sequence Maps. IEEE Robot. Autom. Lett. (RA-L) 4, 1730–1736 (2019)
Milford, M., Wyeth, G.F.: SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights. In: Proceedings of the IEEE International Conference on Robotics & Automation (ICRA) (2012)
Chen, X., Milioto, A., Palazzolo, E., Giguère, P., Behley, J., Stachniss, C.: SuMa++: Efficient LiDAR-based Semantic SLAM. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2019)
Stenborg, E., Toft, C., Hammarstrand, L.: Long-term visual localization using semantically segmented images. In: Proceedings of the IEEE International Conference on Robotics & Automation (ICRA), IEEE, pp. 6484–6490 (2018)
Arshad, S., Kim, G.-W.: Leveraging Semantics in Appearance based Loop Closure Detection for Long-Term Visual SLAM. In: IEEE International Conference on Big Data and Smart Computing (BigComp), IEEE, pp. 371–372 (2023)
Chebrolu, N., Lottes, P., Läbe, T., Stachniss, C.: Robot Localization Based on Aerial Images for Precision Agriculture Tasks in Crop Fields. In: Proceedings of the IEEE International Conference on Robotics & Automation (ICRA), IEEE, pp. 1787–1793 (2019)
Radwan, N., Valada, A., Burgard, W.: Vlocnet++: Deep multitask learning for semantic visual localization and odometry. IEEE Robot. Autom. Lett. (RA-L) 3(4), 4407–4414 (2018)
Toft, C., Stenborg, E., Hammarstrand, L., Brynte, L., Pollefeys, M., Sattler, T., Kahl, F.: Semantic match consistency for long-term visual localization. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 383–399 (2018)
Zeng, A., Song, S., Nießner, M., Fisher, M., Xiao, J., Funkhouser, T.: 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1802–1811 (2017)
Dellaert, F., Fox, D., Burgard, W., Thrun, S.: Monte Carlo Localization for Mobile Robots, booktitle = IEEE International Conference on Robotics and Automation (ICRA). (1999)
Thrun, S., Burgard, W., Fox, D.: Probabilistic Robotics (Intelligent Robotics and Autonomous Agents). The MIT Press, Cambridge, Massachusetts (2005)
Fox, D., Burgard, W., Thrun, S.: Markov localization for mobile robots in dynamic environments. J Artif. Intell. Res. 11, 391–427 (1999)
Krajník, T., Fentanes, J.P., Mozos, O., Duckett, T., Ekekrantz, J., Hanheide, M.: Long-term topological localisation for service robots in dynamic environments using spectral maps. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, pp. 4537–4542 (2014)
He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2017)
Gomez, C., Hernandez, A.C., Derner, E., Barber, R., Babuška, R.: Object-Based Pose Graph for Dynamic Indoor Environments. IEEE Robot. Autom. Lett. (RA-L) 5(4), 5401–5408 (2020)
Funding
This work has partially been funded by the European Union’s Horizon 2020 research and innovation programme under grant agreement No 101017008 (Harmony), also by HEROITEA: Heterogeneous Intelligent Multi-Robot Team for Assistance of Elderly People (RTI2018-095599-B-C21), funded by Spanish Ministerio de Economia y Competitividad, and the RoboCity2030 – DIH-CM project (S2018/NMT-4331, RoboCity2030 – Madrid Robotics Digital Innovation Hub).
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Implementation, data collection and analysis were performed by Clara Gomez. The first draft of the manuscript was written by Clara Gomez and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval
Not applicable.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Gomez, C., Hernandez, A.C., Barber, R. et al. Localization Exploiting Semantic and Metric Information in Non-static Indoor Environments. J Intell Robot Syst 109, 86 (2023). https://doi.org/10.1007/s10846-023-02021-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10846-023-02021-y