Skip to main content
SpringerLink
Log in

IBISCape: A Simulated Benchmark for multi-modal SLAM Systems Evaluation in Large-scale Dynamic Environments

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

The development process of high fidelity SLAM systems depends on their validation upon reliable datasets. Towards this goal, we propose IBISCape, a simulated benchmark that includes data synchronization and acquisition APIs for telemetry from heterogeneous sensors: stereo-RGB/DVS, LiDAR, IMU, and GPS, along with the ground truth scene segmentation, depth maps and vehicle ego-motion. Our benchmark is built upon the CARLA simulator, whose back-end is the Unreal Engine rendering a high dynamic scenery simulating the real world. Moreover, we offer 43 datasets for Autonomous Ground Vehicles (AGVs) reliability assessment, including scenarios for scene understanding evaluation like accidents, along with a wide range of frame quality based on a dynamic weather simulation class integrated with our APIs. We also introduce the first calibration targets to CARLA maps to solve the unknown distortion parameters problem of CARLA simulated DVS and RGB cameras. Furthermore, we propose a novel pre-processing layer that eases the integration of DVS sensor events in any frame-based Visual-SLAM system. Finally, extensive qualitative and quantitative evaluations of the latest state-of-the-art Visual/Visual-Inertial/LiDAR SLAM systems are performed on various IBISCape sequences collected in simulated large-scale dynamic environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Forster, C., Zhang, Z., Gassner, M., Werlberger, M., Scaramuzza, D.: Svo: Semidirect visual odometry for monocular and multicamera systems. IEEE Trans. Robot. 33(2), 249–265 (2017). https://doi.org/10.1109/TRO.2016.2623335

    Article  Google Scholar 

  2. Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R., Furgale, P.: Keyframe-based visual-inertial odometry using nonlinear optimization. Int. J. Robot. Res. 34 (2014). https://doi.org/10.1177/0278364914554813

  3. Forster, C., Carlone, L., Dellaert, F., Scaramuzza, D.: On-manifold preintegration for real-time visual-inertial odometry. IEEE Trans. Robot. 33(1), 1–21 (2017). https://doi.org/10.1109/TRO.2016.2597321

    Article  Google Scholar 

  4. Qin, T., Li, P., Shen, S.: Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Trans. Robot. 34(4), 1004–1020 (2018). https://doi.org/10.1109/TRO.2018.2853729

    Article  Google Scholar 

  5. Kerl, C., Sturm, J., Cremers, D.: Dense visual slam for rgb-d cameras. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (2013)

  6. Alliez, P. etal.: Real-time multi-slam system for agent localization and 3d mapping in dynamic scenarios. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4894–4900 (2020)

  7. Caron, F., Duflos, E., Pomorski, D., Vanheeghe, P.: Gps/imu data fusion using multisensor kalman filtering: introduction of contextual aspects. Information fusion 7(2), 221–230 (2006)

    Article  Google Scholar 

  8. Yang, Y., et al.: icalib: Inertial aided multi-sensor calibration. ICRA - VINS Workshop 2021. Xi’an, China (2021)

  9. PerŠić, J., Petrović, L., Marković, I., Petrović, I.: Spatiotemporal multisensor calibration via gaussian processes moving target tracking. IEEE Trans. Robot. 1–15 (2021). https://doi.org/10.1109/TRO.2021.3061364

  10. Lee, W., Yang, Y., Huang, G.: Efficient multi-sensor aided inertial navigation with online calibration. 2021 IEEE International Conference on Robotics and Automation (ICRA) (2021)

  11. Gehrig, D., Rüegg, M., Gehrig, M., Hidalgo-Carrió, J., Scaramuzza, D.: Combining events and frames using recurrent asynchronous multimodal networks for monocular depth prediction. IEEE Robot. Autom. Lett. 6(2), 2822–2829 (2021)

    Article  Google Scholar 

  12. Gehrig, M., Aarents, W., Gehrig, D., Scaramuzza, D.: Dsec: A stereo event camera dataset for driving scenarios. IEEE Robot. Autom. Lett. PP, 1–8 (2021). https://doi.org/10.1109/LRA.2021.3068942

  13. Li, Y., Yunus, R., Brasch, N., Navab, N., Tombari, F.: Rgb-d slam with structural regularities. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 11581–11587 (2021)

  14. Debeunne, C., Vivet, D.: A review of visual-lidar fusion based simultaneous localization and mapping. Sensors 20(7) (2020). https://www.mdpi.com/1424-8220/20/7/2068. https://doi.org/10.3390/s20072068

  15. Minoda, K., Schilling, F., Wüest, V., Floreano, D., Yairi, T.: Viode: A simulated dataset to address the challenges of visual-inertial odometry in dynamic environments. IEEE Robot. Autom. Lett. 6(2), 1343–1350 (2021). https://doi.org/10.1109/LRA.2021.3058073

    Article  Google Scholar 

  16. Deschaud, J.-E. etal. Paris-carla-3d: A real and synthetic outdoor point cloud dataset for challenging tasks in 3d mapping. Remote Sensing 13(22) (2021). https://www.mdpi.com/2072-4292/13/22/4713. 10.3390/rs13224713

  17. Deschaud, J.-E.: KITTI-CARLA: a KITTI-like dataset generated by CARLA Simulator. arXiv e-prints (2021)

  18. Sekkat, A.R., et al.: Synwoodscape: Synthetic surround-view fisheye camera dataset for autonomous driving. IEEE Robotics and Automation Letters 7(3), 8502–8509 (2022). https://doi.org/10.1109/LRA.2022.3188106

    Article  Google Scholar 

  19. Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: An open urban driving simulator. Proceedings of the 1st Annual Conference on Robot Learning (2017)

  20. Gehrig, D., Loquercio, A., Derpanis, K.G., Scaramuzza, D.: End-to-end learning of representations for asynchronous event-based data. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5633–5643 (2019)

  21. Sturm, J., Engelhard, N., Endres, F., Burgard, W., Cremers, D.: A benchmark for the evaluation of rgb-d slam systems. 2012 IEEE/RSJ international conference on intelligent robots and systems (2012)

  22. Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. Int. J. Rob. Res. 32(11), 1231–1237 (2013)

    Article  Google Scholar 

  23. Blanco-Claraco, J.-L., Moreno-Duenas, F.-A., González-Jiménez, J.: The málaga urban dataset: High-rate stereo and lidar in a realistic urban scenario. Int. J. Rob. Res. 33(2), 207–214 (2014)

    Article  Google Scholar 

  24. Carlevaris-Bianco, N., Ushani, A.K., Eustice, R.M.: University of michigan north campus long-term vision and lidar dataset. Int. J. Rob. Res. 35(9), 1023–1035 (2016)

    Article  Google Scholar 

  25. Burri, M., et al.: The euroc micro aerial vehicle datasets. Int. J. Rob. Res. 35(10), 1157–1163 (2016)

    Article  Google Scholar 

  26. Majdik, A.L., Till, C., Scaramuzza, D.: The Zurich urban micro aerial vehicle dataset. Int. J. Rob. Res. 36(3), 269–273 (2017)

    Article  Google Scholar 

  27. Pfrommer, B., Sanket, N., Daniilidis, K., Cleveland, J.: Penncosyvio: A challenging visual inertial odometry benchmark. 2017 IEEE International Conference on Robotics and Automation (ICRA) (2017)

  28. Schubert, D. etal.: The tum vi benchmark for evaluating visual-inertial odometry. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1680–1687 (2018)

  29. Judd, K.M., Gammell, J.D.: The oxford multimotion dataset: Multiple se (3) motions with ground truth. IEEE Robotics and Automation Letters 4(2), 800–807 (2019)

    Article  Google Scholar 

  30. Jeong, J., Cho, Y., Shin, Y.-S., Roh, H., Kim, A.: Complex urban dataset with multi-level sensors from highly diverse urban environments. Int. J. Rob. Res. 38(6), 642–657 (2019)

    Article  Google Scholar 

  31. Kasper, M., McGuire, S., Heckman, C.: A benchmark for visual-inertial odometry systems employing onboard illumination. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2019)

  32. Delmerico, J., Cieslewski, T., Rebecq, H., Faessler, M., Scaramuzza, D.: Are we ready for autonomous drone racing? the uzh-fpv drone racing dataset. 2019 International Conference on Robotics and Automation (ICRA) (2019)

  33. Zuñiga-Noël, D., Jaenal, A., Gomez-Ojeda, R., Gonzalez-Jimenez, J.: The uma-vi dataset: Visual-inertial odometry in low-textured and dynamic illumination environments. Int. J. Rob. Res. 39(9), 1052–1060 (2020)

    Article  Google Scholar 

  34. Antonini, A., Guerra, W., Murali, V., Sayre-McCord, T., Karaman, S.: The blackbird uav dataset. Int. J. Rob. Res. 39(10–11), 1346–1364 (2020)

    Article  Google Scholar 

  35. Zhang, H., Jin, L., Ye, C.: The vcu-rvi benchmark: Evaluating visual inertial odometry for indoor navigation applications with an rgb-d camera. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6209–6214 (2020). https://doi.org/10.1109/IROS45743.2020.9341713

  36. Klenk, S., Chui, J., Demmel, N., Cremers, D.: Tum-vie: The tum stereo visual-inertial event dataset. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8601–8608 (2021). https://doi.org/10.1109/IROS51168.2021.9636728

  37. Yuan, C., et al.: A novel fault-tolerant navigation and positioning method with stereo-camera/micro electro mechanical systems inertial measurement unit (mems-imu) in hostile environment. Micromachines 9, 626 (2018). https://doi.org/10.3390/mi9120626

    Article  Google Scholar 

  38. Faessler, M. etal.: Autonomous, vision-based flight and live dense 3d mapping with a quadrotor micro aerial vehicle. J. Field Robot. 33(4), 431–450 (2016). https://onlinelibrary.wiley.com/doi/abs/10.1002/rob.21581

  39. Lynen, S., Achtelik, M.W., Weiss, S., Chli, M., Siegwart, R.: A robust and modular multi-sensor fusion approach applied to mav navigation. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (2013)

  40. Mourikis, A.I., Roumeliotis, S.I.: A multi-state constraint kalman filter for vision-aided inertial navigation. Proceedings 2007 IEEE International Conference on Robotics and Automation (2007)

  41. Bloesch, M., Omari, S., Hutter, M., Siegwart, R.: Robust visual inertial odometry using a direct ekf-based approach. 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) (2015)

  42. Qin, T., Li, P., Shen, S.: Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Trans. Robot. 34(4), 1004–1020 (2018)

    Article  Google Scholar 

  43. Leutenegger, S., Lynen, S., Bosse, M., Siegwart, R., Furgale, P.: Keyframe-based visual-inertial odometry using nonlinear optimization. Int. J. Rob. Res. 34(3), 314–334 (2015)

    Article  Google Scholar 

  44. Campos, C., Elvira, R., Rodriguez, J.J.G., M.Montiel, J.M., D.Tardos, J.: Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Trans. Robot. 1–17 (2021). http://dx.doi.org/10.1109/TRO.2021.3075644. https://doi.org/10.1109/tro.2021.3075644

  45. Usenko, V., Demmel, N., Schubert, D., Stueckler, J., Cremers, D.: Visual-inertial mapping with non-linear factor recovery. IEEE Robotics and Automation Letters (RA-L) & Int. Conference on Intelligent Robotics and Automation (ICRA) 5(2), 422–429 (2020). https://doi.org/10.1109/LRA.2019.2961227

    Article  Google Scholar 

  46. Delmerico, J., Scaramuzza, D.: A benchmark comparison of monocular visual-inertial odometry algorithms for flying robots. 2018 IEEE International Conference on Robotics and Automation (ICRA) (2018)

  47. Zhou, Y., Gallego, G., Shen, S.: Event-based stereo visual odometry. IEEE Trans. Robot. 37(5), 1433–1450 (2021). https://doi.org/10.1109/TRO.2021.3062252

    Article  Google Scholar 

  48. Gehrig, D., Gehrig, M., Hidalgo-Carrio, J., Scaramuzza, D.: Video to events: Recycling video datasets for event cameras. IEEE Conf. Comput. Vis. Pattern Recog. (CVPR), pp. 3583–3592 (2020). https://doi.org/10.1109/CVPR42600.2020.00364

  49. Rebecq, H., Gallego, G., Mueggler, E., Scaramuzza, D.: EMVS: Event-based multi-view stereo–3D reconstruction with an event camera in real-time. Int. J. Comput. Vis. 126, 1394–1414 (2018). https://doi.org/10.1007/s11263-017-1050-6

    Article  Google Scholar 

  50. Tomy, A., Paigwar, A., Mann, K.S., Renzaglia, A., Laugier, C.: Fusing Event-based and RGB camera for Robust Object Detection in Adverse Conditions. ICRA 2022 - IEEE International Conference on Robotics and Automation (2022). https://hal.archives-ouvertes.fr/hal-03591717

  51. Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: Events-to-video: Bringing modern computer vision to event cameras. IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2019)

  52. Zhang, J., Singh, S.: Loam: Lidar odometry and mapping in real-time. Robotics: Science and Systems (2014)

  53. Pan, Y., Xiao, P., He, Y., Shao, Z., Li, Z.: Mulls: Versatile lidar slam via multi-metric linear least square. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 11633–11640 (2021). https://doi.org/10.1109/ICRA48506.2021.9561364

  54. Rehder, J., Nikolic, J., Schneider, T., Hinzmann, T., Siegwart, R.: Extending kalibr: Calibrating the extrinsics of multiple imus and of individual axes. 2016 IEEE International Conference on Robotics and Automation (ICRA) (2016)

  55. Muglikar, M., Gehrig, M., Gehrig, D., Scaramuzza, D.: How to calibrate your event camera. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1403–1409 (2021)

  56. Galleani, L., Tavella, P.: The dynamic allan variance. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 56(3), 450–464 (2009). https://doi.org/10.1109/TUFFC.2009.1064

    Article  Google Scholar 

  57. Tomasi, C., Kanade, T.: Detection and tracking of point. Int. J. Comput. Vis. 9, 137–154 (1991)

    Article  Google Scholar 

  58. Chen, W. etal.: An overview on visual slam: From tradition to semantic. Remote Sensing 14(13) (2022). https://www.mdpi.com/2072-4292/14/13/3010. https://doi.org/10.3390/rs14133010

  59. Sironi, A., Brambilla, M., Bourdis, N., Lagorce, X., Benosman, R.: Hats: Histograms of averaged time surfaces for robust event-based object classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1731–1740 (2018)

  60. Yang, H., Shi, J., Carlone, L.: Teaser: Fast and certifiable point cloud registration. IEEE Trans. Robot. 37(2), 314–333 (2020)

    Article  Google Scholar 

  61. Zhou, Y. etal. Semi-dense 3d reconstruction with a stereo event camera. Proceedings of the European conference on computer vision (ECCV), pp. 235–251 (2018)

Download references

Funding

This work is supported by the French Ministry of Higher Education, Research and Innovation (MESRI). Author A.S. has received a Ph. D. grant from MESRI covering this research.

Author information

Authors and Affiliations

  1. Université Paris-Saclay, Univ Evry, IBISC Laboratory, 34 Rue du Pelvoux, Evry, 91020, Essonne, France

    Abanob Soliman, Fabien Bonardi, Désiré Sidibé & Samia Bouchafa

Authors
  1. Abanob Soliman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fabien Bonardi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Désiré Sidibé
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Samia Bouchafa
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. The first draft of the manuscript was written by A.S. and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Abanob Soliman.

Ethics declarations

Ethics approval

The submitted work is original and not have been published elsewhere in any form or language.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Ethics Declarations

This research work is based on computer simulation open source software and did not involve human participants or animals. Hence, Consent to participate and Consent for publication are not applicable.

Competing Interests

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 115249 KB)

Appendix A: Extended Data

Appendix A: Extended Data

We generate data by eight acquisition APIs with four sensor setups mentioned in Table 4 in two groups: 1. calibration and 2. SLAM. SLAM data acquisition APIs run on all CARLA maps with an autopilot for traffic-aligned navigation. On the other hand, calibration APIs run on our modified CARLA-map with manual vehicle control to apply desired motions to collect sequences with basic or complex motions. Both AprilGrid and Checkerboard targets are introduced during acquisition. Half of the calibration sequences are collected using the AprilGrid \(6\times 6\) and the other half using the Checkerboard \(7\times 7\).

In order to operate all sensors in the same acquisition API on multiple frequencies, we develop the following procedure: the core data acquisition concept is that the CARLA world clock ticks with the highest frequency sensor in the setup. After that, the system waits to listen to all sensors sending data at this tick, updates the weather conditions, and waits for a new world tick. This allows the acquisition of all sensors data with its occurrence timestamps. Then, one can apply any synchronization/calibration algorithms on the collected datasets as in [8, 10]. We apply this methodology (see Program 1) to all sensor setups except the RGB-D setup, which requires time-synchronized and registered frames.

figure b

On the contrary, the CARLA world ticks with the lowest frequency sensor in the LiDAR/RGB-D setup with CARLA synchronous_mode acquisition (see Program 2). All the spawned sensors in the setup are stacked in a queue waiting for the world’s tick to start listening to the data. Although all sensors operate with their frequencies, the API reads the measurements of all sensors simultaneously at the timestamp of that CARLA world tick.

figure c

The open source data acquisition APIs and all sequences can be accessed using the Github repository: https://github.com/AbanobSoliman/IBISCape.git.

In the repository there is a complete manual on how to execute the APIs in all setups and options, including a library developed for IBISCape dataset files format to be processed using Robotic Operating System (ROS) based algorithms. Besides the Python based ROS tools, we attach the configuration files for all the assessed algorithms along with the Kalibr calibration results. A more in-detail insights of the IBISCape benchmark is available in the supplementary multimedia file.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Soliman, A., Bonardi, F., Sidibé, D. et al. IBISCape: A Simulated Benchmark for multi-modal SLAM Systems Evaluation in Large-scale Dynamic Environments. J Intell Robot Syst 106, 53 (2022). https://doi.org/10.1007/s10846-022-01753-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-022-01753-7

Keywords

Search

Navigation