Skip to main content
SpringerLink
Log in

Map management for robust long-term visual localization of an autonomous shuttle in changing conditions

  • 1200: Machine Vision Theory and Applications for Cyber Physical Systems
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Changes in appearance present a tremendous problem for the visual localization of an autonomous vehicle in outdoor environments. Data association between the current image and the landmarks in the map can be challenging in cases where the map was built with different environmental conditions. This paper introduces a solution to build and use multi-session maps incorporating sequences recorded in different conditions (day, night, fog, snow, rain, change of season, etc.). During visual localization, we exploit a ranking function to extract the most relevant keyframes from the map. This ranking function is designed to take into account the pose of the vehicle as well as the current environmental condition. In the mapping phase, covering all conditions by constantly adding data to the map leads to a continuous growth in the map size which in turn deteriorates the localization speed and performance. Our map management strategy is an incremental approach that aims to limit the size of the map while keeping it as diverse as possible. Our experiments were performed on real data collected with our autonomous shuttle as well as on a widely used public dataset. The results demonstrate that our keyframe-based ranking function is suitable for long-term scenarios. Our map management algorithm aims to build a map with as much diversity as possible whereas some state of the art approaches tend to filter out the less observed landmarks. This strategy shows a reduction of localization failures while maintaining real-time performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Notes

  1. To download our dataset please visit http://iplt.ip.uca.fr/datasets/ and enter the following username/password for a read-only access to our ftp server: ipltuser/iplt_ro

References

  1. Arandjelovic R, Gronat P, Torii A, Pajdla T, Sivic J (2016) Netvlad: Cnn architecture for weakly supervised place recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5297–5307

  2. Bay H, Tuytelaars T, Van Gool L (2006) Surf: Speeded up robust features. In: European conference on computer vision. pp. 404–417. Springer

  3. Berrio JS, Ward J, Worrall S, Nebot E (2019) Identifying robust landmarks in feature-based maps. In: 2019 IEEE Intelligent vehicles symposium (IV). pp. 1166–1172. IEEE

  4. Bouaziz Y, Royer E, Bresson G, Dhome M (2021) Keyframes retrieval for robust long-term visual localization in changing conditions. In: 2021 IEEE 19Th world symposium on applied machine intelligence and informatics (SAMI). pp. 093–100. IEEE

  5. Bouaziz Y, Royer E, Bresson G, Dhome M (2021) Over two years of challenging environmental conditions for localization: the iplt dataset. In: 18Th international conference on informatics in control, automation and robotics

  6. Bürki M, Cadena C, Gilitschenski I, Siegwart R, Nieto J (2019) Appearance-based landmark selection for visual localization. Journal of Field Robotics 36(6):1041–1073

    Article  Google Scholar 

  7. Bürki M, Dymczyk M, Gilitschenski I, Cadena C, Siegwart R, Nieto J (2018) Map management for efficient long-term visual localization in outdoor environments. In: 2018 IEEE Intelligent vehicles symposium (IV). pp. 682–688. IEEE

  8. Bürki M, Gilitschenski I, Stumm E, Siegwart R, Nieto J (2016) Appearance-based landmark selection for efficient long-term visual localization. In: 2016 IEEE/RSJ International conference on intelligent robots and systems (IROS). pp. 4137–4143. IEEE

  9. Carlevaris-Bianco N, Ushani AK, Eustice RM (2015) University of Michigan North Campus long-term vision and lidar dataset. Int J Robot Res 35 (9):1023–1035

    Article  Google Scholar 

  10. Chen C, Wang B, Lu CX, Trigoni A, Markham A (2020) A survey on deep learning for localization and mapping: Towards the age of spatial machine intelligence. ArXiv 2006.12567

  11. Churchill W, Newman P (2012) Practice makes perfect? managing and leveraging visual experiences for lifelong navigation. In: 2012 IEEE International conference on robotics and automation. pp. 4525–4532. IEEE

  12. Churchill W, Newman P (2013) Experience-based navigation for long-term localisation. The International Journal of Robotics Research 32 (14):1645–1661

    Article  Google Scholar 

  13. Clark R, Wang S, Markham A, Trigoni N, Wen H (2017) Vidloc: a deep spatio-temporal model for 6-dof video-clip relocalization. pp 2652–2660 (07. https://doi.org/10.1109/CVPR.2017.284

  14. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3213–3223

  15. Diaz-Escobar J, Kober V, Gonzalez-Fraga JA (2018) Luift: Luminance invariant feature transform. Math Probl Eng 2018:1–17

    Article  Google Scholar 

  16. Dusmanu M, Rocco I, Pajdla T, Pollefeys M, Sivic J, Torii A, Sattler T (2019) D2-net: a trainable cnn for joint description and detection of local features. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8092–8101

  17. Dymczyk M, Lynen S, Cieslewski T, Bosse M, Siegwart R, Furgale P (2015) The gist of maps-summarizing experience for lifelong localization. In: 2015 IEEE International conference on robotics and automation (ICRA). pp. 2767–2773. IEEE

  18. Dymczyk M, Schneider T, Gilitschenski I, Siegwart R, Stumm E (2016) Erasing bad memories: Agent-side summarization for long-term mapping. In: 2016 IEEE/RSJ International conference on intelligent robots and systems (IROS). pp. 4572–4579. IEEE

  19. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: the kitti dataset. The International Journal of Robotics Research 32(11):1231–1237

    Article  Google Scholar 

  20. Gridseth M, Barfoot TD (2020) Deepmel: Compiling visual multi-experience localization into a deep neural network. In: 2020 IEEE International conference on robotics and automation (ICRA). pp. 1674–1681. IEEE

  21. Halodová L, Dvoráková E, Majer F, Vintr T, Mozos OM, Dayoub F, Krajník T (2019) Predictive and adaptive maps for long-term visual navigation in changing environments. In: 2019 IEEE/RSJ International conference on intelligent robots and systems (IROS). pp. 7033–7039. IEEE

  22. Harris CG, Stephens M, et al. (1988) A combined corner and edge detector. In: Alvey vision conference. vol. 15, pp. 10–5244. Citeseer

  23. Jatzkowski I, Wilke D, Maurer M (2018) A deep-learning approach for the detection of overexposure in automotive camera images. In: 2018 21St international conference on intelligent transportation systems (ITSC). pp. 2030–2035. IEEE

  24. Kendall A, Grimes M, Cipolla R (2015) Posenet: a convolutional network for real-time 6-dof camera relocalization. In: Proceedings of the IEEE international conference on computer vision. pp. 2938–2946

  25. Krajník T, Fentanes JP, Hanheide M, Duckett T (2016) Persistent localization and life-long mapping in changing environments using the frequency map enhancement. In: 2016 IEEE/RSJ International conference on intelligent robots and systems (IROS). pp. 4558–4563. IEEE

  26. Krajník T, Vintr T, Molina S, Fentanes JP, Cielniak G, Mozos OM, Broughton G, Duckett T (2019) Warped hypertime representations for long-term autonomy of mobile robots. IEEE Robotics and Automation Letters 4 (4):3310–3317

    Article  Google Scholar 

  27. Laskar Z, Melekhov I, Kalia S, Kannala J (2017) Camera relocalization by computing pairwise relative poses using convolutional neural network. In: Proceedings of the IEEE international conference on computer vision workshops. pp. 929–938

  28. Lébraly P., Royer E, Ait-Aider O, Deymier C, Dhome M (2011) Fast calibration of embedded non-overlapping cameras. In: 2011 IEEE International conference on robotics and automation. pp. 221–227. IEEE

  29. Linegar C, Churchill W, Newman P (2015) Work smart, not hard: Recalling relevant experiences for vast-scale but time-constrained localisation. In: 2015 IEEE International conference on robotics and automation (ICRA). pp. 90–97. IEEE

  30. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2):91–110

    Article  Google Scholar 

  31. MacTavish K, Paton M, Barfoot TD (2018) Selective memory: Recalling relevant experience for long-term visual localization. Journal of Field Robotics 35 (8):1265–1292

    Article  Google Scholar 

  32. Maddern W, Pascoe G, Linegar C, Newman P (2017) 1 year, 1000km: the oxford robotcar dataset. The International Journal of Robotics Research (IJRR) 36(1):3–15. https://doi.org/10.1177/0278364916679498

    Article  Google Scholar 

  33. Maddern W, Pascoe G, Gadd M, Barnes D, Yeomans B, Newman P (2020) Real-time kinematic ground truth for the oxford robotcar dataset. arXiv:2002.10152

  34. Magnago V, Palopoli L, Passerone R, Fontanelli D, Macii D (2019) Effective landmark placement for robot indoor localization with position uncertainty constraints. IEEE Trans Instrum Meas 68(11):4443–4455

    Article  Google Scholar 

  35. Milford MJ, Wyeth GF (2012) Seqslam: Visual route-based navigation for sunny summer days and stormy winter nights. In: 2012 IEEE International conference on robotics and automation. pp. 1643–1649. IEEE

  36. Mühlfellner P, Bürki M, Bosse M, Derendarz W, Philippsen R, Furgale P (2016) Summary maps for lifelong visual localization. Journal of Field Robotics 33(5):561–590

    Article  Google Scholar 

  37. Mur-Artal R, Montiel JMM, Tardos JD (2015) Orb-slam: a versatile and accurate monocular slam system. IEEE Transactions on Robotics 31(5):1147–1163

    Article  Google Scholar 

  38. Murillo AC, Kosecka J (2009) Experiments in place recognition using gist panoramas. In: 2009 IEEE 12Th international conference on computer vision workshops, ICCV workshops. pp. 2196–2203. IEEE

  39. Naseer T, Oliveira GL, Brox T, Burgard W (2017) Semantics-aware visual localization under challenging perceptual conditions. In: 2017 IEEE International conference on robotics and automation (ICRA). pp. 2614–2620. IEEE

  40. Pascoe G, Maddern W, Tanner M, Piniés P, Newman P (2017) Nid-slam: Robust monocular slam using normalised information distance. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1435–1444

  41. Pepperell E, Corke P, Milford M (2016) Routed roads: Probabilistic vision-based place recognition for changing conditions, split streets and varied viewpoints. The International Journal of Robotics Research 35(9):1057–1179

    Article  Google Scholar 

  42. Rosen DM, Mason J, Leonard JJ (2016) Towards lifelong feature-based mapping in semi-static environments. In: 2016 IEEE International conference on robotics and automation (ICRA). pp. 1063–1070. IEEE

  43. Royer E, Marmoiton F, Alizon S, Ramadasan D, Slade M, Nizard A, Dhome M, Thuilot B, Bonjean F (2016) Lessons learned after more than 1000 km in an autonomous shuttle guided by vision. In: 2016 IEEE 19Th international conference on intelligent transportation systems (ITSC). pp. 2248–2253. IEEE

  44. Schneider T, Dymczyk M, Fehr M, Egger K, Lynen S, Gilitschenski I, Siegwart R (2018) Maplab: an open framework for research in visual-inertial mapping and localization. IEEE Robotics and Automation Letters 3(3):1418–1425

    Article  Google Scholar 

  45. Schönberger JL, Pollefeys M, Geiger A, Sattler T (2018) Semantic visual localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6896–6906

  46. Stenborg E, Sattler T, Hammarstrand L (2020) Using image sequences for long-term visual localization. In: 2020 International conference on 3d vision (3DV). pp. 938–948. https://doi.org/10.1109/3DV50981.2020.00104

  47. Tian Y, Yu X, Fan B, Wu F, Heijnen H, Balntas V (2019) Sosnet: Second order similarity regularization for local descriptor learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 11016–11025

  48. Walch F, Hazirbas C, Leal-Taixe L, Sattler T, Hilsenbeck S, Cremers D (2017) Image-based localization using lstms for structured feature correlation. In: Proceedings of the IEEE international conference on computer vision. pp. 627–637

  49. Yan Z, Sun L, Krajník T, Ruichek Y (2020) Eu long-term dataset with multiple sensors for autonomous driving. In: 2020 IEEE/RSJ International conference on intelligent robots and systems (IROS). pp. 10697–10704. https://doi.org/10.1109/IROS45743.2020.9341406

  50. Yi KM, Trulls E, Lepetit V, Fua P (2016) Lift: Learned invariant feature transform. In: European conference on computer vision. pp. 467–483. Springer

Download references

Acknowledgements

This work has been sponsored by the French government research program “Investissements d’Avenir” through the IMobS3 Laboratory of Excellence (ANR-10-LABX-16-01) and the RobotEx Equipment of Excellence (ANR-10-EQPX-44), by the European Union through the Regional Competitiveness and Employment program 2014-2020 (ERDF - AURA region) and by the AURA region.

Author information

Authors and Affiliations

  1. Institut Pascal, CNRS, Clermont-Ferrand, Institut VEDECOM, Versailles, France

    Youssef Bouaziz

  2. Institut Pascal, CNRS, SIGMA Clermont, Clermont-Ferrand, France

    Eric Royer & Michel Dhome

  3. Institut VEDECOM, Versailles, France

    Guillaume Bresson

Authors
  1. Youssef Bouaziz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eric Royer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guillaume Bresson
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michel Dhome
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Youssef Bouaziz.

Ethics declarations

Conflict of Interests

We declare that the authors have no conflict of interest on this work.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

(MP4 15.5 MB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bouaziz, Y., Royer, E., Bresson, G. et al. Map management for robust long-term visual localization of an autonomous shuttle in changing conditions. Multimed Tools Appl 81, 22449–22480 (2022). https://doi.org/10.1007/s11042-021-11870-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11870-4

Keywords

Search

Navigation