Abstract
We reveal how implementing the homogeneous, multi-scale mapping frameworks observed in the mammalian brain’s mapping systems radically improves the performance of a range of current robotic localization techniques. Roboticists have developed a range of predominantly single- or dual-scale heterogeneous mapping approaches (typically locally metric and globally topological) that starkly contrast with neural encoding of space in mammalian brains: a multi-scale map underpinned by spatially responsive cells like the grid cells found in the rodent entorhinal cortex. Yet the full benefits of a homogeneous multi-scale mapping framework remain unknown in both robotics and biology: in robotics because of the focus on single- or two-scale systems and limits in the scalability and open-field nature of current test environments and benchmark datasets; in biology because of technical limitations when recording from rodents during movement over large areas. New global spatial databases with visual information varying over several orders of magnitude in scale enable us to investigate this question for the first time in real-world environments. In particular, we investigate and answer the following questions: why have multi-scale representations, how many scales should there be, what should the size ratio between consecutive scales be and how does the absolute scale size affect performance? We answer these questions by developing and evaluating a homogeneous, multi-scale mapping framework mimicking aspects of the rodent multi-scale map, but using current robotic place recognition techniques at each scale. Results in large-scale real-world environments demonstrate multi-faceted and significant benefits for mapping and localization performance and identify the key factors that determine performance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Arandjelovic R, Gronat P, Torii A, Pajdla T, Sivic J (2018) NetVLAD: CNN architecture for weakly supervised place recognition. IEEE Trans Pattern Anal Mach Intell 40(6):1437–1451. https://doi.org/10.1109/TPAMI.2017.2711011. arXiv:1511.07247
Arleo A, Gerstner W (2000) Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity. Biol Cybern. https://doi.org/10.1007/s004220000171
Banino A, Barry C, Uria B, Blundell C, Lillicrap T, Mirowski P, Pritzel A, Chadwick MJ, Degris T, Modayil J, Wayne G, Soyer H, Viola F, Zhang B, Goroshin R, Rabinowitz N, Pascanu R, Beattie C, Petersen S, Sadik A, Gaffney S, King H, Kavukcuoglu K, Hassabis D, Hadsell R, Kumaran D (2018) Vector-based navigation using grid-like representations in artificial agents. Nature 557(7705):429–433. https://doi.org/10.1038/s41586-018-0102-6
Bosse M, Newman P, Leonard J, Soika M, Feiten W, Teller S (2003) An Atlas framework for scalable mapping. In: 2003 IEEE international conference on robotics and automation (Cat. No.03CH37422). IEEE, pp 1899–1906. https://doi.org/10.1109/ROBOT.2003.1241872
Browning B (2000) Biologically plausible spatial navigation for a mobile robot. Computer Science and Electrical Engineering
Brun VH, Solstad T, Kjelstrup KB, Fyhn M, Witter MP, Moser EI, Moser MB (2008) Progressive increase in grid scale from dorsal to ventral medial entorhinal cortex. Hippocampus 18(12):1200–1212. https://doi.org/10.1002/hipo.20504
Burak Y, Fiete IR (2009) Accurate path integration in continuous attractor network models of grid cells. PLoS Comput Biol 5(2):e1000291. https://doi.org/10.1371/journal.pcbi.1000291
Casali G, Bush D, Jeffery K (2019) Altered neural odometry in the vertical dimension. Proc Nat Acad Sci USA 116(10):4631. https://doi.org/10.1073/pnas.1811867116
Chen Z, Lowry S, Jacobson A, Hasselmo ME, Milford M (2015) Bio-inspired homogeneous multi-scale place recognition. Neural Netw 72:48–61. https://doi.org/10.1016/j.neunet.2015.10.002
Cummins M, Newman P (2009) Highly scalable appearance-only SLAM. In: Robotics science and systems V
Erdem U, Hasselmo M (2014) A biologically inspired hierarchical goal directed navigation model. J Physiol Paris 108(1):28–37. https://doi.org/10.1016/j.jphysparis.2013.07.002
Furgale P, Barfoot TD (2010) Visual teach and repeat for long-range rover autonomy. J Field Robot 27(5):534–560. https://doi.org/10.1002/rob.20342
Geva-Sagiv M, Las L, Yovel Y, Ulanovsky N (2015) Spatial cognition in bats and rats: from sensory acquisition to multiscale maps and navigation. Nat Rev Neurosci 16(2):94–108. https://doi.org/10.1038/nrn3888
Google (2020) Google maps. https://www.google.com/maps. Accessed 21 Feb 2020
Grisetti G, Tipaldi GD, Stachniss C, Burgard W, Nardi D (2007) Fast and accurate SLAM with Rao–Blackwellized particle filters. Robot Auton Syst. https://doi.org/10.1016/j.robot.2006.06.007
Hafting T, Fyhn M, Molden S, Moser MB, Moser EI (2005) Microstructure of a spatial map in the entorhinal cortex. Nature 436(7052):801–806. https://doi.org/10.1038/nature03721
Hausler S, Jacobson A, Milford MJ (2019) Multi-process fusion: visual place recognition using multiple image processing methods. IEEE Robot Autom Lett. https://doi.org/10.1109/LRA.2019.2898427
Hausler S, Jacobson A, Milford M (2020) Filter early, match late: improving network-based visual place recognition. Institute of Electrical and Electronics Engineers (IEEE), pp 3268–3275 https://doi.org/10.1109/iros40897.2019.8967783. arXiv:1906.12176
Hébert M, Bulla J, Vivien D, Agin V (2017) Are distal and proximal visual cues equally important during spatial learning in mice? A pilot study of overshadowing in the spatial domain. Front Behav Neurosci 11:109. https://doi.org/10.3389/fnbeh.2017.00109
Høydal ØA, Skytøen ER, Andersson SO, Moser MB, Moser EI (2019) Object-vector coding in the medial entorhinal cortex. Nature 568(7752):400–404. https://doi.org/10.1038/s41586-019-1077-7
Hurly TA, Fox TA, Zwueste DM, Healy SD (2014) Wild hummingbirds rely on landmarks not geometry when learning an array of flowers. Anim Cognit 17(5):1157–1165. https://doi.org/10.1007/s10071-014-0748-x
Jacobson A, Scheirer W, Milford M (2017) Déjà vu: scalable place recognition using mutually supportive feature frequencies. In: IEEE international conference on intelligent robots and systems. https://doi.org/10.1109/IROS.2017.8206580
Jacobson A, Chen Z, Milford M (2018) Leveraging variable sensor spatial acuity with a homogeneous, multi-scale place recognition framework. Biol Cybern 112(3):209–225. https://doi.org/10.1007/s00422-017-0745-7
Krichmar JL, Seth AK, Nitz DA, Fleischer JG, Edelman GM (2005) Spatial navigation and causal analysis in a brain-based device modeling cortical-hippocampal interactions. Neuroinformatics. https://doi.org/10.1385/NI:3:3:197
Kuipers B (2007) An intellectual history of the spatial semantic hierarchy. In: Jefferies ME, Yeap WK (eds) Robotics and cognitive approaches to spatial mapping. Springer, Berlin, pp 243–264. https://doi.org/10.1007/978-3-540-75388-9_15
Lever C, Burton S, Jeewajee A, O’Keefe J, Burgess N (2009) Boundary vector cells in the subiculum of the hippocampal formation. J Neurosci 29(31):9771–9777. https://doi.org/10.1523/jneurosci.1319-09.2009
Lowry S, Sunderhauf N, Newman P, Leonard JJ, Cox D, Corke P, Milford MJ (2016) Visual place recognition: a survey. IEEE Trans Robot 32(1):1–19. https://doi.org/10.1109/TRO.2015.2496823. arXiv:1612.00593
Lu F, Milios E (1997) Robot pose estimation in unknown environments by matching 2d range scans. J Intell Robot Syst Theory Appl 18(3):249–275. https://doi.org/10.1023/A:1007957421070
Milford MJ, Wyeth GF (2008) Mapping a suburb with a single camera using a biologically inspired SLAM system. IEEE Trans Robot 24(5):1038–1053. https://doi.org/10.1109/TRO.2008.2004520
Milford M, Wyeth G (2010) Persistent navigation and mapping using a biologically inspired slam system. Int J Robot Res 29(9):1131–1153. https://doi.org/10.1177/0278364909340592
Milford MJ, Wyeth GF (2012) SeqSLAM: Visual route-based navigation for sunny summer days and stormy winter nights. In: Proceedings—IEEE international conference on robotics and automation. IEEE, pp 1643–1649. https://doi.org/10.1109/ICRA.2012.6224623
Milford M, Wyeth G, Prasser D (2004) RatSLAM: a hippocampal model for simultaneous localization and mapping. https://doi.org/10.1109/robot.2004.1307183
Moser EI (2011) A bat’s perspective on navigation. Proc Natl Acad Sci. https://doi.org/10.1073/pnas.1112843108
Mount J, Dawes L, Milford MJ (2019) Automatic coverage selection for surface-based visual localization. IEEE Robot Autom Lett 4(4):3900–3907. https://doi.org/10.1109/lra.2019.2928259
Mur-Artal R, Montiel JM, Tardos JD (2015) ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Trans Robot 31(5):1147–1163. https://doi.org/10.1109/TRO.2015.2463671. arXiv:1502.00956
Naseer T, Burgard W, Stachniss C (2018) Robust visual localization across seasons. IEEE Trans Robot 34(2):289–302. https://doi.org/10.1109/TRO.2017.2788045
Nearmaps (2019) Nearmaps: aerial maps, high resolution aerial imagery. https://www.nearmap.com/au/en. Accessed 01 Aug 2019
O’Keefe J, Conway D (1978) Hippocampal place units in the freely moving rat: why they fire where they fire. Exp Brain Res 31(4):573–590. https://doi.org/10.1007/BF00239813
Oliva A, Torralba A (2006) Building the gist of a scene: the role of global image features in recognition. Prog Brain Res 155B:23–36. https://doi.org/10.1016/S0079-6123(06)55002-2
Sargolini F, Fyhn M, Hafting T, McNaughton BL, Witter MP, Moser MB, Moser EI (2006) Conjunctive representation of position, direction, and velocity in entorhinal cortex. Science 312(5774):758–762. https://doi.org/10.1126/science.1125572
Schoenenberger P, O’Neill J, Csicsvari J (2016) Activity-dependent plasticity of hippocampal place maps. Nat Commun 7(1):11824. https://doi.org/10.1038/ncomms11824
Sherry DF, Grella SL, Guigueno MF, White DJ, Marrone DF (2017) Are there place cells in the avian hippocampus? Brain Behav Evol 90(1):73–80. https://doi.org/10.1159/000477085
Sreenivasan S, Fiete I (2011) Grid cells generate an analog error-correcting code for singularly precise neural computation. Nat Neurosci. https://doi.org/10.1038/nn.2901
Stemmler M, Mathis A, Herz AV (2015) Neuroscience: connecting multiple spatial scales to decode the population activity of grid cells. Sci Adv 1(11):1–12. https://doi.org/10.1126/science.1500816
Stensola H, Stensola T, Solstad T, FrØland K, Moser MB, Moser EI (2012) The entorhinal grid map is discretized—supplementary information 2. Nature 492(7427):72–78. https://doi.org/10.1038/nature11649
Sünderhauf N, Neubert P, Protzel P (2013) Are we there yet? challenging SeqSLAM on a 3000 km journey across all four seasons. In: Workshop on long-term automony at ICRA 2013. IEEE, pp 1–3. https://doi.org/10.1016/j.cell.2007.12.011. http://www.tu-chemnitz.de/etit/proaut/rsrc/openseqslam.pdf
Sünderhauf N, Shirazi S, Dayoub F, Upcroft B, Milford M (2015) On the performance of ConvNet features for place recognition. In: IEEE international conference on intelligent robots and systems, vol 2015-Decem. IEEE, pp 4297–4304. https://doi.org/10.1109/IROS.2015.7353986. arXiv:1501.04158
Taube JS, Muller RU, Ranck JB (1990) Head-direction cells recorded from the postsubiculum in freely moving rats. I. Description and quantitative analysis. J Neurosci 10(2):420–35. https://doi.org/10.1523/JNEUROSCI.10-02-00420.1990
Ulanovsky N (2015) Three-dimensional head-direction coding in the bat brain. Nature 517(7533):159–164. https://doi.org/10.1038/nature14031
Yu L, Jacobson A, Milford M (2017) Rhythmic representations: learning periodic patterns for scalable place recognition at a sub-linear storage cost. https://doi.org/10.1109/LRA.2018.2792144. arXiv:1712.07315
Yu F, Shang J, Hu Y, Milford M (2019) NeuroSLAM: a brain-inspired SLAM system for 3D environments. Biol Cybern. https://doi.org/10.1007/s00422-019-00806-9
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Jean-Marc Fellous.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This research has been supported by the QUT Centre for Robotics, U.S. Office of Naval Research MURI Grant N00014-19-1-2571 and ARC grants FT140101229 and CE140100016. The work of SH was supported by a Research Training Program Stipend.
This article is part of the special Issue entitled ‘Complex Spatial Navigation in Animals, Computational Models and Neuro-inspired Robots’.
Rights and permissions
About this article
Cite this article
Hausler, S., Chen, Z., Hasselmo, M.E. et al. Bio-inspired multi-scale fusion. Biol Cybern 114, 209–229 (2020). https://doi.org/10.1007/s00422-020-00831-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00422-020-00831-z