Abstract
Foveation can be defined as the organic action of directing the gaze towards a visual region of interest to acquire relevant information selectively. With the recent advent of event cameras, we believe that taking advantage of this visual neuroscience mechanism would greatly improve the efficiency of event data processing. Indeed, applying foveation to event data would allow to comprehend the visual scene while significantly reducing the amount of raw data to handle. In this respect, we demonstrate the stakes of neuromorphic foveation theoretically and empirically across several computer vision tasks, namely semantic segmentation and classification. We show that foveated event data have a significantly better trade-off between quantity and quality of the information conveyed than high- or low-resolution event data. Furthermore, this compromise extends even over fragmented datasets. Our code is publicly available online at: https://github.com/amygruel/FoveationStakes_DVS.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Code Availability
The code is publicly available online at: https://github.com/amygruel/FoveationStakes_DVS.
References
Albada S, Rowley A, Senk J, Hopkins M, Schmidt M, Stokes A, Lester D, Diesmann M, Furber S (2018) Performance comparison of the digital neuromorphic hardware spinnaker and the neural network simulation software nest for a full-scale cortical microcircuit model. Front Neurosci 12:291
Alonso I, Murillo A (2019) EV-SegNet: semantic segmentation for event-based Cameras. In: 2019 IEEE, CVF conference on computer vision and pattern recognition workshops (CVPRW)
Amir A, Taba B, Berg D, Melano, T, McKinstry J, Di Nolfo C, Nayak T, Andreopoulos A, Garreau G, Mendoza M et al (2017) A low power, fully event-based gesture recognition system. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7243–7252
Araujo H, Dias J (1997) An introduction to the log-polar mapping. In: Proceedings II workshop on cybernetic vision, vol 1, pp 139–144
Bear M et al (2007) The human eye. Neurosciences, exploring the brain. Wolters Kluwer, Philadelphia
Binas J, Neil D, Liu S-C, Delbruck T (2017) DDD17: end-to-end Davis driving dataset. arXiv:1711.01458 [cs]
Dampfhoffer M, Mesquida T, Valentian A, Anghel L (2022) Are SNNs really more energy-efficient than ANNs? An in-depth hardware-aware study. IEEE Trans Emerg Top Comput Intell 2022:1–11
D’Angelo G, Janotte E, Schoepe T, O’Keeffe J, Milde M, Chicca E, Bartolozzi C (2020) Event-based eccentric motion detection exploiting time difference encoding. Front Neurosci 14:451
Daucé E, Albiges P, Perrinet LU (2020) A dual foveal-peripheral visual processing model implements efficient saccade selection. J Vis 20(8):22–22
Daucé E, Perrinet L (2020) Visual search as active inference. In: Verbelen T, Lanillos P, Buckley CL, De Boom C (eds) Active inference, communications in computer and information science. Springer International Publishing, Berlin, pp 165–178
Davies M et al (2018) Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38(1):82–99
Davison AP, Brüderle D, Eppler J, Kremkow J, Muller E, Pecevski D, Perrinet L, Yger P (2009) PyNN: a common interface for neuronal network simulators. Front Neuroinform 2:388
Delbrück T, Graca R, Paluch M (2021) Feedback control of event cameras. CoRRarXiv:2105.00409
Finateu T, Niwa A, Matolin D, Tsuchimoto K, Mascheroni A, Reynaud E, Mostafalu P, Brady FT, Chotard L, Legoff F, Takahashi H, Wakabayashi H, Oike Y, Posch C (2020) 5.10 a 1280 \(\times \) 720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86 \(\mu \) m pixels, 1.066geps readout, programmable event-rate controller and compressive data-formatting pipeline. In: 2020 IEEE international solid- state circuits conference—(ISSCC), pp 112–114
Furber S, Bogdan P (2020) Spinnaker–a spiking neural network architecture. NOW Publishers INC, Hanover
Gehrig D, Scaramuzza D (2022) Are high-resolution cameras really needed?. arXiv
Geisler WS, Perry JS (1998) Real-time foveated multiresolution system for low-bandwidth video communication. In: Rogowitz BE, Pappas TN (eds) Human vision and electronic imaging III, vol 3299. International Society for Optics and Photonics, SPIE, pp 294–305
Gewaltig M-O, Diesmann M (2007) Nest (neural simulation tool)
Ghosh R, Gupta A, Silva AN, Soares A, Thakor NV (2019) Spatiotemporal filtering for event-based action recognition. CoRRarXiv:1903.07067
Grimaldi A, Boutin V, Ieng S-H, Benosman R, Perrinet L (2022) A robust event-driven approach to always-on object recognition
Gruel A, Hareb D, Martinet J, Linares-Barranco B, Serrano-Gotarredona T (2022a) Neuromorphic foveation applied to semantic segmentation. In: NeuroVision: what can computer vision learn from visual neuroscience? A CVPR 2022 Workshop, New Orleans, United States
Gruel A, Martinet J (2021) Bio-inspired visual attention for silicon retinas based on spiking neural networks applied to pattern classification. In: 2021 international conference on content-based multimedia indexing (CBMI)
Gruel A, Martinet J, Magno M (2023) Simultaneous neuromorphic selection of multiple salient objects for event vision. In: 2023 international joint conference on neural networks (IJCNN)
Gruel A, Martinet J, Serrano-Gotarredona T, Linares-Barranco B (2022b) Event data downscaling for embedded computer vision. In: Proceedings of the 17th international joint conference on computer vision, imaging and computer graphics theory and applications (VISAPP)
Gruel A, Vitale A, Martinet J, Magno M (2022c) Neuromorphic event-based spatio-temporal attention using adaptive mechanisms. In: 2022 IEEE 4th international conference on artificial intelligence circuits and systems (AICAS)
Guo M, Huang J, Chen S (2017) Live demonstration: a 768 \(\times \) 640 pixels 200meps dynamic vision sensor. In: 2017 IEEE international symposium on circuits and systems (ISCAS)
Hao Q, Tao Y, Cao J, Tang M, Cheng Y, Zhou D, Ning Y, Bao C, Cui H (2021) Retina-like imaging and its applications: a brief review. Appl Sci 11(15):7058
Kubendran R, Paul A, Cauwenberghs G (2021) A 256 \(\times \) 256 6.3pj/pixel-event query-driven dynamic vision sensor with energy-conserving row-parallel event scanning. In: 2021 IEEE custom integrated circuits conference (CICC), pp 1–2
Lagorce X, Orchard G, Galluppi F, Shi BE, Benosman RB (2016) Hots: a hierarchy of event-based time-surfaces for pattern recognition. IEEE Trans Pattern Anal Mach Intell 39(7):1346–1359
Land MF (2018) Eyes to see: the astonishing variety of vision in nature. Oxford University Press, Oxford
Li C, Longinotti L, Corradi F, Delbruck T (2019) A 132 by 104 10 micro m-pixel 250 micro w 1kefps dynamic vision sensor with pixel-parallel noise and spatial redundancy suppression. In: 2019 symposium on VLSI circuits, pp C216–C217
Lichtsteiner P, Posch C, Delbruck T (2008) A 128 \(\times \) 128 120 dB 15 \(\mu \)s latency asynchronous temporal contrast vision sensor. IEEE J Solid State Circuit 43(2):566–576
Maass W (1997) Networks of spiking neurons. Neural Netw 10(9):1659–1671
Markram H, Wang Y, Tsodyks M (1998) Differential signaling via the same axon of neocortical pyramidal neurons. Proc Natl Acad Sci USA 95:5323–5328
Martinet J, Lablack A, Lew S, Djeraba C (2009) Gaze based quality assessment of visual media understanding. In: IEEE Pacific-Rim symposium on image and video technology-CVIM’09
Paugam-Moisy H, Bohte SM (2012) Computing with spiking neuron networks. Handbook of natural computing. Springer-Verlag, Berlin
Pehle C, Pedersen JE (2021) Norse—a deep learning library for spiking neural networks. Documentation: https://norse.ai/docs/
Posch C, Serrano-Gotarredona T, Linares-Barranco B, Delbruck T (2014) Retinomorphic event-based vision sensors: bioinspired cameras with spiking output. Proc IEEE 102(10):1470–1484
Pramod R, Katti H, Arun S (2022) Human peripheral blur is optimal for object recognition. Vis Res 200:108083
Rizzo C, Schuman CD, Plank JS (2023) Neuromorphic downsampling of event-based camera output. In: Kudithipudi D, Frenkel C, Cardwell S, Aimone JB (eds) Neuro-inspired computational elements conference, NICE2023, San Antonio, TX, USA, 11–14 Apr, 2023. ACM, pp 26–34
Sarvaiya JN, Patnaik S, Bombaywala S (2009) Image registration using log-polar transform and phase correlation. In: IEEE region 10 annual international conference, proceedings/TENCON, pp 1–5
Serrano-Gotarredona T, Faramarzi F, Linares-Barranco B (2022) Electronically foveated dynamic vision sensor. In: 2022 IEEE international conference on omni-layer intelligent systems (COINS), pp 1–5
Suh Y, Choi S, Ito M, Kim J, Lee Y, Seo J, Jung H, Yeo D-H, Namgung S, Bong J, Yoo S, Shin S-H, Kwon D, Kang P, Kim S, Na H, Hwang K, Shin C, Kim J-S, Park P KJ, Kim J, Ryu H, Park Y (2020) A 1280 \(\times \) 960 dynamic vision sensor with a 4.95-micro m pixel pitch and motion artifact minimization. In: 2020 IEEE international symposium on circuits and systems (ISCAS), pp. 1–5
Traver VJ, Pla F (2003) Designing the lattice for log-polar images. Discrete geometry for computer imagery. Springer, Berlin, pp 164–173
Acknowledgements
This work was supported by the European Union’s ERA-NET CHIST-ERA 2018 research and innovation programme under grant agreement ANR-19-CHR3-0008. The authors are grateful to the OPAL infrastructure from Université Côte d’Azur for providing resources and support.
Funding
This work was supported by the European Union’s ERA-NET CHIST-ERA 2018 research and innovation programme under Grant Agreement ANR-19-CHR3-0008.
Author information
Authors and Affiliations
Contributions
The authors TS-G, JM, AG and BL-B contributed to the conceptualisation and methodology design of the study. The project coordination and administration were handled by AG. JM and Laurent Perrinet carried out the funding acquisition and supervision. Formal analysis and investigation were performed by AG, DH and AG. Results visualisation and presentation were realised by AG. The first draft of the manuscript was written by AG, DH and JM; AG, LP and TS-G added to a second draft by reviewing and editing. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors declares that they have no conflict of interest.
Consent for publication
Not applicable.
Ethical approval
Not applicable.
Additional information
Communicated by Benjamin Lindner
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is published as part of the Special Issue on “What can Computer Vision learn from Visual Neuroscience?".
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Gruel, A., Hareb, D., Grimaldi, A. et al. Stakes of neuromorphic foveation: a promising future for embedded event cameras. Biol Cybern 117, 389–406 (2023). https://doi.org/10.1007/s00422-023-00974-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00422-023-00974-9