Skip to main content
SpringerLink
Log in

Convolutional Spiking Neural Networks for Spatio-Temporal Feature Extraction

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Spiking neural networks (SNNs) can be used in low-power and embedded systems e.g. neuromorphic chips due to their event-based nature. They preserve conventional artificial neural networks (ANNs) properties with lower computation and memory costs. The temporal coding in layers of convolutional SNNs has not yet been studied. In this paper, we exploit the spatio-temporal feature extraction property of convolutional SNNs. Based on our analysis, we have shown that the shallow convolutional SNN outperforms spatio-temporal feature extractor methods such as C3D, ConvLstm, and cascaded Conv and LSTM. Furthermore, we present a new deep spiking architecture to tackle real-world classification and activity recognition tasks. This model is trained with our proposed hybrid training method. The proposed architecture achieved superior performance compared to other SNN methods on NMNIST (99.6%), DVS-CIFAR10 (69.2%), and DVS-Gesture (96.7%). Also, it achieves comparable results compared to ANN methods on UCF-101 (42.1%) and HMDB-51 (21.5%) datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Availability of Data and Materials

All of the data and materials are available under https://github.com/aa-samad/conv_snn.

Code Availability

Code is available under https://github.com/aa-samad/conv_snn.

References

  1. Sharmin S et al (2019) A comprehensive analysis on adversarial robustness of spiking neural networks. IEEE, pp 1–8

  2. Kim H, Leutenegger S, Davison AJ (2016) Real-time 3d reconstruction and 6-dof tracking with an event camera. Springer, pp 349–364

  3. Kueng B, Mueggler E, Gallego G, Scaramuzza D (2016) Low-latency visual odometry using event-based feature tracks. IEEE, pp 16–23

  4. Rebecq H, Horstschäfer T, Gallego G, Scaramuzza D (2016) Evo: a geometric approach to event-based 6-dof parallel tracking and mapping in real time. IEEE Robot Autom Lett 2(2):593–600

    Article  Google Scholar 

  5. Gallego G et al (2017) Event-based, 6-dof camera tracking from photometric depth maps. IEEE Trans Pattern Anal Mach Intell 40(10):2402–2412

    Article  MathSciNet  Google Scholar 

  6. Mueggler E, Huber B, Scaramuzza D (2014) Event-based, 6-dof pose tracking for high-speed maneuvers. IEEE, pp 2761–2768

  7. Wang W et al (2020) Temporal pulses driven spiking neural network for fast object recognition in autonomous driving. arXiv preprint arXiv:2001.09220

  8. Zhang J et al (2022) Spiking transformers for event-based single object tracking, pp 8801–8810

  9. Zhu L et al (2022) Event-based video reconstruction via potential-assisted spiking neural network, pp 3594–3604

  10. Allen JN, Abdel-Aty-Zohdy HS, Ewing RL (2009) Cognitive processing using spiking neural networks. IEEE, pp 56–64

  11. Kasabov N, Capecci E (2015) Spiking neural network methodology for modelling, classification and understanding of eeg spatio-temporal data measuring cognitive processes. Inf Sci 294:565–575

    Article  MathSciNet  Google Scholar 

  12. Zhang X, Xu Z, Henriquez C, Ferrari S (2013) Spike-based indirect training of a spiking neural network-controlled virtual insect. IEEE, pp 6798–6805

  13. Mostafa H (2017) Supervised learning based on temporal coding in spiking neural networks. IEEE Trans Neural Netw Learn Syst 29(7):3227–3235

    Google Scholar 

  14. Wu Y, Deng L, Li G, Zhu J, Shi L (2018) Spatio-temporal backpropagation for training high-performance spiking neural networks. Front Neurosci 12:331

    Article  Google Scholar 

  15. Wu Y et al (2019) Direct training for spiking neural networks: faster, larger, better, vol 33, pp 1311–1318

  16. Hao Y, Huang X, Dong M, Xu B (2020) A biologically plausible supervised learning method for spiking neural networks using the symmetric stdp rule. Neural Netw 121:387–395

    Article  Google Scholar 

  17. Kheradpisheh SR, Ganjtabesh M, Thorpe SJ, Masquelier T (2018) Stdp-based spiking deep convolutional neural networks for object recognition. Neural Netw 99:56–67

    Article  Google Scholar 

  18. Lee C, Panda P, Srinivasan G, Roy K (2018) Training deep spiking convolutional neural networks with stdp-based unsupervised pre-training followed by supervised fine-tuning. Front Neurosci 12:435

    Article  Google Scholar 

  19. Tavanaei A, Maida A (2019) Bp-stdp: approximating backpropagation using spike timing dependent plasticity. Neurocomputing 330:39–47

    Article  Google Scholar 

  20. Cao Y, Chen Y, Khosla D (2015) Spiking deep convolutional neural networks for energy-efficient object recognition. Int J Comput Vis 113(1):54–66

    Article  MathSciNet  Google Scholar 

  21. Diehl PU et al (2015) Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing. IEEE, pp 1–8

  22. Esser SK, Appuswamy R, Merolla P, Arthur JV, Modha DS (2015) Backpropagation for energy-efficient neuromorphic computing, pp 1117–1125

  23. Han B, Srinivasan G, Roy K (2020) Rmp-snn: residual membrane potential neuron for enabling deeper high-accuracy and low-latency spiking neural network, pp 13558–13567

  24. Rueckauer B, Lungu I-A, Hu Y, Pfeiffer M, Liu S-C (2017) Conversion of continuous-valued deep networks to efficient event-driven networks for image classification. Front Neurosci 11:682

    Article  Google Scholar 

  25. Sengupta A, Ye Y, Wang R, Liu C, Roy K (2019) Going deeper in spiking neural networks: Vgg and residual architectures. Front Neurosci 13:95

    Article  Google Scholar 

  26. Stromatias E, Soto M, Serrano-Gotarredona T, Linares-Barranco B (2017) An event-driven classifier for spiking neural networks fed with synthetic or dynamic vision sensor data. Front Neurosci 11:350

    Article  Google Scholar 

  27. Deng L et al (2020) Rethinking the performance comparison between snns and anns. Neural Netw 121:294–307

    Article  Google Scholar 

  28. He W et al (2020) Comparing snns and rnns on neuromorphic vision datasets: similarities and differences. arXiv preprint arXiv:2005.02183

  29. Wu J, Chua Y, Zhang M, Li H, Tan KC (2018) A spiking neural network framework for robust sound classification. Front Neurosci 12:836

    Article  Google Scholar 

  30. Neftci EO, Mostafa H, Zenke F (2019) Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks. IEEE Signal Process Mag 36(6):51–63

    Article  Google Scholar 

  31. Selvaraju RR et al (2017) Grad-cam: visual explanations from deep networks via gradient-based localization, pp 618–626

  32. Deng S, Li Y, Zhang S, Gu S (2022) Temporal efficient training of spiking neural network via gradient re-weighting. arXiv preprint arXiv:2202.11946

  33. Guo Y et al (2022) Recdis-snn: rectifying membrane potential distribution for directly training spiking neural networks, pp 326–335

  34. Na B et al (2022) Autosnn: towards energy-efficient spiking neural networks. PMLR, pp 16253–16269

  35. Orchard G, Jayawant A, Cohen GK, Thakor N (2015) Converting static image datasets to spiking neuromorphic datasets using saccades. Front Neurosci 9:437

    Article  Google Scholar 

  36. Amir A et al (2017) A low power, fully event-based gesture recognition system, pp 7243–7252

  37. Li H, Liu H, Ji X, Li G, Shi L (2017) Cifar10-dvs: an event-stream dataset for object classification. Front Neurosci 11:309

    Article  Google Scholar 

  38. Soomro K, Zamir AR, Shah M (2012) Ucf101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402

  39. Kuehne H, Jhuang H, Garrote E, Poggio T, Serre T (2011) Hmdb: a large video database for human motion recognition. IEEE, pp 2556–2563

  40. Kim J, Bae J, Park G, Zhang D, Kim YM (2021) N-imagenet: towards robust, fine-grained object recognition with event cameras, pp 2146–2156

  41. Fang W (2020) Leaky integrate-and-fire spiking neuron with learnable membrane time parameter. arXiv preprint arXiv:2007.05785

  42. Tavanaei A, Ghodrati M, Kheradpisheh SR, Masquelier T, Maida A (2019) Deep learning in spiking neural networks. Neural Netw 111:47–63

    Article  Google Scholar 

  43. Sainath TN, Vinyals O, Senior A, Sak H (2015) Convolutional, long short-term memory, fully connected deep neural networks. IEEE, pp 4580–4584

  44. Tran D, Bourdev L, Fergus R, Torresani L, Paluri M (2015) Learning spatiotemporal features with 3d convolutional networks, pp 4489–4497

  45. Xingjian S et al (2015) Convolutional lstm network: a machine learning approach for precipitation nowcasting, pp 802–810

  46. Srivastava N, Mansimov E, Salakhudinov R (2015) Unsupervised learning of video representations using lstms, pp 843–852

  47. Liu Q, Ruan H, Xing D, Tang H, Pan G (2020) Effective aer object classification using segmented probability-maximization learning in spiking neural networks, pp 1308–1315

  48. Patino-Saucedo A, Rostro-Gonzalez H, Serrano-Gotarredona T, Linares-Barranco B (2020) Event-driven implementation of deep spiking convolutional neural networks for supervised classification using the spinnaker neuromorphic platform. Neural Netw 121:319–328

    Article  Google Scholar 

  49. Wang Q, Zhang Y, Yuan J, Lu Y (2019) Space-time event clouds for gesture recognition: from rgb cameras to event cameras. IEEE, pp 1826–1835

  50. Sironi A, Brambilla M, Bourdis N, Lagorce X, Benosman R (2018) Hats: histograms of averaged time surfaces for robust event-based object classification, pp 1731–1740

  51. Hara K, Kataoka H, Satoh Y (2018) Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?, pp 6546–6555

Download references

Funding

Not available.

Author information

Authors and Affiliations

  1. Computer Engineering Department, Amirkabir University of Technology, Tehran, Iran

    Ali Samadzadeh, Fatemeh Sadat Tabatabaei Far, Ali Javadi & Ahmad Nickabadi

  2. Department of Computer Science and Engineering, Chalmers University of Technology, Göteburg, Sweden

    Morteza Haghir Chehreghani

Authors
  1. Ali Samadzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fatemeh Sadat Tabatabaei Far
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ali Javadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ahmad Nickabadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Morteza Haghir Chehreghani
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmad Nickabadi.

Ethics declarations

Conflict of interest

Not available.

Consent to Participate

Not applicable.

Consent for Publication

Not applicable.

Ethics Approval

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Samadzadeh, A., Far, F.S.T., Javadi, A. et al. Convolutional Spiking Neural Networks for Spatio-Temporal Feature Extraction. Neural Process Lett 55, 6979–6995 (2023). https://doi.org/10.1007/s11063-023-11247-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11247-8

Keywords

Search

Navigation