Skip to main content

Exploring Lottery Ticket Hypothesis in Spiking Neural Networks

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Abstract

Spiking Neural Networks (SNNs) have recently emerged as a new generation of low-power deep neural networks, which is suitable to be implemented on low-power mobile/edge devices. As such devices have limited memory storage, neural pruning on SNNs has been widely explored in recent years. Most existing SNN pruning works focus on shallow SNNs (2–6 layers), however, deeper SNNs (\(\ge \)16 layers) are proposed by state-of-the-art SNN works, which is difficult to be compatible with the current SNN pruning work. To scale up a pruning technique towards deep SNNs, we investigate Lottery Ticket Hypothesis (LTH) which states that dense networks contain smaller subnetworks (i.e., winning tickets) that achieve comparable performance to the dense networks. Our studies on LTH reveal that the winning tickets consistently exist in deep SNNs across various datasets and architectures, providing up to \(97\%\) sparsity without huge performance degradation. However, the iterative searching process of LTH brings a huge training computational cost when combined with the multiple timesteps of SNNs. To alleviate such heavy searching cost, we propose Early-Time (ET) ticket where we find the important weight connectivity from a smaller number of timesteps. The proposed ET ticket can be seamlessly combined with a common pruning techniques for finding winning tickets, such as Iterative Magnitude Pruning (IMP) and Early-Bird (EB) tickets. Our experiment results show that the proposed ET ticket reduces search time by up to \(38\%\) compared to IMP or EB methods. Code is available at Github.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://github.com/Yanqi-Chen/Gradient-Rewiring

References

  1. Akopyan, F., et al.: TrueNorth: design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 34(10), 1537–1557 (2015)

    Article  Google Scholar 

  2. Bai, Y., Wang, H., Tao, Z., Li, K., Fu, Y.: Dual lottery ticket hypothesis. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=fOsN52jn25l

  3. Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., Maass, W.: Long short-term memory and learning-to-learn in networks of spiking neurons. In: Advances in Neural Information Processing Systems 31 (2018)

    Google Scholar 

  4. Brix, C., Bahar, P., Ney, H.: Successfully applying the stabilized lottery ticket hypothesis to the transformer architecture. arXiv preprint arXiv:2005.03454 (2020)

  5. Burkholz, R., Laha, N., Mukherjee, R., Gotovos, A.: On the existence of universal lottery tickets. arXiv preprint arXiv:2111.11146 (2021)

  6. Chen, G.K., Kumar, R., Sumbul, H.E., Knag, P.C., Krishnamurthy, R.K.: A 4096-neuron 1M-synapse 3.8-pJ/SOP spiking neural network with on-chip STDP learning and sparse weights in 10-nm finfet CMOS. IEEE J. Solid-State Circuits 54(4), 992–1002 (2018)

    Google Scholar 

  7. Chen, T., et al.: The lottery ticket hypothesis for pre-trained BERT networks. In: Advances in Neural Information Processing Systems, vol. 33, pp. 15834–15846 (2020)

    Google Scholar 

  8. Chen, T., et al.: Sparsity winning twice: better robust generalization from more efficient training. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=SYuJXrXq8tw

  9. Chen, Y., Yu, Z., Fang, W., Huang, T., Tian, Y.: Pruning of deep spiking neural networks through gradient rewiring. arXiv preprint arXiv:2105.04916 (2021)

  10. Chen, Y.H., Emer, J., Sze, V.: Eyeriss: a spatial architecture for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH Comput. Archit. News 44(3), 367–379 (2016)

    Article  Google Scholar 

  11. Christensen, D.V., et al.: 2022 roadmap on neuromorphic computing and engineering. Neuromorphic Computing and Engineering (2022)

    Google Scholar 

  12. Comsa, I.M., Fischbacher, T., Potempa, K., Gesmundo, A., Versari, L., Alakuijala, J.: Temporal coding in spiking neural networks with alpha synaptic function. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8529–8533. IEEE (2020)

    Google Scholar 

  13. Davies, M., et al.: Loihi: a neuromorphic manycore processor with on-chip learning. IEEE Micro 38(1), 82–99 (2018)

    Article  Google Scholar 

  14. Deng, L., et al.: Comprehensive SNN compression using ADMM optimization and activity regularization. IEEE Trans. Neural Networks and Learn. Syst. (2021)

    Google Scholar 

  15. Deng, S., Li, Y., Zhang, S., Gu, S.: Temporal efficient training of spiking neural network via gradient re-weighting. In: International Conference on Learning Representations (2022., https://openreview.net/forum?id=_XNtisL32jv

  16. Desai, S., Zhan, H., Aly, A.: Evaluating lottery tickets under distributional shifts. arXiv preprint arXiv:1910.12708 (2019)

  17. Ding, J., Yu, Z., Tian, Y., Huang, T.: Optimal ann-snn conversion for fast and accurate inference in deep spiking neural networks. arXiv preprint arXiv:2105.11654 (2021)

  18. Ding, S., Chen, T., Wang, Z.: Audio lottery: speech recognition made ultra-lightweight, noise-robust, and transferable. In: International Conference on Learning Representations (2022). https://openreview.net/forum?id=9Nk6AJkVYB

  19. Fang, W., et al.: Spikingjelly (2020). https://github.com/fangwei123456/spikingjelly

  20. Fang, W., Yu, Z., Chen, Y., Huang, T., Masquelier, T., Tian, Y.: Deep residual learning in spiking neural networks. Advances in Neural Information Processing Systems 34 (2021)

    Google Scholar 

  21. Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635 (2018)

  22. Frankle, J., Dziugaite, G.K., Roy, D.M., Carbin, M.: Stabilizing the lottery ticket hypothesis. arXiv preprint arXiv:1903.01611 (2019)

  23. Furber, S.B., Galluppi, F., Temple, S., Plana, L.A.: The spinnaker project. Proc. IEEE 102(5), 652–665 (2014)

    Article  Google Scholar 

  24. Girish, S., Maiya, S.R., Gupta, K., Chen, H., Davis, L.S., Shrivastava, A.: The lottery ticket hypothesis for object recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 762–771 (2021)

    Google Scholar 

  25. Guo, W., Fouda, M.E., Yantir, H.E., Eltawil, A.M., Salama, K.N.: Unsupervised adaptive weight pruning for energy-efficient neuromorphic systems. Frontiers in Neuroscience p. 1189 (2020)

    Google Scholar 

  26. Han, S., et al.: Dsd: Dense-sparse-dense training for deep neural networks. arXiv preprint arXiv:1607.04381 (2016)

  27. Han, S., Pool, J., Tran, J., Dally, W.: Learning both weights and connections for efficient neural network. Advances in neural information processing systems 28 (2015)

    Google Scholar 

  28. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

    Google Scholar 

  29. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015)

  30. Izhikevich, E.M.: Simple model of spiking neurons. IEEE Trans. Neural Networks 14(6), 1569–1572 (2003)

    Article  MathSciNet  Google Scholar 

  31. Kalibhat, N.M., Balaji, Y., Feizi, S.: Winning lottery tickets in deep generative models. arXiv preprint arXiv:2010.02350 (2020)

  32. Kim, Y., Li, Y., Park, H., Venkatesha, Y., Panda, P.: Neural architecture search for spiking neural networks. arXiv preprint arXiv:2201.10355 (2022)

  33. Kim, Y., Panda, P.: Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, p. 1638 (2020)

    Google Scholar 

  34. Kim, Y., Panda, P.: Visual explanations from spiking neural networks using interspike intervals. Sci. Rep. 11, 19037 (2021). https://doi.org/10.1038/s41598-021-98448-0 (2021)

  35. Kim, Y., Venkatesha, Y., Panda, P.: Privatesnn: privacy-preserving spiking neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 1192–1200 (2022)

    Google Scholar 

  36. Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

    Google Scholar 

  37. Kundu, S., Pedram, M., Beerel, P.A.: Hire-snn: harnessing the inherent robustness of energy-efficient deep spiking neural networks by training with crafted input noise. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5209–5218 (2021)

    Google Scholar 

  38. Ledinauskas, E., Ruseckas, J., Juršėnas, A., Buračas, G.: Training deep spiking neural networks. arXiv preprint arXiv:2006.04436 (2020)

  39. Lee, C., Sarwar, S.S., Panda, P., Srinivasan, G., Roy, K.: Enabling spike-based backpropagation for training deep neural network architectures. Front. Neurosci. 14 (2020)

    Google Scholar 

  40. Lee, J.H., Delbruck, T., Pfeiffer, M.: Training deep spiking neural networks using backpropagation. Front. Neurosci. 10, 508 (2016)

    Article  Google Scholar 

  41. Lee, N., Ajanthan, T., Torr, P.H.: Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340 (2018)

  42. Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2016)

  43. Li, Y., Deng, S., Dong, X., Gong, R., Gu, S.: A free lunch from ann: towards efficient, accurate spiking neural networks calibration. arXiv preprint arXiv:2106.06984 (2021)

  44. Li, Y., Deng, S., Dong, X., Gu, S.: Converting artificial neural networks to spiking neural networks via parameter calibration. arXiv preprint arXiv:2205.10121 (2022)

  45. Li, Y., Guo, Y., Zhang, S., Deng, S., Hai, Y., Gu, S.: Differentiable spike: Rethinking gradient-descent for training spiking neural networks. Advances in Neural Information Processing Systems 34 (2021)

    Google Scholar 

  46. Liu, S., et al.: Deep ensembling with no overhead for either training or testing: the all-round blessings of dynamic sparsity. arXiv preprint arXiv:2106.14568 (2021)

  47. Liu, Z., Sun, M., Zhou, T., Huang, G., Darrell, T.: Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270 (2018)

  48. Loshchilov, I., Hutter, F.: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)

  49. Martinelli, F., Dellaferrera, G., Mainar, P., Cernak, M.: Spiking neural networks trained with backpropagation for low power neuromorphic implementation of voice activity detection. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8544–8548. IEEE (2020)

    Google Scholar 

  50. Mehta, R.: Sparse transfer learning via winning lottery tickets. arXiv preprint arXiv:1905.07785 (2019)

  51. Morcos, A., Yu, H., Paganini, M., Tian, Y.: One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. In: Advances in Neural Information Processing Systems 32 (2019)

    Google Scholar 

  52. Mostafa, H.: Supervised learning based on temporal coding in spiking neural networks. IEEE Trans. Neural Networks Learn. Syst. 29(7), 3227–3235 (2017)

    Google Scholar 

  53. Movva, R., Zhao, J.Y.: Dissecting lottery ticket transformers: structural and behavioral study of sparse neural machine translation. arXiv preprint arXiv:2009.13270 (2020)

  54. Neftci, E.O., Mostafa, H., Zenke, F.: Surrogate gradient learning in spiking neural networks. IEEE Signal Process. Mag. 36, 61–63 (2019)

    Article  Google Scholar 

  55. Neftci, E.O., Pedroni, B.U., Joshi, S., Al-Shedivat, M., Cauwenberghs, G.: Stochastic synapses enable efficient brain-inspired learning machines. Front. Neurosci. 10, 241 (2016)

    Article  Google Scholar 

  56. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning (2011)

    Google Scholar 

  57. Orchard, G., et al.: Efficient neuromorphic signal processing with loihi 2. In: 2021 IEEE Workshop on Signal Processing Systems (SiPS), pp. 254–259. IEEE (2021)

    Google Scholar 

  58. Parashar, A., Rhu, M., Mukkara, A., Puglielli, A., Venkatesan, R., Khailany, B., Emer, J., Keckler, S.W., Dally, W.J.: Scnn: an accelerator for compressed-sparse convolutional neural networks. ACM SIGARCH Computer Architecture News 45(2), 27–40 (2017)

    Article  Google Scholar 

  59. Paszke, A., et al.: Automatic differentiation in pytorch. In: NIPS-W (2017)

    Google Scholar 

  60. Rathi, N., Panda, P., Roy, K.: Stdp-based pruning of connections and weight quantization in spiking neural networks for energy-efficient recognition. IEEE Trans. Comput. Aided Des. Integr. Circuits Syst. 38(4), 668–677 (2018)

    Article  Google Scholar 

  61. Rathi, N., Roy, K.: Diet-SNN: a low-latency spiking neural network with direct input encoding and leakage and threshold optimization. IEEE Trans. Neural Networks Learn Syst. (2021)

    Google Scholar 

  62. Rathi, N., Srinivasan, G., Panda, P., Roy, K.: Enabling deep spiking neural networks with hybrid conversion and spike timing dependent backpropagation. arXiv preprint arXiv:2005.01807 (2020)

  63. Roy, K., Jaiswal, A., Panda, P.: Towards spike-based machine intelligence with neuromorphic computing. Nature 575(7784), 607–617 (2019)

    Article  Google Scholar 

  64. Schuman, C.D., Kulkarni, S.R., Parsa, M., Mitchell, J.P., Kay, B., et al.: Opportunities for neuromorphic computing algorithms and applications. Nature Comput. Sci. 2(1), 10–19 (2022)

    Article  Google Scholar 

  65. Shi, Y., Nguyen, L., Oh, S., Liu, X., Kuzum, D.: A soft-pruning method applied during training of spiking neural networks for in-memory computing applications. Front. Neurosci. 13, 405 (2019)

    Article  Google Scholar 

  66. Shrestha, S.B., Orchard, G.: Slayer: Spike layer error reassignment in time. arXiv preprint arXiv:1810.08646 (2018)

  67. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ICLR (2015)

    Google Scholar 

  68. Venkatesha, Y., Kim, Y., Tassiulas, L., Panda, P.: Federated learning with spiking neural networks. arXiv preprint arXiv:2106.06579 (2021)

  69. Vischer, M.A., Lange, R.T., Sprekeler, H.: On lottery tickets and minimal task representations in deep reinforcement learning. arXiv preprint arXiv:2105.01648 (2021)

  70. Wang, C., Zhang, G., Grosse, R.: Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376 (2020)

  71. Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. In: Advances in Neural Information Processing Systems 29 (2016)

    Google Scholar 

  72. Wu, H., et al.: Training spiking neural networks with accumulated spiking flow. ijo 1(1) (2021)

    Google Scholar 

  73. Wu, J., Xu, C., Zhou, D., Li, H., Tan, K.C.: Progressive tandem learning for pattern recognition with deep spiking neural networks. arXiv preprint arXiv:2007.01204 (2020)

  74. Wu, Y., Deng, L., Li, G., Zhu, J., Shi, L.: Spatio-temporal backpropagation for training high-performance spiking neural networks. Front. Neurosci. 12, 331 (2018)

    Article  Google Scholar 

  75. Wu, Y., Deng, L., Li, G., Zhu, J., Xie, Y., Shi, L.: Direct training for spiking neural networks: Faster, larger, better. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 1311–1318 (2019)

    Google Scholar 

  76. Wu, Y., Zhao, R., Zhu, J., Chen, F., Xu, M., Li, G., Song, S., Deng, L., Wang, G., Zheng, H., et al.: Brain-inspired global-local learning incorporated with neuromorphic computing. Nat. Commun. 13(1), 1–14 (2022)

    Google Scholar 

  77. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

  78. Yao, M., Gao, H., Zhao, G., Wang, D., Lin, Y., Yang, Z., Li, G.: Temporal-wise attention spiking neural networks for event streams classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10221–10230 (2021)

    Google Scholar 

  79. Yin, R., Moitra, A., Bhattacharjee, A., Kim, Y., Panda, P.: Sata: Sparsity-aware training accelerator for spiking neural networks. arXiv preprint arXiv:2204.05422 (2022)

  80. You, H., et al.: Drawing early-bird tickets: Towards more efficient training of deep networks. arXiv preprint arXiv:1909.11957 (2019)

  81. Yu, H., Edunov, S., Tian, Y., Morcos, A.S.: Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. arXiv preprint arXiv:1906.02768 (2019)

  82. Zhang, Z., Chen, X., Chen, T., Wang, Z.: Efficient lottery ticket finding: less data is more. In: International Conference on Machine Learning, pp. 12380–12390. PMLR (2021)

    Google Scholar 

  83. Zheng, H., Wu, Y., Deng, L., Hu, Y., Li, G.: Going deeper with directly-trained larger spiking neural networks. arXiv preprint arXiv:2011.05280 (2020)

  84. Zhou, H., Lan, J., Liu, R., Yosinski, J.: Deconstructing lottery tickets: Zeros, signs, and the supermask. In: Advances in Neural Information Processing Systems 32 (2019)

    Google Scholar 

Download references

Acknowledgment

We would like to thank Anna Hambitzer for her helpful comments. This work was supported in part by C-BRIC, a JUMP center sponsored by DARPA and SRC, Google Research Scholar Award, the National Science Foundation (Grant#1947826), TII (Abu Dhabi) and the DARPA AI Exploration (AIE) program.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Youngeun Kim .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 286 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kim, Y., Li, Y., Park, H., Venkatesha, Y., Yin, R., Panda, P. (2022). Exploring Lottery Ticket Hypothesis in Spiking Neural Networks. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13672. Springer, Cham. https://doi.org/10.1007/978-3-031-19775-8_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19775-8_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19774-1

  • Online ISBN: 978-3-031-19775-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics