Energy Efficient Spiking Neural Network Design with RRAM Devices

  • Yu Wang
  • Tianqi Tang
  • Boxun Li
  • Lixue Xia
  • Huazhong Yang
Chapter
First Online:

Abstract

Inspired by the human brain’s function and efficiency, neuromorphic computing offers a promising solution for a wide set of cognitive tasks, ranging from brain machine interfaces to real-time classification. The spiking neural network (SNN), which encodes and processes information with bionic spikes, is an emerging neuromorphic model with great potential to drastically promote the energy efficiency of computing systems. However, an energy efficient hardware implementation and the difficulty of training the model significantly limit the application of the spiking neural network. In this chapter, we first introduce the background knowledge of SNN and metal-oxide resistive switching random-access memory (RRAM). Then, we compare different training algorithms of SNN for real-world applications, and demonstrate that the Neural Sampling method is much more effective than other methods. We also explore the performance and energy efficiency by building the SNN-based energy efficient system for real-time classification with RRAM devices. We implement different training algorithms of SNN, including Spiking Time Dependent Plasticity (STDP) and Neural Sampling method. Our RRAM-based SNN systems for these two training algorithms show good power efficiency and recognition performance on real-time classification tasks, e.g., the MNIST digit recognition. Finally, we discuss a possible direction to further improve the classification accuracy by boosting multiple SNNs.

This is a preview of subscription content, log in to check access.

Notes

Acknowledgements

This work was supported by 973 project 2013CB329000, National Science and Technology Major Project (2011ZX03003-003-01, 2013ZX03003013-003) and National Natural Science Foundation of China (Nos. 61373026, 61261160501, 61271269), and Tsinghua University Initiative Scientific Research Program. And we gratefully acknowledge the support from Prof. Shimeng Yu with the help of RRAM model.

References

  1. 1.
    L. Chang, Y.-K. Choi, D. Ha, P. Ranade, S. Xiong, J. Bokor, C. Hu, T.-J. King, Extremely scaled silicon nano-CMOS devices. Proc. IEEE 91 (11), 1860–1873 (2003)CrossRefGoogle Scholar
  2. 2.
    W.A. Wulf, S.A. McKee, Hitting the memory wall: implications of the obvious. ACM SIGARCH Comput. Arch. News 23 (1), 20–24 (1995)CrossRefGoogle Scholar
  3. 3.
    W. Maass, Networks of spiking neurons: the third generation of neural network models. Neural Netw. 10 (9), 1659–1671 (1997)CrossRefGoogle Scholar
  4. 4.
    T. Masquelier, S.J. Thorpe, Unsupervised learning of visual features through spike timing dependent plasticity. PLoS Comput. Biol. 3 (2), e31 (2007)Google Scholar
  5. 5.
    D. Querlioz, W. Zhao, P. Dollfus, J. Klein, O. Bichler, C. Gamrat, Bioinspired networks with nanoscale memristive devices that combine the unsupervised and supervised learning approaches, in 2012 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH) (IEEE, 2012), pp. 203–210Google Scholar
  6. 6.
    P.A. Merolla, J.V. Arthur, R. Alvarezicaza, A.S. Cassidy, J. Sawada, F. Akopyan, B.L. Jackson, N. Imam, C. Guo, Y. Nakamura et al., A million spiking-neuron integrated circuit with a scalable communication network and interface, Science 345 (6197), 668–673 (2014)CrossRefGoogle Scholar
  7. 7.
    B. Govoreanu, G.S. Kar, Y. Chen, V. Paraschiv, 10×10 nm2 HF/HFOx crossbar resistive ram with excellent performance, reliability and low-energy operation, Electron Devices Meeting IEDM Technical Digest. International (2011), pp. 31.6.1–31.6.4Google Scholar
  8. 8.
    B. Li, Y. Shan, M. Hu, Y. Wang, Memristor-based approximated computation, in IEEE International Symposium on Low Power Electronics and Design (2013), pp. 242–247Google Scholar
  9. 9.
    T. Tang, R. Luo, B. Li, H. Li, Energy efficient spiking neural network design with RRAM devices, in International Symposium on Integrated Circuits (2015), pp. 268–271Google Scholar
  10. 10.
    T. Tang, L. Xia, B. Li, R. Luo, Spiking neural network with RRAM: can we use it for real-world application? in Design, Automation and Test in Europe (2015), pp. 860–865Google Scholar
  11. 11.
    R. Wang, T.J. Hamilton, J. Tapson, A. Van Schaik, An FPGA design framework for large-scale spiking neural networks, in Proceedings - IEEE International Symposium on Circuits and Systems (2014), pp. 457–460Google Scholar
  12. 12.
    E. Painkras, L.A. Plana, J. Garside, S. Temple, Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation. IEEE J. Solid-State Circ. 48 (8), 1943–1953 (2013)CrossRefGoogle Scholar
  13. 13.
    S.K. Esser, A. Andreopoulos, R. Appuswamy, P. Datta, D. Barch, A. Amir, J. Arthur, A. Cassidy, M. Flickner, P. Merolla, Cognitive computing systems: algorithms and applications for networks of neurosynaptic cores, in The 2013 International Joint Conference on Neural Networks (IJCNN) (2013), pp. 1–10Google Scholar
  14. 14.
    D. Kuzum, R.G. Jeyasingh, B. Lee, H.-S.P. Wong, Nanoelectronic programmable synapses based on phase change materials for brain-inspired computing. Nano Letters 12 (5), 2179–2186 (2011)CrossRefGoogle Scholar
  15. 15.
    Y. Lecun, C. Cortes, The MNIST database of handwritten digits (1998)Google Scholar
  16. 16.
    E.M. Izhikevich, Which model to use for cortical spiking neurons? IEEE Trans. Neural Netw. 15 (5), 1063–1070 (2004)CrossRefGoogle Scholar
  17. 17.
    G. Indiveri, A low-power adaptive integrate-and-fire neuron circuit, in ISCAS (4) (2003), pp. 820–823Google Scholar
  18. 18.
    S. Yu, B. Gao, Z. Fang, H. Yu, J. Kang, H.-S.P. Wong, Stochastic learning in oxide binary synaptic device for neuromorphic computing. Front. Neurosci. 7,186 (2013)CrossRefGoogle Scholar
  19. 19.
    M. Hu, H. Li, Q. Wu, G.S. Rose, Hardware realization of BSB recall function using memristor crossbar arrays, in Proceedings of the 49th Annual Design Automation Conference (ACM, 2012), pp. 498–503Google Scholar
  20. 20.
    P. Gu, B. Li, T. Tang, S. Yu, Y. Wang, H. Yang, Technological exploration of rram crossbar array for matrix-vector multiplication, in ASP-DAC (2015)Google Scholar
  21. 21.
    B. Li, Y. Wang, Y. Chen, H.H. Li, H. Yang, Ice: inline calibration for memristor crossbar-based computing engine, in Proceedings of the conference on Design, Automation & Test in Europe (European Design and Automation Association, 2014), p. 184Google Scholar
  22. 22.
    S. Song, K.D. Miller, L.F. Abbott, Competitive hebbian learning through spike-timing-dependent synaptic plasticity. Nat. Neurosci. 3 (9), 919–926 (2000)CrossRefGoogle Scholar
  23. 23.
    C.-C. Chang, C.-J. Lin, Libsvm: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2 (3), 27 (2011)Google Scholar
  24. 24.
    F. Ponulak, ReSuMe - new supervised learning method for spiking neural networks, Institute of Control and Information Engineering, Poznan University of Technology. Available online at: http://d1.cie.put.poznan.pl/~{}fp/research.html (2005)
  25. 25.
    J. Hu, H. Tang, K.C. Tan, H. Li, L. Shi, A spike-timing-based integrated model for pattern recognition. Neural Comput. 25 (2), 450–472 (2013)MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    P. O’Connor, D. Neil, S.-C. Liu, T. Delbruck, M. Pfeiffer, Real-time classification and sensor fusion with a spiking deep belief network. Neuromorphic Eng. Syst. Appl 61, 1–10 (2015)Google Scholar
  27. 27.
    T. Tang, R. Luo, B. Li, H. Li, Y. Wang, H. Yang, Energy efficient spiking neural network design with RRAM devices, in 14th International Symposium on Integrated Circuits (ISIC) (IEEE, 2014), pp. 268–271Google Scholar
  28. 28.
    E. Neftci, S. Das, B. Pedroni, K. Kreutz-Delgado, G. Cauwenberghs, Event-driven contrastive divergence for spiking neuromorphic systems (2013). Preprint, arXiv:1311.0966Google Scholar
  29. 29.
    P.U. Diehl, D. Neil, J. Binas, M. Cook, Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing, in International Joint Conference on Neural Networks (2015)Google Scholar
  30. 30.
    T. Tang, L. Xia, B. Li, R. Luo, Y. Wang, Y. Chen, H. Yang, Spiking neural network with RRAM: can we use it for real-world application? In Proceedings of the 2015 Design, Automation & Test in Europe Conference & Exhibition (EDA Consortium, 2015), pp. 860–865Google Scholar
  31. 31.
    B. Li, L. Xia, P. Gu, Y. Wang, H. Yang, Merging the interface: power, area and accuracy co-optimization for RRAM crossbar-based mixed-signal computing system, in Design Automation Conference (2015), pp. 1–6Google Scholar
  32. 32.
    Z.-H. Zhou, Ensemble Methods: Foundations and Algorithms (CRC Press, 2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Yu Wang
    • 1
  • Tianqi Tang
    • 1
  • Boxun Li
    • 1
  • Lixue Xia
    • 1
  • Huazhong Yang
    • 1
  1. 1.Department of Electronic EngineeringTsinghua UniversityBeijingChina

Personalised recommendations