Advertisement

Memristive LSTM Architectures

  • Kazybek Adam
  • Kamilya Smagulova
  • Alex Pappachen JamesEmail author
Chapter
Part of the Modeling and Optimization in Science and Technologies book series (MOST, volume 14)

Abstract

Mainstream standard LSTM architecture that is currently used in Tensorflow library does not use the original architecture. In fact, there are many different architectures of LSTM. One of the more widely used architectures of LSTM is Coupled Input and Forget Gate (CIFG). It is known more as Gated Recurrent Units (GRU). This chapter will introduce the existing architectures of LSTM. Further it will present memristive LSTM architecture implementation in analog hardware. The implementation realizes the standard version of LSTM architecture. Other architecture variations can be easily constructed by rearranging, adding, and deleting the existing analog circuit parts; and adding extra crossbar rows.

References

  1. 1.
    Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780CrossRefGoogle Scholar
  2. 2.
    Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw 18(5–6):602–610CrossRefGoogle Scholar
  3. 3.
    Greff K, Srivastava RK, Koutník J et al (2017) LSTM: a search space Odyssey. IEEE Trans Neural Netw Learn Syst 28(10):2222–2232.  https://doi.org/10.1109/TNNLS.2016.2582924MathSciNetCrossRefGoogle Scholar
  4. 4.
    Gers FA, Schmidhuber J, Cummins F (1999) Learning to forget: continual prediction with LSTM. In: Proceedings of the 9th international conference on artificial neural networks (ICANN), vol 2. pp 850–855Google Scholar
  5. 5.
    Gers FA, Schmidhuber J (2000) Recurrent nets that time and count. In: Proceedings of IEEE-INNS-ENNS international joint conference on neural networks (IJCNN), vol 3. pp 189–194Google Scholar
  6. 6.
    Cho K, van Merrienboer B, Gulcehre C et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078
  7. 7.
    Smagulova K, Krestinskaya O, James AP (2018) Analog Integr Circuit Signal Process 95:467.  https://doi.org/10.1007/s10470-018-1180-yCrossRefGoogle Scholar
  8. 8.
    Smagulova K, Adam K, Krestinskaya O et al (2018) Design of CMOS-memristor circuits for LSTM architecture. In: 2018 IEEE international conference on electron devices and solid state circuits (EDSSC), 1 pp.  https://doi.org/10.1109/EDSSC.2018.8487179
  9. 9.
    Gokmen T, Rasch MJ, Haensch W (2018) Training LSTM networks with resistive cross-point devices. Front Neurosci 12:745.  https://doi.org/10.3389/fnins.2018.00745CrossRefGoogle Scholar
  10. 10.
    Li C, Wang Z, Rao M et al (2018) Long short-term memory networks in memristor crossbars.  https://doi.org/10.1038/s42256-018-0001-4CrossRefGoogle Scholar
  11. 11.
    Adam K, Smagulova K, James AP (2018) Memristive LSTM network hardware architecture for time-series predictive modeling problem. arXiv:1809.03119
  12. 12.
    Adam K, Smagulova K, Krestinskaya O et al (2018) Wafer quality inspection using memristive LSTM, ANN, DNN and HTM. arXiv:1809.10438
  13. 13.
    Krestinskaya O, Salama K, James AP (2018) Learning in memristive neural network architectures using analog backpropagation circuits. IEEE Trans Circuits Syst I: Regul Pap.  https://doi.org/10.1109/TCSI.2018.2866510CrossRefGoogle Scholar
  14. 14.
    Saxena V, Baker RJ (2010) Indirect compensation techniques for three-stage fully-differential op-amps. In: 53rd IEEE international midwest symposium on circuits and systems, Seattle, WA, pp 588–591.  https://doi.org/10.1109/MWSCAS.2010.5548896
  15. 15.
    Hasan R, Taha TM, Yakopcic C (2017) On-chip training of memristor based deep neural networks. In: International joint conference on neural networks (IJCNN), Anchorage, AK, pp 3527–3534.  https://doi.org/10.1109/IJCNN.2017.7966300
  16. 16.
    Ramirez-Angulo J, Thoutam S, Lopez-Martin A et al (2004) Low-voltage CMOS analog four quadrant multiplier based on flipped voltage followers. In: 2004 IEEE international symposium on circuits and systems (IEEE Cat. No.04CH37512), Vancouver, BC, pp I–681.  https://doi.org/10.1109/ISCAS.2004.1328286
  17. 17.
    Brownlee J (2016) Time series prediction with LSTM recurrent neural networks in python with keras. Available at: https://machinelearningmastery.com
  18. 18.
    Olszewski RT (2001) Generalized feature extraction for structural pattern recognition in time-series data (No. CMU-CS-01-108). Carnegie-Mellon University Pittsburgh PA School of Computer ScienceGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Kazybek Adam
    • 1
  • Kamilya Smagulova
    • 1
  • Alex Pappachen James
    • 1
    Email author
  1. 1.Nazarbayev UniversityAstanaKazakhstan

Personalised recommendations