Abstract
Mainstream standard LSTM architecture that is currently used in Tensorflow library does not use the original architecture. In fact, there are many different architectures of LSTM. One of the more widely used architectures of LSTM is Coupled Input and Forget Gate (CIFG). It is known more as Gated Recurrent Units (GRU). This chapter will introduce the existing architectures of LSTM. Further it will present memristive LSTM architecture implementation in analog hardware. The implementation realizes the standard version of LSTM architecture. Other architecture variations can be easily constructed by rearranging, adding, and deleting the existing analog circuit parts; and adding extra crossbar rows.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Graves A, Schmidhuber J (2005) Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Netw 18(5–6):602–610
Greff K, Srivastava RK, Koutník J et al (2017) LSTM: a search space Odyssey. IEEE Trans Neural Netw Learn Syst 28(10):2222–2232. https://doi.org/10.1109/TNNLS.2016.2582924
Gers FA, Schmidhuber J, Cummins F (1999) Learning to forget: continual prediction with LSTM. In: Proceedings of the 9th international conference on artificial neural networks (ICANN), vol 2. pp 850–855
Gers FA, Schmidhuber J (2000) Recurrent nets that time and count. In: Proceedings of IEEE-INNS-ENNS international joint conference on neural networks (IJCNN), vol 3. pp 189–194
Cho K, van Merrienboer B, Gulcehre C et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv:1406.1078
Smagulova K, Krestinskaya O, James AP (2018) Analog Integr Circuit Signal Process 95:467. https://doi.org/10.1007/s10470-018-1180-y
Smagulova K, Adam K, Krestinskaya O et al (2018) Design of CMOS-memristor circuits for LSTM architecture. In: 2018 IEEE international conference on electron devices and solid state circuits (EDSSC), 1 pp. https://doi.org/10.1109/EDSSC.2018.8487179
Gokmen T, Rasch MJ, Haensch W (2018) Training LSTM networks with resistive cross-point devices. Front Neurosci 12:745. https://doi.org/10.3389/fnins.2018.00745
Li C, Wang Z, Rao M et al (2018) Long short-term memory networks in memristor crossbars. https://doi.org/10.1038/s42256-018-0001-4
Adam K, Smagulova K, James AP (2018) Memristive LSTM network hardware architecture for time-series predictive modeling problem. arXiv:1809.03119
Adam K, Smagulova K, Krestinskaya O et al (2018) Wafer quality inspection using memristive LSTM, ANN, DNN and HTM. arXiv:1809.10438
Krestinskaya O, Salama K, James AP (2018) Learning in memristive neural network architectures using analog backpropagation circuits. IEEE Trans Circuits Syst I: Regul Pap. https://doi.org/10.1109/TCSI.2018.2866510
Saxena V, Baker RJ (2010) Indirect compensation techniques for three-stage fully-differential op-amps. In: 53rd IEEE international midwest symposium on circuits and systems, Seattle, WA, pp 588–591. https://doi.org/10.1109/MWSCAS.2010.5548896
Hasan R, Taha TM, Yakopcic C (2017) On-chip training of memristor based deep neural networks. In: International joint conference on neural networks (IJCNN), Anchorage, AK, pp 3527–3534. https://doi.org/10.1109/IJCNN.2017.7966300
Ramirez-Angulo J, Thoutam S, Lopez-Martin A et al (2004) Low-voltage CMOS analog four quadrant multiplier based on flipped voltage followers. In: 2004 IEEE international symposium on circuits and systems (IEEE Cat. No.04CH37512), Vancouver, BC, pp I–681. https://doi.org/10.1109/ISCAS.2004.1328286
Brownlee J (2016) Time series prediction with LSTM recurrent neural networks in python with keras. Available at: https://machinelearningmastery.com
Olszewski RT (2001) Generalized feature extraction for structural pattern recognition in time-series data (No. CMU-CS-01-108). Carnegie-Mellon University Pittsburgh PA School of Computer Science
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Chapter Highlights
Chapter Highlights
-
Long short-term memory (LSTM) unit operation can be split into several computational blocks.
-
Weight-matrix multiplication in LSTM is implemented using a memristor crossbar array.
-
Pointwise (Hadamard) multiplication and activation layer circuits are implemented using TSMC 180nm CMOS technology.
-
Voltage-based memristive crossbar array wins over current based design due to higher accuracy.
-
Memristive devices which exhibit symmetric behavior [9] would bring us closer to achieving the same performance metrics as in the software implementation of LSTM.
-
The work by [10] is an example of applying real memristive crossbar for implementing LSTM. Most of the design uses digital building blocks to realize the hardware.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Adam, K., Smagulova, K., James, A.P. (2020). Memristive LSTM Architectures. In: James, A. (eds) Deep Learning Classifiers with Memristive Networks. Modeling and Optimization in Science and Technologies, vol 14. Springer, Cham. https://doi.org/10.1007/978-3-030-14524-8_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-14524-8_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-14522-4
Online ISBN: 978-3-030-14524-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)