Skip to main content
SpringerLink
Log in

Adaptive learning rule for hardware-based deep neural networks using electronic synapse devices

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a learning rule based on a back-propagation (BP) algorithm that can be applied to a hardware-based deep neural network using electronic devices that exhibit discrete and limited conductance characteristics. This adaptive learning rule, which enables forward, backward propagation, as well as weight updates in hardware, is helpful during the implementation of power-efficient and high-speed deep neural networks. In simulations using a three-layer perceptron network, we evaluate the learning performance according to various conductance responses of electronic synapse devices and weight-updating methods. It is shown that the learning accuracy is comparable to that obtained when using a software-based BP algorithm when the electronic synapse device has a linear conductance response with a high dynamic range. Furthermore, the proposed unidirectional weight-updating method is suitable for electronic synapse devices which have nonlinear and finite conductance responses. Because this weight-updating method can compensate the demerit of asymmetric weight updates, we can obtain better accuracy compared to other methods. This adaptive learning rule, which can be applied to full hardware implementation, can also compensate the degradation of learning accuracy due to the probable device-to-device variation in an actual electronic synapse device.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Poon CS, Zhou K (2011) Neuromorphic silicon neurons and large-scale neural networks: challenges and opportunities. Front Neurosci 5:108. https://doi.org/10.3389/fnins.2011.00108

    Article  Google Scholar 

  2. Suri M, Bichler O, Querlioz D, Cueto O, Perniola L, Sousa V, Vuillaume D, Gamrat C, DeSalvo B (2011) Phase change memory as synapse for ultra-dense neuromorphic systems: application to complex visual pattern extraction. IEEE Electron Devices Meeting. https://doi.org/10.1109/IEDM.2011.6131488

    Article  Google Scholar 

  3. Wright CD, Liu Y, Kohary KI, Aziz MM, Hicken RJ (2011) Arithmetic and biologically-inspired computing using phase-change materials. Adv Mater 23:3408–3413. https://doi.org/10.1002/adma.201101060

    Article  Google Scholar 

  4. Kuzum D, Jeyasingh RG, Lee B, Wong HS (2012) Nanoelectronic programmable synapses based on phase change materials for brain-inspired computing. Nano Lett 12:2179–2186. https://doi.org/10.1021/nl201040y

    Article  Google Scholar 

  5. Jo SH, Chang T, Ebong I, Bhadviya BB, Mazumder P, Lu W (2010) Nanoscale memristor device as synapse in neuromorphic systems. Nano Lett 10:1297–1301. https://doi.org/10.1021/nl904092h

    Article  Google Scholar 

  6. Ohno T, Hasegawa T, Tsuruoka T, Terabe K, Gimzewski JK, Aono M (2011) Short-term plasticity and long-term potentiation mimicked in single inorganic synapses. Nat Mater 10:591–595. https://doi.org/10.1038/nmat3054

    Article  Google Scholar 

  7. Wu Y, Yu S, Wong HS (2012) AlOx-based resistive switching device with gradual resistance modulation for neuromorphic device application. IEEE Memory Workshop. https://doi.org/10.1109/IMW.2012.6213663

    Article  Google Scholar 

  8. Yu S, Gao B, Fang Z (2012) A neuromorphic visual system using RRAM synaptic devices with sub-pJ energy and tolerance to variability: experimental characterization and large-scale modeling. IEEE Electron Devices Meeting. https://doi.org/10.1109/IEDM.2012.6479018

    Article  Google Scholar 

  9. Chanthbouala A, Garcia V, Cherifi RO, Bouzehouane K, Fusil S, Moya X, Xavier S, Yamada H, Deranlot C, Mathur ND, Bibes M, Barthelemy A, Grollier J (2012) A ferroelectric memristor. Nat Mater 11:860–864. https://doi.org/10.1038/nmat3415

    Article  Google Scholar 

  10. Diorio C, Hasler P, Minch A, Mead CA (1996) A single-transistor silicon synapse. IEEE Trans Electron Devices 43:1972–1980. https://doi.org/10.1109/16.543035

    Article  Google Scholar 

  11. Ziegler M, Oberländer M, Schroeder D, Krautschneider WH, Kohlstedt H (2012) Memristive operation mode of floating gate transistors: a two-terminal MemFlash-cell. Appl Phys Lett 101:263504. https://doi.org/10.1063/1.4773300

    Article  Google Scholar 

  12. Kim H, Park J, Kwon M-W, Lee J-H, Park B-G (2016) Silicon-based floating-body synaptic transistor with frequency-dependent short- and long-term memories. IEEE Electron Device Lett 37:249–252. https://doi.org/10.1109/led.2016.2521863

    Article  Google Scholar 

  13. Bi G-q, Poo M-m (1998) Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J Neurosci 18:10464. https://www.ncbi.nlm.nih.gov/pubmed/9852584

  14. Kuzum D, Yu S, Wong HS (2013) Synaptic electronics: materials, devices and applications. Nanotechnology 24:382001. https://doi.org/10.1088/0957-4484/24/38/382001

    Article  Google Scholar 

  15. Burr GW, Shelby RM, di Nolfo C, Jang J-W, Shenoy RS, Narayanan P, Virwani K, Giacometti EU, Kurdi B, Hwang H (2014) Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element. IEEE Electron Devices Meeting. https://doi.org/10.1109/iedm.2014.7047135

    Article  Google Scholar 

  16. Nair MV, Dudek P (2015) Gradient-descent-based learning in memristive crossbar arrays. IEEE Int Joint Conf Neural Netw. https://doi.org/10.1109/IJCNN.2015.7280658

    Article  Google Scholar 

  17. Prezioso M, Merrikh-Bayat F, Hoskins BD, Adam GC, Likharev KK, Strukov DB (2015) Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521:61–64. https://doi.org/10.1038/nature14441

    Article  Google Scholar 

  18. Merrikh-Bayat F, Guo X, Om’mani HA, Do N, Likharev KK, Strukov DB (2015) Redesigning commercial floating-gate memory for analog computing applications. IEEE Int Symp Circuits Syst. https://doi.org/10.1109/ISCAS.2015.7169048

    Article  Google Scholar 

  19. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning internal representations by error propagation. In: Parallel distributed processing: explorations in macrostructure of cognition, vol I. Badford, Cambridge. https://dl.acm.org/citation.cfm?id=104293

  20. Krizhevsky A, Sutskever I, and Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1–9. https://dl.acm.org/citation.cfm?id=2999257

  21. He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR arXiv:abs/1512.03385

  22. Cho K, van Merrienboer B, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. arXiv:1406.1078

  23. Greff K, Srivastava RK, Koutnik J, Steunebrink BR, Schmidhuber J (2017) LSTM: a search space odyssey. IEEE Trans Neural Netw Learn Syst 28(10):2222–2232. https://doi.org/10.1109/TNNLS.2016.2582924

    Article  MathSciNet  Google Scholar 

  24. Deng Y, Ren Z, Kong Y, Bao F, Dai Q (2017) A hierarchical fused fuzzy deep neural network for data classification. IEEE Trans Fuzzy Syst 25:1006–1012. https://doi.org/10.1109/tfuzz.2016.2574915

    Article  Google Scholar 

  25. Deng Y, Bao F, Kong Y, Ren Z, Dai Q (2017) Deep direct reinforcement learning for financial signal representation and trading. IEEE Trans Neural Netw Learn Syst 28:653–664. https://doi.org/10.1109/TNNLS.2016.2522401

    Article  Google Scholar 

  26. Merkel C, Hasan R, Soures N, Kudithipudi D, Taha T, Agarwal S, Marinella M (2016) Neuromemristive systems: boosting efficiency through brain-inspired computing. Computer 49:56–64. https://doi.org/10.1109/MC.2016.312

    Article  Google Scholar 

  27. Neftci EO, Augustine C, Paul S, Detorakis G (2017) Event-driven random back-propagation: enabling neuromorphic deep learning machines. Front Neurosci 11:324. https://doi.org/10.3389/fnins.2017.00324

    Article  Google Scholar 

  28. Merrikh Bayat F, Bagheri Shouraki S, Esmaili Paeen Afrakoti I (2013) Bottleneck of using a single memristive device as a synapse. Neurocomputing 115:166–168. https://doi.org/10.1016/j.neucom.2012.12.027

    Article  Google Scholar 

  29. Jang J-W, Park S, Burr GW, Hwang H, Jeong Y-H (2015) Optimization of conductance change in PrCaMnO based synaptic devices for neuromorphic systems. IEEE Electron Device Lett 36:457–459. https://doi.org/10.1109/LED.2015.2418342

    Article  Google Scholar 

  30. Woo J, Moon K, Song J, Kwak M, Park J, Hwang H (2016) Optimized programming scheme enabling linear potentiation in filamentary HfO2 RRAM synapse for neuromorphic systems. IEEE Trans Electron Devices 63:5064–5067. https://doi.org/10.1109/ted.2016.2615648

    Article  Google Scholar 

  31. Fuller EJ, Gabaly FE, Leonard F, Agarwal S, Plimpton SJ, Jacobs-Gedrim RB, James CD, Marinella MJ, Talin AA (2017) Li-ion synaptic transistor for low power analog computing. Adv Mater 29(4):1604310. https://doi.org/10.1002/adma.201604310

    Article  Google Scholar 

  32. Bae J-H, Lim S, Park B-G, Lee J-H (2017) High-density and near-linear synaptic device based on a reconfigurable gated Schottky diode. IEEE Electron Device Lett 38:1153–1156. https://doi.org/10.1109/LED.2017.2713460

    Article  Google Scholar 

  33. Gokmen T, Vlasov Y (2016) Acceleration of deep neural network training with resistive cross-point devices: design considerations. Front Neurosci 10:333. https://doi.org/10.3389/fnins.2016.00333

    Article  Google Scholar 

  34. Fumarola A, Narayana P, Sanches LL, Sidler S, Jang J, Moon K, Shelby RM, Hwang H, Burr GW (2016) Accelerating machine learning with non-volatile memory: exploring device and circuit tradeoffs. IEEE Int Conf Rebooting Comput. https://doi.org/10.1109/ICRC.2016.7738684

    Article  Google Scholar 

  35. Binas J, Neil D, Indiveri G, Liu S-C, Pfeiffer M (2016) Precise deep neural network computation on imprecise low-power analog hardware. arXiv preprint. arXiv:1606.07786

  36. Schiffmann W, Joost M, Werner R (1994) Optimization of the backpropagation algorithm for training multilayer perceptrons. Technical report, University of Koblenz, Institute of Physics, Rheinau. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.53.6869&rep=rep1&type=pdf

  37. Bichler O, Suri M, Querlioz D, Vuillaume D, DeSalvo B, Gamrat C (2012) Visual pattern extraction using energy-efficient “2-PCM Synapse” neuromorphic architecture. IEEE Trans Electron Devices 59:2206–2214. https://doi.org/10.1109/ted.2012.2197951

    Article  Google Scholar 

  38. MATLAB and Statistics Toolbox Release 2016a. The MathWorks, Inc., Natick

  39. Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86:2278–2324. https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  40. Querlioz D, Bichler O, Dollfus P, Gamrat C (2013) Immunity to device variations in a spiking neural network with memristive nanodevices. IEEE Trans Nanotechnol 12:288–295. https://doi.org/10.1109/TNANO.2013.2250995

    Article  Google Scholar 

  41. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2:1–127. https://doi.org/10.1561/2200000006

    Article  MATH  Google Scholar 

  42. Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. Adv Neural Inf Process Syst 153–160. https://dl.acm.org/citation.cfm?id=2976476

  43. Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. The J Mach Learn Res 15:1929–1958. http://dl.acm.org/citation.cfm?id=2670313

Download references

Acknowledgements

This work was supported by the Korea Institute of Science and Technology (KIST) Institutional Program (Project No. 2E27810-18-P040) and the Brain Korea 21 Plus Project in 2018.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Inter-University Semiconductor Research Center (ISRC), Seoul National University, 1 Gwanak-ro, Gwanak-gu, Seoul, 08826, Republic of Korea

    Suhwan Lim, Jong-Ho Bae, Jai-Ho Eum, Sungtae Lee, Chul-Heung Kim, Dongseok Kwon, Byung-Gook Park & Jong-Ho Lee

Authors
  1. Suhwan Lim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jong-Ho Bae
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jai-Ho Eum
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sungtae Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chul-Heung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dongseok Kwon
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Byung-Gook Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jong-Ho Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jong-Ho Lee.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lim, S., Bae, JH., Eum, JH. et al. Adaptive learning rule for hardware-based deep neural networks using electronic synapse devices. Neural Comput & Applic 31, 8101–8116 (2019). https://doi.org/10.1007/s00521-018-3659-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-018-3659-y

Keywords

Search

Navigation