Advertisement

Efficient parallel implementation of reservoir computing systems

  • M. L. Alomar
  • Erik S. Skibinsky-Gitlin
  • Christiam F. Frasser
  • Vincent Canals
  • Eugeni Isern
  • Miquel Roca
  • Josep L. Rosselló
Original Article
  • 45 Downloads

Abstract

Reservoir computing (RC) is a powerful machine learning methodology well suited for time-series processing. The hardware implementation of RC systems (HRC) may extend the utility of this neural approach to solve real-life problems for which software solutions are not satisfactory. Nevertheless, the implementation of massive parallel-connected reservoir networks is costly in terms of circuit area and power, mainly due to the requirement of implementing synapse multipliers that increase gate count to prohibitive values. Most HRC systems present in the literature solve this area problem by sequencializing the processes, thus loosing the expected fault-tolerance and low latency of fully parallel-connected HRCs. Therefore, the development of new methodologies to implement fully parallel HRC systems is of high interest to many computational intelligence applications requiring quick responses. In this article, we propose a compact hardware implementation for Echo-State Networks (an specific type of reservoir) that reduces the area cost by simplifying the synapses and using linear piece-wise activation functions for neurons. The proposed design is synthesized in a Field-Programmable Gate Array and evaluated for different time-series prediction tasks. Without compromising the overall accuracy, the proposed approach achieves a significant saving in terms of power and hardware when compared with recently published implementations. This technique pave the way for the low-power implementation of fully parallel reservoir networks containing thousands of neurons in a single integrated circuit.

Keywords

Artificial neural networks Recurrent neural networks Reservoir computing Echo sate networks Hardware neural network Field-programmable gate array Time-series prediction 

Notes

Acknowledgements

This work has been partially supported by the Spanish Ministry of Economy and Competitiveness (MINECO), the Regional European Development Funds (FEDER), and the Comunitat Autnoma de les Illes Balears under grant contracts TEC2014-56244-R, TEC2017-84877-R and a fellowship (FPI/1513/2012) financed by the European Social Fund (ESF) and the Govern de les Illes Balears (Conselleria d’Educació, Cultura i Universitats).

References

  1. 1.
    Baptista D, Abreu S, Freitas F, Vasconcelos R, Morgado-Dias F (2013) A survey of software and hardware use in artificial neural networks. Neural Comput Appl 23(3–4):591–599CrossRefGoogle Scholar
  2. 2.
    Misra J, Saha I (2010) Artificial neural networks in hardware: a survey of two decades of progress. Neurocomputing 74(1–3):239–255CrossRefGoogle Scholar
  3. 3.
    Baptista FD, Morgado-Dias F (2017) Automatic general-purpose neural hardware generator. Neural Comput Appl 28(1):25–36CrossRefGoogle Scholar
  4. 4.
    Amir MF, Kim D, Kung J, Lie D, Yalamanchili S, Mukhopadhyay S (2017) NeuroSensor: a 3D image sensor with integrated neural accelerator. In: 2016 SOI-3D-subthreshold microelectronics technology unified conference, S3S 2016Google Scholar
  5. 5.
    Krizhevsky A, Sutskever I, Geoffrey EH (2012) ImageNet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25(NIPS2012):1–9Google Scholar
  6. 6.
    Morro A, Canals V, Oliver A, Alomar ML, Galan-Prado F, Ballester PJ, Rossello JL (2017) A stochastic spiking neural network for virtual screeningGoogle Scholar
  7. 7.
    Li H, Zhang D, Foo SY (2006) A stochastic digital implementation of a neural network controller for small wind turbine systems. IEEE Trans Power Electron 21(5):1502–1507CrossRefGoogle Scholar
  8. 8.
    Chauhan A, Semwal S, Chawhan R (2013) Artificial neural network-based forest fire detection system using wireless sensor network. In: 2013 annual IEEE India conference INDICON, pp 1–6Google Scholar
  9. 9.
    Raghunathan S, Gupta SK, Ward MP, Worth RM, Roy K, Irazoqui PP (2009) The design and hardware implementation of a low-power real-time seizure detection algorithm. J Neural Eng 6(5):056,005CrossRefGoogle Scholar
  10. 10.
    Lee M, Hwang K, Park J, Choi S, Shin S, Sung W (2016) FPGA-based low-power speech recognition with recurrent neural networks. In: IEEE workshop on signal processing systems, SiPS: design and implementation, pp 230–235Google Scholar
  11. 11.
    Basterretxea K, Tarela JM, del Campo I (2002) Digital design of sigmoid approximator for artificial neural networks. Electron Lett 38(1):35–37CrossRefGoogle Scholar
  12. 12.
    Baptista D, Morgado-Dias F (2013) Low-resource hardware implementation of the hyperbolic tangent for artificial neural networks. Neural Comput Appl 23(3–4):601–607CrossRefGoogle Scholar
  13. 13.
    Nascimento I, Jardim R, Morgado-Dias F (2013) A new solution to the hyperbolic tangent implementation in hardware: polynomial modeling of the fractional exponential part. Neural Comput Appl 23(2):363–369CrossRefGoogle Scholar
  14. 14.
    Carrasco-Robles M, Serrano L (2009) Accurate differential tanh(nx) implementation. Int J Circuit Theory Appl 37(5):613–629zbMATHCrossRefGoogle Scholar
  15. 15.
    Nedjah N, De MacEdo Mourelle L (2007) Reconfigurable hardware for neural networks: binary versus stochastic. Neural Comput Appl 16(3):249–255CrossRefGoogle Scholar
  16. 16.
    Lotrič U, Bulić P (2012) Applicability of approximate multipliers in hardware neural networks. Neurocomputing 96:57–65CrossRefGoogle Scholar
  17. 17.
    Lukoševičius M, Jaeger H, Schrauwen B (2012) Reservoir computing trends. KI - Künstliche Intelligenz 26(4):365–371CrossRefGoogle Scholar
  18. 18.
    Antonelo EA, Schrauwen B (2015) On learning navigation behaviors for small mobile robots with reservoir computing architectures. IEEE Trans Neural Netw Learn Syst 26(4):763–780MathSciNetCrossRefGoogle Scholar
  19. 19.
    Jalalvand A, Wallendael GV, Walle RVD (2015) Real-time reservoir computing network-based systems for detection tasks on visual contents. In: Proceedings—7th international conference on computational intelligence, communication systems and networks, CICSyN 2015, pp 146–151Google Scholar
  20. 20.
    Bacciu D, Barsocchi P, Chessa S, Gallicchio C, Micheli A (2014) An experimental characterization of reservoir computing in ambient assisted living applications. Neural Comput Appl 24(6):1451–1464CrossRefGoogle Scholar
  21. 21.
    Lin X, Yang Z, Song Y (2009) Short-term stock price prediction based on echo state networks. Expert Syst Appl 36(3 PART 2):7313–7317CrossRefGoogle Scholar
  22. 22.
    Buteneers P, Verstraeten D, Nieuwenhuyse BV, Stroobandt D, Raedt R, Vonck K, Boon P, Schrauwen B (2013) Real-time detection of epileptic seizures in animal models using reservoir computing. Epilepsy Res 103(2–3):124–134CrossRefGoogle Scholar
  23. 23.
    Alomar ML, Canals V, Perez-Mora N, Martínez-Moll V, Rosselló JL (2016) FPGA-based stochastic echo state networks for time-series forecasting. Comput Intell Neurosci 2016:3917892.  https://doi.org/10.1155/2016/3917892 CrossRefGoogle Scholar
  24. 24.
    Alomar ML, Soriano MC, Escalona-Morán M, Canals V, Fischer I, Mirasso CR, Rosselló JL (2015) Digital implementation of a single dynamical node reservoir computer. IEEE Trans Circuits Syst II Express Br 62(10):977–981CrossRefGoogle Scholar
  25. 25.
    Soriano MC, Ortín S, Keuninckx L, Appeltant L, Danckaert J, Pesquera L, van der Sande G (2015) Delay-based reservoir computing: noise effects in a combined analog and digital implementation. IEEE Trans Neural Netw Learn Syst 26(2):388–393.  https://doi.org/10.1109/TNNLS.2014.2311855 MathSciNetCrossRefGoogle Scholar
  26. 26.
    Ortín S, Soriano MC, Pesquera L, Brunner D, San-Martín D, Fischer I, Mirasso CR, Gutiérrez JM (2015) A unified framework for reservoir computing and extreme learning machines based on a single time-delayed neuron. Sci Rep 5:14945.  https://doi.org/10.1038/srep14945 CrossRefGoogle Scholar
  27. 27.
    Brunner D, Soriano MC, Mirasso CR, Fischer I (2013) Parallel photonic information processing at gigabyte per second data rates using transient states. Nat Commun 4:1364.  https://doi.org/10.1038/ncomms2368 CrossRefGoogle Scholar
  28. 28.
    Hicke K, Escalona-Morán M, Brunner D, Soriano MC, Fischer I, Mirasso CR (2013) Information processing using transient dynamics of semiconductor lasers subject to delayed feedback. IEEE J Sel Top Quantum Electron 19(4):1501,610–1501,610CrossRefGoogle Scholar
  29. 29.
    Bueno J, Maktoobi S, Froehly L, Fischer I, Jacquot M, Larger L, Brunner D (2018) Reinforcement learning in a large-scale photonic recurrent neural network. Optica 5(6):756–760CrossRefGoogle Scholar
  30. 30.
    Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2Activity: recognizing complex activities from sensor data. In: IJCAI international joint conference on artificial intelligence, vol 2015, January, pp 1617–1623Google Scholar
  31. 31.
    Liu Y, Zhang L, Nie L, Yan Y, Rosenblum DS (2016a) Fortune teller: predicting your career path. In: Proceedings of the 30th conference on artificial intelligence (AAAI 2016) (1), pp 201–207Google Scholar
  32. 32.
    Liu Y, Nie L, Liu L, Rosenblum DS (2016b) From action to activity: sensor-based activity recognition. Neurocomputing 181:108–115CrossRefGoogle Scholar
  33. 33.
    Draper NR, Smith H (1998) Applied regression analysis. Technometrics 47(3):706MathSciNetzbMATHGoogle Scholar
  34. 34.
    Chatterjee S, Hadi AS (1986) Influential observations, high leverage points, and outliers in linear regression. Stat Sci 1(3):379–393MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Boukouvalas A, Cornford D, Stehlík M (2014) Optimal design for correlated processes with input-dependent noise. Comput Stat Data Anal 71:1088–1102MathSciNetzbMATHCrossRefGoogle Scholar
  36. 36.
    Lim YC, Liu B (1988) Design of cascade form FIR filters with discrete valued coefficients. IEEE Trans Acoust Speech Signal Process 36(11):1735–1739zbMATHCrossRefGoogle Scholar
  37. 37.
    Marchesi M, Orlandi G, Piazza F, Uncini A (1993) Fast neural networks without multipliers. IEEE Trans Neural Netw 4(1):53–62CrossRefGoogle Scholar
  38. 38.
    Rodan A, Tiño P (2011) Minimum complexity echo state network. IEEE Trans Neural Netw 22(1):131–144CrossRefGoogle Scholar
  39. 39.
    McCulloch WS, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5(4):115–133MathSciNetzbMATHCrossRefGoogle Scholar
  40. 40.
    Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536zbMATHCrossRefGoogle Scholar
  41. 41.
    Mackey MC, Glass L (1977) Oscillation and chaos in physiological control systems. Science (New York, NY) 197(4300):287–9zbMATHCrossRefGoogle Scholar
  42. 42.
    Weigend AS, Gershenfeld NA (1993) Results of the time series prediction competition at the Santa Fe Institute. In: IEEE international conference on neural networks—conference proceedings, vol 1993, January, pp 1786–1793Google Scholar
  43. 43.
    Modeste Nguimdo R, Verschaffelt G, Danckaert J, Van Der Sande G (2015) Simultaneous computation of two independent tasks using reservoir computing based on a single photonic nonlinear node with optical feedback. IEEE Trans Neural Netw Learn Syst 26(12):3301–3307MathSciNetCrossRefGoogle Scholar
  44. 44.
    Benedetto S, Biglieri E (1999) Principles of digital transmission: with wireless applications. Kluwer Academic Publishers, NorwellzbMATHGoogle Scholar
  45. 45.
    Boccato L, Lopes A, Attux R, Von Zuben FJ (2011) An echo state network architecture based on Volterra filtering and PCA with application to the channel equalization problem. In: Proceedings of the international joint conference on neural networks, pp 580–587Google Scholar
  46. 46.
    Lucky RW (1965) Automatic equalization for digital communication. Bell Syst Tech J 44(4):547–588MathSciNetCrossRefGoogle Scholar
  47. 47.
    Gersho A, Lim TL (1981) Adaptive cancellation of intersymbol interference for data transmission. Bell Syst Tech J 60(9):1997–2021CrossRefGoogle Scholar
  48. 48.
    Karam G, Sari H (1989) Analysis of predistortion, equalization, and ISI cancellation techniques in digital radio systems with nonlinear transmit amplifiers. IEEE Trans Commun 37(12):1245–1253CrossRefGoogle Scholar
  49. 49.
    Mathews VJ (1991) Adaptive polynomial filters. IEEE Signal Process Mag 8(3):10–26CrossRefGoogle Scholar
  50. 50.
    Malone J, Wickert MA (2011) Practical Volterra equalizers for wideband satellite communications with TWTA nonlinearities. In: 2011 Digital signal processing and signal processing education meeting, DSP/SPE 2011—proceedings, pp 48–53Google Scholar
  51. 51.
    Chen S, Gibson GJ, Cowan CFN (1990) Adaptive channel equalisation using a polynomial-perceptron structure. IEEE Proc I Commun Speech Vis 137(5):257–264CrossRefGoogle Scholar
  52. 52.
    Patra JC, Pal RN, Baliarsingh R, Panda G (1999) Nonlinear channel equalization for QAM signal constellation using artificial neural networks. IEEE Trans Syst Man Cybern Part B Cybern 29(2):262–271CrossRefGoogle Scholar
  53. 53.
    Patra JC, Meher PK, Chakraborty G (2009) Nonlinear channel equalization for wireless communication systems using Legendre neural networks. Signal Process 89(11):2251–2262zbMATHCrossRefGoogle Scholar
  54. 54.
    Jaeger H, Haas H (2004) Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science 304(5667):78–80CrossRefGoogle Scholar
  55. 55.
    Bauduin M, Smerieri A, Massar S, Horlin F (2015) Equalization of the non-linear satellite communication channel with an Echo state network. In: IEEE vehicular technology conference, vol 2015Google Scholar
  56. 56.
    Mathews VJ, Lee J (1994) Adaptive algorithms for bilinear filtering. Proc SPIE Int Soc Opt Eng 2296:317–327Google Scholar
  57. 57.
    Seth S, Ozturk MC, Principe JC (2007) Signal processing with echo state networks in the complex domain. In: Machine learning for signal processing 17—Proceedings of the 2007 IEEE signal processing society workshop, MLSP, pp 408–412Google Scholar
  58. 58.
    Cribari-Neto F (2004) Asymptotic inference under heteroskedasticity of unknown form. Comput Stat Data Anal 45(2):215–233MathSciNetzbMATHCrossRefGoogle Scholar
  59. 59.
    White H (1980) A heteroskedasticity-consistent covariance matrix and a direct test for heteroskedasticity. Econometrica 48:817–838MathSciNetzbMATHCrossRefGoogle Scholar
  60. 60.
    Vinckier Q, Duport F, Smerieri A, Haelterman M, Massar S (2016) Autonomous bio-inspired photonic processor based on reservoir computing paradigm. In: 2016 IEEE photonics society summer topical meeting series, SUM 2016, pp 183–184Google Scholar
  61. 61.
    Mathews E, Poigné A (2008) An Echo State Network based pedestrian counting system using wireless sensor networks. In: 2008 international workshop on intelligent solutions in embedded systems (WISES 2008), pp 1–14Google Scholar

Copyright information

© Springer-Verlag London Ltd., part of Springer Nature 2018

Authors and Affiliations

  • M. L. Alomar
    • 1
  • Erik S. Skibinsky-Gitlin
    • 1
  • Christiam F. Frasser
    • 1
  • Vincent Canals
    • 1
  • Eugeni Isern
    • 1
  • Miquel Roca
    • 1
  • Josep L. Rosselló
    • 1
  1. 1.Electronics Engineering Group, Department of PhysicsUniversity of Balearic IslandsPalma de MallorcaSpain

Personalised recommendations