On the Interpretation and Characterization of Echo State Networks Dynamics: A Complex Systems Perspective

  • Filippo Maria Bianchi
  • Lorenzo Livi
  • Cesare Alippi
Chapter
First Online:
Part of the Studies in Computational Intelligence book series (SCI, volume 738)

Abstract

In this chapter, we discuss recently developed methods for characterizing the dynamics of recurrent neural networks. Such methods rely on theory and concepts coming from the field of complex systems. We focus on a class of recurrent networks called echo state networks. First, we present a method to analyze and characterize the evolution of its internal state. This allows to provide a qualitative interpretation of the network dynamics. In addition, it allows to assess the stability of the system, a necessary requirement in many practical applications. Successively, we focus on the identification of the onset of criticality in such networks. We discuss an unsupervised method based on Fisher information, which can be used to tune the network hyperparameters. With respect to standard supervised techniques, we show that the proposed approach offers several advantages and is effective on a number of tasks.

Keywords

Echo state networks Criticality Recurrence quantification analysis Fisher information matrix Unsupervised learning 
This is a preview of subscription content, log in to check access.

References

  1. 1.
    Aljadeff, J., Stern, M., Sharpee, T.: Transition to chaos in random networks with cell-type-specific connectivity. Phys. Rev. Lett. 114, 088101 (2015). doi: 10.1103/PhysRevLett.114.088101 CrossRefGoogle Scholar
  2. 2.
    Barzel, B., Barabási, A.-L.: Universality in network dynamics. Nat. Phys. 9(10), 673–681 (2013). doi: 10.1038/nphys2741 CrossRefGoogle Scholar
  3. 3.
    Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994). ISSN 1045-9227. doi: 10.1109/72.279181
  4. 4.
    Berisha, V., Hero, A.Q. III.: Empirical non-parametric estimation of the Fisher information. IEEE Signal Process. Lett. 22(7), 988–992 (2015). ISSN 1070-9908. doi: 10.1109/LSP.2014.2378514
  5. 5.
    Bertschinger, N., Natschläger, T.: Real-time computation at the edge of chaos in recurrent neural networks. Neural Comput. 16(7), 1413–1436 (2004). doi: 10.1162/089976604323057443 CrossRefMATHGoogle Scholar
  6. 6.
    Bianchi, F.M., Livi, L., Alippi, C.: Investigating echo state networks dynamics by means of recurrence analysis. IEEE Trans. Neural Netw. Learn. Syst. 1–13 (2016). doi: 10.1109/TNNLS.2016.2630802
  7. 7.
    Boedecker, J., Obst, O., Lizier, J.T., Mayer, N.M., Asada, M.: Information processing in echo state networks at the edge of chaos. Theory Biosci. 131(3), 205–213 (2012). doi: 10.1007/s12064-011-0146-8 CrossRefGoogle Scholar
  8. 8.
    Charles, A., Yin, D., Rozell, C.: Distributed sequence memory of multidimensional inputs in recurrent networks. arXiv:1605.08346 (2016)
  9. 9.
    De Arcangelis, L., Lombardi, F., Herrmann, H.J.: Criticality in the brain. J. Stat. Mech. Theory Exp. 2014(3), P03026 (2014). doi: 10.1088/1742-5468/2014/03/P03026 MathSciNetCrossRefGoogle Scholar
  10. 10.
    Enel, P., Procyk, E., Quilodran, R., Dominey, P.F.: Reservoir computing properties of neural dynamics in prefrontal cortex. PLoS Comput. Biol. 12(6), e1004967 (2016). doi: 10.1371/journal.pcbi.1004967 CrossRefGoogle Scholar
  11. 11.
    Eroglu, D., Peron, T.K.D.M., Marwan, N., Rodrigues, F.A., da Costa, L.F., Sebek, M., Kiss, I.Z., Kurths, J.: Entropy of weighted recurrence plots. Phys. Rev. E 90(4), 042919 (2014). doi: 10.1103/PhysRevE.90.042919
  12. 12.
    Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990). ISSN 0364-0213. doi: 10.1016/0364-0213(90)90002-E
  13. 13.
    Friedman, J.H., Rafsky, L.C.: Multivariate generalizations of the Wald-Wolfowitz and Smirnov two-sample tests. Ann. Stat. 7(4), 697–717 (1979)MathSciNetCrossRefMATHGoogle Scholar
  14. 14.
    Grigolini, P.: Emergence of biological complexity: criticality, renewal and memory. Chaos, Solitons Fractals (2015). doi: 10.1016/j.chaos.2015.07.025
  15. 15.
    Hammer, B., Micheli, A., Sperduti, A., Strickert, M.: Recursive self-organizing network models. Neural Netw. 17(8–9), 1061–1085 (2004). ISSN 0893-6080. doi: 10.1016/j.neunet.2004.06.009
  16. 16.
    Hidalgo, J., Grilli, J., Suweis, S., Muñoz, M.A., Banavar, J.R., Maritan, A.: Information-based fitness and the emergence of criticality in living systems. Proc. Natl. Acad. Sci. 111(28), 10095–10100 (2014). doi: 10.1073/pnas.1319166111 CrossRefGoogle Scholar
  17. 17.
    Hidalgo, J., Grilli, J., Suweis, S., Maritan, A., Muñoz, M.A.: Cooperation, competition and the emergence of criticality in communities of adaptive systems. J. Stat. Mech. Theory Exp. 2016(3), 033203 (2016). doi: 10.1088/1742-5468/2016/03/033203 MathSciNetCrossRefGoogle Scholar
  18. 18.
    Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks-with an erratum note. Bonn, Germany: German National Research Center for Information Technology GMD Technical Report, vol. 148, p. 34 (2001)Google Scholar
  19. 19.
    Lacasa, L., Nicosia, V., Latora, V.: Network structure of multivariate time series. Sci. Rep. 5, (2015). doi: 10.1038/srep15508
  20. 20.
    Langton, C.G.: Computation at the edge of chaos: Phase transitions and emergent computation. Phys. D Nonlinear Phenom. 42(1), 12–37 (1990). doi: 10.1016/0167-2789(90)90064-V MathSciNetCrossRefGoogle Scholar
  21. 21.
    Legenstein, R., Maass, W.: Edge of chaos and prediction of computational performance for neural circuit models. Neural Netw. 20(3), 323–334 (2007). doi: 10.1016/j.neunet.2007.04.017 CrossRefMATHGoogle Scholar
  22. 22.
    Livi, L., Bianchi, F.M., Alippi, C.: Determination of the edge of criticality in echo state networks through Fisher information maximization. IEEE Trans. Neural Netw. Learn. Syst. 1–12 (2017). doi: 10.1109/TNNLS.2016.2644268
  23. 23.
    Maass, W., Joshi, P., Sontag, E.D.: Computational aspects of feedback in neural circuits. PLoS Comput. Biol. 3(1), e165 (2007). doi: 10.1371/journal.pcbi.0020165.eor MathSciNetCrossRefGoogle Scholar
  24. 24.
    Magnus, J.R., Neudecker, H.: Matrix Differential Calculus with Applications in Statistics and Econometrics. Wiley, New York (1995)MATHGoogle Scholar
  25. 25.
    Manjunath, G., Jaeger, H.: Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks. Neural Comput. 25(3), 671–696 (2013). doi: 10.1162/NECO_a_00411 MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Marichal, R.L., Piñeiro, J.D.: Analysis of multiple quasi-periodic orbits in recurrent neural networks. Neurocomputing 162, 85–95 (2015). doi: 10.1016/j.neucom.2015.04.001 CrossRefGoogle Scholar
  27. 27.
    Marwan, N.: How to avoid potential pitfalls in recurrence plot based data analysis. Int. J. Bifurcat. Chaos 21(04), 1003–1017 (2011). doi: 10.1142/S0218127411029008 MathSciNetCrossRefMATHGoogle Scholar
  28. 28.
    Marwan, N., Kurths, J.: Line structures in recurrence plots. Phys. Lett. A 336(4), 349–357 (2005). doi: 10.1016/j.physleta.2004.12.056 CrossRefMATHGoogle Scholar
  29. 29.
    Marwan, N., Carmen, M., Thiel, R.M., Kurths, J.: Recurrence plots for the analysis of complex systems. Phys. Rep. 438(5), 237–329 (2007). doi: 10.1016/j.physrep.2006.11.001
  30. 30.
    Marwan, N., Schinkel, S., Kurths, J.: Recurrence plots 25 years later-Gaining confidence in dynamical transitions. EPL (Europhys. Lett.) 101(2), 20007 (2013). doi: 10.1209/0295-5075/101/20007 CrossRefGoogle Scholar
  31. 31.
    Massar, M., Massar, S.: Mean-field theory of echo state networks. Phys. Rev. E 87(4), 042809 (2013). doi: 10.1103/PhysRevE.87.042809 CrossRefMATHGoogle Scholar
  32. 32.
    Massobrio, P., de Arcangelis, L., Pasquale, V., Jensen, H.J., Plenz, D.: Criticality as a signature of healthy neural systems. Front. Syst. Neurosci. 9, 22 (2015). doi: 10.3389/fnsys.2015.00022 CrossRefGoogle Scholar
  33. 33.
    Mastromatteo, I., Marsili, M.: On the criticality of inferred models. J. Stat. Mech. Theory Exp. 2011(10), P10012 (2011). doi: 10.1088/1742-5468/2011/10/P10012 CrossRefGoogle Scholar
  34. 34.
    Mora, T., Bialek, W.: Are biological systems poised at criticality? J. Stat. Phys. 144(2), 268–302 (2011). doi: 10.1007/s10955-011-0229-4 MathSciNetCrossRefMATHGoogle Scholar
  35. 35.
    Mora, T., Deny, S., Marre, O.: Dynamical criticality in the collective activity of a population of retinal neurons. Phys. Rev. Lett. 114(7), 078105 (2015). doi: 10.1103/PhysRevLett.114.078105 CrossRefGoogle Scholar
  36. 36.
    Pascanu, R., Mikolov, T., Bengio, Y.: On the difficulty of training recurrent neural networks. arXiv:1211.5063 (2012)
  37. 37.
    Peng, Y., Lei, M., Li, J.-B., Peng, X.-Y.: A novel hybridization of echo state networks and multiplicative seasonal ARIMA model for mobile communication traffic series forecasting. Neural Comput. Appl. 24(3–4), 883–890 (2014)CrossRefGoogle Scholar
  38. 38.
    Prokopenko, M., Lizier, J.T., Obst, O., Wang, X.R.: Relating Fisher information to order parameters. Phys. Rev. E 84(4), 041116 (2011). doi: 10.1103/PhysRevE.84.041116 CrossRefGoogle Scholar
  39. 39.
    Rajan, K., Abbott, L.F., Sompolinsky, H.: Stimulus-dependent suppression of chaos in recurrent neural networks. Phys. Rev. E 82(1), 011903 (2010). doi: 10.1103/PhysRevE.82.011903 CrossRefGoogle Scholar
  40. 40.
    Reinhart, R.F., Steil, J.J.: Regularization and stability in reservoir networks with output feedback. Neurocomputing 90, 96–105 (2012). doi: 10.1016/j.neucom.2012.01.032 CrossRefGoogle Scholar
  41. 41.
    Roli, A., Villani, M., Filisetti, A., Serra, R.: Dynamical criticality: overview and open questions. arXiv:1512.05259 (2015)
  42. 42.
    Rumelhart, D.E., Smolensky, P., McClelland, J.L., Hinton, G.: Sequential thought processes in pdp models. V 2, 3–57 (1986)Google Scholar
  43. 43.
    Scheffer, M., Bascompte, J., Brock, W.A., Brovkin, V., Carpenter, S.R., Dakos, V., Held, H., Van Nes, E.H., Rietkerk, M., Sugihara, G.: Early-warning signals for critical transitions. Nature 461(7260), 53–59 (2009). doi: 10.1038/nature08227 CrossRefGoogle Scholar
  44. 44.
    Scheffer, M., Carpenter, S.R., Lenton, T.M., Bascompte, J., Brock, W., Dakos, V., van De Koppel, J., van De Leemput, I.A., Levin, S.A., van Nes, E.H., Pascual, M., Vandermeer, J.: Anticipating critical transitions. Science 338(6105), 344–348 (2012). doi: 10.1126/science.1225244 CrossRefGoogle Scholar
  45. 45.
    Schiller, U.D., Steil, J.J.: Analyzing the weight dynamics of recurrent learning algorithms. Neurocomputing 63, 5–23 (2005). doi: 10.1016/j.neucom.2004.04.006 CrossRefGoogle Scholar
  46. 46.
    Shen, Y., Wang, J.: An improved algebraic criterion for global exponential stability of recurrent neural networks with time-varying delays. IEEE Trans. Neural Netw. 19(3), 528–531 (2008). ISSN 1045-9227. doi: 10.1109/TNN.2007.911751
  47. 47.
    Steil, J.J.: Memory in backpropagation-decorrelation o(n) efficient online recurrent learning. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds.) Artificial Neural Networks: Formal Models and Their Applications-ICANN 2005, pp. 649–654. Springer, Berlin, Heidelberg (2005)Google Scholar
  48. 48.
    Sussillo, D.: Neural circuits as computational dynamical systems. Curr. Opin. Neurobiol. 25, 156–163 (2014). doi: 10.1016/j.conb.2014.01.008 CrossRefGoogle Scholar
  49. 49.
    Sussillo, D., Barak, O.: Opening the black box: low-dimensional dynamics in high-dimensional recurrent neural networks. Neural Comput. 25(3), 626–649 (2013). doi: 10.1162/NECO_a_00409 MathSciNetCrossRefMATHGoogle Scholar
  50. 50.
    Tiňo, P., Rodan, A.: Short term memory in input-driven linear dynamical systems. Neurocomputing 112, 58–63 (2013). doi: 10.1016/j.neucom.2012.12.041 CrossRefGoogle Scholar
  51. 51.
    Tkačik, G., Bialek, W.: Information processing in living systems. Ann. Rev. Condens. Matter Phys. 7(1), 89–117 (2016). doi: 10.1146/annurev-conmatphys-031214-014803 CrossRefGoogle Scholar
  52. 52.
    Tkačik, G., Mora, T., Marre, O., Amodei, D., Palmer, S.E., Berry, M.J., Bialek, W.: Thermodynamics and signatures of criticality in a network of neurons. Proc. Natl. Acad. Sci. 112(37), 11508–11513 (2015). doi: 10.1073/pnas.1514188112 CrossRefGoogle Scholar
  53. 53.
    Torres, J.J., Marro, J.: Brain performance versus phase transitions. Sci. Rep. 5 (2015). doi: 10.1038/srep12216
  54. 54.
    Toyoizumi, T., Abbott, L.F.: Beyond the edge of chaos: amplification and temporal integration by recurrent networks in the chaotic regime. Phys. Rev. E 84(5), 051908 (2011). doi: 10.1103/PhysRevE.84.051908 CrossRefGoogle Scholar
  55. 55.
    Toyoizumi, T., Aihara, K., Amari, S.-I.: Fisher information for spike-based population decoding. Phys. Rev. Lett. 97(9), 098102 (2006). doi: 10.1103/PhysRevLett.97.098102 CrossRefGoogle Scholar
  56. 56.
    Verstraeten, D., Schrauwen, B.: On the quantification of dynamics in reservoir computing. In: Artificial Neural Networks–ICANN 2009, pp. 985–994. Springer, Berlin (2009). doi: 10.1007/978-3-642-04274-4_101
  57. 57.
    Verstraeten, D., Schrauwen, B., D’Haene, M., Stroobandt, D.: An experimental unification of reservoir computing methods. Neural Netw. 20(3), 391–403 (2007). ISSN 0893-6080. doi: 10.1016/j.neunet.2007.04.003. Echo State Networks and Liquid State Machines
  58. 58.
    Wainrib, G., Touboul, J.: Topological and dynamical complexity of random neural networks. Phys. Rev. Lett. 110, 118101 (2013). doi: 10.1103/PhysRevLett.110.118101 CrossRefGoogle Scholar
  59. 59.
    Wang, X., Lizier, J., Prokopenko, M.: Fisher information at the edge of chaos in random boolean networks. Artif. Life 17(4), 315–329 (2011). ISSN 1064-5462. doi: 10.1162/artl_a_00041
  60. 60.
    Werbos, P.J.: Backpropagation: past and future. Proc. IEEE Int. Conf. Neural Netw. 1, 343–353 (1988). doi: 10.1109/ICNN.1988.23866 CrossRefGoogle Scholar
  61. 61.
    Yildiz, I.B., Jaeger, H., Kiebel, S.J.: Re-visiting the echo state property. Neural Netw. 35, 1–9 (2012). doi: 10.1016/j.neunet.2012.07.005 CrossRefMATHGoogle Scholar
  62. 62.
    Zegers, P.: Fisher information properties. Entropy 17(7), 4918–4939 (2015). doi: 10.3390/e17074918 MathSciNetCrossRefGoogle Scholar
  63. 63.
    Zhang, B., Miller, D.J., Wang, Y.: Nonlinear system modeling with random matrices: echo state networks revisited. IEEE Trans. Neural Netw. Learn. Syst. 23(1), 175–182 (2012). ISSN 2162-237X. doi: 10.1109/TNNLS.2011.2178562
  64. 64.
    Zhang, Y., Wang, J.: Global exponential stability of recurrent neural networks for synthesizing linear feedback control systems via pole assignment. IEEE Trans. Neural Netw. 13(3), 633–644 (2002). ISSN 1045-9227. doi: 10.1109/TNN.2002.1000129

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Filippo Maria Bianchi
    • 1
  • Lorenzo Livi
    • 2
    • 3
  • Cesare Alippi
    • 2
    • 3
  1. 1.Machine Learning Group, Department of Physics and TechnologyUiT the Arctic University of NorwayTromsøNorway
  2. 2.Department of Electronics, Information, and BioengineeringPolitecnico di MilanoMilanItaly
  3. 3.Faculty of InformaticsUniversità della Svizzera ItalianaLuganoSwitzerland

Personalised recommendations