Skip to main content
SpringerLink
Log in

Neural network equations and symbolic dynamics

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

In this paper we provide an up-to-date survey on the study of the complexity of the mosaic solutions on neural network equations. Three types of equations, namely, cellular neural networks (CNNs), multi-layer CNN (MCNNs) and inhomogeneous CNNs (ICNNs) are discuss herein. Such topic strong related to the learning algorithm and training process on neural network equations. Each neural network produces different mosaic solution space, and each mosaic solution space induces an different symbolic dynamics. To understand the complexity (spatial entropy) of the mosaic solution space for a given neural network equation, we need to identify which the underlying symbolic space is, then using the established knowledge of symbolic dynamical systems to compute its spatial entropy. Recently there has been substantial progress in this field. This paper is a comprehensive survey of this field. It provides a summary of the interesting results in this field. It is our hope that the paper will provide a good overview of major results and techniques, and a friendly entry point for anyone who is interested in studying problems in this field.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

Notes

  1. Such coupling is suggested by Chua and Roska. The purpose is to design MCNNs for solving some image processing and pattern recognition problems [25].

References

  1. Abram W, Lagarias JC (2012) Path sets and their symbolic dynamics. arXiv:1207.5004 [math.DS]

  2. Abram W, Lagarias JC (2013) p-adic path set fractals and arithemetics. arXiv:1210.2478 [math.MG]

  3. Adler RL, Marcus B (1979) Topological entropy and equivalence of dynamical systems. Memoris of American Mathematical Society, vol 219, Amarican Mathematical Society, Providence

  4. Akiduki T, Zhong Z, Imamura T, Miyake T (2009) Associative memories with multi-valued cellular neural networks and their application to disease diagnosis. In: IEEE international conference on systems, man and cybernetics, SMC 2009, pp 3824–3829

  5. Alecsandrescu I, Ungureanu P, Goras L (2008) Nonhomogeneous cnns and their use for texture classification. In: 9th symposium on IEEE neural network applications in electrical engineering, NEUREL 2008, pp 153–156

  6. Alsultanny YA, Aqul MM (2003) Pattern recognition using multilayer neural-genetic algorithm. Neurocomputing 51:237–247

    Article  MATH  Google Scholar 

  7. Andreyev Y, Belsky Y, Dmitriev A, Kuminov D (1992) Inhomogeneous cellular neural networks: possibility of functional device design. In: IEEE second international workshop on CNNA-92 proceedings cellular neural networks and their applications, pp 135–140

  8. Arena P, Baglio S, Fortuna L, Manganaro G (1998) Self-organization in a two-layer CNN. IEEE Trans Circuits Syst I Fundam Theory Appl 45:157–162

    Article  Google Scholar 

  9. Ayoub A, Tokes S (2005) A new cnn template for poac peak enhancement. In: Proceedings of the 2005 European conference on circuit theory and design, vol 1, pp 1–157

  10. Ban J-C, Chang C-H (2009) On the monotonicity of entropy for multi-layer cellular neural networks. Int J Bifurc Chaos Appl Sci Eng 19:3657–3670

    Article  MathSciNet  MATH  Google Scholar 

  11. Ban J-C, Chang C-H (2013) Inhomogeneous lattice dynamical systems and the boundary effect. Bound Value Probl 2013(1):249

    Article  MathSciNet  Google Scholar 

  12. Ban J-C, Chang C-H (2013) Solution structure of multi-layer neural networks with initial conditions, (submitted)

  13. Ban J-C, Chang C-H, Lin S-S (2012) The structure of multi-layer cellular neural networks. J Differ Equ 252:4563–4597

    Article  MathSciNet  MATH  Google Scholar 

  14. Ban J-C, Chang C-H, Lin S-S, Lin Y-H (2009) Spatial complexity in multi-layer cellular neural networks. J Differ Equ 246:552–580

    Article  MathSciNet  Google Scholar 

  15. Ban J-C, Hu W-G, Lin S-S, Lin Y-H (2013) Zeta functions for two-dimensional shifts of finite type. Memoirs of the American Mathematical Society 221, no.1037

    Google Scholar 

  16. Ban J-C, Lin S-S (2005) Patterns generation and transition matrices in multi-dimensional lattice models. Discret Contin Dyn Syst 13(3):637

    Article  MathSciNet  MATH  Google Scholar 

  17. Ban J-C, Lin S-S, Shih C-W (2001) Exact number of mosaic patterns in cellular neural networks. Int J Bifurc Chaos Appl Sci Eng 11:1645–1653

    Article  Google Scholar 

  18. Carmona R, Jimenez-Garrido F, Dominguez-Castro R, Espejo S, Rodriguez-Vazquez A (2002) CMOS realization of a 2-layer cnn universal machine chip. In: Proceedings of the 2002 7th IEEE international workshop on cellular neural networks and their applications, CNNA 2002, pp 444–451.

  19. Chow S-N, Mallet-Paret J (1995) Pattern formation and spatial chaos in lattice dynamical systems: I and II. IEEE Trans Circuits Syst I Fundam Theory Appl 42:746–756

    Article  MathSciNet  Google Scholar 

  20. Chow S-N, Mallet-Paret J, Van Vleck ES (1996) Dynamics of lattice differential equations. Int J Bifurc Chaos Appl Sci Eng 6:1605–1621

    Article  MATH  Google Scholar 

  21. Chow S-N, Mallet-Paret J, Van Vleck ES (1996) Pattern formation and spatial chaos in spatially discrete evolution equations. Random Comput Dyn 4:109–178

    MATH  Google Scholar 

  22. Chua LO (1998) Cnn: a paradigm for complexity. World scientific series on nonlinear science, series A, 31. World Scietific, Singapore

  23. Chua LO, Roska T (2002) Cellular neural networks and visual computing. Cambridge University Press, Cambridge

    Book  Google Scholar 

  24. Chua, LO Shi B-E (1991) Multiple layer cellular neural networks: a tutorial. In: Deprettere EF, van der Veen A-F (eds) Algorithms and parallel VLSI architectures, pp 137–168. Elsevier, Amsterdam

  25. Chua LO, Yang L (1988) Cellular neural networks: applications. IEEE Trans Circuits Syst 35:1273–1290

    Article  MathSciNet  Google Scholar 

  26. Chua LO, Yang L (1988) Cellular neural networks: theory. IEEE Trans Circuits Syst 35:1257–1272

    Article  MathSciNet  MATH  Google Scholar 

  27. Civalleri PP, Gilli M, Pandolfi L (1993) On stability of cellular neural networks with delay. In: IEEE transactions on circuits and systems I: fundamental theory and applications 40(3):157–165

  28. Crounse KR, Chua LO (1995) Methods for image processing and pattern formation in cellular neural networks: a tutorial. IEEE Trans Circuits Syst 42:583–601

    Article  MathSciNet  Google Scholar 

  29. Crounse KR, Roska T, Chua LO (1993) Image halftoning with cellular neural networks. IEEE Trans Circuits Syst 40:267–283

    Article  MATH  Google Scholar 

  30. Dahl GE, Yu D, Deng L, Acero A (2012) Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. In: IEEE transactions on audio, speech, and language processing, 20(1):30–42

  31. Dawes JHP (2008) Localised pattern formation with a large-scale mode: slanted snaking. SIAM J Appl Dyn Syst 7:186–206

    Article  MathSciNet  MATH  Google Scholar 

  32. Dawes JHP, Lilley S (2010) Localized states in a model of pattern formation in a vertically vibrated layer. SIAM J Appl Dyn Syst 9:238–260

    Article  MathSciNet  Google Scholar 

  33. Doebeli M, Hauert C, Killingback T (2004) The evolutionary origin of cooperators and defectors. Science 306:859–862

    Article  Google Scholar 

  34. Doebeli M, Killingback T (2003) Metapopulation dynamics with quasi-local competition. Theor Popul Biol 64:397–416

    Article  MATH  Google Scholar 

  35. Dominguez-Castro R, Espejo S, Rodriguez-Vazquez A, Carmona R (1994) A cnn universal chip in cmos technology. In: IEEE proceedings of the third ieee international workshop on cellular neural networks and their applications, CNNA-94, pp 91–96

  36. Einsiedler ML, Ward T (2011) Ergodic theory: with a view towards number theory, vol 259. Springer, Berlin

  37. Espejo S, Carmona R, Dominguez-Castro R, Rodrigúez-Vázquez A (1996) A cnn universal chip in cmos technology. Int J Circuit Theory Appl 24(1):93–109

    Article  Google Scholar 

  38. Espejo S, Carmona R, Domínguez-Castro R, Rodriguez-Vazquez A (1996) A vlsi-oriented continuous-time cnn model. Int J Circuit Theory Appl 24(3):341–356

    Article  Google Scholar 

  39. Fan A-H, Liao L, Ma J-H (2012) Level sets of multiple ergodic averages. Monatsh Math 168:17–26

    Article  MathSciNet  MATH  Google Scholar 

  40. Fukushima K (2013) Artificial vision by multi-layered neural networks: neocognitron and its advances. Neural Netw 37:103–119

    Article  MathSciNet  Google Scholar 

  41. Fukushima K (2013) Training multi-layered neural network neocognitron. Neural Netw 40:18–31

    Article  MATH  Google Scholar 

  42. Furstenberg H (1977) Ergodic behavior of diagonal measures and a theorem of szemeredi on arithmetic progressions. J Anal Math 31:204–256

    Article  MathSciNet  MATH  Google Scholar 

  43. Golubitsky M, Stewart I (2002) The symmetry perspective: from equilibrium to chaos in phase space and physical space. Progress in Mathematics, vol 200. Birkhauser, Basel

  44. Golubitsky M, Stewart I, Török A (2005) Patterns of symmetry in coupled cell networks with multiple arrows. SIAM J Appl Dyn Syst 4:78–100

    Article  MathSciNet  MATH  Google Scholar 

  45. Hauert C, Doebeli M (2004) Spatial structure often inhibits the evolution of cooperation in the snowdrift game. Nature 428:643–646

    Article  Google Scholar 

  46. Hinton G, Deng L, Yu D, Dahl GE, Mohamed A-R, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath TN et al (2012) Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE signal processing magazine 29(6):82–97

    Google Scholar 

  47. Hsu C-H, Juang J, Lin S-S, Lin W-W (2000) Cellular neural networks: local patterns for general template. Int J Bifurc Chaos Appl Sci Eng 10:1645–1659

    Article  MathSciNet  MATH  Google Scholar 

  48. Hsu C-H, Yang T-H (2002) Abundance of mosaic patterns for cnn with spatial variant templates. Int J Bifurc Chaos Appl Sci Eng 12:1321–1332

    Article  MathSciNet  Google Scholar 

  49. Itoh M, Julián P, Chua LO (2001) Rtd-based cellular neural networks with multiple steady states. Int J Bifurc Chaos 11(12):2913–2959

    Article  MATH  Google Scholar 

  50. Juang J, Lin S-S (2000) Cellular neural networks: mosaic pattern and spatial chaos. SIAM J Appl Math 60:891–915

    Article  MathSciNet  MATH  Google Scholar 

  51. Kamio T, Fujisaka H, Morisue M (2002) Associative memories using interaction between multilayer perceptrons and sparsely interconnected neural networks. IEICE Trans Fundam Electron Commun Comput Sci 85(6):1220–1228

    Google Scholar 

  52. Kamio T, Morisue M (2003) A synthesis procedure for associative memories using cellular neural networks with space-invariant cloning template library. In: IEEE proceedings of the international joint conference on neural networks, vol 2, pp 885–890

  53. Kanagawa A, Kawabata H, Takahashi H (1996) Cellular neural networks with multiple-valued output and its application. IEICE Trans Fundam Electron Commun Comput Sci 79(10):1658–1663

    Google Scholar 

  54. Killingback T, Loftus G, Sundaram B (2012) Competitively coupled maps and spatial pattern formation. arXiv:1204.2463

  55. Lin S-S, Shih C-W (1999) Complete stability for standard cellular neural networks. Int J Bifurc Chaos Appl Sci Eng 9:909–918

    Article  MathSciNet  MATH  Google Scholar 

  56. Lin S-S, Yang T-S (2000) Spatial entropy of one-dimensional cellular neural network. Int J Bifurc Chaos 10(09):2129–2140

    Article  MathSciNet  MATH  Google Scholar 

  57. Lind D, Marcus B (1995) An introduction to symbolic dynamics and coding. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  58. Mohamed A-R, Yu D, Deng L (2010) Investigation of full-sequence training of deep belief networks for speech recognition. In: INTERSPEECH, pp 2846–2849

  59. Murugesh V (2010) Image processing applications via time-multiplexing cellular neural network simulator with numerical integration algorithms. Int J Comput Math 87:840–848

    Article  MathSciNet  MATH  Google Scholar 

  60. Roska T, Chua LO (1993) The cnn universal machine: an analogic array computer. In: IEEE transactions on circuits and systems II: analog and digital signal processing, 40(3):163–173

  61. Roska T, Wu C-W, Balsi M, Chua LO (1992) Stability and dynamics of delay-type general and cellular neural networks. In: IEEE transactions on circuits and systems I: fundamental theory and applications, 39(6):487–490

  62. Roska T, Wu C-W, Chua LO (1993) Stability of cellular neural networks with dominant nonlinear and delay-type templates. In: IEEE Transactions on circuits and systems I: fundamental theory and applications, 40(4):270–272

  63. Seide F, Li G, Chen X, Yu D (2011) Feature engineering in context-dependent deep neural networks for conversational speech transcription. In: IEEE Workshop on automatic speech recognition and understanding (ASRU), pp 24–29

  64. Seide F, Li G, Yu D (2011) Conversational speech transcription using context-dependent deep neural networks. In: INTERSPEECH, pp 437–440.

  65. Shih C-W (2001) Complete stability for a class of cellular neural networks. Int J Bifurc Chaos Appl Sci Eng 11:169–178

    Article  MATH  Google Scholar 

  66. Stewart I (2004) Networking opportunity. Nature 427:601–604

    Article  Google Scholar 

  67. Stewart I, Golubitsky M, Pivato M (2003) Symmetry groupoids and patterns of synchrony in coupled cell networks. SIAM J Appl Dyn Syst 2:609–646

    Article  MathSciNet  MATH  Google Scholar 

  68. Török L, Roska T (2004) Stability of multi-layer cellular neural/nonlinear networks. Int J Bifurc Chaos Appl Sci Eng 14:3567–3586

    Article  MATH  Google Scholar 

  69. Walters P (1982) An introduction to ergodic theory. Springer, New York

    Book  MATH  Google Scholar 

  70. Xavier de Souza S, Yalcin ME, SuykensJAK, Vandewalle J (2004) Toward CNN chip-specific robustness. In: IEEE transactions on circuits and systems I: regular papers 51:892–902

  71. Yang L, Chua LO, Krieg KR (1990) VLSI implementation of cellular neural networks. In: IEEE international symposium on circuits and systems, pp 2425–2427

  72. Yang T, Yang L-B (1996) The global stability of fuzzy cellular neural network. In: IEEE transactions on circuits and systems I: fundamental theory and applications, 43(10):880–883

  73. Yang Z, Nishio Y, Ushida A (2001) A two layer cnn in image processing applications. In: Proceedings of the 2001 international symposium on nonlinear theory and its applications, pp 67–70

  74. Yang Z, Nishio Y, Ushida A (2002) Image processing of two-layer CNNs: applications and their stability. IEICE Trans Fundam E85–A:2052–2060

    Google Scholar 

  75. Yokozama M, Kubota Y, Hara T (1998) Effects of competition mode on spatial pattern dynamics in plant communities. Ecol Model 106:1–16

    Article  Google Scholar 

  76. Yokozawa M, Kubota Y, Hara T (1999) Effects of competition mode on the spatial pattern dynamics of wave regeneration in subalpine tree stands. Ecol Model 118:73–86

    Article  Google Scholar 

  77. Yu D, Seide F, Li G, Deng L (2012) Exploiting sparseness in deep neural networks for large vocabulary speech recognition. In: IEEE International conference on acoustics, speech and signal processing (ICASSP), pp 4409–4412

  78. Yu D, Seltzer ML (2011) Improved bottleneck features using pretrained deep neural networks. In: INTERSPEECH, pp 237–240

  79. Zhang Z, Akiduki T, Miyake T, Imamura T (2006) Design of multi-valued cellular neural networks for associative memory. In: IEEE international joint conference on SICE-ICASE, pp 4057–4061

  80. Zhang Z, Liu Z-Q, Kawabata H (1999) Tri-output cellular neural network and its application to diagnosing liver diseases. In: IEEE SMC’99 conference proceedings. 1999 IEEE international conference on systems, man, and cybernetics, vol 3, pp 372–377

  81. Zhang Z, Taniai R, Akiduki T, Imamura T, Miyake T (2009) A new design method of multi-valued cellular neural networks for associative memory. In: IEEE 2009 fourth international conference on innovative computing, information and control (ICICIC), pp 1562–1565

Download references

Acknowledgments

The author thanks for the anonymous referees’ valuable opinions. The suggestions improve this paper and motivate some further works.

Author information

Authors and Affiliations

  1. Department of Applied Mathematics, National Dong Hwa University, Hualien, 970003, Taiwan, ROC

    Jung-Chao Ban

Authors
  1. Jung-Chao Ban
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jung-Chao Ban.

Additional information

Ban is partially supported by the National Science Council, ROC (Contract No. NSC 100-2115-M-259-009-MY2).

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ban, JC. Neural network equations and symbolic dynamics . Int. J. Mach. Learn. & Cyber. 6, 567–579 (2015). https://doi.org/10.1007/s13042-014-0244-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-014-0244-2

Keywords

Search

Navigation