Advertisement

Neural Processing Letters

, Volume 50, Issue 2, pp 1705–1733 | Cite as

Restricted Convolutional Neural Networks

  • Mehran MirkhanEmail author
  • Mohammad Reza Meybodi
Article
  • 122 Downloads

Abstract

In this paper, a new type of convolutional neural network is proposed which is inspired by cellular automata research. This model is referred to as “restricted convolutional neural network” and its characteristic is that the feature maps are not fully connected, i.e. each feature map is only connected to a small neighborhood of previous feature maps. First this model is formally defined. Then it is used for image classification. Two layerwise pretraining methods have been proposed, and some structural variations have been analyzed. The model is tested on both MNIST and CIFAR-10 datasets. Results suggest that this model in some cases can outperform a convolutional neural network with similar architecture.

Keywords

Restricted convolutional neural network Cellular automata Image classification Self-organizing map Boltzmann machine 

Notes

References

  1. 1.
    Baldi P (2012) Autoencoders, unsupervised learning, and deep architectures. In: Proceedings of ICML workshop on unsupervised and transfer learning, pp 37–49Google Scholar
  2. 2.
    Beigy H, Meybodi MR (2004) A mathematical framework for cellular learning automata. Adv Complex Syst 7(03n04):295–319MathSciNetCrossRefGoogle Scholar
  3. 3.
    Chua LO, Yang L (1988) Cellular neural networks: applications. IEEE Trans Circuits Syst 35(10):1273–1290MathSciNetCrossRefGoogle Scholar
  4. 4.
    de Korte A, Brouwers H (2013) A cellular automata approach to chemical reactions: 1 reaction controlled systems. Chem Eng J 228:172–178.  https://doi.org/10.1016/j.cej.2013.04.084 CrossRefGoogle Scholar
  5. 5.
    Gardner M (1970) Mathematical games: the fantastic combinations of john conways new solitaire game life. Sci Am 223(4):120–123CrossRefGoogle Scholar
  6. 6.
    Gehring WJ (2005) New perspectives on eye development and the evolution of eyes and photoreceptors. J Hered 96(3):171–184CrossRefGoogle Scholar
  7. 7.
    Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, CambridgezbMATHGoogle Scholar
  8. 8.
    Guo Y, Yao A, Chen Y (2016) Dynamic network surgery for efficient DNNs. In: Lee DD, Sugiyama M, Luxburg UV, Guyon I, Garnett R (eds) Advances in neural information processing systems 29. Curran Associates, Inc., pp 1379–1387Google Scholar
  9. 9.
    Han S, Pool J, Tran J, Dally W (2015) Learning both weights and connections for efficient neural network. In: Advances in neural information processing systems, pp 1135–1143Google Scholar
  10. 10.
    He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR. arXiv:1512.03385
  11. 11.
    Hinton G (2010) A practical guide to training restricted boltzmann machines. Momentum 9(1):926Google Scholar
  12. 12.
    Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554MathSciNetCrossRefGoogle Scholar
  13. 13.
    Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507MathSciNetCrossRefGoogle Scholar
  14. 14.
    Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward networks are universal approximators. Neural Netw 2(5):359–366.  https://doi.org/10.1016/0893-6080(89)90020-8 CrossRefzbMATHGoogle Scholar
  15. 15.
    Huberman B (1985) Probabilistic cellular automata. In: Nonlinear phenomena in physics. Springer, pp 129–137Google Scholar
  16. 16.
    Jump JR, Kirtane JS (1974) On the interconnection structure of cellular networks. Inf Control 24(1):74–91MathSciNetCrossRefGoogle Scholar
  17. 17.
    Kohonen T (1982) Self-organized formation of topologically correct feature maps. Biol Cybern 43(1):59–69MathSciNetCrossRefGoogle Scholar
  18. 18.
    Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images. https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf; https://www.cs.toronto.edu/~kriz/cifar.html
  19. 19.
    LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324CrossRefGoogle Scholar
  20. 20.
    LeCun Y, Cortes C, Burges CJ (2010) Mnist handwritten digit database. AT&T Labs (Online). http://yann.lecun.com/exdb/mnist
  21. 21.
    Margenstern M (2007) Cellular automata in hyperbolic spaces: theory, vol 1. Archives contemporainesGoogle Scholar
  22. 22.
    McCulloch WS, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5(4):115–133MathSciNetCrossRefGoogle Scholar
  23. 23.
    Molchanov P, Tyree S, Karras T, Aila T, Kautz J (2016) Pruning convolutional neural networks for resource efficient transfer learning. CoRR. arXiv:1611.06440
  24. 24.
    Osindero S, Hinton GE (2008) Modeling image patches with a directed hierarchy of Markov random fields. In: Platt JC, Koller D, Singer Y, Roweis ST (eds) Advances in neural information processing systems 20. Curran Associates, Inc., pp 1121–1128Google Scholar
  25. 25.
    O’Sullivan D (2001) Exploring spatial process dynamics using irregular cellular automaton models. Geogr Anal 33(1):1–18CrossRefGoogle Scholar
  26. 26.
    Rosenblatt F (1958) The perceptron: a probabilistic model for information storage and organization in the brain. Psychol Rev 65(6):386CrossRefGoogle Scholar
  27. 27.
    Sarkar P (2000) A brief history of cellular automata. ACM Comput Surv (CSUR) 32(1):80–107CrossRefGoogle Scholar
  28. 28.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9Google Scholar
  29. 29.
    Toffoli T, Margolus N (1987) Cellular automata machines: a new environment for modeling. MIT Press, CambridgeCrossRefGoogle Scholar
  30. 30.
    Von Neumann J, Burks AW et al (1966) Theory of self-reproducing automata. IEEE Trans Neural Netw 5(1):3–14Google Scholar
  31. 31.
    Wang L, Zhang J, Shao H (2014) Existence and global stability of a periodic solution for a cellular neural network. Commun Nonlinear Sci Numer Simul 19(9):2983–2992MathSciNetCrossRefGoogle Scholar
  32. 32.
    Werbos PJ (1974) Beyond regression: new tools for prediction and analysis in the behavioral sciences. Doctoral dissertation, Applied Mathematics, Harvard University, MAGoogle Scholar
  33. 33.
    Widrow B, Hoff ME (1960) Adaptive switching circuits. Technical report, Stanford University, Stanford Electronics LabsGoogle Scholar
  34. 34.
    Wolfram S (1994) Cellular automata and complexity: collected papers, vol 1. Addison-Wesley, ReadingzbMATHGoogle Scholar
  35. 35.
    Wolfram S (2002) A new kind of science, vol 5. Wolfram Media, ChampaignzbMATHGoogle Scholar
  36. 36.
    Yang T, Yang LB, Wu CW, Chua LO (1996) Fuzzy cellular neural networks: theory. In: 1996 4th IEEE international workshop on cellular neural networks and their applications, 1996. CNNA-96. Proceedings. IEEE, pp 181–186Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer Engineering and Information TechnologyAmirkabir University of TechnologyTehranIran

Personalised recommendations