Skip to main content
SpringerLink
Log in

Artificial Neural Network Models

  • Chapter
Springer Handbook of Computational Intelligence

Part of the book series: Springer Handbooks ((SHB))

Abstract

We outline the main models and developments in the broad field of artificial neural networks (GlossaryTerm

ANN

). A brief introduction to biological neurons motivates the initial formal neuron model – the perceptron. We then study how such formal neurons can be generalized and connected in network structures. Starting with the biologically motivated layered structure of GlossaryTerm

ANN

(feed-forward GlossaryTerm

ANN

), the networks are then generalized to include feedback loops (recurrent GlossaryTerm

ANN

) and even more abstract generalized forms of feedback connections (recursive neuronal networks) enabling processing of structured data, such as sequences, trees, and graphs. We also introduce GlossaryTerm

ANN

models capable of forming topographic lower-dimensional maps of data (self-organizing maps). For each GlossaryTerm

ANN

type we outline the basic principles of training the corresponding GlossaryTerm

ANN

models on an appropriate data collection.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 269.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 349.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Abbreviations

ANN:

artificial neural network

BPTT:

back-propagation through time

DAG:

directed acyclic graph

ESN:

echo state network

FPM:

fractal prediction machine

LSM:

liquid state machine

LSTM:

long short term memory

RBF:

radial basis function

RecNN:

recursive neural network

RNN:

recurrent neural network

RTRL:

real-time recurrent learning

SD:

structured data

SOM:

self-organizing map

SRN:

simple recurrent network

TDNN:

time delay neural network

References

  1. F. Rosenblatt: The perceptron, a probabilistic model for information storage and organization in the brain, Psychol. Rev. 62, 386–408 (1958)

    Article  Google Scholar 

  2. D.E. Rumelhart, G.E. Hinton, R.J. Williams: Learning internal representations by error propagation. In: Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Vol. 1 Foundations, ed. by D.E. Rumelhart, J.L. McClelland (MIT Press/Bradford Books, Cambridge 1986) pp. 318–363

    Google Scholar 

  3. J. Zurada: Introduction to Artificial Neural Systems (West Publ., St. Paul 1992)

    Google Scholar 

  4. K. Hornik, M. Stinchocombe, H. White: Multilayer feedforward networks are universal approximators, Neural Netw. 2, 359–366 (1989)

    Article  Google Scholar 

  5. D.J.C. MacKay: Bayesian interpolation, Neural Comput. 4(3), 415–447 (1992)

    Article  MATH  Google Scholar 

  6. S. Haykin: Neural Networks and Learning Machines (Prentice Hall, Upper Saddle River 2009)

    Google Scholar 

  7. C. Bishop: Neural Networks for Pattern Recognition (Oxford Univ. Press, Oxford 1995)

    MATH  Google Scholar 

  8. T. Sejnowski, C. Rosenberg: Parallel networks that learn to pronounce English text, Complex Syst. 1, 145–168 (1987)

    MATH  Google Scholar 

  9. A. Weibel: Modular construction of time-delay neural networks for speech recognition, Neural Comput. 1, 39–46 (1989)

    Article  MathSciNet  Google Scholar 

  10. J.L. Elman: Finding structure in time, Cogn. Sci. 14, 179–211 (1990)

    Article  Google Scholar 

  11. M.I. Jordan: Serial order: A parallel distributed processing approach. In: Advances in Connectionist Theory, ed. by J.L. Elman, D.E. Rumelhart (Erlbaum, Hillsdale 1989)

    Google Scholar 

  12. Y. Bengio, R. Cardin, R. DeMori: Speaker independent speech recognition with neural networks and speech knowledge. In: Advances in Neural Information Processing Systems II, ed. by D.S. Touretzky (Morgan Kaufmann, San Mateo 1990) pp. 218–225

    Google Scholar 

  13. P.J. Werbos: Generalization of backpropagation with application to a recurrent gas market model, Neural Netw. 1(4), 339–356 (1988)

    Article  Google Scholar 

  14. R.J. Williams, D. Zipser: A learning algorithm for continually running fully recurrent neural networks, Neural Comput. 1(2), 270–280 (1989)

    Article  Google Scholar 

  15. Y. Bengio, P. Simard, P. Frasconi: Learning long-term dependencies with gradient descent is difficult, IEEE Trans. Neural Netw. 5(2), 157–166 (1994)

    Article  Google Scholar 

  16. T. Lin, B.G. Horne, P. Tino, C.L. Giles: Learning long-temr dependencies with NARX recurrent neural networks, IEEE Trans. Neural Netw. 7(6), 1329–1338 (1996)

    Article  Google Scholar 

  17. S. Hochreiter, J. Schmidhuber: Long short-term memory, Neural Comput. 9(8), 1735–1780 (1997)

    Article  Google Scholar 

  18. M. Lukosevicius, H. Jaeger: Overview of Reservoir Recipes, Technical Report, Vol. 11 (School of Engineering and Science, Jacobs University, Bremen 2007)

    MATH  Google Scholar 

  19. A. Graves, M. Liwicki, S. Fernandez, R. Bertolami, H. Bunke, J. Schmidhuber: A novel connectionist system for improved unconstrained handwriting recognition, IEEE Trans. Pattern Anal. Mach. Intell. 31, 5 (2009)

    Article  Google Scholar 

  20. S. Hochreiter, M. Heusel, K. Obermayer: Fast model-based protein homology detection without alignment, Bioinformatics 23(14), 1728–1736 (2007)

    Article  Google Scholar 

  21. H. Jaeger, H. Hass: Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless telecommunication, Science 304, 78–80 (2004)

    Article  Google Scholar 

  22. W. Maass, T. Natschlager, H. Markram: Real-time computing without stable states: A new framework for neural computation based on perturbations, Neural Comput. 14(11), 2531–2560 (2002)

    Article  MATH  Google Scholar 

  23. P. Tino, G. Dorffner: Predicting the future of discrete sequences from fractal representations of the past, Mach. Learn. 45(2), 187–218 (2001)

    Article  MATH  Google Scholar 

  24. M.H. Tong, A. Bicket, E. Christiansen, G. Cottrell: Learning grammatical structure with echo state network, Neural Netw. 20, 424–432 (2007)

    Article  MATH  Google Scholar 

  25. K. Ishii, T. van der Zant, V. Becanovic, P. Ploger: Identification of motion with echo state network, Proc. OCEANS 2004 MTS/IEEE-TECHNO-OCEAN Conf., Vol. 3 (2004) pp. 1205–1210

    Google Scholar 

  26. L. Medsker, L.C. Jain: Recurrent Neural Networks: Design and Applications (CRC, Boca Raton 1999)

    Book  Google Scholar 

  27. J. Kolen, S.C. Kremer: A Field Guide to Dynamical Recurrent Networks (IEEE, New York 2001)

    Google Scholar 

  28. D. Mandic, J. Chambers: Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability (Wiley, New York 2001)

    Book  Google Scholar 

  29. J.B. MacQueen: Some models for classification and analysis if multivariate observations, Proc. 5th Berkeley Symp. Math. Stat. Probab. (Univ. California Press, Oakland 1967) pp. 281–297

    Google Scholar 

  30. M.D. Buhmann: Radial Basis Functions: Theory and Implementations (Cambridge Univ. Press, Cambridge 2003)

    Book  MATH  Google Scholar 

  31. G.-B. Huang, Q.-Y. Zhu, C.-K. Siew: Extreme learning machine: theory and applications, Neurocomputing 70, 489–501 (2006)

    Article  Google Scholar 

  32. T. Kohonen: Self-Organizing Maps, Springer Series in Information Sciences, Vol. 30 (Springer, Berlin, Heidelberg 2001)

    MATH  Google Scholar 

  33. T. Kohonen, E. Oja, O. Simula, A. Visa, J. Kangas: Engineering applications of the self-organizing map, Proc. IEEE 84(10), 1358–1384 (1996)

    Article  Google Scholar 

  34. T. Koskela, M. Varsta, J. Heikkonen, K. Kaski: Recurrent SOM with local linear models in time series prediction, 6th Eur. Symp. Artif. Neural Netw. (1998) pp. 167–172

    Google Scholar 

  35. T. Voegtlin: Recursive self-organizing maps, Neural Netw. 15(8/9), 979–992 (2002)

    Article  Google Scholar 

  36. M. Strickert, B. Hammer: Merge som for temporal data, Neurocomputing 64, 39–72 (2005)

    Article  Google Scholar 

  37. M. Hagenbuchner, A. Sperduti, A. Tsoi: Self-organizing map for adaptive processing of structured data, IEEE Trans. Neural Netw. 14(3), 491–505 (2003)

    Article  MATH  Google Scholar 

  38. A. Sperduti, A. Starita: Supervised neural networks for the classification of structures, IEEE Trans. Neural Netw. 8(3), 714–735 (1997)

    Article  Google Scholar 

  39. P. Frasconi, M. Gori, A. Sperduti: A general framework for adaptive processing of data structures, IEEE Trans. Neural Netw. 9(5), 768–786 (1998)

    Article  Google Scholar 

  40. B. Hammer, A. Micheli, A. Sperduti: Universal approximation capability of cascade correlation for structures, Neural Comput. 17(5), 1109–1159 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  41. A. Micheli: Neural network for graphs: A contextual constructive approach, IEEE Trans. Neural Netw. 20(3), 498–511 (2009)

    Article  MathSciNet  Google Scholar 

  42. B. Hammer, A. Micheli, A. Sperduti, M. Strickert: A general framework for unsupervised processing of structured data, Neurocomputing 57, 3–35 (2004)

    Article  Google Scholar 

  43. M. Hagenbuchner, A. Sperduti, A.-C. Tsoi: Graph self-organizing maps for cyclic and unbounded graphs, Neurocomputing 72(7–9), 1419–1430 (2009)

    Article  Google Scholar 

  44. Y. Bengio, Y. LeCun: Greedy Layer-Wise Training of Deep Network. In: Advances in Neural Information Processing Systems 19, ed. by B. Schölkopf, J. Platt, T. Hofmann (MIT Press, Cambridge 2006) pp. 153–160

    Google Scholar 

  45. D.C. Ciresan, U. Meier, L.M. Gambardella, J. Schmidhuber: Deep big simple neural nets for handwritten digit recognition, Neural Comput. 22(12), 3207–3220 (2010)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, University of Birmingham, Edgbaston, B15 2TT, Birmingham, UK

    Peter Tino

  2. Dep. Computer Science, University of Otago, 133 Union Street East, 9016, Dunedin, New Zealand

    Lubica Benuskova

  3. Dep. Pure and Applied Mathematics, University of Padova, Via Trieste, 63, 351 21, Padova, Italy

    Alessandro Sperduti

Authors
  1. Peter Tino
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lubica Benuskova
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alessandro Sperduti
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter Tino .

Editor information

Editors and Affiliations

  1. Systems Research Inst., Polish Academy of Sciences, ul. Newelska 6, 01-447, Warsaw, Poland

    Janusz Kacprzyk

  2. Dep. Electrical and Computer Engineering, University of Alberta, 116 Street 9107, T6J 2V4, Edmonton, Alberta, Canada

    Witold Pedrycz

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Tino, P., Benuskova, L., Sperduti, A. (2015). Artificial Neural Network Models. In: Kacprzyk, J., Pedrycz, W. (eds) Springer Handbook of Computational Intelligence. Springer Handbooks. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-43505-2_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-43505-2_27

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-43504-5

  • Online ISBN: 978-3-662-43505-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation