Artificial neural networks for pattern recognition

Abstract

This tutorial article deals with the basics of artificial neural networks (ANN) and their applications in pattern recognition. ANN can be viewed as computing models inspired by the structure and function of the biological neural network. These models are expected to deal with problem solving in a manner different from conventional computing. A distinction is made between pattern and data to emphasize the need for developing pattern processing systems to address pattern recognition tasks. After introducing the basic principles of ANN, some fundamental networks are examined in detail for their ability to solve simple pattern recognition tasks. These fundamental networks together with the principles of ANN will lead to the development of architectures for complex pattern recognition tasks. A few popular architectures are described to illustrate the need to develop an architecture specific to a given pattern recognition problem. Finally several issues that still need to be addressed to solve practical problems using ANN approach are discussed.

This is a preview of subscription content, access via your institution.

References

  1. Abu-Mostafa Y S, St. Jaques J M 1985 Information capacity of the Hopfield model.IEEE Trans. Inf. Theor. 31: 461–464

    MATH  Article  Google Scholar 

  2. Ackley D M, Hinton G E, Sejnowski T J 1985 A learning algorithm for Boltzmann machines.Cogn. Sci. 9: 147–169

    Article  Google Scholar 

  3. Ahalt S C, Krishnamurthy A K, Chen P, Melton D E 1990 Competitive learning algorithms for vector quantization.Neural Networks 3: 277–290

    Article  Google Scholar 

  4. Aleksander I, Morton H 1990An introduction to neural computing (London: Chapman and Hall)

    Google Scholar 

  5. Bhat N, McAvoy T 1989 Use of neural nets for dynamic modelling and control of chemical process systems.Proc. Am. Autom. Contr. Conf., Pittsburgh, PA, pp. 1342–1348

  6. Bienenstock E, von der Malsburg Ch 1987 A neural network for the retrieval of superimposed connection patterns.Euro. Phys. Lett. 3: 1243–1249

    Article  Google Scholar 

  7. Bottou L, Soulie F F, Blanchet P, Lienard J S 1990 Speaker independent isolated digit recognition: multilayer perceptrons vs. dynamic time warping.Neural Networks 3: 436–465

    Article  Google Scholar 

  8. Carpenter G A 1989 Neural network models for pattern recognition and associative memory.Neural networks 2: 138–152

    Google Scholar 

  9. Carpenter G A, Grossberg S 1987 art2: Self-organization of stable category recognition codes for analog input patterns.Appl. Opt. 26: 4919–4930

    Google Scholar 

  10. Carpenter G A, Grossberg S 1988 The art of adaptive pattern recognition by a self-organizing neural network.IEEE Comput. 21: 77–88

    Google Scholar 

  11. Cohen M, Grossberg S 1983 Absolute stability of global pattern formation and parallel storage by competitive neural networks.IEEE Trans. Syst., Man Cybern. SMC-13: 815–825

    MathSciNet  Google Scholar 

  12. Cole R, Fanty M, Muthuswamy Y, Gopalakrishna M 1992 Speaker-independent recognition of spoken English letters.Proc. Int. Joint Conf. Neural Networks, San Diego, ca

  13. Collins E, Ghosh S, Scotfield C L 1988 An application of a multiple neural network learning system to emulation of mortgage underwriting judgements.IEEE Int. Conf. Neural Networks (Piscataway, NJ: IEEE Press) 2: 459–466

    Google Scholar 

  14. Cybenko G 1989 Continuous value neural networks with two hidden layers are sufficient.Math. Control. Signal Syst. 2: 303–314

    MATH  Article  MathSciNet  Google Scholar 

  15. Desai M S 1990Noisy pattern retrieval using associative memories. MSEE thesis, University of Louisville, Kentucky

    Google Scholar 

  16. Deuker J, Schwartz D, Wittner B, Solla S, Howard R, Jackel L, Hopfield J 1987 Large automatic learning, rule extraction, and generalization.Complex Syst. 1: 877–922

    Google Scholar 

  17. Dotsenko V S 1988 Neural networks: translation-, rotation- and scale invariant pattern recognitionJ. Phys. A21: L783-L787

    MathSciNet  Google Scholar 

  18. Dutta S, Shekkar S 1988 Bond rating: a non-conservative application of neural networks.IEEE Int. Conf. Neural Networks (Piscataway, NJ: IEEE Press) 2: 443–450

    Google Scholar 

  19. Freeman J A, Skupura D M 1991Neural network algorithms, applications and programming techniques (New York: Addison-Wesley)

    Google Scholar 

  20. Fukushima K 1975 Cognitron: A self-organizing multilayer neural network.Biol. Cybern. 20: 121–136

    Article  Google Scholar 

  21. Fukushima K 1988 A neural network for visual pattern recognition.IEEE Comput. 21: 65–75

    MathSciNet  Google Scholar 

  22. Fukushima K 1991 Handwritten alphanumeric character recognition by the neocognitron.IEEE Trans. Neural Networks 2: 355–365

    Article  Google Scholar 

  23. Fukushima K, Miyake S 1982 Neocognitron: a new algorithm for pattern recognition tolerant of deformations and shifts in position.Pattern Recogn. 15: 455–469

    Article  Google Scholar 

  24. Geman S, Geman D 1984 Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images.IEEE Trans. Pattern Anal. Machine Intell. PAMI-6: 721–741

    Google Scholar 

  25. Gorman R, Sejnowski T 1988 Learned classification of sonar targets using a massively parallel network.IEEE Trans. Acoust. Speech Signal Process. 36: 1135–1140

    MATH  Article  Google Scholar 

  26. Grossberg S 1969 Some networks that can learn, and reproduce any number of complicated space-time patterns.Int. J. Math. Mech. 19: 53–91

    MATH  MathSciNet  Google Scholar 

  27. Grossberg S 1980 How does a brain build a cognitive code?Psychol. Rev. 87: 1–51

    Article  Google Scholar 

  28. Grossberg S 1982Studies of mind & brain (Boston: Reidel)

    Google Scholar 

  29. Grossberg S 1988 Nonlinear neural networks: Principles, mechanisms, and architecture.Neural networks 1: 17–61

    Article  Google Scholar 

  30. Handleman D H, Lane S H, Gelfand J J 1990 Integrating neural networks and knowledge-based systems for intelligent robotic control.IEEE Control Syst. Mag. 10(3): 77–87

    Article  Google Scholar 

  31. Hassoun M H 1989 Dynamic heteroassociative memories.Neural Networks 2: 275–287

    Article  Google Scholar 

  32. Hebb D 1949Organization of the behaviour (New York: Wiley)

    Google Scholar 

  33. Hecht-Nielson R 1987 Counterpropagation networks.Appl. Opt. 26: 4979–4984

    Google Scholar 

  34. Hecht-Nielson R 1990Neurocomputing (Reading, MA: Addison-Wesley)

    Google Scholar 

  35. Hertz J, Krogh A, Richard G P 1991Introduction to the theory of neural computation (New York: Addison-Wesley)

    Google Scholar 

  36. Hinton G E, Sejnowski T J 1986 Learning and relearning in Boltzmann machines. InParallel distributed processing: Explorations in the microstructure of cognition (eds) D E Rumelhart, J L McClelland (Cambridge, MA: MIT Press) 1: 282–317

    Google Scholar 

  37. Hodgkin A L, Huxley A F 1952 A quantitative description of membrane current and its application to conduction and excitation in nerve.J. Physiol. 117: 500–544

    Google Scholar 

  38. Hopfield J J 1982 Neural networks and physical systems with emergent collective computational capabilities.Proc. Natl. Acad. Sci. (USA) 79: 2554–2558

    Article  MathSciNet  Google Scholar 

  39. Hopfield J J, Tank D W 1985 Neural computation of decisions in optimization problems.Biol. Cybern. 52: 141–154

    MATH  MathSciNet  Google Scholar 

  40. Huang Z, Kuh A 1992 A combined self-organizing feature map and multilayer perceptron for isolated word recognition.IEEE Trans. Signal Process. 40: 2651–2657

    Article  Google Scholar 

  41. Hush D R, Horne B G 1993 Progress in supervised neural networks.IEEE Signal Process. Mag. 10: 8–39

    Article  Google Scholar 

  42. Kamp Y, Hasler M 1990Recursive neural networks for associative memory (Chichester: John Wiley & Sons)

    Google Scholar 

  43. Kennedy M P, Chau L O 1988 Neural networks for nonlinear programming.IEEE Trans. Circuits Syst. CAS-35: 554–562

    Article  Google Scholar 

  44. Kirkpatrick S, Gelatt C D Jr, Vecchi M P 1983 Optimization by simulated annealing.Science 220: 671–680

    Article  MathSciNet  Google Scholar 

  45. Kohonen T 1988 An introduction to neural computing.Neural Networks 1: 3–16

    Article  Google Scholar 

  46. Kohonen T 1989Self-organization and associative memory (3rd edn) (Berlin: Springer-Verlag)

    Google Scholar 

  47. Kohonen T 1990 The self-organizing map.Proc. IEEE 78: 1464–1480

    Article  Google Scholar 

  48. Konishi M, Otsuka Y, Matsuda K, Tamura N, Fuki A, Kadoguchi K 1990 Application of a neural network to operation guidance in a blast furnace.3rd European Seminar on Neural Computing: The Marketplace, London

  49. Kosko B 1988 Bidirectional associative memories.IEEE Trans. Syst., Man Cybern. 18: 49–60

    Article  MathSciNet  Google Scholar 

  50. Kosko B 1990 Unsupervised learning in noise.IEEE Trans. Neural Networks 1: 44–57

    Article  Google Scholar 

  51. Kosko B 1992Neural networks and fuzzy systems (Englewood Cliffs, NJ: Prentice-Hall)

    Google Scholar 

  52. Krzyzak A, Dali W, Yuen C Y 1990 Unconstrained handwritten character classification using modified back propagation model. InFrontiers in handwriting recognition (ed.) C Y Suen (Montreal: CENPARMI)

    Google Scholar 

  53. Kung S Y, Hwang J N 1989 Neural network architectures for robotic applications.IEEE Trans. Robotics Autom. 5: 641–657

    Article  Google Scholar 

  54. Kuperstein M, Wang J 1990 Neural controller for adaptive movements with unforeseen payloads.IEEE Trans. Neural Networks 1(1): 137–142

    Article  Google Scholar 

  55. Lang K J, Waibel A H, Hinton G E 1990 A time-delay neural network architecture for isolated word recognition.Neural Networks 3 (1): 23–44

    Article  Google Scholar 

  56. LeCun Y, Boser B, Denker J S, Henderson D, Howard R E, Hubbard W, Jackel L D 1989 Back propagation applied to handwritten zip code recognition.Neural Comput. 1:541–551

    Google Scholar 

  57. Lippmann R P 1987 An introduction to computing with neural nets.IEEE Trans. Acoust. Speech Signal Process. Mag. (April): 4–22

  58. Lippmann R P 1989a Review of neural networks for speech recognition.Neural Comput. 1(1): 1–38

    Article  Google Scholar 

  59. Lippmann R P 1989b Pattern classification using neural networks.IEEE Commun. Mag. (Nov): 47–64

  60. Lisboa P G P 1992Neural networks current applications (London: Chapman & Hall)

    Google Scholar 

  61. Maa C Y, Chin C, Shanblatt M A 1990 A constrained optimization neural net techniques for economic power dispatch.Proc. 1990 (New York: IEEE Press)

    Google Scholar 

  62. Marcus A, van Dam A 1991 User-interface developments for the nineties.IEEE Comput. 24: 49–57

    Google Scholar 

  63. McCulloch W S, Pitts W 1943 A logical calculus of the ideas immanent in nervous activity.Bull. Math. Biophys. 5: 115–133

    MATH  Article  MathSciNet  Google Scholar 

  64. Michel A N, Farrell J A 1990 Associative memories via artificial neural networks.IEEE Control Syst. Mag. (April): 6–17

  65. Minsky M, Papert S A 1988Perceptron (Cambridge, MA: MIT Press)

    Google Scholar 

  66. Muller B, Reinhardt J 1990Neural networks: An introduction (Berlin: Springer-Verlag)

    Google Scholar 

  67. Murakami K, Aibara T 1987 An improvement on the Moore-Penrose generalized inverse associative memory.IEEE Trans. Syst. Man. Cybern. SMC 17: 699–706

    Article  MathSciNet  Google Scholar 

  68. Naidu S R, Zafiriou E, McAvoy T J 1990 Use of neural networks for sensor failure detection in a control system.IEEE Control Syst. Mag. 10 (3): 49–55

    Article  Google Scholar 

  69. Nasrabadi N M, King R A 1988 Image coding using vector quantization: A review.IEEE Trans. Commun. 36: 957–971

    Article  Google Scholar 

  70. Naylor J, Li K P 1988 Analysis of a neural network algorithm for vector quantization of speech parameters.Neural Networks 1 (Suppl): 310

    Google Scholar 

  71. Pal S K, Mitra S 1992 Multilayer perceptron, fuzzy sets, and classification.IEEE Trans. Neural Networks 3: 683–697

    Article  Google Scholar 

  72. Raghu P P, Chouhan H M, Yegnanarayana B 1993 Multispectral image classification using neural network.Proc. Natl. Conf. on Neural Networks, Anna University, Madras (NCNN): 1–10

    Google Scholar 

  73. Rauch H E, Winarske T 1988 Neural networks for routing communications traffic.IEEE Control Syst. Mag. (April): 26–31

  74. Reggia J A, Suttonn G G 1988 III. Self-processing networks and their biomedical implications.Proc. IEEE 76: 680–692

    Article  Google Scholar 

  75. Rosenblatt F 1958 A probabilistic model for information storage and organization in the brain.Psychol. Rev. 65: 386–408

    Article  MathSciNet  Google Scholar 

  76. Rosenblatt F 1962Principles of neurodynamics (Washington, DC: Spartan)

    Google Scholar 

  77. Rumelhart D, McClelland J 1986Parallel distributed processing: Explorations in the microstructure of cognition (Boston: MIT Press) vol. 1

    Google Scholar 

  78. Rumelhart D E, Zipser D 1986 Feature discovery by competitive learning.Parallel and distributed processing (eds) J L McClelland, D E Rumelhart 1: 151-193

  79. Scalia F, Marconi L, Ridella S, Arrigo P, Mansi C, Mela G S 1988 An example of back propagation: diagnosis of dyspepsia.Ist IEE Conf. Neural Networks (IEE Conf. Publ.) 313: 332–540

    Google Scholar 

  80. Schalkoft R 1992Pattern recognition — Statistical, structural and neural approaches (New York: John Wiley & Sons)

    Google Scholar 

  81. Seibert M, Waxman A 1989 Spreading activation layers, visual saccades, and invariant representations for neural pattern recognition systems.Neural Networks 2: 9–27

    Article  Google Scholar 

  82. Sejnowski T, Rosenberg C 1987 Parallel networks that learn to pronounce English text.Complex Syst. 1: 145–168

    MATH  Google Scholar 

  83. Shea P M, Lin V 1989 Detection of explosives in checked airline baggage using an artificial neural system.Int. Joint. Conf. on Neural Networks 2: 31–34

    Article  Google Scholar 

  84. Simpson K P 1990Artificial neural systems (New York: Pergamon)

    Google Scholar 

  85. Simpson K P 1992Foundations of neural networks in artificial neural networks (eds) Edgar Sanchez-Sinencio, Clifford Lau (New York: IEEE Press)

    Google Scholar 

  86. Szu H 1986 Fast simulated annealing. InNeural networks for computing (ed.) J S Denker (New York: Snowbird)

    Google Scholar 

  87. Tagliarini G A, Page E W 1988 A neural network solution to the concentrator assignment problem.Neural information processing systems (ed.) D Z Anderson (New York: Am. Inst. Phys.)

    Google Scholar 

  88. von der Malsburg Ch 1973 Self-organization of orientation sensitive cells in the striate cortex.Kybernetik 14: 85–100

    Article  Google Scholar 

  89. Waibel A 1989 Modular construction of time-delay neural networks for speech recognition.Neural Comput. 1: 39–46

    Google Scholar 

  90. Wasserman P D 1988 Combined backpropagation/cauchy machine.Neural networks: Abstracts of the first INNS Meeting, Boston (Elmsford, NY: Pergamon) 1: 556

    Google Scholar 

  91. White H 1988 Economic prediction using neural networks: the case of IBM daily stock returns.Neural networks: Abstracts of the First INNS Meeting, Boston (Elmsford, NY: Pergamon) 1: 451–458

    Google Scholar 

  92. Widrow B, Hoff M E 1960 Adaptive switching circuits.IRE WESCON Convention Record (4): 96–104

  93. Willshaw D J, von der Malsburg Ch 1976 How patterned neural connections can be set up by self-organization.Proc. R. Soc. London B194: 431–445

    Google Scholar 

  94. Zurada J M 1992Introduction to artificial neural systems (St. Paul, MN: West)

    Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to B Yegnanarayana.

Additional information

This paper is mostly a consolidation of work reported by several researchers in the literature, some of which is cited in the references. The author has borrowed several ideas and illustrations from the references quoted in this paper.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Yegnanarayana, B. Artificial neural networks for pattern recognition. Sadhana 19, 189–238 (1994). https://doi.org/10.1007/BF02811896

Download citation

Keywords

  • Artificial neural network
  • pattern recognition
  • biological neural network