Soft Computing

, Volume 11, Issue 2, pp 169–183

Multilayer Feedforward Neural Network Based on Multi-valued Neurons (MLMVN) and a Backpropagation Learning Algorithm

Original Paper

Abstract

A multilayer neural network based on multi-valued neurons (MLMVN) is considered in the paper. A multi-valued neuron (MVN) is based on the principles of multiple-valued threshold logic over the field of the complex numbers. The most important properties of MVN are: the complex-valued weights, inputs and output coded by the kth roots of unity and the activation function, which maps the complex plane into the unit circle. MVN learning is reduced to the movement along the unit circle, it is based on a simple linear error correction rule and it does not require a derivative. It is shown that using a traditional architecture of multilayer feedforward neural network (MLF) and the high functionality of the MVN, it is possible to obtain a new powerful neural network. Its training does not require a derivative of the activation function and its functionality is higher than the functionality of MLF containing the same number of layers and neurons. These advantages of MLMVN are confirmed by testing using parity n, two spirals and “sonar” benchmarks and the Mackey–Glass time series prediction.

Keywords

Feedforward complex-valued neural network Derivative free backpropagation learning 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Aizenberg I, Aizenberg N, Vandewalle J (2000) Multi-valued and universal binary neurons: theory, learning, applications. Kluwer, BostonGoogle Scholar
  2. 2.
    Aizenberg I, Bregin T, Butakoff C, Karnaukhov V, Merzlyakov N, Milukova O (2002) Type of blur and blur parameters identification using neural network and its application to image restoration. In: Dorronsoro JR (eds) Lecture notes in computer science, 2415. Springer, Berlin Heidelberg New York, pp 1231–1236Google Scholar
  3. 3.
    Aizenberg I, Myasnikova E, Samsonova M, Reinitz J (2002) Temporal classification of Drosophila segmentation gene expression patterns by the multi-valued neural recognition method. J Math Biosci 176(1):145–159CrossRefMathSciNetMATHGoogle Scholar
  4. 4.
    Aizenberg NN, Ivaskiv Yu L, Pospelov DA (1971) About one generalization of the threshold function (in Russian). The reports of the Academy of Sciences of the USSR. Doklady Akademii Nauk SSSR 196:1287–1290Google Scholar
  5. 5.
    Aizenberg NN, Ivaskiv Yu L (1977) Multiple-valued threshold logic (in Russian). Naukova Dumka, KievGoogle Scholar
  6. 6.
    Aizenberg NN, Aizenberg IN (1992) CNN based on multi-valued neuron as a model of associative memory for gray-scale images. In: Proceedings of the second IEEE International workshop on cellular neural networks and their applications, Technical University Munich, Germany, 14–16 October, 1992, pp 36–41Google Scholar
  7. 7.
    Aoki H, Kosugi Y (2000) An image storage system using complex-valued associative memory. In: Proceedings of the 15th international conference on pattern recognition, vol 2. IEEE Computer Society Press, pp 626–629Google Scholar
  8. 8.
    Aoki H, Watanabe E, Nagata A, Kosugi Y (2001) image association for endoscopic positional identification using complex-valued associative memories. In: Mira J, Prieto A (eds) Bio-inspired applications of connectionism. Lecture notes in computer science, 2085. Springer, Berlin Heidelberg New York, pp 369–374Google Scholar
  9. 9.
    Chen J-H, Chen C-S (2002) Fuzzy kernel perceptron. IEEE Trans Neural Netw 13:1364–1373CrossRefGoogle Scholar
  10. 10.
    Cover TM (1965) Geometrical and statistical properties of systems of linear inequalities with application in pattern recognition. IEEE Trans Electron Comput 14:326–334CrossRefMATHGoogle Scholar
  11. 11.
    Fahlman JD, Lebiere C (1987) Predicting the Mackey–Glass time series. Phys Rev Lett 59:845–847CrossRefMathSciNetGoogle Scholar
  12. 12.
    Franco L, Cannas SA (2001) Generalization properties of modular networks: implementing the parity function. IEEE Trans Neural Netw 12:1306–1313CrossRefGoogle Scholar
  13. 13.
    Funahashi KI (1989) On the approximate realization of continuous mappings by neural networks. Neural Netw 2:183–192CrossRefGoogle Scholar
  14. 14.
    Fung H, Li LK (2001) Minimal feedforward parity networks using threshold gates. Neural Comput 13:319–326CrossRefMATHGoogle Scholar
  15. 15.
    Georgiou GM, Koutsougeras C (1992) Complex domain backpropagation. IEEE Trans Circuits Syst CAS-II 39:330–334CrossRefMATHGoogle Scholar
  16. 16.
    Gorman RP, Sejnowski TJ (1988) Analysis of hidden units in a layered network trained to classify sonar targets. Neural Netw 1:75–89CrossRefGoogle Scholar
  17. 17.
    Haykin S (1999) Neural networks: a comprehensive foundation, 2nd edn. Prentice Hall, Englewood CliffMATHGoogle Scholar
  18. 18.
    Hecht-Nielsen R (1988) Kolmogorov mapping neural network existence theorem. In: Proceedings of the 1st IEEE international conference on neural networks, vol 3. IEEE Computer Society Press, pp 11–13Google Scholar
  19. 19.
    Hecht-Nielsen R (1990) Neurocomputing. Addison Wesley, New YorkGoogle Scholar
  20. 20.
    Hirose A (ed) (2003) Complex valued neural networks. Theories and applications. World Scientific, SingaporeMATHGoogle Scholar
  21. 21.
    Hornik K, Stinchcombe M, White H (1989) Multilayer feedforward neural networks are universal approximators. Neural Netw 2:259–366CrossRefGoogle Scholar
  22. 22.
    Impagliazzo R, Paturi R, Saks ME (1997) Size-depth tradeoffs for threshold circuits. SIAM J Comput 26:693–707CrossRefMathSciNetMATHGoogle Scholar
  23. 23.
    Islam MM, Yao X, Murase K (2003) A constructive algorithm for training cooperative neural networks ensembles. IEEE Trans Neural Netw 14:820–834CrossRefGoogle Scholar
  24. 24.
    Jang J-SR (1993) ANFIS: adaptive-network-based fuzzy inference system. IEEE Trans Syst, Man Cybern 23:665–685CrossRefGoogle Scholar
  25. 25.
    Jankowski S, Lozowski A, Zurada JM (1996) Complex-valued multistate neural associative memory. IEEE Trans Neural Netw 7:1491–1496CrossRefGoogle Scholar
  26. 26.
    Kim D, Kim C (1997) Forecasting time series with genetic fuzzy predictor ensemble. IEEE Trans Neural Netw 5:523–535Google Scholar
  27. 27.
    Kolmogorov AN (1957) On the representation of continuous functions of many variables by superposition of continuous functions and addition (in Russian). The Reports of the Academy of Sciences of the USSR. Doklady Akademii Nauk SSSR 114:953–956MathSciNetMATHGoogle Scholar
  28. 28.
    Lee S-H, Kim I (1994) Time series analysis using fuzzy learning. In: Proceedings of the international conference on neural information processing, Seoul, Korea, vol 6, pp 1577–1582Google Scholar
  29. 29.
    Leung H, Haykin S (1991) The complex backpropagation algorithm. IEEE Trans Signal Process 39:2101–2104CrossRefGoogle Scholar
  30. 30.
    Mackey MC, Glass L (1977) Oscillation and chaos in physiological control systems. Science 197:287–289CrossRefGoogle Scholar
  31. 31.
    Mizutani E, Dreyfus SE, Jang J-SR (2000) On dynamic programming-like recursive gradient formula for alleviating hidden-node saturation in the parity problem. In: Proceedings of the international workshop on intelligent systems resolutions – the 8th Bellman continuum, Hsinchu, Taiwan, pp 100–104Google Scholar
  32. 32.
    Mizutani E, Dreyfus SE (2002) MLP’s hidden-node saturations and insensitivity to initial weights in two classification benchmark problems: parity and two-spirals. In: Proceedings of the 2002 international joint conference on neural networks (IJCNN’02), pp 2831–2836Google Scholar
  33. 33.
    Muezzinoglu MK, Guzelis C, Zurada JM (2003) A new design method for the complex-valued multistate hopfield associative memory. IEEE Trans Neural Netw 14:891–899CrossRefGoogle Scholar
  34. 34.
    Müller K-R, Mika S, Rätsch G, Tsuda K, Shölkopf B (2001) An introduction to kernel-based learning algorithms. IEEE Trans Neural Netw 12:181–201CrossRefGoogle Scholar
  35. 35.
    Nitta T (1997) An extension of the backpropagation algorithm to complex numbers. Neural Netw 10:1391–1415CrossRefGoogle Scholar
  36. 36.
    Paul S, Kumar S (2002) Subsethood-product fuzzy neural inference system (SuPFuNIS). IEEE Trans Neural Netw 13:578–599CrossRefGoogle Scholar
  37. 37.
    Rumelhart DE, McClelland JL (1986) Parallel distributed processing: explorations in the microstructure of cognition. MIT Press, CambridgeGoogle Scholar
  38. 38.
    Russo M (2000) Genetic fuzzy learning. IEEE Trans Evol Comput 4:259–273CrossRefGoogle Scholar
  39. 39.
    Siegelman H, Sontag E (1991) Neural nets are universal computing devices. Research Report SYCON-91–08. Rutgers Center for Systems and Control. Rutgers UniversityGoogle Scholar
  40. 40.
    Vapnik V (1995) The nature of statistical learning theory. Springer, Berlin Heidelberg New YorkMATHGoogle Scholar
  41. 41.
    Yao X, Liu Y (1997) A new evolutionary system for evolving artificial neural networks. IEEE Trans Neural Netw 8:694–713CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2006

Authors and Affiliations

  1. 1.Department of Computer and Information SciencesTexas A&M University–TexarkanaTexarkanaUSA
  2. 2.Department of Computer Science-1University of DortmundDortmundGermany

Personalised recommendations