Nonlinear Feature Extraction: Deterministic Neural Networks
Independent feature extraction, i.e independent component analysis, was formulated in Chapter 4 as a search for an invertible, volume preserving map which statistically decorrelates the output components in the case of an arbitrary, possibly non-Gaussian input distribution. The same chapter discussed in detail the case where input-output maps are linear, i.e. matrices. The first extension to nonlinear transformation was carried out in Chapter 5 where stochastic neural networks were used to perform statistical decorrelation of Boolean outputs. This chapter further extends nonlinear independent feature extraction by introducing a very general class of nonlinear input-output deterministic maps whose architecture guarantees bijectivity and volume preservation. The criteria for evaluating statistical dependence are those defined in Chapter 4: the cumulant expansion method and the minimization of the mutual information among output components. Atick and Redlich [6.1] and especially the two papers of Redlich [6.2]–[6.3] use similar information theoretic concepts and reversible cellular automata architectures in order to define how nonlinear decorrelation can be performed. Taylor and Coombes [6.4] presented an extension of Oja’s learning rule for polynomial, i.e. higher order neural networks.
KeywordsInput Space Output Space Novelty Detection Chaotic Time Series Stochastic Neural Network
Unable to display preview. Download preview PDF.
- [6.10]C. Beck and F. Schlögl: Thermodynamics of Chaotic Systems. Cambridge Nonlinear Science Series, University Press, Cambridge, 1993.Google Scholar
- [6.18]A. Lapedes and R. Farber: Nonlinear Signal Processing Using Neural Networks: Prediction and System Modeling. Tech. Rep. n LA-UR-87-2662, Los Alamos National Laboratory, Los Alamos, NM, 1987.Google Scholar
- [6.19]A. Weigend, D. Rumelhart and B. Huberman: Back-Propagation, Weight Elimination and Time Series Prediction. In Connectionist Models, Proc. 1990, Touretzky, Elman, Sejnowski and Hinton eds., 105–116, 1990.Google Scholar
- [6.20]G. Deco and B. Schürmann: Recurrent Neural Networks Capture the Dynamical Invariance of Chaotic Time Series. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 77-A (11), 1840–1845, 1994.Google Scholar
- [6.29]R. Abraham and J. Marsden: Theoretical Mechanics. The Benjamin-Cummings Publishing Company, Inc., London, 1978.Google Scholar
- [6.31]Feng Kang, Qin Meng-zhao: The Symplectic Methods for the Computation of Hamiltonian Equations. In: Zhu You-lan, Guo Ben-yu, eds., Numerical Methods for Partial Differential Equations. Lecture Notes in Mathematics., 1297, 1–35. Springer-Verlag, New York, 1985.Google Scholar
- [6.35]A. Papoulis: Probability, Random Variables, and Stochastic Processes. Third Edition, McGraw-Hill, New York, 1991.Google Scholar
- [6.39]R. De Valois, H. Morgan and D. Snodderly: Psycophysical Studies of Monkey Vision: III Spatial Luminance Contrast Sensitivity Test of Macaque Retina and Human Observers. Invest. Opthalmol. Vis. Sci., 14, 75–81, 1974.Google Scholar