Nonlinear Feature Extraction: Deterministic Neural Networks
Independent feature extraction, i.e independent component analysis, was formulated in Chapter 4 as a search for an invertible, volume preserving map which statistically decorrelates the output components in the case of an arbitrary, possibly non-Gaussian input distribution. The same chapter discussed in detail the case where input-output maps are linear, i.e. matrices. The first extension to nonlinear transformation was carried out in Chapter 5 where stochastic neural networks were used to perform statistical decorrelation of Boolean outputs. This chapter further extends nonlinear independent feature extraction by introducing a very general class of nonlinear input-output deterministic maps whose architecture guarantees bijectivity and volume preservation. The criteria for evaluating statistical dependence are those defined in Chapter 4: the cumulant expansion method and the minimization of the mutual information among output components. Atick and Redlich [6.1] and especially the two papers of Redlich [6.2]–[6.3] use similar information theoretic concepts and reversible cellular automata architectures in order to define how nonlinear decorrelation can be performed. Taylor and Coombes [6.4] presented an extension of Oja’s learning rule for polynomial, i.e. higher order neural networks.
KeywordsEntropy Covariance Bromide Retina Assure
Unable to display preview. Download preview PDF.
- [6.10]C. Beck and F. Schlögl: Thermodynamics of Chaotic Systems. Cambridge Nonlinear Science Series, University Press, Cambridge, 1993.Google Scholar
- [6.18]A. Lapedes and R. Farber: Nonlinear Signal Processing Using Neural Networks: Prediction and System Modeling. Tech. Rep. n LA-UR-87-2662, Los Alamos National Laboratory, Los Alamos, NM, 1987.Google Scholar
- [6.19]A. Weigend, D. Rumelhart and B. Huberman: Back-Propagation, Weight Elimination and Time Series Prediction. In Connectionist Models, Proc. 1990, Touretzky, Elman, Sejnowski and Hinton eds., 105–116, 1990.Google Scholar
- [6.20]G. Deco and B. Schürmann: Recurrent Neural Networks Capture the Dynamical Invariance of Chaotic Time Series. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 77-A (11), 1840–1845, 1994.Google Scholar
- [6.29]R. Abraham and J. Marsden: Theoretical Mechanics. The Benjamin-Cummings Publishing Company, Inc., London, 1978.Google Scholar
- [6.31]Feng Kang, Qin Meng-zhao: The Symplectic Methods for the Computation of Hamiltonian Equations. In: Zhu You-lan, Guo Ben-yu, eds., Numerical Methods for Partial Differential Equations. Lecture Notes in Mathematics., 1297, 1–35. Springer-Verlag, New York, 1985.Google Scholar
- [6.35]A. Papoulis: Probability, Random Variables, and Stochastic Processes. Third Edition, McGraw-Hill, New York, 1991.Google Scholar
- [6.39]R. De Valois, H. Morgan and D. Snodderly: Psycophysical Studies of Monkey Vision: III Spatial Luminance Contrast Sensitivity Test of Macaque Retina and Human Observers. Invest. Opthalmol. Vis. Sci., 14, 75–81, 1974.Google Scholar