Supervised and Unsupervised Learning in Linear Networks
We give an overview of the main facts on supervised and unsupervised learning in networks of linear units and present several new results and open questions. In the case of back-propagation, the complete structure of the landscape of the error function and its connections to known statistical techniques such as linear regression, principal component analysis and discriminant analysis have been established. Here, we examine the dynamical aspects of the learning process, how in certain cases the spectral properties of a covariance matrix are learnt according to the order defined by the eigenvalues, and the effects of noise. In the low noise limit, we prove that the strategy adopted by the networks are unchanged whereas in the high noise limit, the solution adopted is one of complete redundancy. In the case of unsupervised learning, several algorithms based on various hebbian and anti-hebbian mechanisms are reviewed together with the structure of their fixed points. We show that three “symmetric” algorithms suggested in the literature (Oja, 1982; Williams, 1985; Baldi, 1988) are in fact equivalent. Results of simulations are presented.
KeywordsGradient Descent Unsupervised Learning Hide Unit Fact Equivalent Linear Network
Unable to display preview. Download preview PDF.
- Baldi, P. (1988). Linear learning: Landscapes and algorithms. In D. S. Touretzky (Ed.), Advances in neural information processing systems 1, Morgan Kaufman. Palo Alto, CA.Google Scholar
- Baldi, P. and Hornik, K. (1990). Back-propagation and unsupervised learning in linear networks. In Y. Chauvin and D. E. Rumelhart (Eds.) Back-propagation: Theory, architectures and applications. Lawrence Erlbaum Ass. To Appear.Google Scholar
- Chauvin, Y. (1989). Principal component analysis by gradient descent on a constrained linear hebbian cell. Proceedings of the 1989 IJCNN Conference, 1, 373–380, Washington D. C.Google Scholar
- Gallinari, P., Thiria, S. and Folgelman Soulie, F. (1988). Multilayer perceptrons and data analysis. Proceedings of the 1988 IJCNN Conference, 391–399, San Diego, CA.Google Scholar
- Linsker, R. (1988). Self-organization in a perceptual network. Computer, March, 105–117.Google Scholar
- Williams, R. J. (1985). Feature discovery through error-correction learning. Technical Report 8501. Institute for Cognitive Science, UCSD, La Jolla, CA.Google Scholar