An Information-Theoretic Approach to Neural Computing

1. Front Matter

   Pages i-xiii

   PDF

2. Introduction

   1. Introduction

      • Gustavo Deco, Dragan Obradovic

      Pages 1-5

3. Preliminaries of Information Theory and Neural Networks

   1. Preliminaries of Information Theory and Neural Networks

      • Gustavo Deco, Dragan Obradovic

      Pages 7-37

4. Unsupervised Learning

   1. Front Matter

      Pages 39-39

      PDF

   2. Linear Feature Extraction: Infomax Principle

      • Gustavo Deco, Dragan Obradovic

      Pages 41-63

   3. Independent Component Analysis: General Formulation and Linear Case

      • Gustavo Deco, Dragan Obradovic

      Pages 65-107

   4. Nonlinear Feature Extraction: Boolean Stochastic Networks

      • Gustavo Deco, Dragan Obradovic

      Pages 109-133

   5. Nonlinear Feature Extraction: Deterministic Neural Networks

      • Gustavo Deco, Dragan Obradovic

      Pages 135-166

5. Supervised Learning

   1. Front Matter

      Pages 167-167

      PDF

   2. Supervised Learning and Statistical Estimation

      • Gustavo Deco, Dragan Obradovic

      Pages 169-186

   3. Statistical Physics Theory of Supervised Learning and Generalization

      • Gustavo Deco, Dragan Obradovic

      Pages 187-217

   4. Composite Networks

      • Gustavo Deco, Dragan Obradovic

      Pages 219-224

   5. Information Theory Based Regularizing Methods

      • Gustavo Deco, Dragan Obradovic

      Pages 225-241

6. Back Matter

   Pages 243-261

   PDF

Neural networks provide a powerful new technology to model and control nonlinear and complex systems. In this book, the authors present a detailed formulation of neural networks from the information-theoretic viewpoint. They show how this perspective provides new insights into the design theory of neural networks. In particular they show how these methods may be applied to the topics of supervised and unsupervised learning including feature extraction, linear and non-linear independent component analysis, and Boltzmann machines. Readers are assumed to have a basic understanding of neural networks, but all the relevant concepts from information theory are carefully introduced and explained. Consequently, readers from several different scientific disciplines, notably cognitive scientists, engineers, physicists, statisticians, and computer scientists, will find this to be a very valuable introduction to this topic.

• calculus
• complex system
• control
• information theory
• learning
• neural networks
• supervised learning

• Corporate Research and Development, Siemens AG, Munich, Germany

  Gustavo Deco, Dragan Obradovic

Policies and ethics