Accelerated learning in Boltzmann Machines using mean field theory
The learning process in Boltzmann Machines is computationally intractible. We present a new approximate learning algorithm for Boltzmann Machines, which is based on mean field theory and the linear response theorem. The computational complexity of the algorithm is cubic in the number of neurons.
In the absence of hidden units, we show how the weights can be directly computed from the fixed point equation of the learning rules. We show that the solutions of this method are close to the optimal and give a significant improvement over the naive mean field approach.
KeywordsPartition Function Firing Rate Linear Response Exact Method Boltzmann Distribution
Unable to display preview. Download preview PDF.
- 1.D. Ackley, G. Hinton, and T. Sejnowski. A learning algorithm for Boltzmann Machines. Cognitive Science, 9:147–169, 1985.Google Scholar
- 2.C. Itzykson and J-M. Drouffe. Statistical field theory. Cambridge monographs on mathematical physics. Cambridge University Press, Cambridge, UK, 1989.Google Scholar
- 3.C. Peterson and J.R. Anderson. A mean field theory learning algorithm for neural networks. Complex systems, 1:995–1019, 1987.Google Scholar
- 4.G.E. Hinton. Deterministic Boltzmann learning performs steepest descent in weightspace. Neural Computation, 1:143–150, 1989.Google Scholar
- 5.H.J. Kappen and F.B. Rodríguez. Efficient learning in Boltzmann Machines using linear response theory. Neural Computation, page Submitted, 1997.Google Scholar
- 6.G. Parisi. Statistical Field Theory. Frontiers in Physics. Addison-Wesley, 1988.Google Scholar
- 7.S. Kullback. Information theory and statistics. Wiley, New York, 1959.Google Scholar