Joint European Conference on Machine Learning and Knowledge Discovery in Databases

ECML PKDD 2015: Machine Learning and Knowledge Discovery in Databases pp 483-497

An Empirical Investigation of Minimum Probability Flow Learning Under Different Connectivity Patterns

  • Daniel Jiwoong Im
  • Ethan Buchman
  • Graham W. Taylor
Conference paper

DOI: 10.1007/978-3-319-23528-8_30

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9284)
Cite this paper as:
Im D.J., Buchman E., Taylor G.W. (2015) An Empirical Investigation of Minimum Probability Flow Learning Under Different Connectivity Patterns. In: Appice A., Rodrigues P., Santos Costa V., Soares C., Gama J., Jorge A. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2015. Lecture Notes in Computer Science, vol 9284. Springer, Cham

Abstract

Energy-based models are popular in machine learning due to the elegance of their formulation and their relationship to statistical physics. Among these, the Restricted Boltzmann Machine (RBM), and its staple training algorithm contrastive divergence (CD), have been the prototype for some recent advancements in the unsupervised training of deep neural networks. However, CD has limited theoretical motivation, and can in some cases produce undesirable behaviour. Here, we investigate the performance of Minimum Probability Flow (MPF) learning for training RBMs. Unlike CD, with its focus on approximating an intractable partition function via Gibbs sampling, MPF proposes a tractable, consistent, objective function defined in terms of a Taylor expansion of the KL divergence with respect to sampling dynamics. Here we propose a more general form for the sampling dynamics in MPF, and explore the consequences of different choices for these dynamics for training RBMs. Experimental results show MPF outperforming CD for various RBM configurations.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Daniel Jiwoong Im
    • 1
  • Ethan Buchman
    • 1
  • Graham W. Taylor
    • 1
  1. 1.School of EngineeringUniversity of GuelphGuelphCanada

Personalised recommendations