Abstract
Sum-product networks allow to model complex variable interactions while still granting efficient inference. However, most learning algorithms proposed so far are explicitly or implicitly restricted to the image domain, either by assuming variable neighborhood or by assuming that dependent variables are related by their magnitudes over the training set. In this paper, we introduce a novel algorithm, learning the structure and parameters of sum-product networks in a greedy bottom-up manner. Our algorithm iteratively merges probabilistic models of small variable scope to larger and more complex models. These merges are guided by statistical dependence test, and parameters are learned using a maximum mutual information principle. In experiments our method competes well with the existing learning algorithms for sum-product networks on the task of reconstructing covered image regions, and outperforms these when neither neighborhood nor correlations by magnitude can be assumed.
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
Poon, H., Domingos, P.: Sum-product networks: A new deep architecture. In: Proceedings of the Twenty-Seventh Conference on Uncertainty in Artificial Intelligence, pp. 337–346 (2011)
Darwiche, A.: A differential approach to inference in bayesian networks. ACM 50(3), 280–305 (2003)
Lowd, D., Domingos, P.: Learning arithmetic circuits. In: Twenty Fourth Conference on Uncertainty in Artificial Intelligence, pp. 383–392 (2008)
Poon, H., Domingos, P.: (2011), http://alchemy.cs.washington.edu/spn/
Gens, R., Domingos, P.: Discriminative learning of sum-product networks. Advances in Neural Information Processing Systems 25, 3248–3256 (2012)
Coates, A., Lee, H., Ng, A.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (2011)
Dennis, A., Ventura, D.: Learning the architecture of sum-product networks using clustering on variables. Advances in Neural Information Processing Systems 25, 2042–2050 (2012)
Gens, R., Domingos, P.: Learning the structure of sum-product networks. In: Proceedings of ICML, pp. 873–880 (2013)
Lowd, D., Rooshenas, A.: Learning markov networks with arithmetic circuits. In: Proceedings of AISTATS, pp. 406–414 (2013)
Hinton, G., Salakhutdinov, R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19, 153–160 (2007)
Bengio, Y.: Learning Deep Architectures for AI. Foundations and Trends in Machine Learning, vol. 2 (2009)
Koller, D., Friedman, N.: Probabilistic Graphical Models: Principles and Techniques. MIT Press (2009)
Margaritis, D., Thrun, S.: A bayesian multiresolution independence test for continuous variables. In: 17th Conference on Uncertainty in Artificial Intelligence, pp. 346–353 (2001)
Tishby, N., Pereira, F.C., Bialek, W.: The information bottleneck method. In: Proc. Allerton Conf. on Communication, Control, and Computing, pp. 368–377 (1999)
Slonim, N., Tishby, N.: Agglomerative information bottleneck. In: Advances in Neural Information Processing Systems (NIPS), pp. 617–623. MIT Press (1999)
Geiger, B.C., Kubin, G.: Signal enhancement as minimization of relevant information loss. In: Proc. ITG Conf. on Systems, Communication and Coding, Munich, pp. 1–6 (2013); extended version available: arXiv:1205.6935 [cs.IT]
Samaria, F., Harter, A.: Parameterisation of a stochastic model for human face identification. In: Proceedings of the 2nd IEEE Workshop on Applications of Computer Vision, pp. 138–142 (1994)
Park, J.: Map complexity results and approximation methods. In: Proceedings of the Conference on Uncertainty in Artificial Intelligence, pp. 338–396 (2002)
Frey, B., Dueck, D.: Clustering by passing messages between data points. Science 315, 972–976 (2007)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Peharz, R., Geiger, B.C., Pernkopf, F. (2013). Greedy Part-Wise Learning of Sum-Product Networks. In: Blockeel, H., Kersting, K., Nijssen, S., Železný, F. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2013. Lecture Notes in Computer Science(), vol 8189. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40991-2_39
Download citation
DOI: https://doi.org/10.1007/978-3-642-40991-2_39
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40990-5
Online ISBN: 978-3-642-40991-2
eBook Packages: Computer ScienceComputer Science (R0)