Distribution-Free Learning of Bayesian Network Structure

  • Xiaohai Sun
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5212)


We present an independence-based method for learning Bayesian network (BN) structure without making any assumptions on the probability distribution of the domain. This is mainly useful for continuous domains. Even mixed continuous-categorical domains and structures containing vectorial variables can be handled. We address the problem by developing a non-parametric conditional independence test based on the so-called kernel dependence measure, which can be readily used by any existing independence-based BN structure learning algorithm. We demonstrate the structure learning of graphical models in continuous and mixed domains from real-world data without distributional assumptions. We also experimentally show that our test is a good alternative, in particular in case of small sample sizes, compared to existing tests, which can only be used in purely categorical or continuous domains.


graphical models independence tests kernel methods 


  1. 1.
    Heckerman, D., Meek, C., Cooper, G.: A Bayesian approach to causal discovery. In: Glymour, C., Cooper, G. (eds.) Computation, Causation, and Discovery, pp. 141–165. MIT Press, Cambridge (1999)Google Scholar
  2. 2.
    Cooper, G.: The computational complexity of probabilistic inference using Bayesian belief networks. Journal of Artificial Intelligence 42(3–4), 393–405 (1990)zbMATHCrossRefGoogle Scholar
  3. 3.
    Chickering, D., Heckerman, D., Meek, C.: Large-sample learning of Bayesian networks is NP-hard. Journal of Machine Learning Research 5, 1287–1330 (2004)MathSciNetGoogle Scholar
  4. 4.
    Spirtes, P., Glymour, C., Scheines, R.: Causation, prediction, and search. Lecture notes in statistics. Springer, New York (1993)zbMATHGoogle Scholar
  5. 5.
    Pearl, J.: Causality: Models, reasoning, and inference. Cambridge University Press, Cambridge (2000)zbMATHGoogle Scholar
  6. 6.
    Margaritis, D.: A Bayesian multiresolution independence test for continuous variables. In: Proceedings of the 17th conference on uncertainty in artificial intelligence, Pittsburgh, PA, pp. 346–353 (2001)Google Scholar
  7. 7.
    Margaritis, D.: Distribution-free learning of Bayesian network structure in continuous domains. In: Proceedings of the 20th National Conference on Artificial Intelligence, Seattle, WA, pp. 825–830 (2005)Google Scholar
  8. 8.
    Kraskov, A., Stögbauer, H., Grassberger, P.: Estimating mutual information. Physical Review E 69(6), 66138 (2000)CrossRefGoogle Scholar
  9. 9.
    Schölkopf, B., Smola, A.: Learning with kernels. MIT Press, Cambridge (2002)Google Scholar
  10. 10.
    Sun, X., Janzing, D., Schölkopf, B., Fukumizu, K.: A kernel-based causal learning algorithm. In: Ghahramani, Z. (ed.) Proceedings of the 24th International Conference on Machine Learning, Corvallis, OR, pp. 855–862 (2007)Google Scholar
  11. 11.
    Fukumizu, K., Gretton, A., Sun, X., Schölkopf, B.: Kernel measures of conditional dependence. In: Platt, J., Koller, D., Singer, Y., Roweis, S. (eds.) Proceedings of the 21th Neural Information Processing Systems Conference, pp. 489–496. MIT Press, Cambridge (2007)Google Scholar
  12. 12.
    Gretton, A., Bousquet, O., Smola, A., Schölkopf, B.: Measuring statistical dependence with Hilbert-Schmidt norms. In: Jain, S., Simon, H.U., Tomita, E. (eds.) ALT 2005. LNCS (LNAI), vol. 3734, pp. 63–77. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  13. 13.
    Good, P.: Permutation tests: A practical guide to resampling methods for testing hypothesis. Birkhäuer, Boston (1994)Google Scholar
  14. 14.
    Fine, S., Scheinberg, K.: Efficient SVM training using low-rank kernel representations. Journal of Machine Learning Research 2, 243–264 (2001)CrossRefGoogle Scholar
  15. 15.
    Gretton, A., Fukumizu, K., Teo, C., Song, L., et al.: A kernel statistical test of independence. In: Platt, J., Koller, D., Singer, Y., Roweis, S. (eds.) Proceedings of the 21th Neural Information Processing Systems Conference, pp. 585–592. MIT Press, Cambridge (2007)Google Scholar
  16. 16.
    Halkin, H., Sheiner, L., Peck, C., Melmon, K.: Determinants of the renal clearance of digoxin. Clinical Pharmacology and Therapeutics 17(4), 385–394 (1975)Google Scholar
  17. 17.
    Altman, D.G.: Practical statistics for medical research. Chapman and Hall, London (1991)Google Scholar
  18. 18.
    Edwards, D.: Introduction to graphical modelling. Springer, New York (2000)zbMATHGoogle Scholar
  19. 19.
    Jelliffe, R., Blankenhorn, D.: Improved method of digitalis therapy in patients with reduced renal function. Circulation 35, 11–150 (1967)Google Scholar
  20. 20.
    Pearl, J.: Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann, San Mateo (1988)Google Scholar
  21. 21.
    Shachter, R.: Probabilistic inference and influence diagrams. Operations Research 36(4), 589–604 (1988)zbMATHCrossRefGoogle Scholar
  22. 22.
    Geiger, D.: Graphoids: A qualitative framework for probabilistic inference. PhD thesis, Cognitive Systems Laboratory, Department of Computer Science, University of California, Los Angeles, CA (1990)Google Scholar
  23. 23.
    Martín, E.: Ignorable common information, null sets and Basu’s first theorem. The Indian Journal of Statistics 67(4), 674–698 (2005)MathSciNetGoogle Scholar
  24. 24.
    Florens, J., Mouchart, M., Rolin, J.: Elements of Bayesian Statistics. Marcel Dekker, New York (1990)zbMATHGoogle Scholar
  25. 25.
    Vandaele, W.: Participation in illegitimate activites: Erlich revisited. In: Blumstein, A., Cohen, J., Nagin, D. (eds.) Deterrence and incapacitation, pp. 270–335. National Academy of Sciences, Washington (1978)Google Scholar
  26. 26.
    Margaritis, D.: Distribution-free learning of graphical model structure in continuous domains. Technical Report TR-ISU-CS-04-06, Computer Science, Iowa State University (2004)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Xiaohai Sun
    • 1
  1. 1.Max Planck Institute for Biological CyberneticsTübingenGermany

Personalised recommendations