Probabilistic modelling, inference and learning using logical theories

Article

Abstract

This paper provides a study of probabilistic modelling, inference and learning in a logic-based setting. We show how probability densities, being functions, can be represented and reasoned with naturally and directly in higher-order logic, an expressive formalism not unlike the (informal) everyday language of mathematics. We give efficient inference algorithms and illustrate the general approach with a diverse collection of applications. Some learning issues are also considered.

Keywords

Probabilistic modelling Probabilistic inference Learning Higher-order logic 

Mathematics Subject Classifications (2000)

03B15 03B48 68T37 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Allison, L.: Models for machine learning and data mining in functional programming. J. Funct. Program. 15(1), 15–32 (2005)MATHCrossRefMathSciNetGoogle Scholar
  2. 2.
    Andrews, P.B.: An Introduction to Mathematical Logic and Type Theory: To Truth Through Proof. Academic, London (1986)MATHGoogle Scholar
  3. 3.
    Baral, C., Gelfond, M., Rushton, J.N.: Probabilistic reasoning with answer sets. In: Proceedings of the 7th International Conference on Logic Programming and Nonmonotonic Reasoning, pp. 21–33, Fort Lauderdale, 6–8 January 2004Google Scholar
  4. 4.
    Boole, G.: An Investigation of the Laws of Thought on which are founded the Mathematical Theories of Logic and Probabilities. Walton and Maberly, London (1854)Google Scholar
  5. 5.
    Boole, G.: Studies in Logic and Probability. Watts, London (1952)MATHGoogle Scholar
  6. 6.
    Church, A.: A formulation of the simple theory of types. J. Symb. Log. 5, 56–68 (1940)MATHCrossRefMathSciNetGoogle Scholar
  7. 7.
    Clark, K.: Negation as failure. In: Gallaire, H., Minker, J. (eds.) Logic and Databases, pp. 293–322. Plenum, New York (1978)Google Scholar
  8. 8.
    Cole, J.J., Gray, M., Lloyd, J.W., Ng, K.S.: Personalisation for user agents. In: Proceedings of the 4th International Joint Conference on Autonomous Agents and Multi Agent Systems, pp. 603–610, Utrecht, 25–29 July 2005Google Scholar
  9. 9.
    De Raedt, L., Kersting, K.: Probabilistic logic learning. SIGKDD Explorations 5(1), 31–48 (2003)CrossRefGoogle Scholar
  10. 10.
    de Salvo Braz, R.: Lifted First-Order Probabilistic Inference. PhD thesis, University of Illinois at Urbana-Champaign (2007)Google Scholar
  11. 11.
    de Salvo Braz, R., Amir, E., Roth, D.: Lifted first-order probabilistic inference. In: Proceedings of the 19th International Joint Conference on Artificial Intelligence, pp. 1319–1325, Edinburgh, 30 July–5 August 2005Google Scholar
  12. 12.
    de Salvo Braz, R., Amir, E., Roth, D.: Lifted first-order probabilistic inference. In: Getoor, L., Taskar, B. (eds.) Introduction to Statistical Relational Learning, chapter 15. MIT, Cambridge (2007)Google Scholar
  13. 13.
    Dekhtyar, A., Subrahmanian, V.S.: Hybrid probabilistic programs. J. Log. Program. 43(3), 187–250 (2000)MATHCrossRefMathSciNetGoogle Scholar
  14. 14.
    Dowek, G.: Higher-order unification and matching. In: Handbook of automated reasoning, pp. 1009–1062. Elsevier Science, Amsterdam (2001)CrossRefGoogle Scholar
  15. 15.
    Ebbinghaus, H.D., Flum, J., Thomas, W.: Mathematical Logic. Springer, New York (1984)MATHGoogle Scholar
  16. 16.
    Enderton, H.B.: A Mathematical Introduction to Logic, 2nd edn. Harcourt Academic, San Diego (2001)MATHGoogle Scholar
  17. 17.
    Erwig, M., Kollmansberger, S.: Probabilistic functional programming in Haskell. J. Funct. Program. 16(1), 21–34 (2006)MATHCrossRefGoogle Scholar
  18. 18.
    Fagin, R., Halpern, J.Y.: Reasoning about knowledge and probability. J. ACM 41(2), 340–367 (1994)MATHCrossRefMathSciNetGoogle Scholar
  19. 19.
    Farmer, W.M.: The seven virtues of simple type theory. J. Appl. Logic 6(3), 267–286 (2008)MATHCrossRefMathSciNetGoogle Scholar
  20. 20.
    Gelfond, M., Lifschitz, V.: Classical negation in logic programs and disjunctive databases. New Gener. Comput. 9, 365–387 (1991)Google Scholar
  21. 21.
    Goodman, N., Mansinghka, V.K., Roy, D., Bonawitz, K., Tenenbaum, J.B.: Church: a language for generative models. In: McAllester, D.A., Myllymäki, P. (eds.) Proceedings of the 24th Conference in Uncertainty in Artificial Intelligence, pp. 220–229. AUAI, Corvallis (2008)Google Scholar
  22. 22.
    Gyftodimos, E., Flach, P.A.: Combining Bayesian networks with higher-order data representations. In: 6th International Symposium on Intelligent Data Analysis, pp. 145–156, Madrid, 8–10 September 2005Google Scholar
  23. 23.
    Hailperin, T.: Sentential Probability Logic. Lehigh University Press, Bethlehem (1996)MATHGoogle Scholar
  24. 24.
    Hájek, A.: Probability, logic and probability logic. In: Goble, L. (ed.) The Blackwell Guide to Philosophical Logic, chapter 16, pp. 362–384. Blackwell, Oxford (2001)Google Scholar
  25. 25.
    Halpern, J.Y.: An analysis of first-order logics of probability. Artif. Intell. 46(3), 311–350 (1990)MATHCrossRefGoogle Scholar
  26. 26.
    Halpern, J.Y.: Reasoning about Uncertainty. MIT, Cambridge (2003)MATHGoogle Scholar
  27. 27.
    Henkin, L.: Completeness in the theory of types. J. Symb. Log. 15(2), 81–91 (1950)MATHCrossRefMathSciNetGoogle Scholar
  28. 28.
    Kalman, R.E.: A new approach to linear filtering and prediction problems. Trans. ASME J. Basic Eng. 82(D), 35–45 (1960)Google Scholar
  29. 29.
    Kersting, K., De Raedt, L.: Bayesian logic programming: theory and tool. In: Getoor, L., Taskar, B. (eds.) Introduction to Statistical Relational Learning, chapter 10. MIT, Cambridge (2007)Google Scholar
  30. 30.
    Kschischang, F.R., Frey, B.J., Loeliger, H.-A.: Factor graphs and the sum-product algorithm. IEEE Trans. Inf. Theory 47(2) (2001)Google Scholar
  31. 31.
    Lakshmanan, L.V.S., Sadri, F.: Modeling uncertainty in deductive databases. In: Karagiannis, D. (ed.) Proceedings of the International Conference on Database and Expert Systems Applications, DEXA’94, pp. 724–733, Athens, 7–9 September 1994Google Scholar
  32. 32.
    Leivant, D.: Higher-order logic. In: Gabbay, D.M., Hogger, C.J., Robinson, J.A., Siekmann, J. (eds.) Handbook of Logic in Artificial Intelligence and Logic Programming, vol. 2, pp. 230–321. Oxford University Press, Oxford (1994)Google Scholar
  33. 33.
    Lloyd, J.W.: Programming in an integrated functional and logic language. J. Funct. Logic Program. 3 (1999)Google Scholar
  34. 34.
    Lloyd, J.W.: Knowledge Representation, Computation, and Learning in Higher-order Logic. http://rsise.anu.edu.au/~jwl/ (2002)
  35. 35.
    Lloyd, J.W.: Logic for Learning: Learning Comprehensible Theories from Structured Data. Springer, New York (2003)MATHGoogle Scholar
  36. 36.
    Lloyd, J.W.: Knowledge Representation and Reasoning in Modal Higher-order Logic. http://rsise.anu.edu.au/~jwl (2007)
  37. 37.
    Lloyd, J.W., Ng, K.S.: Learning modal theories. In: Muggleton, S., Otero, R., Tamaddoni-Nezhad, A. (eds.) Proceedings of the 16th International Conference on Inductive Logic Programming. LNAI, vol. 4455, pp. 320–334 (2007)Google Scholar
  38. 38.
    Lloyd, J.W., Ng, K.S., Veness, J.: Modal Functional Logic Programming. http://rsise.anu.edu.au/~kee (2007)
  39. 39.
    Lloyd, J.W., Sears, T.D.: An architecture for rational agents. In: Declarative Agent Languages and Technologies (DALT-2005). LNAI, vol. 3904, pp. 51–71. Springer, New York (2006)CrossRefGoogle Scholar
  40. 40.
    Milch, B., Marthi, B., Russell, S., Sontag, D., Ong, D.L., Kolobov, A.: Blog: probabilistic models with unknown objects. In: Kaelbling, L.P., Saffiotti, A. (eds.) Proceedings of the 19th International Joint Conference on Artificial Intelligence, pp. 1352–1359 (2005)Google Scholar
  41. 41.
    Milch, B., Russell, S.: First-order probabilistic languages: into the unknown. In: Muggleton, S., Otero, R., Tamaddoni-Nezhad, A. (eds.) Proceedings of the 16th International Conference on Inductive Logic Programming. LNAI, vol. 4455, pp. 10–24 (2007)Google Scholar
  42. 42.
    Muggleton, S.: Stochastic logic programs. In: De Raedt, L. (ed.) Advances in Inductive Logic Programming, pp. 254–264. IOS, Amsterdam (1996)Google Scholar
  43. 43.
    Ng, K.S.: Learning Comprehensible Theories from Structured Data. PhD thesis, Computer Sciences Laboratory, The Australian National University (2005)Google Scholar
  44. 44.
    Ng, K.S., Lloyd, J.W.: Probabilistic reasoning in a classical logic. J. Appl. Logic (2008). doi:10.1016/j.jal.2007.11.008 Google Scholar
  45. 45.
    Ng, R.T., Subrahmanian, V.S.: Probabilistic logic programming. Inf. Comput. 101(2), 150–201 (1992)MATHCrossRefMathSciNetGoogle Scholar
  46. 46.
    Nilsson, N.J.: Probabilistic logic. Artif. Intell. 28(1), 71–88 (1986)MATHCrossRefMathSciNetGoogle Scholar
  47. 47.
    Park, S., Pfenning, F., Thrun, S.: A probabilistic language based upon sampling functions. In: Conference Record of the 32nd Annual ACM Symposium on Principles of Programming Languages (2005)Google Scholar
  48. 48.
    Pfeffer, A.: The design and implementation of IBAL: a general-purpose probabilistic language. In: Getoor, L., Taskar, B. (eds.) Introduction to Statistical Relational Learning, chapter 14. MIT, Cambridge (2007)Google Scholar
  49. 49.
    Poole, D.: Logic, knowledge representation, and Bayesian decision theory. In: Proceedings of the 1st International Conference on Computational Logic. LNCS, vol. 1861, pp. 70–86 (2000)Google Scholar
  50. 50.
    Poole, D.: First-order probabilistic inference. In: Proceedings of the 18th International Joint Conference on Artificial Intelligence, pp. 985–991 (2003)Google Scholar
  51. 51.
    Ramsey, N., Pfeffer, A.: Stochastic lambda calculus and monads of probability distributions. In: Conference Record of the 29th Annual ACM Symposium on Principles of Programming Languages (2002)Google Scholar
  52. 52.
    Richardson, M., Domingos, P.: Markov logic networks. Mach. Learn. 62, 107–136 (2006)CrossRefGoogle Scholar
  53. 53.
    Sato, T., Kameya, Y.: PRISM: a language for symbolic-statistical modeling. In: Proceedings of the 15th International Joint Conference on Artificial Intelligence, pp. 1330–1335 (1997)Google Scholar
  54. 54.
    Shapiro, S.: Classical logic II—higher-order logic. In: Goble, L. (ed.) The Blackwell Guide to Philosophical Logic, pp. 33–54. Blackwell, Oxford (2001)Google Scholar
  55. 55.
    Shirazi, A., Amir, E.: Probabilistic modal logic. In: Holte, R.C., Howe, A. (eds.) Proceedings of the 22nd AAAI Conference on Artificial Intelligence, pp. 489–495 (2007)Google Scholar
  56. 56.
    Thompson, S.: Type Theory and Functional Programming. Addison-Wesley, Reading (1991)MATHGoogle Scholar
  57. 57.
    van Benthem, J., Doets, K.: Higher-order logic. In: Gabbay, D.M., Guenther, F. (eds.) Handbook of Philosophical Logic, vol. 1, pp. 275–330. Reidel, Dordrecht (1983)Google Scholar
  58. 58.
    Williamson, J.: Probability logic. In: Gabbay, D., Johnson, R., Ohlbach, H.J., Woods, J. (eds.) Handbook of the Logic of Inference and Argument: The Turn Toward the Practical. Studies in Logic and Practical Reasoning, vol. 1, pp. 397–424. Elsevier, Amsterdam (2002)CrossRefGoogle Scholar
  59. 59.
    Williamson, J.: Philosophies of probability. In: Irvine, A., (ed.) Handbook of the Philosophy of Mathematics. Handbook of the Philosophy of Science, vol. 4. Elsevier, Amsterdam (2009, in press)Google Scholar
  60. 60.
    Zhang, N.L., Poole, D.: A simple approach to Bayesian network computations. In: Proceedings of the 10th Biennial Canadian Artificial Intelligence Conference, pp. 171–178 (1994)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.National ICT Australia and The Australian National UniversityActonAustralia
  2. 2.College of Engineering and Computer ScienceThe Australian National UniversityActonAustralia
  3. 3.National ICT Australia and University of New South WalesKensingtonAustralia

Personalised recommendations