Theory of Computing Systems

, Volume 45, Issue 4, pp 650–674 | Cite as

Between Order and Chaos: The Quest for Meaningful Information

Open Access


The notion of meaningful information seems to be associated with the sweet spot between order and chaos. This form of meaningfulness of information, which is primarily what science is interested in, is not captured by both Shannon information and Kolmogorov complexity. In this paper I develop a theoretical framework that can be seen as a first approximation to a study of meaningful information. In this context I introduce the notion of facticity of a data set. I discuss the relation between thermodynamics and algorithmic complexity theory in the context of this problem. I prove that, under adequate measurement conditions, the free energy of a system in the world is associated with the randomness deficiency of a data set with observations about this system. These insights suggest an explanation of the efficiency of human intelligence in terms of helpful distributions. Finally I give a critical discussion of Schmidhuber’s views specifically his notion of low complexity art, I defend the view that artists optimize facticity instead.


Meaningful information Learning as compression MDL Two-part code optimization Randomness deficiency Thermodynamics Free energy Algorithmic esthetics 


  1. 1.
    Adriaans, P.W.: Using MDL for grammar induction. In: Sakakibara, Y., Kobayashi, S., Sato, K., Tomita, T.N.E. (eds.) Grammatical Inference: Algorithms and Applications, 8th International Colloquium, ICGI 2006, Tokyo, Japan. Springer, Berlin (2006) Google Scholar
  2. 2.
    Adriaans, P.W.: The philosophy of learning. In: Adriaans, P.W., van Benthem, J. (eds.) Handbook of the Philosophy of Information. Handbook of the Philosophy of Science, series edited by D.M. Gabbay, P. Thagard and J. Woods (2008) Google Scholar
  3. 3.
    Adriaans, P.W., Vitányi, P.M.B.: The power and perils of MDL. In: Proc. IEEE International Symposium on Information Theory (ISIT), Nice, France, 24–29 June, pp. 2216–2220 (2007) Google Scholar
  4. 4.
    Bais, F.A., Farmer, J.D.: The physics of information. In: Adriaans, P.W., van Benthem, J. (eds.) Handbook of the Philosophy of Information. Handbook of the Philosophy of Science, series edited by D.M. Gabbay, P. Thagard and J. Woods (2008) Google Scholar
  5. 5.
    Barron, A., Rissanen, J., Yu, B.: The minimum description length principle in coding and modeling. IEEE Trans. Inf. Theory 44(6), 2743–2760 (1998) MATHCrossRefMathSciNetGoogle Scholar
  6. 6.
    Chaitin, G.J.: Algorithmic Information Theory. Cambridge University Press, Cambridge (1987) Google Scholar
  7. 7.
    Chater, N., Vitányi, P.: Simplicity: a unifying principle in cognitive science? Trends Cogn. Sci. 7(1), 19–22 (2003) CrossRefGoogle Scholar
  8. 8.
    Cilibrasi, R., Vitányi, P.: Clustering by compression. IEEE Trans. Inf. Theory 51(4), 1523–1545 (2005) CrossRefGoogle Scholar
  9. 9.
    Cilibrasi, R., Vitányi, P.M.B.: Automatic meaning discovery using Google. (2004)
  10. 10.
    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (2006) MATHGoogle Scholar
  11. 11.
    Crutchfield, J.P., Young, K.: Inferring statistical complexity. Phys. Rev. Lett. 63(2), 105–108 (1989) CrossRefMathSciNetGoogle Scholar
  12. 12.
    Curnéjols, A., Miclet, L.: Apprentissage Artificiel, Concepts et Algorithmes. Eyrolles, Paris (2003) Google Scholar
  13. 13.
    Dalkilic, M.M., Clark, W.T., Costello, J.C., Radiovojac, P.: Using compression to identify classes of inauthenic texts. In: Proceedings of the 2006 SIAM Conference on Data Mining (2007).
  14. 14.
    Domingos, P.: The role of Occam’s Razor in knowledge discovery. Data Min. Knowl. Discov. 3(4), 409–425 (1999) CrossRefGoogle Scholar
  15. 15.
    Geusebroek, J.M., Smeulders, A.W.M.: A six-stimulus theory for stochastic texture. Int. J. Comput. Vis. 62(1/2), 7–16 (2005) CrossRefGoogle Scholar
  16. 16.
    Grünwald, P.D.: The Minimum Description Length Principle. MIT Press, Cambridge (2007) Google Scholar
  17. 17.
    Grünwald, P.D., Langford, J.: Suboptimal behavior of Bayes and MDL in classification under misspecification. Mach. Learn. 66(2–3), 119–149 (2007). doi: 10.1007/s10994-007-0716-7 CrossRefGoogle Scholar
  18. 18.
    Hume, D.: An Enquiry Concerning Human Understanding, The Harvard Classics, vol. XXXVII, Part 3. PF Collier, Toronto (1909–1914) Google Scholar
  19. 19.
    Hutter, M.: Universal algorithmic intelligence: a mathematical top→down approach. In: Goertzel, B., Pennachin, C. (eds.) Artificial General Intelligence. Cognitive Technologies, pp. 227–290. Springer, Berlin (2007) CrossRefGoogle Scholar
  20. 20.
    Li, M., Vitányi, P.M.B.: An Introduction to Kolmogorov Complexity and Its Applications, 2nd edn. Springer, New York (1997) MATHGoogle Scholar
  21. 21.
    Lloyd, S.: Ultimate physical limits to computation. Nature 406, 1047–1054 (2000) CrossRefGoogle Scholar
  22. 22.
    Locke, J.: An Essay Concerning Human Understanding. Dent/Dutton, London/New York (1961) Google Scholar
  23. 23.
    Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997) MATHGoogle Scholar
  24. 24.
    Ramachandran, V.S., Hirstein, W.: The science of art, a neurological theory of aesthetic experience. J. Conscious. Stud. 6(6–7), 15–51 (1999) Google Scholar
  25. 25.
    Schmidhuber, J.: Low-complexity art, Leonardo. J. Int. Soc. Arts Sci. Technol. 30(2), 97–103 (1997) Google Scholar
  26. 26.
    Schmidhuber, J.: Completely self-referential optimal reinforcement learners. In: Duch, W., (eds.) Proceedings of the International Conference on Artificial Neural Networks ICANN’05. LNCS, vol. 3697, pp. 223–233. Springer, Berlin (2005) Google Scholar
  27. 27.
    Schmidhuber, J.: Simple algorithmic principles of discovery, subjective beauty, selective attention, curiosity and creativity. In: V. Corruble, M. Takeda, E. Suzuki (eds.) DS 2007. LNAI, vol. 4755, pp. 26–38 (2007) Google Scholar
  28. 28.
    Solomonoff, R.J.: The discovery of algorithmic probability. J. Comput. Syst. Sci. 55(1), 73–88 (1997) MATHCrossRefMathSciNetGoogle Scholar
  29. 29.
    Vereshchagin, N.K., Vitányi, P.M.B.: Kolmogorov’s structure functions and model selection. IEEE Trans. Inf. Theory 50(12), 3265–3290 (2004) CrossRefGoogle Scholar
  30. 30.
    Wolff, J.G.: Unifying Computing and Cognition, the SP Theory and Its Applications., Menai Bridge (2006) Google Scholar
  31. 31.
    Wolfram, S.: A New Kind of Science. Wolfram Media, Champaign (2002) MATHGoogle Scholar

Copyright information

© The Author(s) 2009

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of AmsterdamAmsterdamThe Netherlands

Personalised recommendations