Between Order and Chaos: The Quest for Meaningful Information

Abstract

The notion of meaningful information seems to be associated with the sweet spot between order and chaos. This form of meaningfulness of information, which is primarily what science is interested in, is not captured by both Shannon information and Kolmogorov complexity. In this paper I develop a theoretical framework that can be seen as a first approximation to a study of meaningful information. In this context I introduce the notion of facticity of a data set. I discuss the relation between thermodynamics and algorithmic complexity theory in the context of this problem. I prove that, under adequate measurement conditions, the free energy of a system in the world is associated with the randomness deficiency of a data set with observations about this system. These insights suggest an explanation of the efficiency of human intelligence in terms of helpful distributions. Finally I give a critical discussion of Schmidhuber’s views specifically his notion of low complexity art, I defend the view that artists optimize facticity instead.

References

  1. 1.

    Adriaans, P.W.: Using MDL for grammar induction. In: Sakakibara, Y., Kobayashi, S., Sato, K., Tomita, T.N.E. (eds.) Grammatical Inference: Algorithms and Applications, 8th International Colloquium, ICGI 2006, Tokyo, Japan. Springer, Berlin (2006)

    Google Scholar 

  2. 2.

    Adriaans, P.W.: The philosophy of learning. In: Adriaans, P.W., van Benthem, J. (eds.) Handbook of the Philosophy of Information. Handbook of the Philosophy of Science, series edited by D.M. Gabbay, P. Thagard and J. Woods (2008)

  3. 3.

    Adriaans, P.W., Vitányi, P.M.B.: The power and perils of MDL. In: Proc. IEEE International Symposium on Information Theory (ISIT), Nice, France, 24–29 June, pp. 2216–2220 (2007)

  4. 4.

    Bais, F.A., Farmer, J.D.: The physics of information. In: Adriaans, P.W., van Benthem, J. (eds.) Handbook of the Philosophy of Information. Handbook of the Philosophy of Science, series edited by D.M. Gabbay, P. Thagard and J. Woods (2008)

  5. 5.

    Barron, A., Rissanen, J., Yu, B.: The minimum description length principle in coding and modeling. IEEE Trans. Inf. Theory 44(6), 2743–2760 (1998)

    MATH  Article  MathSciNet  Google Scholar 

  6. 6.

    Chaitin, G.J.: Algorithmic Information Theory. Cambridge University Press, Cambridge (1987)

    Google Scholar 

  7. 7.

    Chater, N., Vitányi, P.: Simplicity: a unifying principle in cognitive science? Trends Cogn. Sci. 7(1), 19–22 (2003)

    Article  Google Scholar 

  8. 8.

    Cilibrasi, R., Vitányi, P.: Clustering by compression. IEEE Trans. Inf. Theory 51(4), 1523–1545 (2005)

    Article  Google Scholar 

  9. 9.

    Cilibrasi, R., Vitányi, P.M.B.: Automatic meaning discovery using Google. http://www.citebase.org/abstract?id=oai:arXiv.org:cs/0412098 (2004)

  10. 10.

    Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, New York (2006)

    MATH  Google Scholar 

  11. 11.

    Crutchfield, J.P., Young, K.: Inferring statistical complexity. Phys. Rev. Lett. 63(2), 105–108 (1989)

    Article  MathSciNet  Google Scholar 

  12. 12.

    Curnéjols, A., Miclet, L.: Apprentissage Artificiel, Concepts et Algorithmes. Eyrolles, Paris (2003)

    Google Scholar 

  13. 13.

    Dalkilic, M.M., Clark, W.T., Costello, J.C., Radiovojac, P.: Using compression to identify classes of inauthenic texts. In: Proceedings of the 2006 SIAM Conference on Data Mining (2007). http://www.siam.org/meetings/sdm06/proceedings.htm

  14. 14.

    Domingos, P.: The role of Occam’s Razor in knowledge discovery. Data Min. Knowl. Discov. 3(4), 409–425 (1999)

    Article  Google Scholar 

  15. 15.

    Geusebroek, J.M., Smeulders, A.W.M.: A six-stimulus theory for stochastic texture. Int. J. Comput. Vis. 62(1/2), 7–16 (2005)

    Article  Google Scholar 

  16. 16.

    Grünwald, P.D.: The Minimum Description Length Principle. MIT Press, Cambridge (2007)

    Google Scholar 

  17. 17.

    Grünwald, P.D., Langford, J.: Suboptimal behavior of Bayes and MDL in classification under misspecification. Mach. Learn. 66(2–3), 119–149 (2007). doi:10.1007/s10994-007-0716-7

    Article  Google Scholar 

  18. 18.

    Hume, D.: An Enquiry Concerning Human Understanding, The Harvard Classics, vol. XXXVII, Part 3. PF Collier, Toronto (1909–1914)

    Google Scholar 

  19. 19.

    Hutter, M.: Universal algorithmic intelligence: a mathematical top→down approach. In: Goertzel, B., Pennachin, C. (eds.) Artificial General Intelligence. Cognitive Technologies, pp. 227–290. Springer, Berlin (2007)

    Chapter  Google Scholar 

  20. 20.

    Li, M., Vitányi, P.M.B.: An Introduction to Kolmogorov Complexity and Its Applications, 2nd edn. Springer, New York (1997)

    MATH  Google Scholar 

  21. 21.

    Lloyd, S.: Ultimate physical limits to computation. Nature 406, 1047–1054 (2000)

    Article  Google Scholar 

  22. 22.

    Locke, J.: An Essay Concerning Human Understanding. Dent/Dutton, London/New York (1961)

    Google Scholar 

  23. 23.

    Mitchell, T.M.: Machine Learning. McGraw-Hill, New York (1997)

    MATH  Google Scholar 

  24. 24.

    Ramachandran, V.S., Hirstein, W.: The science of art, a neurological theory of aesthetic experience. J. Conscious. Stud. 6(6–7), 15–51 (1999)

    Google Scholar 

  25. 25.

    Schmidhuber, J.: Low-complexity art, Leonardo. J. Int. Soc. Arts Sci. Technol. 30(2), 97–103 (1997)

    Google Scholar 

  26. 26.

    Schmidhuber, J.: Completely self-referential optimal reinforcement learners. In: Duch, W., (eds.) Proceedings of the International Conference on Artificial Neural Networks ICANN’05. LNCS, vol. 3697, pp. 223–233. Springer, Berlin (2005)

    Google Scholar 

  27. 27.

    Schmidhuber, J.: Simple algorithmic principles of discovery, subjective beauty, selective attention, curiosity and creativity. In: V. Corruble, M. Takeda, E. Suzuki (eds.) DS 2007. LNAI, vol. 4755, pp. 26–38 (2007)

  28. 28.

    Solomonoff, R.J.: The discovery of algorithmic probability. J. Comput. Syst. Sci. 55(1), 73–88 (1997)

    MATH  Article  MathSciNet  Google Scholar 

  29. 29.

    Vereshchagin, N.K., Vitányi, P.M.B.: Kolmogorov’s structure functions and model selection. IEEE Trans. Inf. Theory 50(12), 3265–3290 (2004)

    Article  Google Scholar 

  30. 30.

    Wolff, J.G.: Unifying Computing and Cognition, the SP Theory and Its Applications. CognitionResearch.org.uk, Menai Bridge (2006)

    Google Scholar 

  31. 31.

    Wolfram, S.: A New Kind of Science. Wolfram Media, Champaign (2002)

    MATH  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Pieter Adriaans.

Additional information

This project is supported by a BSIK grant from the Dutch Ministry of Education, Culture and Science (OC&W) and is part of the ICT innovation program of the Ministry of Economic Affairs (EZ).

Rights and permissions

Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

Reprints and Permissions

About this article

Cite this article

Adriaans, P. Between Order and Chaos: The Quest for Meaningful Information. Theory Comput Syst 45, 650–674 (2009). https://doi.org/10.1007/s00224-009-9173-y

Download citation

Keywords

  • Meaningful information
  • Learning as compression
  • MDL
  • Two-part code optimization
  • Randomness deficiency
  • Thermodynamics
  • Free energy
  • Algorithmic esthetics