Skip to main content

The Speed Prior: A New Simplicity Measure Yielding Near-Optimal Computable Predictions

Part of the Lecture Notes in Computer Science book series (LNAI,volume 2375)

Abstract

Solomonoff’s optimal but noncomputable method for inductive inference assumes that observation sequences x are drawn from an recursive prior distribution μ(x). Instead of using the unknown μ(x) he predicts using the celebrated universal enumerable prior M(x) which for all x exceeds any recursive μ(x), save for a constant factor independent of x. The simplicity measure M(x) naturally implements “Occam’s razor” and is closely related to the Kolmogorov complexity of x. However, M assigns high probability to certain data x that are extremely hard to compute. This does not match our intuitive notion of simplicity. Here we suggest a more plausible measure derived from the fastest way of computing data. In absence of contrarian evidence, we assume that the physical world is generated by a computational process, and that any possibly infinite sequence of observations is therefore computable in the limit (this assumption is more radical and stronger than Solomonoff’s). Then we replace M by the novel Speed Prior S, under which the cumulative a priori probability of all data whose computation through an optimal algorithm requires more than O(n) resources is 1/n. We show that the Speed Prior allows for deriving a computable strategy for optimal prediction of future y, given past x. Then we consider the case that the data actually stem from a nonoptimal, unknown computational process, and use Hutter’s recent results to derive excellent expected loss bounds for S-based inductive inference. We conclude with several nontraditional predictions concerning the future of our universe.

Keywords

  • Inductive Inference
  • Minimum Description Length
  • Kolmogorov Complexity
  • Anthropic Principle
  • Universal Turing Machine

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This paper is based on section 6 of TR IDSIA-20-00, Version 2.0: http://www.idsia.ch/~juergen/toesv2/ http://arXiv.org/abs/quant-ph/0011122 (public physics archive)

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/3-540-45435-7_15
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-540-45435-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.00
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. C. H. Bennett and D. P. DiVicenzo. Quantum information and computation. Nature, 404(6775):256–259, 2000.

    CrossRef  Google Scholar 

  2. H. J. Bremermann. Minimum energy requirements of information transfer and computing. International Journal of Theoretical Physics, 21:203–217, 1982.

    MATH  CrossRef  MathSciNet  Google Scholar 

  3. B. Carter. Large number coincidences and the anthropic principle in cosmology. In M. S. Longair, editor, Proceedings of the IAU Symposium 63, pages 291–298. Reidel, Dordrecht, 1974.

    Google Scholar 

  4. G. J. Chaitin. On the length of programs for computing finite binary sequences: statistical considerations. Journal of the ACM, 16:145–159, 1969.

    MATH  CrossRef  MathSciNet  Google Scholar 

  5. H. Everett III. ‘Relative State’ formulation of quantum mechanics. Reviews of Modern Physics, 29:454–462, 1957.

    CrossRef  MathSciNet  Google Scholar 

  6. P. Gács. On the relation between descriptional complexity and algorithmic probability. Theoretical Computer Science, 22:71–93, 1983.

    MATH  CrossRef  MathSciNet  Google Scholar 

  7. D. F. Galouye. Simulacron 3. Bantam, 1964.

    Google Scholar 

  8. M. Hutter. Convergence and error bounds of universal prediction for general alphabet. Proceedings of the 12th European Conference on Machine Learning (ECML-2001), (TR IDSIA-07-01, cs.AI/0103015), 2001.

    Google Scholar 

  9. M. Hutter. General loss bounds for universal sequence prediction. In C. E. Brodley and A. P. Danyluk, editors, Proceedings of the 18th International Conference on Machine Learning (ICML-2001), pages 210–217. Morgan Kaufmann, 2001. TR IDSIA-03-01, IDSIA, Switzerland, Jan 2001, cs.AI/0101019.

    Google Scholar 

  10. M. Hutter. Towards a universal theory of artificial intelligence based on algorithmic probability and sequential decisions. Proceedings of the 12 th European Conference on Machine Learning (ECML-2001), (TR IDSIA-14-00, cs.AI/0012011), 2001.

    Google Scholar 

  11. M. Hutter. The fastest and shortest algorithm for all well-defined problems. International Journal of Foundations of Computer Science, (TR IDSIA-16-00, cs.CC/0102018), 2002. In press.

    Google Scholar 

  12. A. N. Kolmogorov. Three approaches to the quantitative definition of information. Problems of Information Transmission, 1:1–11, 1965.

    Google Scholar 

  13. L. G. Kraft. A device for quantizing, grouping, and coding amplitude modulated pulses. M.Sc. Thesis, Dept. of Electrical Engineering, MIT, Cambridge, Mass., 1949.

    Google Scholar 

  14. L. A. Levin. Universal sequential search problems. Problems of Information Transmission, 9(3):265–266, 1973.

    Google Scholar 

  15. M. Li and P. M. B. Vitányi. An Introduction to Kolmogorov Complexity and its Applications (2nd edition). Springer, 1997.

    Google Scholar 

  16. S. Lloyd. Ultimate physical limits to computation. Nature, 406:1047–1054, 2000.

    CrossRef  Google Scholar 

  17. J. Rissanen. Stochastic complexity and modeling. The Annals of Statistics, 14(3):1080–1100, 1986.

    MATH  MathSciNet  CrossRef  Google Scholar 

  18. C. Schmidhuber. Strings from logic. Technical Report CERN-TH/2000-316, CERN, Theory Division, 2000. http://xxx.lanl.gov/abs/hep-th/0011065.

  19. J. Schmidhuber. Discovering solutions with low Kolmogorov complexity and high generalization capability. In A. Prieditis and S. Russell, editors, Machine Learning: Proceedings of the Twelfth International Conference, pages 488–496. Morgan Kaufmann Publishers, San Francisco, CA, 1995.

    Google Scholar 

  20. J. Schmidhuber. A computer scientist’s view of life, the universe, and everything. In C. Freksa, M. Jantzen, and R. Valk, editors, Foundations of Computer Science: Potential-Theory-Cognition, volume 1337, pages 201–208. Lecture Notes in Computer Science, Springer, Berlin, 1997.

    Google Scholar 

  21. J. Schmidhuber. Discovering neural nets with low Kolmogorov complexity and high generalization capability. Neural Networks, 10(5):857–873, 1997.

    CrossRef  Google Scholar 

  22. J. Schmidhuber. Algorithmic theories of everything. Technical Report IDSIA-20-00, quant-ph/0011122, IDSIA, Manno (Lugano), Switzerland, 2000.

    Google Scholar 

  23. J. Schmidhuber. Hierarchies of generalized Kolmogorov complexities and nonenumerable universal measures computable in the limit. International Journal of Foundations of Computer Science, 2002. In press.

    Google Scholar 

  24. R. J. Solomonoff. A formal theory of inductive inference. Part I. Information and Control, 7:1–22, 1964.

    CrossRef  MathSciNet  MATH  Google Scholar 

  25. R. J. Solomonoff. Complexity-based induction systems. IEEE Transactions on Information Theory, IT-24(5):422–432, 1978.

    CrossRef  MathSciNet  Google Scholar 

  26. G. ’t Hooft. Quantum gravity as a dissipative deterministic system. Technical Report SPIN-1999/07/gr-gc/9903084, http://xxx.lanl.gov/abs/gr-qc/9903084, Institute for Theoretical Physics, Univ. of Utrecht, and Spinoza Institute, Netherlands, 1999. Also published in Classical and Quantum Gravity 16, 3263.

    Google Scholar 

  27. C. S. Wallace and D. M. Boulton. An information theoretic measure for classification. Computer Journal, 11(2): 185–194, 1968.

    MATH  Google Scholar 

  28. K. Zuse. Rechnender Raum. Friedrich Vieweg & Sohn, Braunschweig, 1969.

    MATH  Google Scholar 

  29. A. K. Zvonkin and L. A. Levin. The complexity of finite objects and the algorithmic concepts of information and randomness. Russian Math. Surveys, 25(6):83–124, 1970.

    MATH  CrossRef  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2002 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Schmidhuber, J. (2002). The Speed Prior: A New Simplicity Measure Yielding Near-Optimal Computable Predictions. In: Kivinen, J., Sloan, R.H. (eds) Computational Learning Theory. COLT 2002. Lecture Notes in Computer Science(), vol 2375. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45435-7_15

Download citation

  • DOI: https://doi.org/10.1007/3-540-45435-7_15

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-43836-6

  • Online ISBN: 978-3-540-45435-9

  • eBook Packages: Springer Book Archive