Skip to main content

Is There an Elegant Universal Theory of Prediction?

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNAI,volume 4264)


Solomonoff’s inductive learning model is a powerful, universal and highly elegant theory of sequence prediction. Its critical flaw is that it is incomputable and thus cannot be used in practice. It is sometimes suggested that it may still be useful to help guide the development of very general and powerful theories of prediction which are computable. In this paper it is shown that although powerful algorithms exist, they are necessarily highly complex. This alone makes their theoretical analysis problematic, however it is further shown that beyond a moderate level of complexity the analysis runs into the deeper problem of Gödel incompleteness. This limits the power of mathematics to analyse and study prediction algorithms, and indeed intelligent systems in general.


  • Prediction Algorithm
  • Kolmogorov Complexity
  • Input String
  • Computable Sequence
  • Short Program

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
USD   39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Barzdin, J.M.: Prognostication of automata and functions. Information Processing 71, 81–84 (1972)

    MathSciNet  Google Scholar 

  2. Calude, C.S.: Information and Randomness, 2nd edn. Springer, Berlin (2002)

    MATH  Google Scholar 

  3. Chaitin, G.J.: Gödel’s theorem and information. International Journal of Theoretical Physics 22, 941–954 (1982)

    CrossRef  MathSciNet  Google Scholar 

  4. Dawid, A.P.: Comment on The impossibility of inductive inference. Journal of the American Statistical Association 80(390), 340–341 (1985)

    CrossRef  MathSciNet  Google Scholar 

  5. Feder, M., Merhav, N., Gutman, M.: Universal prediction of individual sequences. IEEE Trans. on Information Theory 38, 1258–1270 (1992)

    CrossRef  MATH  MathSciNet  Google Scholar 

  6. Gödel, K.: Über formal unentscheidbare Sätze der principia mathematica und verwandter systeme I. Monatshefte für Matematik und Physik 38, 173–198 (1931); English translation by Mendelsohn, E.: On undecidable propositions of formal mathematical systems. In: Davis, M. (ed.) The undecidable, New York, pp. 39–71. Raven Press, Hewlitt (1965)

    Google Scholar 

  7. Mark Gold, E.: Language identification in the limit. Information and Control 10(5), 447–474 (1967)

    CrossRef  MATH  Google Scholar 

  8. Hutter, M.: Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability, p. 300. Springer, Berlin (2005),

    MATH  Google Scholar 

  9. Hutter, M.: On the foundations of universal sequence prediction. In: Cai, J.-Y., Cooper, S.B., Li, A. (eds.) TAMC 2006. LNCS, vol. 3959, pp. 408–420. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  10. Li, M., Vitányi, P.M.B.: An introduction to Kolmogorov complexity and its applications, 2nd edn. Springer, Heidelberg (1997)

    MATH  Google Scholar 

  11. Poland, J., Hutter, M.: Convergence of discrete MDL for sequential prediction. In: Shawe-Taylor, J., Singer, Y. (eds.) COLT 2004. LNCS (LNAI), vol. 3120, pp. 300–314. Springer, Heidelberg (2004)

    CrossRef  Google Scholar 

  12. Rissanen, J.J.: Fisher Information and Stochastic Complexity. IEEE Trans. on Information Theory 42(1), 40–47 (1996)

    CrossRef  MATH  MathSciNet  Google Scholar 

  13. Solomonoff, R.J.: A formal theory of inductive inference: Part 1 and 2. Inform. Control 7(1–22), 224–254 (1964)

    CrossRef  MATH  MathSciNet  Google Scholar 

  14. Solomonoff, R.J.: Complexity-based induction systems: comparisons and convergence theorems. IEEE Trans. Information Theory IT-24, 422–432 (1978)

    CrossRef  MATH  MathSciNet  Google Scholar 

  15. Sutton, R., Barto, A.: Reinforcement learning: An introduction. MIT Press, Cambridge, MA (1998)

    Google Scholar 

  16. V’yugin, V.V.: Non-stochastic infinite and finite sequences. Theoretical computer science 207, 363–382 (1998)

    CrossRef  MATH  MathSciNet  Google Scholar 

  17. Wallace, C.S., Boulton, D.M.: An information measure for classification. Computer Jrnl. 11(2), 185–194 (1968)

    MATH  Google Scholar 

  18. Willems, F.M.J., Shtarkov, Y.M., Tjalkens, T.J.: The context-tree weighting method: Basic properties. IEEE Transactions on Information Theory 41(3) (1995)

    Google Scholar 

Download references

Author information

Authors and Affiliations


Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Legg, S. (2006). Is There an Elegant Universal Theory of Prediction?. In: Balcázar, J.L., Long, P.M., Stephan, F. (eds) Algorithmic Learning Theory. ALT 2006. Lecture Notes in Computer Science(), vol 4264. Springer, Berlin, Heidelberg.

Download citation

  • DOI:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-46649-9

  • Online ISBN: 978-3-540-46650-5

  • eBook Packages: Computer ScienceComputer Science (R0)