Abstract
Solomonoff’s inductive learning model is a powerful, universal and highly elegant theory of sequence prediction. Its critical flaw is that it is incomputable and thus cannot be used in practice. It is sometimes suggested that it may still be useful to help guide the development of very general and powerful theories of prediction which are computable. In this paper it is shown that although powerful algorithms exist, they are necessarily highly complex. This alone makes their theoretical analysis problematic, however it is further shown that beyond a moderate level of complexity the analysis runs into the deeper problem of Gödel incompleteness. This limits the power of mathematics to analyse and study prediction algorithms, and indeed intelligent systems in general.
Keywords
- Prediction Algorithm
- Kolmogorov Complexity
- Input String
- Computable Sequence
- Short Program
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, access via your institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Barzdin, J.M.: Prognostication of automata and functions. Information Processing 71, 81–84 (1972)
Calude, C.S.: Information and Randomness, 2nd edn. Springer, Berlin (2002)
Chaitin, G.J.: Gödel’s theorem and information. International Journal of Theoretical Physics 22, 941–954 (1982)
Dawid, A.P.: Comment on The impossibility of inductive inference. Journal of the American Statistical Association 80(390), 340–341 (1985)
Feder, M., Merhav, N., Gutman, M.: Universal prediction of individual sequences. IEEE Trans. on Information Theory 38, 1258–1270 (1992)
Gödel, K.: Über formal unentscheidbare Sätze der principia mathematica und verwandter systeme I. Monatshefte für Matematik und Physik 38, 173–198 (1931); English translation by Mendelsohn, E.: On undecidable propositions of formal mathematical systems. In: Davis, M. (ed.) The undecidable, New York, pp. 39–71. Raven Press, Hewlitt (1965)
Mark Gold, E.: Language identification in the limit. Information and Control 10(5), 447–474 (1967)
Hutter, M.: Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability, p. 300. Springer, Berlin (2005), http://www.idsia.ch/~marcus/ai/uaibook.htm
Hutter, M.: On the foundations of universal sequence prediction. In: Cai, J.-Y., Cooper, S.B., Li, A. (eds.) TAMC 2006. LNCS, vol. 3959, pp. 408–420. Springer, Heidelberg (2006)
Li, M., Vitányi, P.M.B.: An introduction to Kolmogorov complexity and its applications, 2nd edn. Springer, Heidelberg (1997)
Poland, J., Hutter, M.: Convergence of discrete MDL for sequential prediction. In: Shawe-Taylor, J., Singer, Y. (eds.) COLT 2004. LNCS (LNAI), vol. 3120, pp. 300–314. Springer, Heidelberg (2004)
Rissanen, J.J.: Fisher Information and Stochastic Complexity. IEEE Trans. on Information Theory 42(1), 40–47 (1996)
Solomonoff, R.J.: A formal theory of inductive inference: Part 1 and 2. Inform. Control 7(1–22), 224–254 (1964)
Solomonoff, R.J.: Complexity-based induction systems: comparisons and convergence theorems. IEEE Trans. Information Theory IT-24, 422–432 (1978)
Sutton, R., Barto, A.: Reinforcement learning: An introduction. MIT Press, Cambridge, MA (1998)
V’yugin, V.V.: Non-stochastic infinite and finite sequences. Theoretical computer science 207, 363–382 (1998)
Wallace, C.S., Boulton, D.M.: An information measure for classification. Computer Jrnl. 11(2), 185–194 (1968)
Willems, F.M.J., Shtarkov, Y.M., Tjalkens, T.J.: The context-tree weighting method: Basic properties. IEEE Transactions on Information Theory 41(3) (1995)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Legg, S. (2006). Is There an Elegant Universal Theory of Prediction?. In: Balcázar, J.L., Long, P.M., Stephan, F. (eds) Algorithmic Learning Theory. ALT 2006. Lecture Notes in Computer Science(), vol 4264. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11894841_23
Download citation
DOI: https://doi.org/10.1007/11894841_23
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-46649-9
Online ISBN: 978-3-540-46650-5
eBook Packages: Computer ScienceComputer Science (R0)