Optimality Issues of Universal Greedy Agents with Static Priors

  • Laurent Orseau
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6331)

Abstract

Finding the universal artificial intelligent agent is the old dream of AI scientists. Solomonoff Induction was one big step towards this, giving a universal solution to the general problem of Sequence Prediction, by defining a universal prior distribution. Hutter defined AIXI, which extends the latter to the Reinforcement Learning framework, where almost all if not all AI problems can be formulated. However, new difficulties arise, because the agent is now active, whereas it is only passive in the Sequence Prediction case. This makes proving AIXI’s optimality difficult. In fact, we prove that the current definition of AIXI can sometimes be only suboptimal in a certain sense, and we generalize this result to infinite horizon agents and to any static prior distribution.

Keywords

AIXI Universal Artificial Intelligence Solomonoff Induction Reinforcement Learning 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Cohen, J.D., McClure, S.M., Yu, A.J.: Should I stay or should I go? How the human brain manages the trade-off between exploitation and exploration. Philosophical Transactions of the Royal Society B: Biological Sciences 362(1481), 933–942 (2007)CrossRefGoogle Scholar
  2. 2.
    Hutter, M.: A theory of universal artificial intelligence based on algorithmic complexity. Arxiv (April 2000), http://arxiv.org/abs/cs/0004001
  3. 3.
    Hutter, M.: Self-optimizing and pareto-optimal policies in general environments based on bayes-mixtures. In: Kivinen, J., Sloan, R.H. (eds.) COLT 2002. LNCS (LNAI), vol. 2375, pp. 364–379. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  4. 4.
    Hutter, M.: Universal Artificial Intelligence: Sequential Decisions Based On Algorithmic Probability. Springer, Heidelberg (2005)MATHGoogle Scholar
  5. 5.
    Hutter, M.: Universal algorithmic intelligence: A mathematical top-down approach. In: Artificial General Intelligence, pp. 227–290. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  6. 6.
    Kraft, L.G.: A device for quantizing, grouping, and coding amplitude modulated pulses. Ph.D. thesis, MIT, Electrical Engineering Department, Cambridge, MA (1949)Google Scholar
  7. 7.
    Li, M., Vitanyi, P.: An Introduction to Kolmogorov Complexity and Its Applications. Springer, New York (2008)MATHCrossRefGoogle Scholar
  8. 8.
    Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach, 2nd edn. Prentice-Hall, Englewood Cliffs (2003)Google Scholar
  9. 9.
    Schmidhuber, J.: The speed prior: A new simplicity measure yielding near-optimal computable predictions. In: Kivinen, J., Sloan, R.H. (eds.) COLT 2002. LNCS (LNAI), vol. 2375, pp. 216–228. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  10. 10.
    Solomonoff, R.: Complexity-based induction systems: comparisons and convergence theorems. IEEE Transactions on Information Theory 24(4), 422–432 (1978)MATHCrossRefMathSciNetGoogle Scholar
  11. 11.
    Solomonoff, R.J.: A formal theory of inductive inference. Part I. Information and Control 7, 1–22 (1964)MATHCrossRefMathSciNetGoogle Scholar
  12. 12.
    Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998); a Bradford BookGoogle Scholar
  13. 13.
    Veness, J., Ng, K.S., Hutter, M., Silver, D.: A monte carlo AIXI approximation. Arxiv (September 2009), http://arxiv.org/abs/0909.0801

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Laurent Orseau
    • 1
  1. 1.UMR AgroParisTech 518 / INRAParisFrance

Personalised recommendations