Self-Modification and Mortality in Artificial Agents

  • Laurent Orseau
  • Mark Ring
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6830)


This paper considers the consequences of endowing an intelligent agent with the ability to modify its own code. The intelligent agent is patterned closely after AIXI [1], but the environment has read-only access to the agent’s description. On the basis of some simple modifications to the utility and horizon functions, we are able to discuss and compare some very different kinds of agents, specifically: reinforcement-learning, goal-seeking, predictive, and knowledge-seeking agents. In particular, we introduce what we call the “Simpleton Gambit” which allows us to discuss whether these agents would choose to modify themselves toward their own detriment.


Self-Modifying Agents AIXI Universal Artificial Intelligence Reinforcement Learning Prediction Real world assumptions 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Hutter, M.: Universal Artificial Intelligence: Sequential Decisions Based On Algorithmic Probability. Springer, Heidelberg (2005)zbMATHGoogle Scholar
  2. 2.
    Hutter, M.: On universal prediction and bayesian confirmation. Theoretical Computer Science 384(1), 33–48 (2007)MathSciNetzbMATHCrossRefGoogle Scholar
  3. 3.
    Li, M., Vitanyi, P.: An Introduction to Kolmogorov Complexity and Its Applications. Springer, New York (2008)zbMATHCrossRefGoogle Scholar
  4. 4.
    Orseau, L.: Optimality issues of universal greedy agents with static priors. In: Hutter, M., Stephan, F., Vovk, V., Zeugmann, T. (eds.) ALT 2010. LNCS, vol. 6331, pp. 345–359. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Ring, M., Orseau, L.: Delusion, survival, and intelligent agents. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), pp. 11–20. Springer, Heidelberg (2011)Google Scholar
  6. 6.
    Schmidhuber, J.: Ultimate cognition à la Gödel. Cognitive Computation 1(2), 177–193 (2009)CrossRefGoogle Scholar
  7. 7.
    Solomonoff, R.: A formal theory of inductive inference. Part I. Information and Control 7, 1–22 (1964)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Solomonoff, R.: Complexity-based induction systems: comparisons and convergence theorems. IEEE Transactions on Information Theory 24(4), 422–432 (1978)MathSciNetzbMATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Laurent Orseau
    • 1
  • Mark Ring
    • 2
  1. 1.UMR AgroParisTech 518 / INRAParisFrance
  2. 2.IDSIA / University of Lugano / SUPSIManno-LuganoSwitzerland

Personalised recommendations