Self-Modification and Mortality in Artificial Agents
This paper considers the consequences of endowing an intelligent agent with the ability to modify its own code. The intelligent agent is patterned closely after AIXI , but the environment has read-only access to the agent’s description. On the basis of some simple modifications to the utility and horizon functions, we are able to discuss and compare some very different kinds of agents, specifically: reinforcement-learning, goal-seeking, predictive, and knowledge-seeking agents. In particular, we introduce what we call the “Simpleton Gambit” which allows us to discuss whether these agents would choose to modify themselves toward their own detriment.
KeywordsSelf-Modifying Agents AIXI Universal Artificial Intelligence Reinforcement Learning Prediction Real world assumptions
Unable to display preview. Download preview PDF.
- 5.Ring, M., Orseau, L.: Delusion, survival, and intelligent agents. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), pp. 11–20. Springer, Heidelberg (2011)Google Scholar