Delusion, Survival, and Intelligent Agents
This paper considers the consequences of endowing an intelligent agent with the ability to modify its own code. The intelligent agent is patterned closely after AIXI with these specific assumptions: 1) The agent is allowed to arbitrarily modify its own inputs if it so chooses; 2) The agent’s code is a part of the environment and may be read and written by the environment. The first of these we call the “delusion box”; the second we call “mortality”. Within this framework, we discuss and compare four very different kinds of agents, specifically: reinforcement-learning, goal-seeking, prediction-seeking, and knowledge-seeking agents. Our main results are that: 1) The reinforcement-learning agent under reasonable circumstances behaves exactly like an agent whose sole task is to survive (to preserve the integrity of its code); and 2) Only the knowledge-seeking agent behaves completely as expected.
KeywordsSelf-Modifying Agents AIXI Universal Artificial Intelligence Reinforcement Learning Prediction Real world assumptions
Unable to display preview. Download preview PDF.
- 3.Orseau, L.: Optimality issues of universal greedy agents with static priors. In: ALT 2010, vol. 6331, pp. 345–359. Springer, Heidelberg (2010)Google Scholar
- 4.Orseau, L., Ring, M.: Self-modification and mortality in artificial agents. In: Schmidhuber, J., Thórisson, K.R., Looks, M. (eds.) AGI 2011. LNCS (LNAI), pp. 1–10. Springer, Heidelberg (2011)Google Scholar