Problems of Self-reference in Self-improving Space-Time Embedded Intelligence
By considering agents to be a part of their environment, Orseau and Ring’s space-time embedded intelligence  is a better fit to the real world than the traditional agent framework. However, a self-modifying AGI that sees future versions of itself as an ordinary part of the environment may run into problems of self-reference. We show that in one particular model based on formal logic, naive approaches either lead to incorrect reasoning that allows an agent to put off an important task forever (the procrastination paradox), or fail to allow the agent to justify even obviously safe rewrites (the Löbian obstacle). We argue that these problems have relevance beyond our particular formalism, and discuss partial solutions.
KeywordsPartial Solution Future Version Naive Approach Safe Action Incompleteness Theorem
Unable to display preview. Download preview PDF.
- 1.Christiano, P., Yudkowsky, E., Herreshoff, M., Barasz, M.: Definability of truth in probabilistic logic (2013), http://intelligence.org/files/DefinabilityOfTruthInProbabilisticLogic-EarlyDraft.pdf
- 2.Fallenstein, B.: An infinitely descending sequence of sound theories each proving the next consistent (2013), https://intelligence.org/files/ConsistencyWaterfall.pdf
- 3.Fallenstein, B.: Procrastination in probabilistic logic (2014), https://intelligence.org/files/ProbabilisticLogicProcrastinates.pdf
- 4.Goertzel, B.: Golem: Toward an agi meta-architecture enabling both goal preservation and radical self-improvement (2010), http://goertzel.org/GOLEM.pdf
- 5.Hutter, M.: Universal Artificial Intelligence: Sequential Decisions based on Algorithmic Probability. Springer, Berlin (2005)Google Scholar
- 6.Legg, S., Hutter, M.: A formal measure of machine intelligence. In: Proc. 15th Annual Machine Learning Conference of Belgium and the Netherlands (Benelearn 2006), Ghent, Belgium, pp. 73–80 (2006)Google Scholar
- 8.Muehlhauser, L., Orseau, L.: Laurent Orseau on Artificial General Intelligence (interview) (2013), http://intelligence.org/2013/09/06/laurent-orseau-on-agi/
- 9.Neumann, L.J., Morgenstern, O.: Theory of games and economic behavior, vol. 60. Princeton University Press, Princeton (1947)Google Scholar
- 11.Robinson, H.: Dualism. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Winter 2012 edition (2012)Google Scholar
- 13.Yudkowsky, E.: The procrastination paradox (2013), https://intelligence.org/files/ProcrastinationParadox.pdf
- 14.Yudkowsky, E., Herreshoff, M.: Tiling agents for self-modifying AI, and the Löbian obstacle (2013)Google Scholar