The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents
- Nick Bostrom
- … show all 1 hide
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
This paper discusses the relation between intelligence and motivation in artificial agents, developing and briefly arguing for two theses. The first, the orthogonality thesis, holds (with some caveats) that intelligence and final goals (purposes) are orthogonal axes along which possible artificial intellects can freely vary—more or less any level of intelligence could be combined with more or less any final goal. The second, the instrumental convergence thesis, holds that as long as they possess a sufficient level of intelligence, agents having any of a wide range of final goals will pursue similar intermediary goals because they have instrumental reasons to do so. In combination, the two theses help us understand the possible range of behavior of superintelligent agents, and they point to some potential dangers in building such an agent.
- Bostrom, N (2003) Are you living in a computer simulation?. Philosophical Quarterly 53: pp. 243-255 CrossRef
- Bostrom, N (2006) What is a singleton?. Linguistic and Philosophical Investigations 5: pp. 48-54
- Bostrom, N. (2012). Information hazards: A Typology of potential harms from knowledge. Review of Contemporary Philosophy, 10, 44–79. (www.nickbostrom.com/information-hazards.pdf).
- Chalmers, D (2010) The singularity: A philosophical analysis. Journal of Consciousness Studies 17: pp. 7-65
- Chislenko, A. (1997). Technology as extension of human functional architecture. Extropy Online. (project.cyberpunk.ru/idb/technology_as_extension.html).
- de Blanc, P. (2011). Ontological crises in artificial agent’s value systems. Manuscript. The singularity institute for artificial intelligence. (arxiv.org/pdf/1105.3821v1.pdf).
- Dewey, D Learning what to value. In: Schmidhuber, J, Thorisson, KR, Looks, M eds. (2011) Proceedings of the 4th conference on artificial general intelligence, AGI 2011. Springer, Heidelberg, pp. 309-314
- Forgas, J eds. (2009) The psychology of attitudes and attitude change. Psychology Press, London
- Lewis, D (1988) Desire as belief. Mind 97: pp. 323-332 CrossRef
- Omohundro, S. (2008a). The basic AI drives. In P. Wang, B. Goertzel, and S. Franklin (eds.). Proceedings of the First AGI Conference, Vol. 171. Frontiers in Artificial Intelligence and Applications. Amsterdam: IOS Press.
- Omohundro, S. (2008b). The nature of self-improving artificial intelligence. Manuscript. (selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf).
- Omohundro, S. (2012). Rationally-shaped artificial intelligence. In Eden, A. et al. (eds.). The singularity hypothesis: A scientific and philosophical assessment (Springer, forthcoming).
- Parfit, D. (1984). Reasons and persons. (pp. 123–124). Reprinted and corrected edition, 1987. Oxford: Oxford University Press.
- Parfit, D (2011) On what matters. Oxford University Press, Oxford
- Russell, S, Norvig, P (2010) Artificial intelligence: A modern approach. Prentice Hall, New Jersey
- Sandberg, A., & Bostrom, N. (2008). Whole brain emulation: A roadmap. Technical Report 2008–3. Oxford: Future of Humanity Institute, Oxford University. (www.fhi.ox.ac.uk/Reports/2008-3.pdf).
- Shulman, C. (2010). Omohundro’s “basic AI drives” and catastrophic risks. Manuscript. (singinst.org/upload/ai-resource-drives.pdf).
- Sinhababu, N (2009) The humean theory of motivation reformulated and defended. Philosophical Review 118: pp. 465-500 CrossRef
- Smith, M (1987) The humean theory of motivation. Mind 46: pp. 36-61 CrossRef
- Weizenbaum, J (1976) Computer power and human reason: From judgment to calculation. W. H. Freeman, San Francisco
- Yudkowsky, E. (2008). Artificial intelligence as a positive and negative factor in global risk. In N. Bostrom, & M. Cirkovic (Eds.), Global catastrophic risks. (pp. 308–345; quote from p. 310). Oxford: Oxford University Press.
- Yudkowsky, E Complex value systems are required to realize valuable futures. In: Schmidhuber, J, Thorisson, KR, Looks, M eds. (2011) Proceedings of the 4th conference on artificial general intelligence, AGI 2011. Springer, Heidelberg, pp. 388-393
- The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents
Minds and Machines
Volume 22, Issue 2 , pp 71-85
- Cover Date
- Print ISSN
- Online ISSN
- Springer Netherlands
- Additional Links
- Artificial intelligence
- Instrumental reason
- Intelligent agent
- Nick Bostrom (1)
- Author Affiliations
- 1. Faculty of Philosophy, Oxford Martin School, Future of Humanity Institute, Oxford University, Oxford, UK