Complex Value Systems in Friendly AI

  • Eliezer Yudkowsky
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6830)


A common reaction to first encountering the problem statement of Friendly AI (”Ensure that the creation of a generally intelligent, self-improving, eventually superintelligent system realizes a positive outcome”) is to propose a simple design which allegedly suffices; or to reject the problem by replying that ”constraining” our creations is undesirable or unnecessary. This paper briefly presents some of the reasoning which suggests that Friendly AI is solvable, but not simply or trivially so, and that a wise strategy would be to invoke detailed learning of and inheritance from human values as a basis for further normalization and reflection.


Friendly AI machine ethics anthropomorphism 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Rinkworks: ComputerStupidities:Programming,
  2. 2.
    Kurzweil, R.: The Singularity is Near: When Humans Transcend Biology. Viking, New York (2005)Google Scholar
  3. 3.
    Omohundro, S.: The basic AI drives. In: Wang, P., Goertzel, B., Franklin, S. (eds.) Proceedings of the First AGI Conference, pp. 483–492. IOS Press, Amsterdam (2008)Google Scholar
  4. 4.
    Schmidhuber, J.: Gödel machines: Fully Self-Referential Optimal Universal Self-Improvers. In: Goertzel, B., Pennachin, C. (eds.) Artificial General Intelligence, pp. 119–226. Springer, Heidelberg (2006)Google Scholar
  5. 5.
    Hibbard, B.: Super-intelligent machines. ACM SIGGRAPH Computer Graphics 35(1) (2001)Google Scholar
  6. 6.
    Hibbard, B.: Message to the SL4 email list, archived at (2004),
  7. 7.
    McDermott, D.: Artificial intelligence meets natural stupidity. SIGART Newsletter 57, 4–9 (1976)Google Scholar
  8. 8.
    Frankena, W.: Ethics, 2nd edn. Prentice Hall, Englewood Cliffs (1973)Google Scholar
  9. 9.
    Tarleton, N.: Coherent extrapolated volition: A meta-level approach to machine ethics,

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Eliezer Yudkowsky
    • 1
  1. 1.Singularity Institute for Artificial IntelligenceSan FranciscoUSA

Personalised recommendations