A fundamental problem in artificial intelligence is that nobody really knows what intelligence is. The problem is especially acute when we need to consider artificial systems which are significantly different to humans. In this paper we approach this problem in the following way: we take a number of well known informal definitions of human intelligence that have been given by experts, and extract their essential features. These are then mathematically formalised to produce a general measure of intelligence for arbitrary machines. We believe that this equation formally captures the concept of machine intelligence in the broadest reasonable sense. We then show how this formal definition is related to the theory of universal optimal learning agents. Finally, we survey the many other tests and definitions of intelligence that have been proposed for machines.
This is a preview of subscription content,to check access.
Access this article
Similar content being viewed by others
Albus, J. S. (1991). Outline for a theory of intelligence. IEEE Transactions Systems, Man and Cybernetics, 21(3), 473–509.
Alvarado, N., Adams, S., Burbeck, S., & Latta, C. (2002). Beyond the Turing test: Performance metrics for evaluating a computer simulation of the human mind. In Performance metrics for intelligent systems workshop. Gaithersburg, MD, North-Holland.
Anastasi, A. (1992). What counselors should know about the use and interpretation of psychological tests. Journal of Counseling and Development, 70(5), 610–615.
Asohan, A. (2003). Leading humanity forward. The Star, October 14.
Bell, T. C., Cleary, J. G., & Witten, I. H. (1990). Text compression. Englewood Cliffs, NJ: Prentice Hall.
Bertsekas, D. P., & Tsitsiklis, J. N. (1996). Neuro-dynamic programming. Belmont, MA: Athena Scientific.
Binet, A. (1911). Les idees modernes sur les enfants. Paris: Flammarion.
Binet, A., & Simon, T. (1905). Methodes nouvelles por le diagnostic du niveai intellectuel des anormaux. L’Année Psychologique, 11, 191–244.
Bingham, W. V. (1937). Aptitudes and aptitude testing. New York: Harper & Brothers.
Block, N. (1981). Psychologism and behaviorism. Philosophical Review, 90, 5–43.
Boring, E. G. (1923). Intelligence as the tests test it. New Republic, 35, 35–37.
Bringsjord, S., & Schimanski, B. (2003). What is artificial intelligence? Psychometric AI as an answer. Eighteenth International Joint Conference on Artificial Intelligence, 18, 887–893.
Calude, C. S. (2002). Information and randomness (2nd ed.). Berlin: Springer.
Carroll, J. B. (1993). Human cognitive abilities: A survey of factor-analytic studies. New York: Cambridge University Press.
Cattell, R. B. (1987). Intelligence: Its structure, growth, and action. New York: Elsevier.
Chaitin, G. J. (1982). Gödel’s theorem and information. International Journal of Theoretical Physics, 22, 941–954.
Dowe, D. L., & Hajek, A. R. (1998). A non-behavioural, computational extension to the Turing test. In International conference on computational intelligence & multimedia applications (ICCIMA ’98) (pp. 101–106). Gippsland, Australia.
Drever, J. (1952). A dictionary of psychology. Harmondsworth: Penguin Books.
Edmonds, B. (2006). The social embedding of intelligence—towards producing a machine that could pass the turing test. In The Turing test sourcebook: Philosophical and methodological issues in the quest for the thinking computer. Dordrecht: Kluwer.
Eisner, J. (1991). Cognitive science and the search for intelligence. Invited paper presented to the Socratic Society, University of Cape Town.
Fiévet, C. (2005). Mesurer l’intelligence d’une machine. In Le Monde de l’intelligence (Vol. 1, pp. 42–45). Paris: Mondeo publishing.
Fogel, D. B. (1995). Review of computational intelligence: Imitating life. Proceedings of the IEEE, 83(11).
Ford, K. M., & Hayes, P. J. (1998). On computational wings: Rethinking the goals of artificial intelligence. Scientific American, 9, (4), 78–83.
French, R. M. (1990). Subcognition and the limits of the Turing test. Mind, 99, 53–65.
Gardner, H. (1993). Frames of mind: Theory of multiple intelligences. London: Fontana Press.
Goertzel, B. (2006). The hidden pattern. Brown Walker Press.
Gottfredson, L. S. (1997). Mainstream science on intelligence: An editorial with 52 signatories, history, and bibliography. Intelligence, 24(1), 13–23.
Gottfredson, L. S. (1997). Why g matters: The complexity of everyday life. Intelligence, 24(1), 79–132.
Gottfredson, L. S. (2002). g: Highly general and highly practical. In R. J. Sternberg & E. L. Grigorenko (Eds.), The general factor of intelligence: How general is it? (pp. 331–380). Hillsdale, NJ: Erlbaum.
Gould, S. J. (1981). The Mismeasure of man. New York: W. W. Norton & Company.
Graham-Rowe, D. (2005). Spotting the bots with brains. In New scientist magazine (Vol. 2512, p. 27).
Gregory, R. L. (1998). The Oxford companion to the mind. Oxford, UK: Oxford University Press.
Gudwin, R. R. (2000). Evaluating intelligence: A computational semiotics perspective. In IEEE international conference on systems, man and cybernetics (pp. 2080–2085). Nashville, TN.
Guilford, J. P. (1967). The nature of human intelligence. New York: McGraw-Hill.
Gunderson, K. (1971). Mentality and machines. Garden City, NY: Doubleday and company.
Harnad, S. (1989). Minds, machines and Searle. Journal of Theoretical and Experimental Artificial Intelligence, 1, 5–25.
Haugeland, J. (1981). Mind design: Philosophy, psychology, and artificial intelligence. MIT Press: Bradford Books.
Henmon, C. V. A. (1921). The measurement of intelligence. School and Society, 13, 151–158.
Herman, L. M., & Pack, A. A. (1994). Animal intelligence: Historical perspectives and contemporary approaches. In R. Sternberg (Ed.), Encyclopedia of human intelligence (pp. 86–96). New York: Macmillan.
Hernández-Orallo, J. (2000a). Beyond the Turing test. Journal of Logic, Language and Information, 9(4), 447–466.
Hernández-Orallo, J. (2000b). On the computational measurement of intelligence factors. In Performance metrics for intelligent systems workshop (pp. 1–8). Gaithersburg, MD.
Hernández-Orallo, J., & Minaya-Collado, N. (1998). A formal definition of intelligence based on an intensional variant of Kolmogorov complexity. In Proceedings of the international symposium of engineering of intelligent systems (EIS’98) (pp. 146–163). ICSC Press.
Herrnstein, R. J., & Murray, C. (1996). The bell curve: Intelligence and class structure in American life. New York: Free Press.
Horn, J. (1970). Organization of data on life-span development of human abilities. In R. Goulet & P. B. Baltes (Eds.), Life-span developmental psychology: Research and theory. New York: Academic Press.
Horst, J. (2002). A native intelligence metric for artificial systems. In Performance metrics for intelligent systems workshop. Gaithersburg, MD
Hsu, F. H., Campbell, M. S., & Hoane, A. J. (1995). Deep blue system overview. In Proceedings of the 1995 international conference on supercomputing (pp. 240–244).
Hutchens, J. L. (1996). How to pass the Turing test by cheating. http://www.cs.umbc.edu/471/current/papers/hutchens.pdf
Hutter, M. (2001a). Towards a universal theory of artificial intelligence based on algorithmic probability and sequential decisions. In Proceedings of the 12th Eurpean conference on machine learning (ECML-2001) (pp. 226–238).
Hutter, M. (2001b). Universal sequential decisions in unknown environments. In Proceedings of the 5th European workshop on reinforcement learning (EWRL-5), 27 (pp. 25–26).
Hutter, M. (2005). Universal artificial intelligence: Sequential decisions based on algorithmic probability. Berlin: Springer, 300 pp., http://www.hutter1.net/ai/uaibook.htm
Hutter, M. (2006a). General discounting versus average reward. In Proceedings of the 17th international conference on algorithmic learning theory (ALT-06) vol 4264 of LNAI (pp. 244–258). Barcelona.
Hutter, M. (2006b). The human knowledge compression prize. http://prize.hutter1.net
Hutter, M. (2007a). On universal prediction and Bayesian confirmation. Theoretical Computer Science, 384(1), 33–48.
Hutter, M. (2007b). Universal algorithmic intelligence: A mathematical top-down approach. In Artificial general intelligence (pp. 227–290). Berlin: Springer.
Johnson, W. L. (1992). Needed: A new test of intelligence. SIGARTN: SIGART Newsletter (ACM Special Interest Group on Artificial Intelligence), 3(4), 7–9.
Johnson-Laird, P. N., & Wason, P. C. (1977). A theoretical analysis of insight into a reasoning task. In P. N. Johnson-Laird & P. C. Wason (Eds.) Thinking: Readings in cognitive science (pp. 143–157). Cambridge, UK: Cambridge University Press.
Kaufman, A. S. (2000). Tests of intelligence. In R. J. Sternberg (Ed.), Handbook of intelligence. Cambridge, UK: Cambridge University Press.
Kurzweil, R. (2000). The age of spiritual machines: When computers exceed human intelligence. East Rutherford, NJ: Penguin.
Legg, S., & Hutter, M. (2004). Ergodic MDPs admit self-optimising policies. Technical Report IDSIA-21-04, IDSIA.
Legg, S., & Hutter, M. (2004). A taxonomy for abstract environments. Technical Report IDSIA-20-04, IDSIA.
Legg, S., & Hutter, M. (2005). A universal measure of intelligence for artificial agents. In Proceedings of the 21st international joint conference on artificial intelligence (IJCAI-2005) (pp. 1509–1510). Edinburgh.
Legg, S., & Hutter, M. (2006a). A formal definition of intelligence for artificial systems. In Proceedings anniversary summit of artificial intelligence. Monte Verita, Switzerland.
Legg, S., & Hutter, M. (2006b). A formal measure of machine intelligence. In Annual machine learning conference of Belgium and The Netherlands (Benelearn’06). Ghent.
Legg, S., & Hutter, M. (2007). A collection of definitions of intelligence. In B. Goertzel & P. Wang (Eds.), Advances in artificial general intelligence: Concepts, architectures and algorithms vol 157 of frontiers in artificial intelligence and applications. (pp. 17–24). Amsterdam, NL: IOS press. Online version: http://www.vetta.org/shane/intelligence.html.
Levin, L. A. (1973). Universal sequential search problems. Problems of Information Transmission, 9, 265–266.
Li, M., & Vitányi, P. M. B. (1997). An introduction to Kolmogorov complexity and its applications (2nd ed.). Berlin: Springer.
Loebner, H. G. (1990). The Loebner prize—the first Turing test. http://www.loebner.net/Prizef/loebner-prize.html
Looks, M., Goertzel, B., & Pennachin, C. (2004). Novamente: An integrative architecture for general intelligence. In AAAI fall symposium, achieving human-level intelligence.
Macphail, E. M. (1985). Vertebrate intelligence: The null hypothesis. In L. Weiskrantz (Ed.), Animal intelligence (pp. 37–50). Oxford: Clarendon.
Mahoney, M. V. (1999). Text compression as a test for artificial intelligence. In AAAI/IAAI.
Masum, H., Christensen, S., & Oppacher, F. (2002). The Turing ratio: Metrics for open-ended tasks. In GECCO 2002: Proceedings of the genetic and evolutionary computation conference (pp. 973–980). New York: Morgan Kaufmann Publishers.
McCarthy, J. (2004). What is artificial intelligence? www-formal.stanford.edu/jmc/whatisai/whatisai.html
Minsky, M. (1985). The society of mind. New York: Simon and Schuster.
Müller, M. (2006). Stationary algorithmic probability. Technical report, TU Berlin, Berlin, http://arXiv.org/abs/cs/0608095
Neisser, U., Boodoo, G., Bouchard, T. J. Jr., Boykin, A. W., Brody, N., Ceci, S. J., Halpern, D. F., Loehlin, J. C., Perloff, R., Sternberg, R. J., & Urbina, S. (1996). Intelligence: Knowns and unknowns. American Psychologist, 51(2), 77–101.
Newell, A., & Simon, H. A. (1976). Computer science as empirical enquiry: Symbols and search. Communications of the ACM 19, 3, 113–126.
Poole, D., Mackworth, A., & Goebel, R. (1998). Computational intelligence: A logical approach. New York: Oxford University Press.
Raven, J. (2000). The Raven’s progressive matrices: Change and stability over culture and time. Cognitive Psychology, 41, 1–48.
Reznikova, Zh. I., & Ryabko, B. Ya. (1986). Analysis of the language of ants by information-theoretic methods. Problems of Information Transmission, 22, 245–249.
Sanghi, P., & Dowe, D. L. (2003). A computer program capable of passing I.Q. tests. In Proceedings of the 4th ICCS international conference on cognitive science (ICCS’03) (pp. 570–575). Sydney, NSW, Australia.
Saygin, A., Cicekli, I., & Akman, V. (2000). Turing test: 50 years later. Minds and Machines, 10(4), 463–518.
Schmidhuber, J. (2002). The speed prior: A new simplicity measure yielding near-optimal computable predictions. In Proceedings of the 15th annual conference on computational learning theory (COLT 2002), Lecture notes in artificial intelligence (pp. 216–228). Sydney, Australia: Springer.
Schweizer, P. (1998). The truly total Turing test. Minds and Machines, 8, 263–272.
Searle, J. (1980). Minds, brains, and programs. Behavioral & Brain Sciences, 3, 417–458.
Shieber, S. (1994). Lessons from a restricted Turing test. CACM: Communications of the ACM, 37(6), 70–78.
Simonton, D. K. (2003). An interview with Dr. Simonton. In J. A. Plucker (Ed.), Human intelligence: Historical influences, current controversies, teaching resources. http://www.indiana.edu/∼intell
Slatter, J. (2001). Assessment of children: Cognitive applications (4th ed.). San Diego: Jermone M. Satler Publisher Inc.
Slotnick, B. M., & Katz, H. M. (1974). Olfactory learning-set formation in rats. Science, 185, 796–798.
Smith, W. D. (2006). Mathematical definition of “intelligence” (and consequences). http://math.temple.edu/∼wds/homepage/works.html
Spearman, C. E. (1927). The abilities of man, their nature and measurement. New York: Macmillan.
Stern, W. L. (1912). Psychologischen Methoden der Intelligenz-Prüfung. Leipzig: Barth.
Sternberg, R. J. (1985). Beyond IQ: A triacrchi theory of human intelligence. New York: Cambridge University Press.
Sternberg, R. J. (Ed.) (2000). Handbook of intelligence. Cambridge University Press.
Sternberg, R. J. (2003). An interview with Dr. Sternberg. In J. A. Plucker (Ed.), Human intelligence: Historical influences, current controversies, teaching resources. http://www.indiana.edu/z∼intell
Sternberg, R. J., & Berg, C. A. (1986). Quantitative integration: Definitions of intelligence: A comparison of the 1921 and 1986 symposia. In R. J. Sternberg & D. K. Detterman (Eds.), What is intelligence? Contemporary wiewpoints on its nature and definition (pp. 155–162). Norwood, NJ: Ablex.
Sternberg, R. J., & Grigorenko, E. L. (Eds.). (2002). Dynamic testing: The nature and measurement of learning potential. Cambridge, UK: Cambridge University Press.
Sutton, R., & Barto, A. (1998). Reinforcement learning: An introduction. Cambridge MA: MIT Press.
Terman, L. M., & Merrill, M. A. (1950). The Stanford-Binet intelligence scale. Boston: Houghton Mifflin.
Thurstone, L. L. (1938). Primary mental abilities. Chicago: University of Chicago Press.
Treister-Goren, A., Dunietz, J., & Hutchens, J. L. (2000). The developmental approach to evaluating artificial intelligence—a proposal. In Performance metrics for intelligence systems.
Treister-Goren, A., & Hutchens, J. L. (2001). Creating AI: A unique interplay between the development of learning algorithms and their education. In Proceeding of the first international workshop on epigenetic robotics.
Turing, A. M. (1950). Computing machinery and intelligence. Mind, LIX(236), 433–460.
Voss, P. (2005). Essentials of general intelligence: The direct path to AGI. In B. Goertzel & C. Pennachin (Eds.), Artificial general intelligence. Berlin: Springer.
Wallace, C. S. (2005). Statistical and inductive inference by minimum message length. Berlin: Springer.
Wang, P. (1995). On the working definition of intelligence. Technical Report 94, Center for Research on Concepts and Cognition, Indiana University.
Watt, S. (1996). Naive psychology and the inverted Turing test. Psycoloquy, 7(14).
Wechsler, D. (1958). The measurement and appraisal of adult intelligence (4 ed.). Baltimore: Williams & Wilkinds.
Wolpert, D. H., & Macready, W. G. (1997). No free lunch theorems for optimization. IEEE Transaction on Evolutionary Computation, 1(1), 67–82.
Zentall, T. R. (1997). Animal memory: The role of instructions. Learning and Motivation, 28, 248–267.
Zentall, T. R. (2000). Animal intelligence. In R. J. Sternberg (Ed.), Handbook of intelligence. Cambridge, UK: Cambridge University Press.
This work was supported by the Swiss NSF grant 200020-107616.
About this article
Cite this article
Legg, S., Hutter, M. Universal Intelligence: A Definition of Machine Intelligence. Minds & Machines 17, 391–444 (2007). https://doi.org/10.1007/s11023-007-9079-x