Skip to main content
SpringerLink
Log in
Book cover

International Conference on Artificial General Intelligence

AGI 2022: Artificial General Intelligence pp 452–463Cite as

Artificial Open World for Evaluating AGI: A Conceptual Design

  • 680 Accesses

Part of the Lecture Notes in Computer Science book series (LNAI,volume 13539)

Abstract

How to evaluate Artificial General Intelligence (AGI) is a critical problem that is discussed and unsolved for a long period. In the research of narrow AI, this seems not a severe problem, since researchers in that field focus on some specific problems as well as one or some aspects of cognition, and the criteria for evaluation are explicitly defined. By contrast, an AGI agent should solve problems that are never-encountered by both agents and developers. However, once a developer tests and debugs the agent with a problem, the never-encountered problem becomes the encountered problem, as a result, the problem is solved by the developers to some extent, exploiting their experience, rather than the agents. This conflict, as we call the trap of developers’ experience, leads to that this kind of problems is probably hard to become an acknowledged criterion. In this paper, we propose an evaluation method named Artificial Open World, aiming to jump out of the trap. The intuition is that most of the experience in the actual world should not be necessary to be applied to the artificial world, and the world should be open in some sense, such that developers are unable to perceive the world and solve problems by themselves before testing, though after that they are allowed to check all the data. The world is generated in a similar way as the actual world, and a general form of problems is proposed. A metric is proposed aiming to quantify the progress of research. This paper describes the conceptual design of the Artificial Open World, though the formalization and the implementation are left to the future.

Keywords

  • Evaluation
  • Artificial Open World
  • Artificial General Intelligence

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Adams, S.S., Banavar, G., Campbell, M.: I-athlon: towards a multidimensional turing test. AI Mag. 37(1), 78–84 (2016)

    Google Scholar 

  2. Campbell, M., Hoane Jr, A.J., Hsu, F.H.: Deep blue. Artif. Intell. 134(1–2), 57–83 (2002)

    Google Scholar 

  3. Chollet, F.: On the measure of intelligence. arXiv preprint arXiv:1911.01547 (2019)

  4. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)

    Google Scholar 

  5. Genesereth, M., Björnsson, Y.: The international general game playing competition. AI Mag. 34(2), 107–107 (2013)

    Google Scholar 

  6. Goertzel, B.: Artificial general intelligence: concept, state of the art, and future prospects. J. Artif. Gen. Intell. 5(1), 1 (2014)

    CrossRef  Google Scholar 

  7. Goertzel, B., Bugaj, S.V.: AGI preschool: a framework for evaluating early-stage human-like AGIs. In: Proceedings of AGI, vol. 9, pp. 31–36 (2009)

    Google Scholar 

  8. Goertzel, B., Pennachin, C.: Artificial General Intelligence, vol. 2. Springer, Cham (2007). https://doi.org/10.1007/978-3-540-68677-4

    CrossRef  MATH  Google Scholar 

  9. Hart, D., Goertzel, B.: Opencog: a software framework for integrative artificial general intelligence. In: AGI, pp. 468–472 (2008)

    Google Scholar 

  10. Hofstadter, D.R.: Gdel, Escher, Bach: An Eternal Golden Braid. Basic Books, 20th anniversary (edn.) (1999)

    Google Scholar 

  11. Legg, S., Hutter, M., et al.: A collection of definitions of intelligence. Front. Artif. Intell. Appl. 157, 17 (2007)

    Google Scholar 

  12. Leibo, J.Z., et al.: Scalable evaluation of multi-agent reinforcement learning with melting pot. In: International Conference on Machine Learning, pp. 6187–6199. PMLR (2021)

    Google Scholar 

  13. Schrenk, M.: Metaphysics of Science: A Systematic and Historical Introduction. Routledge (2016)

    Google Scholar 

  14. Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)

    CrossRef  Google Scholar 

  15. Team, O.E.L., et al.: Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2107.12808 (2021)

  16. Wang, P.: Non-axiomatic reasoning system: exploring the essence of intelligence. Ph.D. thesis, Indiana University (1995)

    Google Scholar 

  17. Wang, P.: The evaluation of AGI systems. In: Proceedings of the Third Conference on Artificial General Intelligence, vol. 11, pp. 164–169. Citeseer (2010)

    Google Scholar 

  18. Wang, P.: Non-axiomatic Logic: A Model of Intelligent Reasoning. World Scientific (2013)

    Google Scholar 

  19. Wang, P.: On defining artificial intelligence. J. Artif. Gen. Intell. 11(2), 73–86 (2020)

    Google Scholar 

  20. Wang, P., Goertzel, B.: Theoretical Foundations of Artificial General Intelligence, vol. 4. Springer, Cham (2012). https://doi.org/10.2991/978-94-91216-62-6

    CrossRef  Google Scholar 

  21. Wang, P., Hammer, P.: Issues in temporal and causal inference. In: Bieger, J., Goertzel, B., Potapov, A. (eds.) AGI 2015. LNCS (LNAI), vol. 9205, pp. 208–217. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21365-1_22

    CrossRef  Google Scholar 

  22. Wray, R., Lebiere, C.: Metrics for cognitive architecture evaluation. In: Proceedings of the AAAI-07 Workshop on Evaluating Architectures for Intelligence, pp. 60–66 (2007)

    Google Scholar 

  23. Xu, B., Zhan, X., Ren, Q.: The gap between intelligence and mind. In: Goertzel, B., Iklé, M., Potapov, A. (eds.) AGI 2021. LNCS (LNAI), vol. 13154, pp. 292–305. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-93758-4_31

    CrossRef  Google Scholar 

Download references

Contributions and Acknowledgements

Bowen Xu proposes the main idea and writes this paper; Quansheng Ren, who reviews and modifies the paper, points out the key idea that the complexity of the world stems from agents’ behaviors. We thank Pei Wang for sharing some pieces of literature on evaluating AGI. We thank those who review this paper. The work was sponsored by Zhejiang Lab (No. 2021RD0AB01).

Author information

Authors and Affiliations

  1. School of Electronics, Peking University, Beijing, 100871, China

    Bowen Xu & Quansheng Ren

Authors
  1. Bowen Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Quansheng Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Bowen Xu or Quansheng Ren .

Editor information

Editors and Affiliations

  1. SingularityNET Foundation, Amsterdam, The Netherlands

    Ben Goertzel

  2. SingularityNET Foundation, Amsterdam, The Netherlands

    Matt Iklé

  3. SingularityNET Foundation, Amsterdam, The Netherlands

    Alexey Potapov

  4. Russian Academy of Sciences, Novosibirsk, Russia

    Denis Ponomaryov

Rights and permissions

Reprints and Permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, B., Ren, Q. (2023). Artificial Open World for Evaluating AGI: A Conceptual Design. In: Goertzel, B., Iklé, M., Potapov, A., Ponomaryov, D. (eds) Artificial General Intelligence. AGI 2022. Lecture Notes in Computer Science(), vol 13539. Springer, Cham. https://doi.org/10.1007/978-3-031-19907-3_43

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19907-3_43

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19906-6

  • Online ISBN: 978-3-031-19907-3

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation