Minds and Machines

, Volume 27, Issue 3, pp 521–543 | Cite as

Algorithmic Iteration for Computational Intelligence



Machine awareness is a disputed research topic, in some circles considered a crucial step in realising Artificial General Intelligence. Understanding what that is, under which conditions such feature could arise and how it can be controlled is still a matter of speculation. A more concrete object of theoretical analysis is algorithmic iteration for computational intelligence, intended as the theoretical and practical ability of algorithms to design other algorithms for actions aimed at solving well-specified tasks. We know this ability is already shown by current AIs, and understanding its limits is an essential step in qualifying claims about machine awareness and Super-AI. We propose a formal translation of algorithmic iteration in a fragment of modal logic, formulate principles of transparency and faithfulness across human and machine intelligence, and consider the relevance to theoretical research on (Super)-AI as well as the practical import of our results.


Artificial intelligence Introspection Machine awareness Algorithm design Algorithm execution 



Thanks to Raf Pasmans, who unexpectedly prompted my first thoughts on this topic. Thanks to the participants at the ‘Debate on Moral and Philosophical Aspects of Artificial Intelligence’ at Ghent University and at the ‘Conference on Artificial Intelligence and Contemporary Society: The Role of Information’ at University of A Coruña, where early versions of this paper were presented. I am especially grateful to Wenceslao Gonzalez, Luciano Floridi, Patrick Allo and Nikos Gkorogiannis, whose observations and comments helped address some conceptual and formal issues. Two anonymous referees’ critiques have helped improving the final version of this paper.


  1. Adelson, B., & Soloway, E. (2007). A model of software design. International Journal of Intelligent Systems, 1(3), 195–213, 1986 (republished).Google Scholar
  2. Aleksander, I. L. (2007). Machine consciousness. In M. Velmans & S. Schneider (Eds.), The Blackwell companion to consciousness (pp. 87–98). New York: Blackwell Publishing.CrossRefGoogle Scholar
  3. Arcuri, A., & Yao, X. (2014). Co-evolutionary automatic programming for software development. Information Sciences, 259, 412–432.CrossRefGoogle Scholar
  4. Armstrong, D. M. (1981). The nature of mind and other essays. Ithaca: Cornell University Press.Google Scholar
  5. Baeten, J. C. M., Luttik, B., & van Tilburg, P. J. A. (2013). Reactive turing machines. Information and Computation, 231, 143–166.MathSciNetCrossRefMATHGoogle Scholar
  6. Balzer, R. (1985). A 15 year perspective on automatic programming. IEEE Transactions on Software Engineering, 11(11), 1257–1268.CrossRefGoogle Scholar
  7. Barcan, R. C. (1946). A functional calculus of first order based on strict implication. Journal of Symbolic Logic, 11, 1–16.MathSciNetCrossRefMATHGoogle Scholar
  8. Barstow, D. (1984). A perspective on automatic programming. AI Magazine, 5(1), 5–27.Google Scholar
  9. Biermann, A. W. (1985). Automatic programming: A tutorial on formal methodologies. Journal of Symbolic Computation, 1, 119–142.CrossRefMATHGoogle Scholar
  10. Bolander, T. (2002). Maximal introspection of agents. Electronic Notes in Theoretical Computer Science, 70(5), 183–198.CrossRefMATHGoogle Scholar
  11. Bolander, T. (2003). Logical theories for agent introspection. Ph.D. thesis, informatics and mathematical modelling, Technical University of Denmark.Google Scholar
  12. Bolander, T., & Andersen, M. B. (2011). Epistemic planning for single and multi-agent systems. Journal of Applied Non-Classical Logics, 21(1), 9–34.MathSciNetCrossRefMATHGoogle Scholar
  13. Bolander, T. (2015). Self-reference. The Stanford encyclopedia of philosophy. In E. N. Zalta (Ed.), Spring Edition.
  14. Bostrom, N. (2014). Superintelligence: Paths, dangers, strategies. New York: Oxford University Press.Google Scholar
  15. Byrne, A. (2005). Introspection. Philosophical Topics, 33(1), 79–104.CrossRefGoogle Scholar
  16. Chalmers, D. (1996). The conscious mind: In search of a fundamental theory. Oxford: Oxford University Press.Google Scholar
  17. Cox, M. T., & Ram, A. (1999). Introspective multistrategy learning: On the construction of learning strategies. Artificial Intelligence, 112(1–2), 1–55.CrossRefMATHGoogle Scholar
  18. Curry, Haskell B., & Feys, R. (1958). Combinatory logic (Vol. I). North-Holland: Amsterdam.MATHGoogle Scholar
  19. Daylight, E. G. (2015). Towards a historical notion of ‘Turing—The Father of Computer Science’. History and Philosophy of Logic, 36(3), 205–228.
  20. De Bruijn, N. (1983). Automath, a language for mathematics. Department of Mathematics, Eindhoven University of Technology, TH-report 68-WSK-05, 1968. Reprinted in revised form, with two pages commentary. In Automation and Reasoning, vol. 2, Classical papers on computational logic 1967–1970, pp. 159–200. Berlin: Springer.Google Scholar
  21. De Grave, K. (Ed.) (2015). Formalism and intuition in software development. A conversation with Michael A. Jackson conducted by Edgar G. Daylight and Bas van Vlijmen. Conversations, Issue 5, Lonely Scholar.Google Scholar
  22. Evans, G. (1982). The varieties of reference. Oxford: Oxford University Press.Google Scholar
  23. Fallenstein, B., & Soares, N. (2015). Vingean reflection: Reliable reasoning for self-improving agents. Technical Report 2015-2, Machine Intelligence Research Institute.Google Scholar
  24. Fischer, M. J., & Ladner, R. E. (1977). Propositional modal logic of programs. In STOC ’77 Proceedings of the ninth annual ACM symposium on theory of computing, pp. 286–294 .Google Scholar
  25. Floridi, L. (2008). The method of levels of abstractions. Minds and Machines, 8(3), 303–329.CrossRefGoogle Scholar
  26. Floridi, L. (2015). Singularitarians, aitheists, and why the problem with artificial intelligence is H.A.L. (humanity at large), not HAL. APA Newsletter, 14(2), 7–11.Google Scholar
  27. Floridi, L., Fresco, N., & Primiero, G. (2015). On malfunctioning software. Synthese, 192(4), 1199–1220.CrossRefGoogle Scholar
  28. Fox, S., & Leake, D. B. (2001). Introspective reasoning for index refinement in case-based reasoning. Journal of Experimental and Theoretical Artificial Intelligence, 13(1), 63–88.CrossRefMATHGoogle Scholar
  29. Fresco, N. (2013). Information processing as an account of concrete digital computation. Philosophy and Technology, 26(1), 31–60.CrossRefGoogle Scholar
  30. Fresco, N., & Primiero, G. (2013). Miscomputation. Philosophy and Technology, 26(3), 253–272.CrossRefGoogle Scholar
  31. Gamez, D. (2008). Progress in machine consciousness. Consciousness and Cognition, 17, 887–910.CrossRefGoogle Scholar
  32. Gelfond, M. (1991). Strong introspection. AAAI, 1991, 386–391.MathSciNetGoogle Scholar
  33. Gertler, B. Self-knowledge. The Stanford Encyclopedia of Philosophy (Summer 2015 Edition). In E. N. Zalta (Ed.)
  34. Gilbert, S., & Lynch, N. (2002). Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. Newsletter ACM SIGACT News, 33(2), 51–59.CrossRefGoogle Scholar
  35. Goel, A. K., Morse, E. L., Raja, A., Scholtz, J., & Stasko, J. T. (2009). Computational explanations for report generation in intelligence analysis. ExaCt, 37–47, 2009.Google Scholar
  36. Goertzel, B., & Pennachin, C. (Eds.). (2007). Artificial general intelligence. Berlin: Springer.MATHGoogle Scholar
  37. Good, I. J. (1965) Speculations concerning the First ultraintelligent machine. In F. L. Alt and M. Rubinoff (eds.), Advances in computers (Vol. 6, pp. 31–88).Google Scholar
  38. Gurevich, Y. (2012). What is an algorithm? SOFSEM 2012: Theory and practice of computer science. Lecture Notes in Computer Science, (Vol. 7147, pp. 31–42).Google Scholar
  39. Halpern, J. Y., & Vardi, M. (1994). Algorithmic knowledge. In R. Fagin (Ed.), Proceedings of the 5th conference on theoretical aspects of reasoning about knowledge (pp. 255–266). Morgan Kaufmann.Google Scholar
  40. Harel, D., Kozen, D., & Tiuryn, J. (2000). Dynamic logic. London: MIT Press.MATHGoogle Scholar
  41. Hibbard, B. (2012). Decision support for safe AI design. In J. Bach, B. Goertzel and M. Iklé (Eds.), Artificial general intelligence. Lecture notes in artificial intelligence (Vol. 7716, pp. 117–25). New York: Springer.Google Scholar
  42. Hill, R. (2016). What an algorithm is. Philosophy and Technology, 29(1), 35–59.CrossRefGoogle Scholar
  43. Hintikka, J. (1975). Impossible possible worlds vindicated. Journal of Philosophical Logic, 4, 475–484.MathSciNetCrossRefMATHGoogle Scholar
  44. Hoare, C. A. R. (1969). An axiomatic basis for computer programming. Communications of the ACM, 12, 576–580.CrossRefMATHGoogle Scholar
  45. Howard, W. A. (1980). The formulae-as-types notion of construction. In Seldin, Jonathan P., Hindley, J. Roger, To H.B. Curry: Essays on combinatory logic, lambda calculus and formalism (pp. 479–490). Boston, MA: Academic Press (original paper manuscript from 1969).Google Scholar
  46. Jeffries, R., Turner, A. A., & Polson, P. G. (1981). The processes involved in designing software. In J. R. Anderson (Ed.), Cognitive skills and their acquisition, ch. 8. Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  47. Kant, E. (1985). Understanding and automating algorithm design. IEEE Transactions on Software Engineering, SE-11, 1243–1253.Google Scholar
  48. Kleinberg, J., & Tardos, E. (2005). Algorithm design. Reading, MA: Pearson Addison-Wesley.Google Scholar
  49. Konolige, K. (1985). A computational theory of belief introspection. IJCAI, 85, 503–508.Google Scholar
  50. Lakemeyer, G. (1996). Limited reasoning in first-order knowledge bases with full introspection. Artificial Intelligence, 84, 209–255.MathSciNetCrossRefGoogle Scholar
  51. van Leeuwen, J. J., & Wiedermann, J. (2001). Beyond the Turing limit: Evolving interactive systems. In L. Pacholski, P. Ruzicka (Eds.), SOFSEM, in Lecture Notes in Computer Science (vol. 2234, pp. 90–109).Google Scholar
  52. McKay, T., Nelson, M. (2014). Propositional attitude reports. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy (Spring Edition).
  53. Menzel, C. A. (2016). The Stanford encyclopedia of philosophy (Summer Edition), E. N. Zalta (ed.).
  54. Moschovakis, Y. N. (1994). Sense and denotation as algorithm and value. In J. Oikkonen and J. Vaananen (Eds.), Lecture notes in logic (Vol. 2, pp. 210–249). Berlin: SpringerGoogle Scholar
  55. Moschovakis, Y. N. (2001). What is an algorithm? In B. Engquist and W. Schmid (Eds.), Mathematics unlimited—2001 and beyond (pp. 919–936). Berlin: Springer.Google Scholar
  56. Moschovakis, Y. (2006). A logical calculus of meaning and synonymy. Linguistics and Philosophy, 29, 27–89.CrossRefGoogle Scholar
  57. Nilsson, N. (2009). The quest for artificial intelligence: A history of ideas and achievements. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  58. Piccinini, G. (2007). Computing mechanisms. Philosophy of Science, 74, 501–526.MathSciNetCrossRefGoogle Scholar
  59. Piccinini, G. (2015). Physical computation—A mechanistic account. Oxford: Oxford University Press.CrossRefMATHGoogle Scholar
  60. Pratt, V. R. (1976) Semantical considerations on Floyd-Hoare Logic. In SFCS ’76 proceedings of the 17th annual symposium on foundations of computer science, pp. 109–121Google Scholar
  61. Primiero, G. (2015). Realist consequence, epistemic inference, computational correctness. In A. Koslow & A. Buchsbaum (Eds.), The road to universal logic, Part of the series Studies in Universal Logic (Vol. 2, pp. 573–588). Springer, Birkhauser.Google Scholar
  62. Primiero, G. (2016). Information in the philosophy of computer science. In L. Floridi (Ed.), The Routledge handbook of philosophy of information, ch. 10 (pp. 90–106).Google Scholar
  63. Rapaport, W. J. (1999). Implementation is semantic interpretation. The Monist, 82(1), 109–130.MathSciNetCrossRefGoogle Scholar
  64. Rapaport, W. J. (2005). Implementation as semantic interpretation: Further thoughts. Journal of Experimental and Theoretical Artificial Intelligence, 17(4), 385–417.CrossRefGoogle Scholar
  65. Schwitzgebel, E. (2014). Introspection. In E. N. Zalta (Ed.), The Stanford encyclopedia of philosophy (Summer Edition).
  66. Shoemaker, S. (1994). Self-knowledge and ‘Inner-Sense’. Philosophy and Phenomenological Research, 54, 249–314. Reprinted in The First Person Perspective and other Essays, OUP, 1996.Google Scholar
  67. Sonntag, D. (2008). On introspection, metacognitive control and augmented data mining live cycles. 0807.4417[CoRRabs].Google Scholar
  68. Sørensen, M. H., & Urzyczyn, P. (2006). Lectures on the Curry-Howard isomorphism. Studies in Logic and the Foundations of Mathematics (Vol. 149). London: Elsevier.Google Scholar
  69. Sotala, K., Yampolskiy, R. (2015). Responses to catastrophic AGI risk: A survey. Physica Scripta, 90. Royal Swedish Academy of Sciences, IOP.Google Scholar
  70. Stalnaker, R. (1991). The problem of logical omniscience I. Synthese, 89, 425–440.MathSciNetCrossRefGoogle Scholar
  71. van Ditmarsch, H., & French, T. (2011). On the interactions of awareness and certainty. Australasian Conference on Artificial Intelligence, pp. 727–738.Google Scholar
  72. Voss, P. (2007). Essentials of general intelligence: The direct path to artificial general intelligence In B. Goertzel & C. Pennachin (Eds.), Artificial General Intelligence (pp. 131–157). Heidelberg: Springer.Google Scholar
  73. Wiedermann, J. (2012). A computability argument against superintelligence. Cognitive Computation, 4(3), 236–245.CrossRefGoogle Scholar
  74. Wright, C. (2000). Self-knowledge: The Wittgensteinean legacy. In C. Wright, B. Smith, & C. MacDonald (Eds.), Knowing our own minds. Oxford: Oxford University Press.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceMiddlesex UniversityLondonUK

Personalised recommendations