Cognitive Computation

, Volume 4, Issue 3, pp 365–382 | Cite as

Using Human–Computer Interfaces to Investigate ‘Mind-As-It-Could-Be’ from the First-Person Perspective

  • Tom FroeseEmail author
  • Keisuke Suzuki
  • Yuta Ogai
  • Takashi Ikegami


There is a growing community of researchers who are interested in establishing a science of the experiential or ‘lived’ aspects of the human mind. This shift from cognitive science to consciousness science presents a profound challenge to synthetic approaches. To be sure, symbolic artificial intelligence constituted the original foundation of cognitive science; subsequent progress in robotics has helped to pioneer a new understanding of the mind as essentially embodied, situated, and dynamical, while artificial life has informed the concept of biological self-organization. However, with regard to the development of a science of the experienced mind, the relevance of these synthetic approaches still remains uncertain. We propose to address the challenge of first-person experience by designing new human–computer interfaces, which aim to artificially mediate a participant’s sensorimotor loop such that novel kinds of experience can emerge for the user. The advantage of this synthetic approach is that computer interface technology enables us to systematically vary the ways in which participants experience the world and thereby allows us to systematically investigate ‘mind-as-it-could-be’ from the first-person perspective. We illustrate the basic principles of this method by drawing on examples from our research in sensory substitution, virtual reality, and interactive installation.


Consciousness Enaction Situatedness Embodiment Human–computer interface Technology Artificial life 



Froese’s research was financially supported by a Grant-in-Aid of the Japanese Society for the Promotion of Science. We thank Yuki Sato for his help in the design of Fig. 2.


  1. 1.
    Froese T, Suzuki K, Wakisaka S, Ogai Y, Ikegami T. From artificial life to artificial embodiment: using human-computer interfaces to investigate the embodied mind ‘as-it-could-be’ from the first-person perspective. In: Kazakov D, Tsoulas G, editors. Proceedings of AISB’11: computing & philosophy. NewYork: Society for the Study of Artificial Intelligence and the Simulation of Behavior; 2011. p. 43–50.Google Scholar
  2. 2.
    Clark A. Mindware: an introduction to the philosophy of cognitive science. New York: Oxford University Press; 2001.Google Scholar
  3. 3.
    Froese T. From cybernetics to second-order cybernetics: a comparative analysis of their central ideas. Constr Found. 2010;5(2):75–85.Google Scholar
  4. 4.
    Froese T, Di Paolo EA. The enactive approach: theoretical sketches from cell to society. Pragmat Cogn. 2011;19(1):1–36.CrossRefGoogle Scholar
  5. 5.
    Stewart J, Gapenne O, Di Paolo EA, editors. Enaction: toward a new paradigm for cognitive science. Cambridge: MIT Press; 2010.Google Scholar
  6. 6.
    Thompson E. Mind in life: biology, phenomenology, and the sciences of mind. Cambridge: The Belknap Press of Harvard University Press; 2007.Google Scholar
  7. 7.
    Froese T. From second-order cybernetics to enactive cognitive science: varela’s turn from epistemology to phenomenology. Syst Res Behav Sci. 2011;28:631–45.CrossRefGoogle Scholar
  8. 8.
    Boden MA. Mind as machine: a history of cognitive science. Oxford: Oxford University Press; 2006.Google Scholar
  9. 9.
    Froese T, Ziemke T. Enactive artificial intelligence: investigating the systemic organization of life and mind. Artif Intell. 2009;173(3–4):366–500.Google Scholar
  10. 10.
    Froese T, Stewart J. Life after ashby: ultrastability and the autopoietic foundations of biological individuality. Cybern Hum Knowing. 2010;17(4):83–106.Google Scholar
  11. 11.
    Hanna R, Thompson E. The mind-body-body problem. Theoria et Historia Scientarum. 2003;7(1):24–44.Google Scholar
  12. 12.
    Weber A, Varela FJ. Life after Kant: natural purposes and the autopoietic foundations of biological individuality. Phenomenol Cogn Sci. 2002;1:97–125.CrossRefGoogle Scholar
  13. 13.
    Froese T. Breathing new life into cognitive science. Avant J Philos Interdiscip Vanguard. 2011;2(1):95–111.Google Scholar
  14. 14.
    Froese T, Di Paolo EA. Sociality and the life-mind continuity thesis. Phenomenol Cogn Sci. 2009;8(4):439–63.CrossRefGoogle Scholar
  15. 15.
    Thompson E. Life and mind: from autopoiesis to neurophenomenology. A tribute to Francisco Varela. Phenomenol Cogn Sci. 2004;3(4):381–98.CrossRefGoogle Scholar
  16. 16.
    Stewart J. Cognition = life: implications for higher-level cognition. Behav Process. 1996;35:311–26.CrossRefGoogle Scholar
  17. 17.
    Varela FJ, Thompson E, Rosch E. The embodied mind: cognitive science and human experience. Cambridge: MIT Press; 1991.Google Scholar
  18. 18.
    Froese T. On the role of AI in the ongoing paradigm shift within the cognitive sciences. In: Lungarella M, Iida F, Bongard J, Pfeifer R, editors. 50 years of artificial intelligence: essays dedicated to the 50th anniversary of artificial intelligence. Berlin: Springer; 2007. p. 63–75.Google Scholar
  19. 19.
    Wheeler M. Reconstructing the cognitive world: the next step. Cambridge: The MIT Press; 2005.Google Scholar
  20. 20.
    Pfeifer R, Bongard JC. How the body shapes the way we think: a new view of intelligence. Cambridge: MIT Press; 2007.Google Scholar
  21. 21.
    Beer RD. The dynamics of active categorial perception in an evolved model agent. Adapt Behav. 2003;11(4):209–43.CrossRefGoogle Scholar
  22. 22.
    Clark A. Being there: putting brain, body, and world together again. Cambridge: The MIT Press; 1997.Google Scholar
  23. 23.
    Brooks RA. Intelligence without representation. Artif Intell. 1991;47(1–3):139–60.CrossRefGoogle Scholar
  24. 24.
    Bedau MA. Artificial life: organization, adaptation and complexity from the bottom up. Trends Cogn Sci. 2003;7(11):505–12.PubMedCrossRefGoogle Scholar
  25. 25.
    Froese T, Virgo N, Ikegami T. Life as a process of open-ended becoming: analysis of a minimal model. In: Lenaerts T, Giacobini M, Bersini H, Bourgine P, Dorigo M, Doursat R, editors. Advances in artificial life, ECAL 2011: proceedings of the eleventh European conference on the synthesis and simulation of living systems. Cambridge: The MIT Press; 2011. p. 250–7.Google Scholar
  26. 26.
    Egbert MD, Barandiaran XE. Quantifying normative behaviour and precariousness in adaptive agency. In: Lenaerts T, Giacobini M, Bersini H, Bourgine P, Dorigo M, Doursat R, editors. Advances in artificial life, ECAL 2011: proceedings of the eleventh European conference on the synthesis and simulation of living systems. Cambridge: The MIT Press; 2011. p. 210–7.Google Scholar
  27. 27.
    Hanczyc MM, Ikegami T. Chemical basis for minimal cognition. Artif Life. 2010;16:233–43.PubMedCrossRefGoogle Scholar
  28. 28.
    Suzuki K, Ikegami T. Shapes and self-movement in protocell systems. Artif Life. 2009;15(1):59–70.PubMedCrossRefGoogle Scholar
  29. 29.
    Varela FJ, Maturana HR, Uribe R. Autopoiesis: the organization of living systems, its characterization and a model. BioSystems. 1974;5:187–96.CrossRefGoogle Scholar
  30. 30.
    Wheeler M. Cognition’s Coming Home: The Reunion of Life and Mind. In: Husbands P, Harvey I, editors. Fourth European conference on artificial life. Cambridge: The MIT Press; 1997. p. 10–9.Google Scholar
  31. 31.
    Stewart J. Life = cognition: the epistemological and ontological significance of artificial life. In: Varela FJ, Bourgine P, editors. Toward a practice of autonomous systems: proceedings of the first European conference on artificial life. Cambridge: MIT Press; 1992. p. 475–83.Google Scholar
  32. 32.
    Di Paolo EA, Iizuka H. How (not) to model autonomous behaviour. BioSystems. 2008;91:409–23.PubMedCrossRefGoogle Scholar
  33. 33.
    Ikegami T, Suzuki K. From homeostatic to homeodynamic self. BioSystems. 2008;91(2):388–400.PubMedCrossRefGoogle Scholar
  34. 34.
    Harvey I. Homeostasis and rein control: from daisyworld to active perception. In: Pollack J, Bedau MA, Husbands P, Ikegami T, Watson RA, editors. Artificial life IX: proceedings of the ninth international conference on the simulation and synthesis of living systems. Cambridge: The MIT Press; 2004. p. 309–14.Google Scholar
  35. 35.
    Di Paolo EA. Organismically-inspired robotics: Homeostatic adaptation and teleology beyond the closed sensorimotor loop. In: Murase K, Asakura T, editors. Dynamical systems approach to embodiment and sociality. Adelaide: Adv Knowl Int; 2003. p. 19–42.Google Scholar
  36. 36.
    Depraz N, Varela FJ, Vermersch P. On becoming aware: a pragmatics of experiencing. Amsterdam: John Benjamins Publishing; 2003.Google Scholar
  37. 37.
    Searle JR. Minds, brains, and programs. Behav Brain Sci. 1980;3(3):417–24.CrossRefGoogle Scholar
  38. 38.
    Dreyfus HL. What computers can’t do: a critique of artificial reason. New York: Harper and Row; 1972.Google Scholar
  39. 39.
    Chalmers DJ. Facing up to the problem of consciousness. J Conscious Stud. 1995;2(3):200–19.Google Scholar
  40. 40.
    Penrose R. The Emperor’s new mind: concerning computers, minds and the laws of physics. Oxford: Oxford University Press; 1989.Google Scholar
  41. 41.
    Bishop JM. A cognitive computation fallacy? cognition. Computations and panpsychism. Cogn Comput. 2009;1:221–33.CrossRefGoogle Scholar
  42. 42.
    Froese T, Gould C, Barrett A. Re-Viewing from within: a commentary on first- and second-person methods in the science of consciousness. Constr Found. 2011;6(2):254–69.Google Scholar
  43. 43.
    Froese T, Gould C, Seth AK. Validating and calibrating first- and second-person methods in the science of consciousness. J Conscious Stud. 2011;18(2):38–64.Google Scholar
  44. 44.
    Froese T, Gallagher S. Phenomenology and Artificial Life: toward a Technological Supplementation of Phenomenological Methodology. Husserl Stud. 2010;26(2):83–106.CrossRefGoogle Scholar
  45. 45.
    Ikegami T. Simulating active perception and mental imagery with embodied chaotic itinerancy. J Conscious Stud. 2007;14(7):111–25.Google Scholar
  46. 46.
    Kiverstein J. Could a robot have a subjective point of view? J Conscious Stud. 2007;14(7):128–40.Google Scholar
  47. 47.
    Ziemke T. What’s life got to do with it? In: Chella A, Manzotti R, editors. Artificial consciousness. Exeter: Imprint Academic; 2007. p. 48–66.Google Scholar
  48. 48.
    Torrance S. Two conceptions of machine phenomenality. J Conscious Stud. 2007;14(7):154–66.Google Scholar
  49. 49.
    O’Regan JK, Noë A. A sensorimotor account of vision and visual consciousness. Behav Brain Sci. 2001;24(5):939–1031.PubMedCrossRefGoogle Scholar
  50. 50.
    Noë A. Action in perception. Cambridge: The MIT Press; 2004.Google Scholar
  51. 51.
    Bach-y-Rita P, Collins CC, Saunders FA, White B, Scadden L. Vision substitution by tactile image projection. Nature. 1969;221(5184):963–4.PubMedCrossRefGoogle Scholar
  52. 52.
    Bach-y-Rita P. Sensory substitution and qualia. In: Noë A, Thompson E, editors. Vision and mind: selected readings in the philosophy of perception. Cambridge: The MIT Press; 2002. p. 497–514.Google Scholar
  53. 53.
    Block N. Tactile sensation via spatial perception. Trends Cogn Sci. 2003;7(7):285–6.PubMedCrossRefGoogle Scholar
  54. 54.
    Prinz J. Putting the brakes on enactive perception. Psyche. 2006;12(1):1–19.Google Scholar
  55. 55.
    Lenay C, Gapenne O, Hanneton S, Marque C, Genouëlle C. Sensory substitution: limits and perspectives. In: Hatwell Y, Streri A, Gentaz E, editors. Touching for knowing: cognitive psychology of haptic manual perception. Amsterdam: John Benjamins; 2003. p. 275–92.Google Scholar
  56. 56.
    Auvray M, Myin E. Perception with compensatory devices: from sensory substitution to sensorimotor extension. Cogn Sci. 2009;33:1036–58.PubMedCrossRefGoogle Scholar
  57. 57.
    Guarniero G. Experience of tactile vision. Perception. 1974;3(1):101–4.PubMedCrossRefGoogle Scholar
  58. 58.
    Froese T, Spiers A. Toward a phenomenological pragmatics of enactive perception. Enactive/07: proceedings of the 4th international conference on enactive interfaces. Grenoble, France: Association ACROE; 2007. p. 105–8.Google Scholar
  59. 59.
    Nagel T. What is it like to be a bat? Philos Rev. 1974;83(4):435–40.CrossRefGoogle Scholar
  60. 60.
    Petitmengin C. Describing one’s subjective experience in the second person: an interview method for the science of consciousness. Phenomenol Cogn Sci. 2006;5(3–4):229–69.CrossRefGoogle Scholar
  61. 61.
    Jackson F. Epiphenomenal qualia. Philos Q. 1982;32(127):127–36.CrossRefGoogle Scholar
  62. 62.
    Varela FJ. The re-enchantment of the concrete: some biological ingredients for a nouvelle cognitive science. In: Steels L, Brooks R, editors. The artificial life route to artificial intelligence. Hove: Lawrence Erlbaum Associates; 1995. p. 11–22.Google Scholar
  63. 63.
    Ihde D. Husserl’s Galileo needed a telescope! Philos Technol. 2011;24(1):69–82.CrossRefGoogle Scholar
  64. 64.
    Stiegler B. Technics and time, 1: the fault of epimetheus. Stanford: Stanford University Press; 1998.Google Scholar
  65. 65.
    De Preester H. Technology and the body: the (im)possibilities of re-embodiment. Found Sci. 2011;16:119–37.CrossRefGoogle Scholar
  66. 66.
    Visell Y. Tactile sensory substitution: models for enaction in HCI. Interact Comput. 2009;21(1–2):38–53.CrossRefGoogle Scholar
  67. 67.
    Gillespie RB, O’Modhrain S, editors. Embodied cognition as a motivating perspective for haptic interaction design: a position paper. IEEE World Haptics Conference (WHC); 2011; Istanbul, Turkey: IEEE.Google Scholar
  68. 68.
    Khatchatourov A, Stewart J, Lenay C, editors. Towards an enactive epistemology of technics. ENACTIVE/07; 2007; Grenoble, France: Association ACROE.Google Scholar
  69. 69.
    Froese T, McGann M, Bigge W, Spiers A, Seth AK. The enactive torch: a new tool for the science of perception. IEEE Trans Haptics. 2012.
  70. 70.
    Di Paolo EA, Noble J, Bullock S. Simulation models as opaque thought experiments. In: Bedau MA, McCaskill JS, Packard NH, Rasmussen S, editors. Artificial life VII: proceedings of the seventh international conference on artificial life. Cambridge: MIT Press; 2000. p. 497–506.Google Scholar
  71. 71.
    Beer RD. Dynamical approaches to cognitive science. Trends Cogn Sci. 2000;4(3):91–9.PubMedCrossRefGoogle Scholar
  72. 72.
    Newell A, Simon HA. Human problem solving. Englewood Cliffs: Prentice-Hall; 1972.Google Scholar
  73. 73.
    Dreyfus HL. Why Heideggerian AI failed and how fixing it would require making it more Heideggerian. Philos Psychol. 2007;20(2):247–68.CrossRefGoogle Scholar
  74. 74.
    Chrisley R, Parthemore J. Synthetic phenomenology: exploiting embodiment to specify the non-conceptual content of visual experience. J Conscious Stud. 2007;14(7):44–58.Google Scholar
  75. 75.
    Gallagher S. How the body shapes the mind. New York: Oxford University Press; 2005.CrossRefGoogle Scholar
  76. 76.
    LeGrand D. The bodily self: the sensori-motor roots of pre-reflective self-consciousness. Phenomenol Cogn Sci. 2006;5(1):89–118.CrossRefGoogle Scholar
  77. 77.
    Sheets-Johnstone M. The primacy of movement, expanded. 2nd ed. Amsterdam: John Benjamins Publishing Company; 2011.Google Scholar
  78. 78.
    De Preester H, Tsakiris M. Body-extension versus body-incorporation: is there a need for a body-model? Phenomenol Cogn Sci. 2009;8(3):307–19.CrossRefGoogle Scholar
  79. 79.
    Thompson E, Stapleton M. Making sense of sense-making: reflections on enactive and extended mind theories. Topoi. 2009;28(1):23–30.CrossRefGoogle Scholar
  80. 80.
    Clark A. Natural-Born Cyborgs: minds, technologies, and the future of human intelligence. New York: Oxford University Press; 2003.Google Scholar
  81. 81.
    Clark A. Supersizing the mind: embodiment, action, and cognitive extension. New York: Oxford University Press; 2008.Google Scholar
  82. 82.
    Di Paolo EA. Extended life. Topoi. 2009;28(1):9–21.CrossRefGoogle Scholar
  83. 83.
    Clark A. Spreading the Joy? Why the machinery of consciousness is (probably) still in the head. Mind. 2009;118:963–93.CrossRefGoogle Scholar
  84. 84.
    Froese T. Exploring mind-as-it-could-be: from artificial life to artificial embodiment. Workshop on key issues in sensory augmentation research; Brighton, UK: University of Sussex; 2009. p 1–2.Google Scholar
  85. 85.
    Auvray M, Hanneton S, Lenay C, O’Regan JK. There is something out there: distal attribution in sensory substitution, twenty years later. J Integr Neurosci. 2005;4(4):505–21.PubMedCrossRefGoogle Scholar
  86. 86.
    Bach-y-Rita P, Kercel SW. Sensory substitution and the human-machine interface. Trends Cogn Sci. 2003;7(12):541–6.PubMedCrossRefGoogle Scholar
  87. 87.
    Lenggenhager B, Tadi T, Metzinger T, Blanke O. Video Ergo Sum: manipulating bodily self-consciousness. Science. 2007;317:1096–9.PubMedCrossRefGoogle Scholar
  88. 88.
    Sanchez-Vives MV, Slater M. From presence to consciousness through virtual reality. Nat Rev Neurosci. 2005;6(4):332–9.PubMedCrossRefGoogle Scholar
  89. 89.
    Reid K, Flowers P, Larkin M. Exploring lived experience. Psychologist. 2005;18(1):20–3.Google Scholar
  90. 90.
    Varela FJ, Shear J. First-person methodologies: what, why, how? J Conscious Stud. 1999;6(2–3):1–14.Google Scholar
  91. 91.
    Petitmengin C. Editorial introduction: ten years of viewing from within. J Conscious Stud. 2009;16(10–12):7–19.Google Scholar
  92. 92.
    Proulx MJ, Stoerig P, Ludowig E, Knoll I. Seeing ‘where’ through the ears: effects of learning-by-doing and long-term sensory deprivation on localization based on image-to-sound substitution. PLoS One. 2008;3(3):e1840.PubMedCrossRefGoogle Scholar
  93. 93.
    Auvray M, Lenay C, Stewart J. Perceptual interactions in a minimalist virtual environment. New Ideas Psychol. 2009;27(1):32–47.CrossRefGoogle Scholar
  94. 94.
    Ward J, Meijer P. Visual experiences in the blind induced by an auditory sensory substitution device. Conscious Cogn. 2010;19(1):492–500.PubMedCrossRefGoogle Scholar
  95. 95.
    Rohde M. Enaction, embodiment, evolutionary robotics: simulation models for a post-cognitivist science of mind. Amsterdam: Atlantis Press; 2010.CrossRefGoogle Scholar
  96. 96.
    Ogai Y. Constructive Research of Active Perception by Cognitive Experiment and Simulation Using Neural Networks [Dissertation]. Tokyo, Japan: University of Tokyo; 2011.Google Scholar
  97. 97.
    Ogai Y, Uno R, Ikegami T. From active perception to language: analysis of onomatopoeias using a tactile display. The third international symposium on Mobiligence. Hyogo, Japan; 2009. p. 382–6.Google Scholar
  98. 98.
    Ogai Y, Uno R, Ikegami T. Shokkan wo kousei suru jikken kara akutibu tacchi wo kangaeru. In: Shinohara K, Uno R, editors. Chikazuku oto to imi: Onomatope kenkyu no shatei. Japan: Hituzi Syobo; 2012 (in press).Google Scholar
  99. 99.
    Ogai Y, Ikegami T. An investigation of active touch using neural networks and a tactile display—constructing tactile sensations represented by onomatopoeias. IEICE Technical Report. 2009;109:17–21.Google Scholar
  100. 100.
    Dawkins R. the blind watchmaker: why the evidence of evolution reveals a universe without design. New York: W. W. Norton & Company, Inc.; 1986.Google Scholar
  101. 101.
    Takagi H. Interactive evolutionary computation: fusion of the capacities of EC optimization and human evaluation. Proc IEEE. 2001;89(9):1275–96.CrossRefGoogle Scholar
  102. 102.
    Coelho C, Tichon J, Hine TJ, Wallis G, Riva G. Media presence and inner presence: the sense of presence in virtual reality technologies. In: Riva G, Anguera MT, Wiederhold BK, Mantovani F, editors. From communication to presence: cognition, emotions and culture towards the ultimate communicative experience festschrift in honor of Luigi Anolli. Amsterdam: IOS Press; 2006. p. 25–45.Google Scholar
  103. 103.
    Heeter C. Being there: the subjective experience of presence. Presence: teleoperators and Virtual. Environments. 1992;1(2):262–71.Google Scholar
  104. 104.
    Noë A. Is the visual world a grand illusion? J Conscious Stud. 2002;9(5–6):1–12.Google Scholar
  105. 105.
    Suzuki K, Wakisaka S, Fujii N. Substituting “here and now”—using virtual reality technology. Toward a science of consciousness. Stockholm: Sweden; 2011. p. 120.Google Scholar
  106. 106.
    Brown P. The mechanization of art. In: Husbands P, Holland O, Wheeler M, editors. The mechanical mind in history. Cambridge: MIT Press; 2008. p. 259–82.Google Scholar
  107. 107.
    Ikegami T. Studying a self-sustainable system by making a mind time machine. Workshop on self-sustaining systems (S3). Tokyo, Japan: University of Tokyo; 2010. p. 1–8.Google Scholar
  108. 108.
    Ikegami T, Ogai Y. Self-organization of subjective time and sustainable autonomy in mind time machine. In: Fellermann H, Dörr M, Hanczyc MM, Laursen LL, Maurer S, Merkle D, et al., editors. Artificial Life XII: Proceedings of the twelfth international conference on the synthesis and simulation of living systems. Cambridge, MA: The MIT Press; 2010. p. 624–5.Google Scholar
  109. 109.
    Ikegami T. A design for living technology: experiments with the mind time machine. Artificial Life. 2012 (in press).Google Scholar
  110. 110.
    McGann M, Froese T, Bigge W, Spiers A, Seth AK. The use of a distal-to-tactile sensory substitution interface does not lead to extension of body image. BIO Web of Conferences. 2011;1:00060.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Tom Froese
    • 1
    Email author
  • Keisuke Suzuki
    • 2
  • Yuta Ogai
    • 3
  • Takashi Ikegami
    • 1
  1. 1.Ikegami LaboratoryUniversity of TokyoTokyoJapan
  2. 2.Sackler Centre for Consciousness ScienceUniversity of SussexBrightonUK
  3. 3.Department of Electronics and MechatronicsTokyo Polytechnic UniversityTokyoJapan

Personalised recommendations