SELFHOOD: An Evolutionary and Interactive Experience Synthesizing Images and Sounds

  • Jonatas Manzolli
  • Artemis MoroniEmail author
  • Guilherme A. Valarini
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11265)


The SELFHOOD installation was conceived aiming to instigate a reflection on the self through a practical and interactive experience. A representation of each participant is created in a form of a cloud of points and a sound drone, suggesting their selves. The dynamics of the visitors’ movements is sonified in such way that colours and sound textures are fused in a surrounding hexaphonic system. CromaCrono≈, the system for immersive improvisation that produces digitally synthesized sounds in real time, is described. Philosophical concepts concerning notions of the Self are presented. We propose that the notion of Presence can be induced by virtual and/or physical sources of stimulation governed by a number of principles that underlie human experience, creativity, and discovery. The methodological point of view is that the notion of Presence indicates that there are essential inputs for the construction of self-referral agents.


Evolutionary systems Interactivity Multimodal installation Presence Self Sound synthesis 



We thank Thomas Taichi Okubo for his commitment and dedication. We thank Georges Schwarzstein for his careful review and appropriate suggestions. Okubo and Valarini were supported by the PIBIC/CNPq program and Manzolli is supported by projects 470358/2014-9 and 305065/2014-9 also from CNPq. The sensory devices used on SELHOOD installation were provided by project FAPESP 2013/26453-1.


  1. 1.
    Mura, A., Manzolli, J., Verschure, P. F., Rezazadeh, B., Groux, S. L., Wierenga, S.: re(PER)curso: a mixed reality chronicle. In: Proceedings of the 2008 ACM SIGGRAPH Conference (2008)Google Scholar
  2. 2.
    Inderbitzin, M., Wierenga, S., Väljama, A., Bernardet, U., Verschure, P.F.M.J.: Social cooperation and competition in the mixed reality space eXperience Induction Machine (XIM). Virtual Reality 13, 153–158 (2009)CrossRefGoogle Scholar
  3. 3.
    Papachristodoulou, P., Betella, A., Manzolli, J., Verschure, P.F.M.J.: Augmenting the navigation of complex data sets using sonification: a case study with BrainX3. In: Virtual Reality (VR), IEEE (2015)Google Scholar
  4. 4.
    Miranda, E.R., Wanderley, M.M.: New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. A-R Editions, Middleton (2006)Google Scholar
  5. 5.
    Bernhardt, D., Robinson, P.: Detecting affect from non-stylised body motions. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 59–70. Springer, Heidelberg (2007). Scholar
  6. 6.
    Bernhardt, D., Robinson, P.: Interactive control of music using emotional body expressions. In: Human Factors in Computing Systems, pp. 3117—3122 (2008)Google Scholar
  7. 7.
    Castelli, F., Happé, F., Frith, U., Frith, C.: Movement and mind: a functional imaging study of perception and interpretation of complex intentional movement patterns. NeuroImage 12(3), 314–325 (2000)CrossRefGoogle Scholar
  8. 8.
    Reynolds, C.W.: Flocks, herds, and schools: a distributed behavioral model. In: SIGGRAPH Conference Proceedings, vol. 21, pp. 25–34 (1987)CrossRefGoogle Scholar
  9. 9.
    Colton, S., Wiggins G.A.: Computational creativity: the final frontier? In: 20th European Conference on Artificial Intelligence (2012)Google Scholar
  10. 10.
    Sternberg, R.S. (ed.): Handbook of Creativity. Cambridge University Press, Cambridge (1999)Google Scholar
  11. 11.
    Pope, R.: Creativity: Theory, History, Practice. Routledge, New York (2005)Google Scholar
  12. 12.
    Boden, M.: What is creativity? In: Boden, M. (ed.) Dimensions of Creativity, pp. 75—117. MIT Press, Cambridge (2005)Google Scholar
  13. 13.
    Bernardet, U.: The eXperience induction machine: a new paradigm for mixed-reality interaction design and psychological experimentation. In: Dubois, M., Gray, P., Nigay, L. (eds.) The Engineering of Mixed Reality Systems. HCIS, pp. 357–379. Springer, London (2010). Scholar
  14. 14.
    Wasserman, K., Manzolli, J., Eng, K., Verschure, P.F.M.J.: Live soundscape composition based on synthetic emotions: using music to communicate between an interactive exhibition and its visitors. IEEE Multimedia 10, 82–90 (2003)CrossRefGoogle Scholar
  15. 15.
    Prescott, T.J., Verschure, P.F.M.J., Lepora, N.: A Handbook of Research in Biomimetic and Biohybrid Systems. Oxford University Press, Oxford, In Press (To appear)Google Scholar
  16. 16.
    Strawson, G.: The self. J. Conscious. Stud. 4, 405–428 (1997)Google Scholar
  17. 17.
    Descartes, R.: The Philosophical Writings of Descartes, vols. 1 and 2, translated by J. Cottingham et al. Cambridge University Press, Cambridge (1985)Google Scholar
  18. 18.
    Hume, D.: Dialogues Concerning Natural Religion. In: Smith, N.K. (ed.). Nelson, Edinburgh (1947)Google Scholar
  19. 19.
    Fichte, J.G.: The Science of Knowledge, translated by Heath P. and Lachs, J. CUP, Cambridge (1982)Google Scholar
  20. 20.
    James, W.: The Principles of Psychology. Dover, New York (1950)Google Scholar
  21. 21.
    James, W.: Psychology: Briefer Course. Harvard University Press, Cambridge (1984)Google Scholar
  22. 22.
    Husserl, E.: Cartesian Meditations, translated by D. Cairns. Nijhoff, The Hague (1973)CrossRefGoogle Scholar
  23. 23.
    Ginsberg, A.: Statement to the Burning Bush. Burning Bush II, San Francisco (1963)Google Scholar
  24. 24.
    Nozick, R.: Philosophical Explanations. Clarendon Press, Oxford (1981)Google Scholar
  25. 25.
    Kenny, A.: The Self. Marquette University Press, Marquette (1988)Google Scholar
  26. 26.
    Dennett, D.: Consciousness Explained. Little, Brown, Boston (1991)Google Scholar
  27. 27.
    Wöllner, C. (ed.): Body, Sound and Space in Music and Beyond: Multimodal Explorations. Routledge, New York (2017)Google Scholar
  28. 28.
    Krausz, M., Dutton, D., Bardsley, K.: The Idea of Creativity. Brill Academic Publishers, Leiden (2009)Google Scholar
  29. 29.
    Sanchez-Vives, M.V., Slater, M.: From presence to consciousness through virtual reality. Nat. Rev. Neurosci. 6(4), 332–339 (2005)CrossRefGoogle Scholar
  30. 30.
    Anderson, M.L.: Embodied cognition: a field guide. Artif. Intell. 149(1), 91–130 (2003)MathSciNetCrossRefGoogle Scholar
  31. 31.
    Moroni, A., Manzolli, J., Valarini, G.: SELFHOOD Installation.
  32. 32.
    Moroni, A., Manzolli, J.: SELFHOOD—cloudiness of the selves.
  33. 33.
    Castellano, G., Villalba, S.D., Camurri, A.: Recognising human emotions from body movement and gesture dynamics. In: Paiva, A.C.R., Prada, R., Picard, R.W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 71–82. Springer, Heidelberg (2007). Scholar
  34. 34.
    Le Groux, S., Manzolli, J., Verschure, P.F.M.J.: Disembodied and collaborative musical interaction in the Multimodal Brain Orchestra. In: Proceedings of NIME, pp. 309–314 (2010)Google Scholar
  35. 35.
    Heppner, F.H.: Avian flight formations. Bird-Banding 45(2), 160–169 (1974)CrossRefGoogle Scholar
  36. 36.
    Manzolli, J.: continuaMENTE: Integrating percussion, audiovisual and improvisation. In: Proceedings of the International Computer Music Conference, Belfast, Ireland, (2008)Google Scholar
  37. 37.
    Moroni, A., Manzolli, J.: From evolutionary composition to robotic sonification. In: Di Chio, C., et al. (eds.) EvoApplications 2010. LNCS, vol. 6025, pp. 401–410. Springer, Heidelberg (2010). Scholar
  38. 38.
    Moroni, A., Von Zuben, F., Manzolli, J.: ArTbitration: human-machine interaction in artistic domains. Leonardo (Oxford) 35, 185–188 (2002)CrossRefGoogle Scholar
  39. 39.
    Wassermann, K.C., Manzolli, J., Eng, K., Verschure, P.: Live soundscape composition based on synthetic emotions: using music to communicate between an interactive exhibition and its visitors. IEEE Multimedia 10, 82–90 (2003)CrossRefGoogle Scholar
  40. 40.
    Verschure, P., Manzolli, J.: Computational modeling of mind and music. In: Language, Music, and the Brain. ESF Reports, MIT Press (2013)Google Scholar
  41. 41.
    Manzolli, J.: Multimodal generative installations and the creation of new art form based on interactive narratives. In: GA2015—XVIII Generative Art Conference, (2015).
  42. 42.
    Manzolli, J.: CromaCrono ≈ .
  43. 43.
    Reynolds, C.W.: Flocks, herds and schools: a distributed behavioral model. ACM SIGGRAPH Comput. Graph. 21(4), 25–34 (1987)CrossRefGoogle Scholar
  44. 44.
    Karplus, K., Strong, A.: Digital synthesis of plucked string and drum timbres. Comput. Music J. 7(2), 43–55 (1983)CrossRefGoogle Scholar
  45. 45.
    Macedo, L., Cardoso, A., Reisenzein, R., Lorini, E., Castelfranchi C.: Artificial surprise. In: Handbook of Research on Synthetic Emotions Sociable Robotics New Applications in Affective Computing Artificial Intelligence. IGI Global, Hershey, USA (2009)Google Scholar
  46. 46.
    Paraense, A.L.O., et al.: The cognitive systems toolkit and the CST reference cognitive architecture. Biol. Inspired Cogn. Archit. J. 17, 32–48 (2016)Google Scholar
  47. 47.
    Gudwin, R., Paraense, A., de Paula, S.M., Fróes, E., Gibaut, W., Castro, E., Raizer, K.: The multipurpose enhanced cognitive architecture (MECA). Biol. Inspired Cogn. Archit. J. 22, 20–34 (2017)Google Scholar
  48. 48.
    Dourish, P.: Where the Action Is: Foundations of Embodied Interaction. MIT Press, Cambridge (2001)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Jonatas Manzolli
    • 1
  • Artemis Moroni
    • 2
  • Guilherme A. Valarini
    • 2
  1. 1.IA, NICS/University of Campinas – UNICAMPCampinasBrazil
  2. 2.CTI Renato ArcherCampinasBrazil

Personalised recommendations