Sensory Augmentation: Toward a Dialogue Between the Arts and Sciences

  • Alwin de RooijEmail author
  • Michel van Dartel
  • Antal Ruhl
  • Hanna Schraffenberger
  • Bente van Melick
  • Mathijs Bontje
  • Mischa Daams
  • Michel Witter
Conference paper
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 229)


People sense the world by exploiting correlations between their physical actions and the changing sensory input that results from those actions. Interfaces that translate non-human sensor data to signals that are compatible with the human senses can therefore augment our abilities to make sense of the world. This insight has recently sparked an increase in projects that explore sensemaking and the creation of novel human experiences across scientific and artistic disciplines. However, there currently exists no constructive dialogue between artists and scientists that conduct research on this topic. In this position paper, we identify the theory and practice of sensory augmentation as a domain that could benefit from such a dialogue. We argue that artistic and scientific methods can complement each other within research on sensory augmentation and identify six thematic starting points for a dialogue between the arts and sciences. We conducted a case study to explore these conjectures, in which we instigated such a dialogue on a small scale. The case study revealed that the six themes we identified as relevant for a dialogue on sensory augmentation emerge rather spontaneously in such a dialogue and that such an exchange may facilitate progress on questions that are central to the theory and practice of sensory augmentation. Overall, this position paper contributes preliminary evidence for the potential of, and a starting point for, a dialogue between the arts and sciences that advances our understanding of sensory augmentation and the development of applications that involve it.


Art-science collaboration Cognitive science Interfaces New media art Sensory augmentation Sensory substitution 


  1. 1.
    Gombrich, E.H.: Art and Illusion. Phaidon, London (1977)Google Scholar
  2. 2.
    Schwartzman, M.: See Yourself Sensing: Redefining Human Perception. Black Dog Publishing, London (2011)Google Scholar
  3. 3.
    Noë, A.: Action in Perception. MIT Press, Cambridge (2004)Google Scholar
  4. 4.
    Marr, D.: Vision. W.H. Freeman, San Francisco (1982)Google Scholar
  5. 5.
    O’Regan, J.K., Noë, A.: A sensorimotor account of vision and visual consciousness. Behav. Brain Sci. 24(5), 883–917 (2001)Google Scholar
  6. 6.
    Noë, A.: Strange Tools: Art and Human Nature. Hill and Wang, New York (2015)Google Scholar
  7. 7.
    Manning, E., Massumi, B.: Thought in the Act. University of Minnesota Press, Minneapolis (2014)CrossRefGoogle Scholar
  8. 8.
    Van Dartel, M.F.: Aesthetics in the Wild: Art and Design Practices and Pedagogies after the Situated Turn. Avans, Breda (2016)Google Scholar
  9. 9.
    Biggs, M., Karlsson, H.: Research in the Arts. Routledge, New York (2012)Google Scholar
  10. 10.
    Froese, T., McGann, M., Bigge, W., Spiers, A., Seth, A.K.: The enactive torch: a new tool for the science of perception. IEEE Trans. Haptics 5(4), 365–375 (2012)CrossRefGoogle Scholar
  11. 11.
    Stratton, G.M.: Vision without inversion of the retinal image. Psychol. Rev. 4, 463–481 (1897)CrossRefGoogle Scholar
  12. 12.
    Kohler, I.: The Formation and Transformation of the Perceptual World. Psychological Issues, vol. 3. International University Press (1964). Monograph 12Google Scholar
  13. 13.
    Miyauchi, S., et al.: Adaptation to left-right reversed vision rapidly activates ipsilateral visual cortex in humans. J. Physiol. Paris 98, 207–219 (2004)CrossRefGoogle Scholar
  14. 14.
    Bach-y-Rita, P., Collins, C.C., Saunders, S.A., White, B., Scadden, L.: Vision substitution by tactile image projection. Nature 221, 963–964 (1969)CrossRefGoogle Scholar
  15. 15.
    Kercel, S.W., Bach-y-Rita, P.: Noninvasive coupling of electronically generated data into the human nervous system. In: Akay, M. (ed.) Wiley Encyclopedia of Biomedical Engineering, pp. 1960–1974. Wiley (2006)Google Scholar
  16. 16.
    Kaspar, K., König, S., Schwandt, J., König, P.: The experience of new sensorimotor contingencies by sensory augmentation. Conscious. Cogn. 28, 47–63 (2014)CrossRefGoogle Scholar
  17. 17.
    Ward, J., Meijer, P.: Visual experiences in the blind induced by an auditory sensory substitution device. Conscious. Cogn. 19, 492–500 (2010)CrossRefGoogle Scholar
  18. 18.
    Deroy, O., Auvray, M.: Reading the world through the skin and ears: a new perspective on sensory substitution. Front. Psychol. 3, 457 (2012)CrossRefGoogle Scholar
  19. 19.
    Sha, X.W.: Poiesis and Enchantment in Topological Matter. MIT Press, Cambridge (2013)Google Scholar
  20. 20.
    Novich, N.: Sound-to-touch sensory substitution and beyond. Ph.D. thesis. Rice University (2015)Google Scholar
  21. 21.
    De Cupere, P.: Blind smell stick (2012). Accessed 2 Nov 2016
  22. 22.
    De Rooij, A., Broekens, J., Lamers, M.H.: Abstract expressions of affect. Int. J. Synth. Emot. 4(1), 1–31 (2013)CrossRefGoogle Scholar
  23. 23.
    Evers, F.: The Academy of the Senses: Synesthetics in Science, Art and Education. ArtScience Interfaculty Press, The Hague (2012)Google Scholar
  24. 24.
    Kandinsky, W.: Composition VIII (1924). Accessed 2 Nov 2016
  25. 25.
    Ramachandran, V.S., Hubbard, E.M.: Synaesthesia: a window into perception, thought and language. J. Conscious. Stud. 8(12), 3–34 (2001)Google Scholar
  26. 26.
    Maidenbaum, S.: Practical sensory substitution in real and virtual worlds: development, accessibility and neuroscience. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, pp. 211–214 (2015)Google Scholar
  27. 27.
    Van Dartel, M., Misker, J., Nigten, A., Van der Ster, J.: Virtual reality and augmented reality art explained in terms of sensory-motor coordination. In: Proceedings of 4th International Conference on Enactive Interfaces (2007)Google Scholar
  28. 28.
    Maidenbaum, S., Chebat, D.R., Levy-Tzedek, S., Namer-Furstenberg, R., Amedi, A.: The effect of expanded sensory range via the EyeCane sensory substitution device on the characteristics of visionless virtual navigation. Multi Sens. Res. 27(5–6), 379–397 (2014)CrossRefGoogle Scholar
  29. 29.
    Froese, T., Spiers, A.: Toward a Phenomenological Pragmatics of Enactive Perception. Cognitive Science Research Papers, 593 (2004)Google Scholar
  30. 30.
    O’Rourke, K.: Walking and Mapping. MIT Press, Cambridge (2013)Google Scholar
  31. 31.
    Studio Eyal Burstein: Eye Candy Can Ltd. (2007). Accessed 2 Nov 2016
  32. 32.
    Wallas, G.: The Art of Thought. Harcourt Brace, New York (1926)Google Scholar
  33. 33.
    Grand Illusions. Reversing goggles. Accessed 19 Apr 2017
  34. 34.
    de Rooij, A., Jones, S.: Mood and creativity: an appraisal tendency perspective. In: Proceedings of the 9th ACM Conference on Creativity and Cognition, pp. 362–365 (2013)Google Scholar

Copyright information

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2018

Authors and Affiliations

  • Alwin de Rooij
    • 1
    Email author
  • Michel van Dartel
    • 2
  • Antal Ruhl
    • 2
  • Hanna Schraffenberger
    • 1
  • Bente van Melick
    • 1
  • Mathijs Bontje
    • 1
  • Mischa Daams
    • 2
  • Michel Witter
    • 2
  1. 1.Tilburg UniversityTilburgThe Netherlands
  2. 2.Avans University of Applied SciencesDen BoschThe Netherlands

Personalised recommendations