In Search for an Integrated Design Basis for Audio and Haptics

  • Antti Pirhonen
  • Kai Tuuri
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5270)


Audio and haptics as interaction modalities share properties, which make them highly appropriate to be handled within a single conceptual framework. This paper outlines such framework, gaining ingredients from the literature concerning cross-modal integration and embodied cognition. The resulting framework is bound up with a concept of physical embodiment, which has been introduced within several scientific disciplines to reveal the role of bodily experience and the corresponding mental imagery as the core of meaning-creation. In addition to theoretical discussion, the contribution of the proposed approach in design is outlined.


haptics audio integration multimodal embodiment 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Card, S.K., Moran, T.P., Newell, A.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Hillsdale (1983)Google Scholar
  2. 2.
    Gardner, H.G.: The Mind’s New Science. Howard Gardner, New York (1985)Google Scholar
  3. 3.
    Broadbent, D.E.: Perception and Communication. Pergamon, London (1958)Google Scholar
  4. 4.
    Deutsch, J.A., Deutsch, D.: Attention: Some Theoretical Considerations. Psychological Review 70(1), 80–90 (1963)CrossRefGoogle Scholar
  5. 5.
    Wickens, C.D.: Processing Resources in Attention. In: Parasuraman, R., Davies, D.R. (eds.) Varieties of Attention, pp. 63–102. Academic Press, Orlando, F (1984)Google Scholar
  6. 6.
    Pirhonen, A.: Semantics of Sounds and Images - Can They Be Paralleled? In: Proc. of International Conference on Auditory Display Schulich School of Music, pp. 319–325. McGill University Montreal, Canada (2007)Google Scholar
  7. 7.
    Pirhonen, A., Palomäki, H.: Sonification of Directional and Emotional Content: Description of Design Challenges. In: Proc. of International Conference on Auditory Display, IRCAM, Paris (2008)Google Scholar
  8. 8.
    McLuhan, M.: Understanding Media: the Extensions of Man. New American Library, New York (1966)Google Scholar
  9. 9.
    Shannon, C.E., Weaver, W.: The Mathematical Theory of Communication. The University of Illinois Press, Urbana (1949)zbMATHGoogle Scholar
  10. 10.
    Edwards, A.D.N.: Redundancy and Adaptability. In: Edwards, A.D.N., Holland, S. (eds.) Multimedia Interface Design in Education. NATO ASI Series F: Computer and System Sciences, vol. 76, pp. 145–155. Springer, Heidelberg (1992)Google Scholar
  11. 11.
    Johnson, M., Rohrer, T.: We Are Live Creatures: Embodiment,American Pragmatism, and the Cognitive Organism. In: Zlatev, J., Ziemke, T., Frank, R., Dirven, R. (eds.) Body, Language, and Mind, vol. 1, pp. 17–54. Mouton de Gruyter, Berlin (2007)Google Scholar
  12. 12.
    Lakoff, G., Johnson, M.: Philosophy in the Flesh: The Embodied Mind and its Challenge to Western Thought. Basic Books, New York (1999)Google Scholar
  13. 13.
    Gallese, V., Lakoff, G.: The Brain’s Concepts: The Role of the Sensory-Motor System in Reason and Language. Cognitive Neuropsychology 22, 455–479 (2005)CrossRefGoogle Scholar
  14. 14.
    Liberman, A.M., Mattingly, I.G.: The Motor Theory of Speech Perception Revised. Cognition 21, 136 (1985)Google Scholar
  15. 15.
    Wilson, M., Knoblich, G.: The Case for Motor Involvement in Perceiving Conspecifics. Psychological Bulletin 1(3), 460–473 (2005)Google Scholar
  16. 16.
    Godoy, R.I.: Motor-Mimetic Music Cognition. Leonardo 36(4), 317–319 (2003)CrossRefGoogle Scholar
  17. 17.
    Gallese, V.: Embodied Simulation: From Mirror Neuron Systems to Interpersonal Relations. In: Empathy and Fairness (Novartis Foundation Symposium), vol. 278, pp. 3–19. Wiley, Chichester (2006)CrossRefGoogle Scholar
  18. 18.
    Johnson, M.: The Body in the Mind: The Bodily Basis of Meaning, Imagination, and Reason. University of Chicago, Chicago (1987)Google Scholar
  19. 19.
    Godøy, R.I.: Gestural Imagery in the Service of Musical Imagery. In: Camurri, A., Volpe, G. (eds.) GW 2003. LNCS (LNAI), vol. 2915, pp. 55–62. Springer, Heidelberg (2004)Google Scholar
  20. 20.
    Tagg, P.: Towards a Sign Typology of Music. In: Dalmonte, R., Baroni, M. (eds.) Secondo Convegno Europeo di Analisi Musicale, pp. 369–378. Universitá Degli Studi di Trento (1992)Google Scholar
  21. 21.
    Dourish, P.: Where the Action is: The Foundations of Embodied Interaction. MIT Press, Cambridge (2001)Google Scholar
  22. 22.
    Pirhonen, A.: To Simulate or to Stimulate? In Search of the Power of Metaphor in Design. In: Pirhonen, A., Isomäki, H., Roast, C., Saariluoma, P. (eds.) Future Interaction Design, pp. 105–123. Springer, London (2005)CrossRefGoogle Scholar
  23. 23.
    Jensenius, A.: Action-Sound: Developing Methods and Tools to Study Music-Related Body Movement. Ph.D. Thesis. Department of Musicology, University of Oslo (2007)Google Scholar
  24. 24.
    Godøy, R. I.. Imagined action, excitation, and resonance. In: Godøy, R. I. and Jørgensen, H. (eds.): Musical Imagery. Swets and Zeitlinger, Lisse (2001) 237-250Google Scholar
  25. 25.
    Tuuri, K., Eerola, T.: Could Function-Specific Prosodic Cues Be Used as a Basis for Non-Speech User Interface Sound Design. In: Proc. of International Conference on Auditory Display, IRCAM, Paris (2008)Google Scholar
  26. 26.
    Csikszentmihalyi, M.: Creativity: Flow and the psychology of discovery and invention. HarperCollins, New York (1996)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Antti Pirhonen
    • 1
  • Kai Tuuri
    • 1
  1. 1.Department of Computer Science and Information SystemsUniversity of JyväskyläFinland

Personalised recommendations