Skip to main content

Mapping for meaning: the embodied sonification listening model and its implications for the mapping problem in sonic information design

Journal on Multimodal User Interfaces volume 14, pages 143–151 (2020)Cite this article

Abstract

This is a theoretical paper that considers the mapping problem, a foundational issue which arises when designing a sonification, as it applies to sonic information design. We argue that this problem can be addressed by using models from the field of embodied cognitive science, including embodied image schema theory, conceptual metaphor theory and conceptual blends, and from research which treats sound and musical structures using these models, when mapping data to sound. However, there are currently very few theoretical frameworks for applying embodied cognition principles in a sonic information design context. This article describes one such framework, the embodied sonification listening model, which provides a theoretical description of sonification listening in terms of conceptual metaphor theory.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1

References

  1. Faste T, Faste H (2012) Demystifying “design research”: design is not research, research is design. Presented at Industrial Designers Society of America Education Symposium, Boston, Massachusetts

  2. Flowers JH (2005) Thirteen years of reflection on auditory graphing: promises, pitfalls, and potential new directions. Faculty Publications, Department of Psychology, 430

  3. Worrall D (2019) Intelligible sonifications. In: Sonification design: from data to intelligible soundfields. Springer, Berlin

  4. Truax B (1984) Acoustic communication. Ablex, Norwood

    Google Scholar 

  5. O’Callaghan C (2007) Sounds: a philosophical theory. Oxford University Press, Oxford

    Book  Google Scholar 

  6. Kahn D (1999) Noise, water, meat: a history of sound in the arts. MIT press, Cambridge

    Book  Google Scholar 

  7. LaBelle B (2010) Acoustic territories: sound culture and everyday life. A&C Black, London

    Google Scholar 

  8. Cox C (2011) Beyond representation and signification: toward a sonic materialism. J Vis Cult 10(2):145–161

    Article  Google Scholar 

  9. Roddy S, Bridges B (2016) Sounding human with data: the role of embodied conceptual metaphors and aesthetics in representing and exploring data sets. In: The proceedings of the MusTWork 2016 the music technology workshop

  10. Roddy S, Bridges B (2018) Addressing the mapping problem in sonic information design through embodied image schemata, conceptual metaphors and conceptual blending. J Sonic Stud 17

  11. Worrall D (2013) Understanding the need for micro-gestural inflections in parameter-mapping sonification. Georgia Institute of Technology, Atlanta

    Google Scholar 

  12. Worrall D (2014) Can micro-gestural inflections be used to improve the soniculatory effectiveness of parameter mapping sonifications? Organ Sound 19(1):52–59

    Article  Google Scholar 

  13. Grond F, Berger J (2011) Parameter mapping sonification. In: Hermann T, Hunt A, Neuhoff JG (eds) The sonification handbook. Logos Publishing House, Berlin, pp 363–397

    Google Scholar 

  14. Ryle G (1949) The concept of mind. Hutchinson, London

    Google Scholar 

  15. Searle JR (1980) Minds, brains, and programs. Behav Brain Sci 3(03):417–424

    Article  Google Scholar 

  16. Harnad S (1990) The symbol grounding problem. Phys D Nonlinear Phenom 42(1):335–346

    Article  Google Scholar 

  17. Dreyfus HL (1965) Alchemy and artificial intelligence. The Rand Corporation, Santa Monica, Research Report

  18. Polanyi M (2012) Personal knowledge. Routledge, London

    Book  Google Scholar 

  19. Varela FJ, Thompson E, Rosch E (1991) The embodied mind: cognitive science and human experience. MIT Press, Cambridge

    Book  Google Scholar 

  20. Johnson M (1987) The body in the mind: the bodily basis of meaning, imagination, and reason. University of Chicago Press, Chicago

    Book  Google Scholar 

  21. Lakoff G, Johnson M (1980) Metaphors we live by. University of Chicago Press, Chicago

    Google Scholar 

  22. Fauconnier G, Turner M (2002) The way we think. Conceptual blending and the mind’s hidden complexities. Basic Books, New York

  23. Imaz M, Benyon D (2007) Designing with blends: conceptual foundations of human–computer interaction and software engineering methods. MIT Press, Cambridge

    Google Scholar 

  24. Hurtienne J (2009) Image schemas and design for intuitive use. Exploring new guidance for user interface design. (Doctoral Thesis). Retrieved from Deposit Once TU Berlin Reposityory for Research and Data Publications. TU Berlin identifier opus3 2970

  25. Waterworth J, Riva G (2014) Feeling present in the physical world and in computer-mediated environments. Palgrave Macmillan, London

    Book  Google Scholar 

  26. Bødker S, Klokmose CN (2016). Dynamics, multiplicity and conceptual blends in HCI. In: Proceedings of the 2016 CHI conference on human factors in computing systems. ACM, pp 2538–2548

  27. Dourish P (2004) Where the action is: the foundations of embodied interaction. MIT press, Cambridge

    Google Scholar 

  28. Serafin S, Franinovic K, Hermann T, Lemaitre G, Rinott M, Rocchesso D (2011) Sonic interaction design. In: Hermann T, Hunt A, Neuhoff JG (eds) The Sonification Handbook. Logos Publishing. House, Berlin, pp 87–110

    Google Scholar 

  29. Diniz N, Deweppe A, Demey M, Leman M (2010) A framework for music-based interactive sonification. In: 16th International conference on auditory display (ICAD-2010)

  30. Diniz N, Coussement P, Deweppe A, Demey M, Leman M (2012) An embodied music cognition approach to multilevel interactive sonification. J Multimodal User Interfaces 5(3–4):211–219

    Article  Google Scholar 

  31. Dyer J, Stapleton P, Rodger MW (2015) Sonification as concurrent augmented feedback for motor skill learning and the importance of mapping design. Open Psychol J 8(3):1–11

    Google Scholar 

  32. Dyer J, Stapleton P, Rodger M (2017) Transposing musical skill: sonification of movement as concurrent augmented feedback enhances learning in a bimanual task. Psychol Res 81(4):850–862

    Article  Google Scholar 

  33. Peres SC, Verona D, Nisar T, Ritchey P (2017) Towards a systematic approach to real-time sonification design for surface electromyography. Displays 47:25–31

    Article  Google Scholar 

  34. Cox A (2001) The mimetic hypothesis and embodied musical meaning. Musicae Scientiae 5(2):195–212

    Article  Google Scholar 

  35. Brower C (2000) A cognitive theory of musical meaning. J Music Theory 44(2):323–379

    Article  Google Scholar 

  36. Adlington R (2003) Moving beyond motion: metaphors for changing sound. J R Music Assoc 128(2):297–318

    Article  Google Scholar 

  37. Zbikowski LM (2002) Conceptualizing music: cognitive structure, theory, and analysis. Oxford University Press, Oxford

    Book  Google Scholar 

  38. Larson S (2012) Musical forces: motion, metaphor, and meaning in music. Indiana University Press, Bloomington

    Google Scholar 

  39. Kendall GS (2010) Meaning in electroacoustic music and the everyday mind. Organ Sound 15(1):63–74

    Article  Google Scholar 

  40. Kendall GS (2014) The feeling blend: feeling and emotion in electroacoustic art. Organ Sound 19(2):192

    Article  Google Scholar 

  41. Graham R, Bridges B (2014) Gesture and embodied metaphor in spatial music performance systems design. In: NIME, pp 581–584

  42. Graham R, Bridges B (2014) Strategies for spatial music performance: the practicalities and aesthetics of responsive systems design. Divergence Press

  43. Smalley D (1996) The listening imagination: listening in the electroacoustic era. Contemp Music Rev 13(2):77–107

    MathSciNet  Article  Google Scholar 

  44. Smalley D (1997) Spectromorphology: explaining sound-shapes. Organ Sound 2(02):107–126

    Article  Google Scholar 

  45. Godøy RI (2006) Gestural-sonorous objects: embodied extensions of Schaeffer’s conceptual apparatus. Organ Sound 11(02):149–157

    Article  Google Scholar 

  46. Graham R, Manzione C, Bridges B, Brent W (2017) Exploring pitch and timbre through 3d spaces: embodied models in virtual reality as a basis for performance systems design. In: New interfaces for musical expression proceedings

  47. Grey JM, Gordon JW (1978) Perceptual effects of spectral modifications on musical timbres. J Acoust Soc Am 63(5):1493–1500

    Article  Google Scholar 

  48. Vickers P, Hogg B (2006) Sonification ab-straite/sonification concr`ete: An ‘æsthetic perspectivespace’ for classifying auditory displays in the ars musica domain. In: 12th International conference on auditory display (ICAD-2010), pp 210–216

  49. Schaeffer P, Reibel G, Ferreyra B, Chiarucci H, Bayle F, Tanguy A et al (1967) Solfège de l’objet sonore. In: INA/GRM

  50. Chion M, Gorbman C, Murch W (1994) Audio-vision

  51. Gaver WW (1989) The SonicFinder: an interface that uses auditory icons. Hum Comput Interact 4:67–94

    Article  Google Scholar 

  52. Roddy S (2015) Embodied sonification. Doctoral Dissertation. Trinity College Dublin. Ireland

  53. Walker BN, Nees MA (2011) Theory of Sonification. In: Hermann T, Hunt A, Neuhoff JG (eds) The Sonification Handbook. Logos Publishing House, Berlin, pp 9–39

    Google Scholar 

  54. Koestler A (1967) The ghost in the machine. Hutchinson & Co., London

    Google Scholar 

  55. Talmy L (2008) The fundamentals of spatial systems. In: Hampe B (ed) From perception to meaning: image schemas in cognitive linguistics. Walter de Greuyter, Berlin

    Google Scholar 

  56. White M (2003) Metaphor and economics: the case of growth. Engl Specif Purp 22(2):131–151

    Article  Google Scholar 

  57. Walker BN (2002) Magnitude estimation of conceptual data dimensions for use in sonification. J Exp Psychol Appl 8(4):211

    Article  Google Scholar 

  58. Kövecses Z (2010) Metaphor and culture. Acta Universitatis Sapientiae Philologica 2(2):197–220

    Google Scholar 

  59. Chung SF (2005). MARKET metaphors: Chinese, English and Malay. In: Proceedings of the 19th Pacific Asia conference on language, information and computation, pp 71–81

  60. Charteris-Black J, Ennis T (2001) A comparative study of metaphor in Spanish and English financial reporting. Engl Specif Purp 20(3):249–266

    Article  Google Scholar 

  61. Polli A (2012) Soundscape, sonification, and sound activism. AI Soc 27(2):257–268

    Article  Google Scholar 

  62. Jeon M, Lee JH, Sterkenburg J, Plummer C (2015) Cultural differences in preference of auditory emoticons: USA and South Korea. Georgia Institute of Technology, Atlanta

    Google Scholar 

Download references

Acknowledgements

This publication has been funded by an Irish Research Council Government of Ireland Postdoctoral Fellowship Award (Grant no. 14887). This publication has emanated from research supported in part by a research grant from Science Foundation Ireland (SFI) and is co- funded under the European Regional Development Fund under Grant No. 13/RC/2077.

Author information

Authors and Affiliations

  1. Electronic and Electrical Engineering, Trinity College Dublin, Dublin 2, Ireland

    Stephen Roddy

  2. School of Arts and Humanities, Ulster University, Magee Campus, Derry/Londonderry, BT48 7JL, Northern Ireland

    Brian Bridges

Authors
  1. Stephen Roddy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian Bridges
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen Roddy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Roddy, S., Bridges, B. Mapping for meaning: the embodied sonification listening model and its implications for the mapping problem in sonic information design. J Multimodal User Interfaces 14, 143–151 (2020). https://doi.org/10.1007/s12193-020-00318-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-020-00318-y

Keywords

  • Auditory display
  • Sonification
  • Conceptual metaphor
  • Image schema
  • Conceptual blending