Skip to main content
SpringerLink
Log in

Sound design inducing attention in the context of audiovisual immersive environments

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Sound design has been a fundamental component of audiovisual storytelling in linear media. However, with recent technological developments and the shift towards non-linear and immersive media, things are rapidly changing. More sensory information is available and, at the same time, the user is gaining agency upon the narrative, being offered the possibility of navigating or making other decisions. These new characteristics of immersive environments bring new challenges to storytelling in interactive narratives and require new strategies and techniques for audiovisual narrative progression. Can technology offer an immersive environment where the user has the sensation of agency, of choice, where her actions are not mediated by evident controls but subliminally induced in a way that it is ensured that a narrative is being followed? Can sound be a subliminal element that induces attentional focus on the most relevant elements for the narrative, inducing storytelling and biasing search in an immersive non-linear audiovisual environment? Herein, we present a literature review that has been guided by this prospect. With these questions in view, we present our exploration process in finding possible answers and potential solution paths. We point out that consistency, in terms of coherency across sensory modalities and emotional matching may be a critical aspect. Finally, we consider that this review may open up new paths for experimental studies that could, in the future, provide new strategies in the practice of sound design in the context of non-linear media.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adams E (2014) Fundamentals of game design, 3rd edn. Pearson Education, New Riders

    Google Scholar 

  2. Albert G (2012) Immersion as category of audiovisual experience: from long beach to hollywood. Worlds of Audiovision

  3. Alves V, Roque L (2009) A proposal of soundscape design guidelines for user experience enrichment. Audio Mostly 2009:27–32

    Google Scholar 

  4. Andersen RA, Snyder LH, Bradley DC, Xing J (1997) Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annu Rev Neurosci 20(1):303–330

    Article  Google Scholar 

  5. Arsenault D (2005) Dark waters: spotlight on immersion. In: Game-On North America 2005 conference: eurosis. Elsevier, Ghent, pp 50–52

  6. Asutay E, Västfjäll D, Tajadura-Jimenez A, Genell A, Bergman P, Kleiner M (2012) Emoacoustics: a study of the psychoacoustical and psychological dimensions of emotional sound design. J Audio Eng Soc 60(1/2):21–28

    Google Scholar 

  7. Bach DR, Neuhoff JG, Perrig W, Seifritz E (2009) Looming sounds as warning signals: the function of motion cues. Int J Psychophysiol 74(1):28–33

    Article  Google Scholar 

  8. Begault DR (1992) Perceptual effects of synthetic reverberation on three-dimensional audio systems. J Audio Eng Soc 40(11):895–904

    Google Scholar 

  9. Begault DR, Trejo LJ (2000) 3-d sound for virtual reality and multimedia. AP Professional

  10. Blood AJ, Zatorre RJ (2001) Intensely pleasurable responses to music correlate with activity in brain regions implicated in reward and emotion. Proc Natl Acad Sci 98(20):11818–11823

    Article  Google Scholar 

  11. Bradley MM (2008) Natural selective attention: orienting and emotion. Psychophysiology 46:1–11

    Article  Google Scholar 

  12. Bregman AS (1990) Auditory scene analysis: the perceptual organization of sound. MIT Press

  13. Brown E, Cairns P (2004) A grounded investigation of game immersion. In: CHI’04 extended abstracts on Human factors in computing systems. ACM, pp 1297–1300

  14. Van der Burg E, Olivers CN, Bronkhorst AW, Theeuwes J (2008) Pip and pop: nonspatial auditory signals improve spatial visual search. J Exp Psychol Hum Percept Perform 34(5):1053

    Article  Google Scholar 

  15. Callejas A, Lupiáñez J, Tudela P (2004) The three attentional networks: on their independence and interactions. Brain Cogn 54(3):225–227

    Article  Google Scholar 

  16. Callejas A, Lupianez J, Funes MJ, Tudela P (2005) Modulations among the alerting, orienting and executive control networks. Exp Brain Res 167(1):27–37

    Article  Google Scholar 

  17. Chen L, Vroomen J (2013) Intersensory binding across space and time: a tutorial review. Attention, Perception, & Psychophysics 75(5):790–811

    Article  Google Scholar 

  18. Chion M (1994) Audio-vision: sound on screen. Columbia University Press

  19. Cho J, Yi E, Cho G (2001) Physiological responses evoked by fabric sounds and related mechanical and acoustical properties. Text Res J 71(12):1068–1073

    Article  Google Scholar 

  20. Chueng P, Marsden P (2003) Designing auditory spaces: the role of expectation. In: Proceedings of 10th international conference on human computer interaction. Citeseer, pp 616–620

  21. Chun MM, Golomb JD, Turk-Browne NB (2011) A taxonomy of external and internal attention. Annu Rev Psychol 62:73–101

    Article  Google Scholar 

  22. Colby CL, Goldberg ME (1999) Space and attention in parietal cortex. Annu Rev Neurosci 22(1):319–349

    Article  Google Scholar 

  23. Collins K (2011) Making gamers cry: mirror neurons and embodied interaction with game sound. In: Proceedings of the 6th audio mostly conference: a conference on interaction with sound. ACM, pp 39–46

  24. Compton RJ (2003) The interface between emotion and attention: a review of evidence from psychology and neuroscience. Behav Cogn Neurosci Rev 2(2):115–129

    Article  Google Scholar 

  25. Dakic V (2009) Sound design for film and television

  26. Darken RP, Bernatovich D, Lawson JP, Peterson B (1999) Quantitative measures of presence in virtual environments: the roles of attention and spatial comprehension. CyberPsychol Behav 2(4):337–347

    Article  Google Scholar 

  27. Demarque TC, Lima E (2013) Auditory hallucination: audiological perspective for horror games. SBC–Proceedings of SBGames 2013

  28. Doyle J (2013) Subtlety of sound: a study of foley art. Senior honors projects, University of Rhode Island

  29. Driver J, Spence C (1998) Cross-modal links in spatial attention. Philos Trans R Soc London B: Biol Sci 353(1373):1319–1331

    Article  Google Scholar 

  30. Driver J, Spence C (1998) Crossmodal attention. Curr Opin Neurobiol 8(2):245–253

    Article  Google Scholar 

  31. Dura MT (2006) The phenomenology of the music-listening experience. Arts Educ Polic Rev 107(3):25–32

    Article  Google Scholar 

  32. Ekman I (2008) Psychologically motivated techniques for emotional sound in computer games. In: Proc. audio mostly 2008, 3rd conference on interaction with sound, Piteå, pp 20–26

  33. Ermi L, Mäyrä F (2005) Fundamental components of the gameplay experience: analysing immersion. Worlds in Play: International Perspectives on Digital Games Research 37(2):37–53

    Google Scholar 

  34. Fan J, Gu X, Guise KG, Liu X, Fossella J, Wang H, Posner MI (2009) Testing the behavioral interaction and integration of attentional networks. Brain Cogn 70(2):209–220

    Article  Google Scholar 

  35. Freeman J, Lessiter J (2001) Here, there and everywhere: the effects of multichannel audio on presence. In: Proceedings of the seventh international conference on audio display. GIT, Helsinki, pp 231–234

  36. Garner T, Grimshaw M, Nabi DA (2010) A preliminary experiment to assess the fear value of preselected sound parameters in a survival horror game. In: Proceedings of the 5th audio mostly conference: a conference on interaction with sound. ACM, p 10

  37. Garner TA, Grimshaw M (2013) Psychophysiological assessment of fear experience in response to sound during computer video gameplay. In: IADIS HCCI Conference

  38. Gaver WW (1993) How do we hear in the world? Explorations in ecological acoustics. Ecol Psychol 5(4):285–313

    Article  MathSciNet  Google Scholar 

  39. Grani F, Overholt D, Erkut C, Gelineck S, Triantafyllidis G, Nordahl R, Serafin S (2015) Spatial sound and multimodal interaction in immersive environments. In: Proceedings of the audio mostly 2015 on interaction with sound, pp 1–5

  40. Grau O (2003) Virtual art: from illusion to immersion. MIT Press

  41. Grimshaw M (2007) The resonating spaces of first-person shooter games. In: Proceedings of the 5th international conference on game design and technology

  42. Grimshaw M, Charlton J, Jagger R (2011) First-person shooters: immersion and attention. Eludamos J Comput Game Culture 5(1):29–44

    Google Scholar 

  43. Handel S (2006) Perceptual coherence: hearing and seeing. Oxford University Press

  44. Hendrix C, Barfield W (1996) Presence within virtual environments as a function of visual display parameters. Presence: Teleoperators Virt Environ 5(3):274–289

    Article  Google Scholar 

  45. Ihde D (2007) Listening and voice: phenomenologies of sound, 2nd edn. Suny Press

  46. Iordanescu L, Guzman-Martinez E, Grabowecky M, Suzuki S (2008) Characteristic sounds facilitate visual search. Psychon Bull Rev 15(3):548–554

    Article  Google Scholar 

  47. Iuppa N, Borst T (2012) Story and simulations for serious games: tales from the trenches. CRC Press

  48. James W (1890) The principles of psychology, vol 1. Henry Holt, New York

    Book  Google Scholar 

  49. Jennett C, Cox AL, Cairns P, Dhoparee S, Epps A, Tijs T, Walton A (2008) Measuring and defining the experience of immersion in games. Int J Human-Comput Stud 66(9):641–661

    Article  Google Scholar 

  50. Jennett CI (2010) Is game immersion just another form of selective attention? an empirical investigation of real world dissociation in computer game immersion. PhD thesis, UCL (University College London)

  51. Kitamura MS, Watanabe K, Kitagawa N (2016) Positive emotion facilitates audiovisual binding. Front Integr Neurosci 9:66

    Article  Google Scholar 

  52. Larsson P, Västfjäll D, Kleiner M (2004) Perception of self-motion and presence in auditory virtual environments. In: Proceedings of seventh annual workshop presence, pp 252– 258

  53. Larsson P, Västfjäll D, Olsson P, Kleiner M (2007) When what you hear is what you see: presence and auditory-visual integration in virtual environments. In: Proceedings of the 10th annual international workshop on presence, pp 11–18

  54. Lipscomb SD (1999) Cross-modal integration: synchronization of auditory and visual components in simple and complex media. J Acoust Soc Am 105(2):1274

    Article  Google Scholar 

  55. McMahan A (2003) Immersion, engagement and presence. Video Game Theory Reader 67:86

    Google Scholar 

  56. Mole C (2008) Attention and consciousness. J Conscious Stud 15(4):86–104

    Google Scholar 

  57. Moncrieff S, Dorai C, Venkatesh S (2001) Affect computing in film through sound energy dynamics. In: Proceedings of the ninth ACM international conference on multimedia. ACM, pp 525– 527

  58. Murray JH (2017) Hamlet on the holodeck: the future of narrative in cyberspace. MIT press

  59. Nilsson NC, Nordahl R, Serafin S (2016) Immersion revisited: a review of existing definitions of immersion and their relation to different theories of presence. Hum Technol, 12

  60. Noesselt T, Bergmann D, Hake M, Heinze HJ, Fendrich R (2008) Sound increases the saliency of visual events. Brain Res 1220:157–163

    Article  Google Scholar 

  61. Ozawa K, Ohtake S, Suzuki Y, Sone T (2003) Effects of visual information on auditory presence. Acoust Sci Technol 24(2):97–99

    Article  Google Scholar 

  62. Parker JR, Heerema J (2008) Audio interaction in computer mediated games. Int J Comput Games Technol 2008:1

    Article  Google Scholar 

  63. Pellegrini RS (2001) Quality assessment of auditory virtual environments. In: Hiipakka J, Zacharov N, Takala T (eds) Proceedings of the 7th international conference on auditory display. Laboratory of acoustics and audio signal processing and the telecommunications software and multimedia laboratory, pp 161–168

  64. Petersen SE, Posner MI (2012) The attention system of the human brain: 20 years after. Ann Rev Neurosci 35:73–89

    Article  Google Scholar 

  65. Posner MI (ed) (2012) Cognitive neuroscience of attention, 2nd edn. Guilford Press, New York

    Google Scholar 

  66. Posner MI, Boies SJ (1971) Components of attention. Psychol Rev 78(5):391–408

    Article  Google Scholar 

  67. Posner MI, Petersen SE (1990) The attention system of the human brain. Annu Rev Neurosci 13(1):25–42

    Article  Google Scholar 

  68. Raymond J (2009) Interactions of attention, emotion and motivation. Prog Brain Res 176:293–308

    Article  Google Scholar 

  69. Raz A, Buhle J (2006) Typologies of attentional networks. Nat Rev Neurosci 7(5):367–379

    Article  Google Scholar 

  70. Rooney B, Benson C, Hennessy E (2012) The apparent reality of movies and emotional arousal: a study using physiological and self-report measures. Poetics 40(5):405–422

    Article  Google Scholar 

  71. Rueda MR, Fan J, McCandliss BD, Halparin JD, Gruber DB, Lercari LP, Posner MI (2004) Development of attentional networks in childhood. Neuropsychologia 42(8):1029–1040

    Article  Google Scholar 

  72. Ryan ML (2003) Narrative as virtual reality: immersion and interactivity in literature and electronic media. Johns Hopkins University Press

  73. Salselas I, Penha R (2019) The role of sound in inducing storytelling in immersive environments. In: Proceedings of the 14th international audio mostly conference: a journey in sound on ZZZ. ACM, pp 191–198

  74. Schutz M, Lipscomb S (2007) Hearing gestures, seeing music: vision influences perceived tone duration. Perception 36(6):888–897

    Article  Google Scholar 

  75. Serafin G, Serafin S (2004) Sound design to enhance presence in photorealistic virtual reality. In: Georgia Institute of Technology

  76. Slater M (2009) Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Philos Trans R Soc London: Biol Sci 364(1535):3549–3557

    Article  Google Scholar 

  77. Sodnik J, Tomazic S, Grasset R, Duenser A, Billinghurst M (2006) Spatial sound localization in an augmented reality environment. In: Proceedings of the 18th Australia conference on computer-human interaction: design: activities, artefacts and environments, pp 111–118

  78. Sonnenschein D (2001) Sound design: the expressive power of music, voice, and sound effects in cinema. Michael Wiese Productions

  79. Spence C, Driver J (1997) Audiovisual links in exogenous covert spatial orienting. Percept Psychophys 59 (1):1–22

    Article  Google Scholar 

  80. Spence CJD (1997) Engineering psychology and cognitive ergonomics: volume two – job design and product design, chap Audiovisual Links in Attention Implications for Interface Design, Routledge

  81. Tajadura-Jiménez A, Larsson P, Väljamäe A, Västfjäll D, Kleiner M (2010) When room size matters: acoustic influences on emotional responses to sounds. Emotion 10(3):416

    Article  Google Scholar 

  82. Västfjäll D (2003) The subjective sense of presence, emotion recognition, and experienced emotions in auditory virtual environments. CyberPsychol Behav 6(2):181–188

    Article  Google Scholar 

  83. Västfjäll D, Larsson P, Kleiner M (2002) Emotion and auditory virtual environments: affect-based judgments of music reproduced with virtual reverberation times. CyberPsychol Behav 5(1):19–32

    Article  Google Scholar 

  84. Viers R (2011) Sound effects bible. Michael Wiese Productions

  85. Visch VT, Tan ES, Molenaar D (2010) The emotional and cognitive effect of immersion in film viewing. Cogn Emotion 24(8):1439–1445

    Article  Google Scholar 

  86. Watzl S (2017) Structuring mind: the nature of attention and how it shapes consciousness. Oxford University Press

  87. Wenzel E, Wightman F, Kistler D, Foster S (1988) Acoustic origins of individual differences in sound localization behavior. J Acoust Soc Am 84(S1):S79–S79

    Article  Google Scholar 

  88. Witmer BG, Singer MJ (1998) Measuring presence in virtual environments: a presence questionnaire. Presence 7(3):225–240

    Article  Google Scholar 

  89. Wu W (2011) What is conscious attention? Philos Phenomenol Res 82(1):93–120

    Article  Google Scholar 

  90. Yewdall DL (2012) Practical art of motion picture sound. Routledge

  91. Yiend J (2010) The effects of emotion on attention: a review of attentional processing of emotional information. Cogn Emotion 24(1):3–47

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. FEUP, INESC TEC, CITAR, Porto, Portugal

    Inês Salselas

  2. ESMAE-IPP, INESC TEC, Porto, Portugal

    Rui Penha

  3. FEUP, INESC TEC, Porto, Portugal

    Gilberto Bernardes

Authors
  1. Inês Salselas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui Penha
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gilberto Bernardes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Inês Salselas.

Ethics declarations

Conflict of interests

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Salselas, I., Penha, R. & Bernardes, G. Sound design inducing attention in the context of audiovisual immersive environments. Pers Ubiquit Comput 25, 737–748 (2021). https://doi.org/10.1007/s00779-020-01386-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-020-01386-3

Keywords

Search

Navigation