Skip to main content

Spatial Soundscape Superposition and Multimodal Interaction

  • Chapter
  • First Online:
The Technology of Binaural Understanding

Part of the book series: Modern Acoustics and Signal Processing ((MASP))

Abstract

Contemporary listeners are exposed to overlaid cacophonies of sonic sources, both intentional and incidental.  Such soundscape superposition can be usefully characterized by where such combination actually occurs: in the air, at the ears of listeners, in the auditory imagery subjectively evoked by such events, or in whatever audio equipment is used to mix, transmit, and display such signals. This chapter regards superposition of spatial soundscapes: physically, perceptually, and procedurally. Physical (acoustic) superposition describes such aspects as configuration of personal sound transducers, panning among multiple sources, speaker arrays, and the modern challenge of how to integrate and exploit mobile devices and “smart speakers.”  Perceptual (subjective and psychological) superposition describes such aspects as binaural image formation, auditory objects and spaces, and multimodal sensory interpretation. Procedural (logical and cognitive) superposition describes higher-level relaxation of insistence upon literal auralization, leveraging idiom and convention to enhance practical expressiveness, metaphorical mappings between real objects and virtual position such as separation of direction and distance; range -compression and -indifference; layering of soundscapes;  audio windowing, narrowcasting, and multipresence as strategies for managing privacy; and mixed reality deployments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 249.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    A “sink” is the dual of a source, used instead of “listener” to distinguish it from an actual human, including allowing designation of multiple sinks for a single user, as explained in Sect.  4.8 below.

  2. 2.

    Similar plug-ins are also offered by other companies, including Facebook (https://facebookincubator.github.io/facebook-360-spatial-workstation/), Microsoft (https://docs.microsoft.com/en-us/azure/cognitive-services/acoustics/what-is-acoustics), and Yamaha (https://research.yamaha.com/ja/technologies/vireal/).

References

  • Alam, S., M. Cohen, J. Villegas, and A. Ahmed. 2009. Narrowcasting in SIP: Articulated privacy control. In SIP Handbook: Services, Technologies, and Security of Session Initiation Protocol, ed. S.A. Ahson, and M. Ilyas, 323–345. Boca Raton: CRC Press, Taylor & Francis. Chap. 14. https://doi.org/10.1201/9781315218939.

  • Bailey, R. 2007. Spatial emphasis of game audio: How to create theatrically enhanced audio. In Audio Anecdotes III, ed. K. Greenebaum, and R. Barzel, 399–406. Wellesley: A K Peters/CRC Press. https://doi.org/10.1201/9781439864869.

  • Bauck, J.L., and D.H. Cooper. 1996. Generalized transaural stereo and applications. Journal of the Audio Engineering Society 44 (9): 683–705. http://www.aes.org/e-lib/browse.cfm?elib=7888.

  • Begault, D.R. 1994. 3-D Sound for Virtual Reality and Multimedia. Boston: Academic Press. ISBN 978-0120847358

    Google Scholar 

  • Blauert, J. 2012. A perceptionist’s view on psychoacoustics. Arch. Acoust. 37 (3): 365–371. https://doi.org/10.2478/v10168-012-0046-z.

    Article  Google Scholar 

  • Blauert, J. 2017. “Reading the World with Two Ears” Keynote at Int. Congress on Sound and Vibration, London, https://www.youtube.com/watch?v=p1kDtgqmTdw.

  • Blauert, J., D. Kolossa, K. Obermayer, and K. Adiloğlu. 2013. Further challenges and the road ahead. Modern Acoustics and Signal Processing, 477–501. Berlin: Springer. https://doi.org/10.1007/978-3-642-37762-4_18. Chap. 18.

    Chapter  Google Scholar 

  • Bregman, A.S. 1990. Auditory Scene Analysis: The Perceptual Organization of Sound. Cambridge: MIT Press.

    Book  Google Scholar 

  • Brimijoin, W.O., and M.A. Akeroyd. 2012. The role of head movements and signal spectrum in an auditory front/back illusion. i-Perception 3 (3): 179–182. https://doi.org/10.1068/i7173sas.

  • Broll, W., I. Lindt, I. Herbst, J. Ohlenburg, A.-K. Braun, and R. Wetzel. 2008. Towards next-gen mobile AR games. Computer Graphics and Animation 28 (4): 40–48. https://doi.org/10.1109/MCG.2008.85.

  • Cabrera, D., H. Sato, W. Martens, and D. Lee. 2009. Binaural measurement and simulation of the room acoustical response from a person’s mouth to their ears. Acoustics Australia 37: 98–103.

    Google Scholar 

  • Choi, J.-W., B.J. Cho, and I. Shin. 2016. Toward the holographic reconstruction of sound fields using smart sound devices. IEEE MultiMedia 23 (3): 64–74. https://doi.org/10.1109/MMUL.2016.46.

    Article  Google Scholar 

  • Choueiri, E. 2018. Binaural audio through loudspeakers, in Roginska and Geluso. https://doi.org/10.4324/9781315707525. Chap. 5.

  • Cohen, M. 1993. Throwing, pitching, and catching sound: Audio windowing models and modes. IJMMS: Journal of Person-Computer Interaction 39 (2): 269–304. https://doi.org/10.1006/imms.1993.1062.

    Article  Google Scholar 

  • Cohen, M. 1998. Quantity of presence: Beyond person, number, and pronouns. In Cyberworlds, ed. T.L. Kunii, and A. Luciani, 289–308. Tokyo: Springer. https://doi.org/10.1007/978-4-431-67941-7_19. Chap. 19.

  • Cohen, M. 2000. Exclude and include for audio sources and sinks: Analogs of mute & solo are deafen & attend. Presence: Teleoperators and Virtual Environments 9 (1): 84–96. https://doi.org/10.1162/105474600566637.

    Article  Google Scholar 

  • Cohen, M. 2015. Hierarchical narrowcasting. In Proceedings of HCII: International Conference on Human-Computer Interaction– DAPI: International Conference on Distributed, Ambient and Pervasive Interactions, ed. N. Streitz and P. Markopoulos, 274–286. Los Angeles: LNCS 9189. https://doi.org/10.1007/978-3-319-20804-6_25.

  • Cohen, M. 2016. Dimensions of spatial sound and interface styles of audio augmented reality: Whereware, wearware, & everyware. In Fundamentals of Wearable Computers and Augmented Reality, ed. W. Barfield, 277–308. Mahwah: CRC Press. https://doi.org/10.1201/b18703. Chap. 12.

  • Cohen, M., and O.N.N. Fernando. 2009. Awareware: Narrowcasting attributes for selective attention, privacy, and multipresence. In Awareness Systems: Advances in Theory, Methodology and Design, ed. P. Markopoulos and W. Mackay, 259–289. London: Springer. https://doi.org/10.1007/978-1-84882-477-5. Chap. 11.

  • Cohen, M., O.N.N. Fernando, U.C. Dumindawardana, and M. Kawaguchi. 2009. Duplex narrowcasting operations for multipresent groupware avatars on mobile devices. IJWMC: International Journal of Wireless and Mobile Computing 3 (4): 280–287. https://doi.org/10.1504/IJWMC.2009.029348.

    Article  Google Scholar 

  • Cohen, M., and H. Kojima. 2018. Multipresence and autofocus for interpreted narrowcasting. In AES: Audio Engineering Society International Conference on Spatial Reproduction—Aesthetics and Science, Tokyo.

    Google Scholar 

  • Cohen, M., W.L. Martens. 2020. Spatial soundscape superposition, Part II: Signals and systems. Acoustical Science and Technology 41.1 (Jan. 2020). ed. by Masato Akagi, Masashi Unoki, and Yoshifumi Chisaki. JASJ 76 (1): 297–307. ISSN: 1347-5177, 1346-3969, 0369-4232. https://doi.org/10.1250/ast.41.297.

  • Cohen, M., and J. Villegas. 2016. Applications of audio augmented reality: Wearware, everyware, anyware, & awareware. In Fundamentals of Wearable Computers and Augmented Reality, 2nd ed, ed. W. Barfield, 309–330. Mahwah: CRC Press. https://www.taylorfrancis.com/books/9780429192395. Chap. 13.

  • Collins, K. (ed.). 2008. Game Sound. Cambridge: MIT Press. https://doi.org/10.7551/mitpress/7909.001.0001. ISBN 978-0-262-03378-7.

  • Crum, P. 2019. Here come the hearables: Technology tucked inside your ears will augment your daily life. IEEE Spectrum 56 (5): 38–43. https://doi.org/10.1109/MSPEC.2019.8701198.

    Article  Google Scholar 

  • Francombe, J., J. Woodcock, R.J. Hughes, R. Mason, A. Franck, C. Pike, T. Brookes, W.J. Davies, P.J.B. Jackson, T.J. Cox, F.M. Fazi, and A. Hilton. 2018. Qualitative evaluation of media device orchestration for immersive spatial audio reproduction. Journal of the Audio Engineering Society 66 (6): 414–429.

    Google Scholar 

  • Hartmann, W.M. 1999. Signals, Sound, and Sensation. New York: AIP Press.

    Book  Google Scholar 

  • Herder, J., and M. Cohen. 2002. The helical keyboard: Perspectives for spatial auditory displays and visual music. JNMR: Journal of New Music Research 31 (3): 269–281. https://doi.org/10.1076/jnmr.31.3.269.14180.

  • Hugonnet, C., and P. Walder. 1998. Stereophonic Sound Recording: Theory and Practice. Chichester: Wiley. ISBN 978-0471974871.

    Google Scholar 

  • Inoue, A., Y. Ikeda, K. Yatabe, and Y. Oikawa. 2017. Three-dimensional sound-field visualization system using head mounted display and stereo camera. In Proceedings of ASA Meetings on Acoustics, vol. 29. https://doi.org/10.1121/2.0000381.

  • Jekosch, U. 2005. Assigning meaning to sounds—semiotics in the context of product-sound design. In Communication Acoustics, ed. J. Blauert. Berlin: Springer. https://doi.org/10.1007/3-540-27437-5_8. Chap. 8.

  • Jo, H., W.L. Martens, Y. Park, and S. Kim. 2010. Confirming the perception of virtual source elevation effects created using 5.1 channel surround sound playback. In VRCAI: Proceedings of International Conference on Virtual-Reality Continuum and Its Applications in Industry, 103–110. Seoul: ACM. https://doi.org/10.1145/1900179.1900200.

  • Jot, J.-M. 1999. Real-time spatial processing of sounds for music, multimedia and interactive human-computer interfaces. Multimedia Systems 7 (1): 55–69.

    Article  Google Scholar 

  • Kawaura, J., Y. Suzuki, F. Asano, and T. Sone. 1991. Sound localization in headphone reproduction by simulating transfer functions from the sound source to the external ear. Journal of the Acoustical Society of Japan (E) 12 (5): 203–216. https://doi.org/10.1250/ast.12.203.

    Article  Google Scholar 

  • Kendall, G. 2010. Spatial perception and cognition in multichannel audio for electroacoustic music. Organised Sound 15 (3): 228–238. https://doi.org/10.1017/S1355771810000336.

    Article  Google Scholar 

  • Kim, C., R. Mason, and T. Brookes. 2013. Head movements made by listeners in experimental and real-life listening activities. Journal of Audio Engineering Society 61 (6): 425–438. http://www.aes.org/e-lib/browse.cfm?elib=16833.

  • Kleiner, M., B.-I. Dalenbäck, and P. Svensson. 1993. Auralization— an overview. Journal of Audio Engineering Society 41 (11): 861–875. http://www.aes.org/e-lib/browse.cfm?elib=6976.

  • Kondo, H.M., D. Pressnitzer, I. Toshima, and M. Kashino. 2012. Effects of self-motion on auditory scene analysis. Proceedings of the National Academy of Sciences 109 (17): 6775–6780.

    Article  ADS  Google Scholar 

  • Lackner, J.R. 1977. Induction of nystagmus in stationary subjects with a rotating sound field. Aviation, Space and Environmental Medicine 48 (2): 129–131.

    Google Scholar 

  • Lackner, J.R. 1983. Influence of posture on the spatial localization of sound. Journal of Audio Engineering Society 31 (9): 650–661. http://www.aes.org/e-lib/browse.cfm?elib=18987.

  • Leung, J., D. Alais, and S. Carlile. 2008. Compression of auditory space during rapid head turns. Proceedings of the National Academy of Sciences 105 (17): 6492–6497. https://doi.org/10.1073/pnas.0710837105.

    Article  ADS  Google Scholar 

  • Lossius, T., P. Baltazar, and T. de la Hogue. 2009. DBAP—Distance-based amplitude panning. In Proceedings of the International Computer Music Conference, ICMC, (Aug. 16–21, 2009) Montréal, Quebec, Canada. https://hdl.handle.net/2027/spo.bbp2372.2009.111.

  • Lyon, E., ed. 2016. Computer Music J.: High-Density Loudspeaker Arrays, Part 1: Institutions, 40, https://doi.org/10.1162/COMJ_e_00388.

  • Lyon, E., ed. 2017. Computer Music J.: High-Density Loudspeaker Arrays, Part 2: Spatial Perception and Creative Practice, 41, https://doi.org/10.1162/COMJ_a_00403.

  • Macpherson, E.A. 2013. Cue weighting and vestibular mediation of temporal dynamics in sound localization via head rotation. In Proceedings of Meetings on Acoustics, Vol. 19, p. 050131. https://doi.org/10.1121/1.4799913.

  • Martens, W., S. Sakamoto, L. Miranda, and D. Cabrera. 2013. Dominance of head-motion-coupled directional cues over other cues during walking depends upon source spectrum. In Proceedings of Meetings on Acoustics, Vol. 19, p. 050129. https://doi.org/10.1121/1.4800124.

  • Martens, W.L., D. Cabrera, and S. Kim. 2011. The ‘phantom walker’ illusion: Evidence for the dominance of dynamic interaural over spectral directional cues during walking. In Principles and Applications of Spatial Hearing, ed. Y. Suzuki, D. Brungart, Y. Iwaya, K. Iida, D. Cabrera, and H. Kato, 81–102. Singapore: World Scientific. https://doi.org/10.1142/7674.

  • Martens, W.L., and M. Cohen. 2020. Spatial soundscape superposition, Part I: Subject motion and scene sensibility. In Acoustical Science and Technology 41.1 (Jan. 2020). ed. by Masato Akagi, Masashi Unoki, and Yoshifumi Chisaki. JASJ 76 (1): 288–296. ISSN: 1347-5177, 1346-3969, 0369-4232. https://doi.org/10.1250/ast.41.288.

  • Martens, W.L., Y. Han. 2018. Discrimination of auditory spatial diffuseness facilitated by head rolling while listening to ‘with-height’ versus ‘without-height’ multichannel loudspeaker reproduction. In Proceedings of Audio Engineering Society International Conference on Spatial Reproduction, Tokyo. http://www.aes.org/e-lib/browse.cfm?elib=19608.

  • Marui, A., and W.L. Martens. 2006. Spatial character and quality assessment of selected stereophonic image enhancements for headphone playback of popular music. In AES: Audio Engineering Society Conv. (\(120\)th Conv.), Paris. http://www.aes.org/e-lib/browse.cfm?elib=13622.

  • Middlebrooks, J.C., J.Z. Simon, A.N. Popper, and R.R. Fay (eds.). 2017. The Auditory System at the Cocktail Party. Cham: Springer. https://doi.org/10.1007/978-3-319-51662-2.

  • Milgram, P., and H. Colquhoun Jr. 1999. A taxonomy of real and virtual world display integration. In Mixed Reality: Merging Real and Virtual Worlds, ed. Y. Ohta and H. Tamura, 5–30. Omsha: Springer. Chap. 1. ISBN 978-3-642-87514-4

    Google Scholar 

  • Ochiai, Y., T. Hoshi, and I. Suzuki. 2017. Holographic whisper: Rendering audible sound spots in three-dimensional space by focusing ultrasonic waves. In Proceedings of CHI Conference on Human Factors in Computing Systems, 4314–4325. New York. https://doi.org/10.1145/3025453.3025989.

  • Pastore, M.T., Y. Zhou, and W.A. Yost. 2020. Cross-modal and cognitive processes in sound localization. In The Technology of Binaural Understanding, eds. J. Blauert and J. Braasch, 315–350. Cham, Switzerland: Springer. Chap. 12. https://doi.org/10.1007/978-3-030-00386-9_12.

  • Pereira, F., and W.L. Martens. 2018. Psychophysical validation of binaurally processed sound superimposed upon environmental sound via an unobstructed pinna and an open-ear-canal earspeaker. In Proceedings of Audio Engineering Society International Conference on Spatial Reproduction, Tokyo. http://www.aes.org/e-lib/browse.cfm?elib=19626.

  • Perrott, D.R., H. Ambarsoom, and J. Tucker. 1987. Changes in head position as a measure of auditory localization performance: Auditory psychomotor coordination under monaural and binaural listening conditions. Journal of the Acoustical Society of America 82 (5): 1637–1645.

    Article  ADS  Google Scholar 

  • Plenge, G. 1974. On the difference between localization and lateralization. Journal of the Acoustical Society of America 56: 944–951. https://doi.org/10.1121/1.1903353.

    Article  ADS  Google Scholar 

  • Pulkki, V. 1997. Virtual source positioning using vector base amplitude panning. Journal of the Audio Engineering Society 45 (6): 456–466.

    Google Scholar 

  • Pulkki, V., T. Lokki, and D. Rocchesso. 2011. Spatial effects. In DAFX: Digital Audio Effects, 2nd ed, ed. U. Zölzer, 139–184. West Sussex: Wiley. https://doi.org/10.1002/9781119991298.ch5. Chap. 5.

  • Ranaweera, R., M. Cohen, and M. Frishkopf. 2015. Narrowcasting and multipresence for music auditioning and conferencing in social cyberworlds. Presence: Teleoperators and Virtual Environments 24 (3): 220–242, https://doi.org/10.1162/PRES_a_00232.

  • Roginska, A., and P. Geluso (eds.). 2018. Immersive Sound: The Art and Science of Binaural and Multi-channel Audio. Routledge: Taylor & Francis. https://doi.org/10.4324/9781315707525.

  • Satongar, D., C. Pike, Y.W. Lam, and A.I. Tew. 2015. The influence of headphones on the localization of external loudspeaker sources. Journal of the Audio Engineering Society 63 (10): 3–19. https://doi.org/10.17743/jaes.2015.0072.

    Article  Google Scholar 

  • Seldess, Z. 2014. “MIAP: Manifold-interface Amplitude Panning in Max/MSP and Pure Data” in Audio Engineering Society Convention 137, Los Angeles, http://www.aes.org/e-lib/browse.cfm?elib=17435.

  • Shaw, E.A.G. 1982. 1979 Rayleigh medal lecture: The elusive connection. In Localization of Sound: Theory and Applications, ed. R. Gatehouse, 13–29. Groton: Amphora Press.

    Google Scholar 

  • Sodnik, J., and S. Tomažič. 2015. Spatial Auditory Human-Computer Interfaces. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-22111-3.

  • Streicher, R., and F.A. Everest. 2006. The New Stereo Soundbook, 3rd ed. Pasadena: Audio Engineering Associates. ISBN 978-0-9665162-1-0.

    Google Scholar 

  • Sullenberger, R.M., S. Kaushik, and C.M. Wynn. 2019. Photoacoustic communications: Delivering audible signals via absorption of light by atmospheric \(\rm H_{2}\rm \rm O\rm \). Optics Letters 44 (3): 622–625. https://doi.org/10.1364/OL.44.000622.

  • Suzuki, Y., A. Honda, Y. Iwaya, M. Ohuchi, and S. Sakamoto. 2020. Binaural display supporting active listening: Perceptual bases and welfare applications initial proposal: Training of spatial perception with binaural displays supporting active listening. In The Technology of Binaural Understanding, ed. J. Blauert and J. Braasch. Berlin: Springer and ASA Press, 665–695. Chap. 22. https://doi.org/10.1007/978-3-030-00386-9_22.

  • Tachi, S. 2015. Telexistence, 2nd ed. Singapore: World Scientific Publishing Company.

    Book  Google Scholar 

  • Tanno, K., A. Saji, and J. Huang. 2014. A 3d sound generation system with horizontally arranged five-channel loudspeakers. IEICE Transactions on Information and Systems 2 (J97-D(5)): 1044–1052.

    Google Scholar 

  • Thurlow, W.R., J.W. Mangles, and P.S. Runge. 1967. Head movements during sound localization. Journal of the Acoustical Society of America 42 (2): 489–493. https://doi.org/10.1121/1.1910605.

    Article  ADS  Google Scholar 

  • Thurlow, W.R., and P.S. Runge. 1967. Effect of induced head movements on localization of direction of sounds. Journal of the Acoustical Society of America 42 (2): 480–488. https://doi.org/10.1121/1.1910604.

    Article  ADS  Google Scholar 

  • Toole, F.E. 1969. In-head localization of acoustic images. Journal of the Acoustical Society of America 48: 943–949. https://doi.org/10.1121/1.1912233.

    Article  ADS  Google Scholar 

  • Villegas, J., and M. Cohen. 2010. Hrir: Modulating range in headphone-reproduced spatial audio. In VRCAI: Proceedings of International Conference on Virtual-Reality Continuum and Its Applications in Industry, Seoul. https://doi.org/10.1145/1900179.1900198.

  • Wallach, H. 1940. The role of head movements and vestibular and visual cues in sound localization. Journal of Experimental Psychology 27: 339–368. https://doi.org/10.1037/h0054629.

    Article  Google Scholar 

  • Wang, D. 2017. Deep learning reinvents the hearing aid. IEEE Spectrum 54 (3): 32–37. https://doi.org/10.1109/MSPEC.2017.7864754.

    Article  Google Scholar 

  • Wenzel, E.M., D.R. Begault, and M. Godfroy-Cooper. 2018. Perception of spatial sound. In Roginska and Geluso (2018), 5–39. https://doi.org/10.4324/9781315707525. Chap. 1.

  • Wersényi, G., and J. Wilson. 2015. Evaluation of head movements in short-term measurements and recordings with human subjects using head-tracking sensors. Acta Technica Jaurinensis 8 (3): 218–229.

    Article  Google Scholar 

  • Wolfe, J. 2018. From idea to acoustics and back again: the creation and analysis of information in music. Substantia 1: 77–91. https://doi.org/10.13128/Substantia-42.

Download references

Acknowledgements

We thank Yôiti Suzuki for his valuable comments and suggestions. This chapter has been reviewed by two anonymous experts.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Michael Cohen .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Cohen, M., Martens, W.L. (2020). Spatial Soundscape Superposition and Multimodal Interaction. In: Blauert, J., Braasch, J. (eds) The Technology of Binaural Understanding. Modern Acoustics and Signal Processing. Springer, Cham. https://doi.org/10.1007/978-3-030-00386-9_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-00386-9_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-00385-2

  • Online ISBN: 978-3-030-00386-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics