From Signal to Substance and Back: Insights from Environmental Sound Research to Auditory Display Design

  • Brian Gygi
  • Valeriy Shafiro
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5954)

Abstract

A persistent concern in the field of auditory display design has been how to effectively use environmental sounds, which are naturally occurring familiar non-speech, non-musical sounds. Environmental sounds represent physical events in the everyday world, and thus they have a semantic content that enables learning and recognition. However, unless used appropriately, their functions in auditory displays may cause problems. One of the main considerations in using environmental sounds as auditory icons is how to ensure the identifiability of the sound sources. The identifiability of an auditory icon depends on both the intrinsic acoustic properties of the sound it represents, and on the semantic fit of the sound to its context, i.e., whether the context is one in which the sound naturally occurs or would be unlikely to occur. Relatively recent research has yielded some insights into both of these factors. A second major consideration is how to use the source properties to represent events in the auditory display. This entails parameterizing the environmental sounds so the acoustics will both relate to source properties familiar to the user and convey meaningful new information to the user. Finally, particular considerations come into play when designing auditory displays for special populations, such as hearing impaired listeners who may not have access to all the acoustic information available to a normal hearing listener, or to elderly or other individuals whose cognitive resources may be diminished. Some guidelines for designing displays for these populations will be outlined.

Keywords

Environmental Sounds Auditory Display Auditory Icons Special Populations 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Gygi, B.: Studying environmental sounds the Watson way. J. Acoust. Soc. Am. 115(5), 2574 (2004)Google Scholar
  2. 2.
    Mynatt, E.: Designing with auditory icons. In: Proceedings of the 2nd International Conference on Auditory Display (ICAD 1994), Santa Fe, NM, U.S, pp. 109–119 (1994)Google Scholar
  3. 3.
    Zhao, F., Stephens, S.D.G., Sim, S.W., Meredith, R.: The use of qualitative questionnaires in patients having and being considered for cochlear implants. Clin. Otolaryngol. 22, 254–259 (1997)CrossRefGoogle Scholar
  4. 4.
    Warren, R.M.: Auditory perception and speech evolution. Annals of the New York Academy of Sciences, Origins and Evolution of Language 280, 708–717 (1976)CrossRefGoogle Scholar
  5. 5.
    Gaver, W.W.: Auditory icons: using sound in computer interfaces. Hum. Comput. Interact. 2(2), 167–177 (1986)CrossRefGoogle Scholar
  6. 6.
    Gaver, W.W.: The SonicFinder: An Interface that Uses Auditory Icons. Hum. Comput. Interact. 4(1), 67–94 (1989)CrossRefMathSciNetGoogle Scholar
  7. 7.
    Kunkler-Peck, A.J., Turvey, M.T.: Hearing shape. J. Exp. Psychol. Human 26(1), 279–294 (2000)CrossRefGoogle Scholar
  8. 8.
    Carello, C., Anderson, K.L., Kunkler-Peck, A.J.: Perception of object length by sound. Psychol. Sci. 9(3), 211–214 (1998)CrossRefGoogle Scholar
  9. 9.
    Li, X., Logan, R.J., Pastore, R.E.: Perception of acoustic source characteristics: Walking sounds. J. Acoust. Soc. Am. 90(6), 3036–3049 (1991)CrossRefGoogle Scholar
  10. 10.
    Keller, P., Stevens, C.: Meaning From Environmental Sounds: Types of Signal-Referent Relations and Their Effect on Recognizing Auditory Icons. J. Exp. Psychol.-Appl. 10(1), 3–12 (2004)CrossRefGoogle Scholar
  11. 11.
    Absar, R., Guastavino, C.: Usability of Non-Speech Sounds in User Interfaces. In: Proceedings of the 14th International Conference on Auditory Display Paris, France, pp. 1–8 (2008)Google Scholar
  12. 12.
    Fernstroem, M., Brazil, E., Bannon, L.: HCI Design and Interactive Sonification for Fingers and Ears. IEEE MultiMedia 12(2), 36–44 (2005)CrossRefGoogle Scholar
  13. 13.
    Watson, C.S., Kidd, G.R.: Factors in the design of effective auditory displays. In: Proceedings of the 2nd International Conference on Auditory Display (ICAD 1994), Santa Fe, NM, U.S, pp. 293–303 (1994)Google Scholar
  14. 14.
    Mynatt, E.: Designing with auditory icons. In: Proceedings of the 2nd International Conference on Auditory Display (ICAD 1994), pp. 109–119. International Community for Auditory Display, Santa Fe, NM, U.S (1994)Google Scholar
  15. 15.
    Gaver, W.W.: Using and creating auditory icons. In: Kramer, G. (ed.) SFI studies in the sciences of complexity, pp. 417–446. Addison Wesley, Longman (1992)Google Scholar
  16. 16.
    Ballas, J.: Delivery of information through sound. In: Kramer, G. (ed.) SFI studies in the sciences of complexity, pp. 79–94. Addison Wesley, Longman (1992)Google Scholar
  17. 17.
    Ballas, J.A.: What is that sound? Some implications for sound design. In: Design Sonore, Paris, pp. 1–12 (2002)Google Scholar
  18. 18.
    Lucas, P.A.: An evaluation of the communicative ability of auditory icons and earcons. In: Proceedings of the 2nd International Conference on Auditory Display (ICAD 1994), Santa Fe, NM, U.S, pp. 121–128 (1994)Google Scholar
  19. 19.
    Lass, N.J., Eastman, S.K., Parrish, W.C., Ralph, D.: Listeners’ identification of environmental sounds. Percept. Motor. Skill. 55(1), 75–78 (1982)Google Scholar
  20. 20.
    Miller, J.D., Tanis, D.C.: Recognition memory for common sounds. Psychon. Sci. 23(4), 307–308 (1973)Google Scholar
  21. 21.
    Lawrence, D.M., Banks, W.P.: Accuracy of recognition memory for common sounds. Bull. Psychonom. Soc. 1(5A), 298–300 (1973)Google Scholar
  22. 22.
    Vanderveer, N.J.: Ecological acoustics: Human perception of environmental sounds. Dissertation Abstracts International 40(9-B), 4543 (1980)Google Scholar
  23. 23.
    Ballas, J.A.: Common factors in the identification of an assortment of brief everyday sounds. J. Exp. Psychol. Human 19(2), 250–267 (1993)CrossRefGoogle Scholar
  24. 24.
    Fabiani, M., Kazmerski, V.A., Cycowicz, Y.M.: Naming norms for brief environmental sounds: Effects of age and dementia. Psychophys. 33(4), 462–475 (1996)CrossRefGoogle Scholar
  25. 25.
    Gygi, B., Kidd, G.R., Watson, C.S.: Spectral-temporal factors in the identification of environmental sounds. J. Acoust. Soc. Am. 115(3), 1252–1265 (2004)CrossRefGoogle Scholar
  26. 26.
    Marcell, M.M., Borella, D., Greene, M., Kerr, E., Rogers, S.: Confrontation naming of environmental sounds. J. Clin. Exp. Neuropsyc. 22(6), 830–864 (2000)CrossRefGoogle Scholar
  27. 27.
    Myers, L.L., Letowski, T.R., Abouchacra, K.S., Kalb, J.T., Haas, E.C.: Detection and recognition of octave-band sound effects. J Am. Acad. Otolayrn 7, 346–357 (1996)Google Scholar
  28. 28.
    Shafiro, V.: Identification of environmental sounds with varying spectral resolution. Ear. Hear. 29(3), 401–420 (2008)CrossRefGoogle Scholar
  29. 29.
    Shafiro, V., Gygi, B., Cheng, M.-Y., Mulvey, M., Holmes, B.: Perception of speech and environmental sounds in cochlear implant patients. J. Acoust. Soc. Am. 123(5), 3303 (2008)CrossRefGoogle Scholar
  30. 30.
    Guillaume, A., Pellieux, L., Chastres, V., Blancard, C.: How long does it take to identify everyday sounds. In: ICAD 2004 -Tenth Meeting of the International Conference on Auditory Display, Sydney, Australia, pp. ICAD04-1–ICAD04-4 (2004)Google Scholar
  31. 31.
    Gygi, B.: From acoustics to perception: How to listen to meaningful sounds in a meaningful way. J. Acoust. Soc. Am. 113(4), 2326 (2003)Google Scholar
  32. 32.
    Warren, W.H., Verbrugge, R.R.: Auditory perception of breaking and bouncing events: A case study in ecological acoustics. J. Exp. Psychol. Human. 10(5), 704–712 (1984)CrossRefGoogle Scholar
  33. 33.
    Gaver, W.W.: What in the world do we hear? An ecological approach to auditory event perception. Ecol. Psychol. 5(1), 1–29 (1993)CrossRefMathSciNetGoogle Scholar
  34. 34.
    Mynatt, E.D.: Transforming graphical interfaces into auditory interfaces for blind users. Hum. Comput. Interact. 12(1), 7–45 (1997)CrossRefGoogle Scholar
  35. 35.
    Bilger, R.C., Nuetzel, J.M., Rabinowitz, W.M., Rzeczkowski, C.: Standardization of a test of speech perception in noise. J. Speech. Hear. Res. 27, 32–48 (1984)Google Scholar
  36. 36.
    Hafter, E.R., Saberi, K.: A level of stimulus representation model for auditory detection and attention. J. Acoust. Soc. Am. 110(3), 1489 (2001)CrossRefGoogle Scholar
  37. 37.
    Schlauch, R.S., Hafter, E.R.: Listening bandwidths and frequency uncertainty in pure-tone signal detection. J. Acoust. Soc. Am. 90, 1332–1339 (1991)CrossRefGoogle Scholar
  38. 38.
    Watson, C.S., Foyle, D.C.: Central factors in the discrimination and identification of complex sounds. J. Acoust. Soc. Am. 78(1), 375–380 (1985)CrossRefGoogle Scholar
  39. 39.
    Ballas, J.A., Mullins, T.: Effects of context on the identification of everyday sounds. Hum. Perform. 4(3), 199–219 (1991)CrossRefGoogle Scholar
  40. 40.
    Gygi, B., Shafiro, V.: The Incongruency Advantage for Environmental Sounds Presented in Natural Auditory Scenes. Submitted (2008)Google Scholar
  41. 41.
    Gordon, R.D.: Attentional allocation during the perception of scenes. J. Exp. Psychol. Human. 30, 760–777 (2004)CrossRefGoogle Scholar
  42. 42.
    Lucas, P.A.: An evaluation of the communicative ability of auditory icons and earcons. In: Proceedings of the 2nd International Conference on Auditory Display (ICAD 1994), pp. 121–128. International Community for Auditory Display, Santa Fe, NM, U.S (1994)Google Scholar
  43. 43.
    Petrie, H., Morley, S.: The use of non-speech sounds in non-visual interfaces to the MS Windows GUI for blind computer users. In: Proceedings of the 5th International Conference on Auditory Display (ICAD 1998), pp. 1–5. University of Glasgow, U.K (1998)Google Scholar
  44. 44.
    Kidd, G.R., Watson, C.S.: The perceptual dimensionality of environmental sounds. Noise Cont. Eng. J. 51(4), 216–231 (2003)CrossRefGoogle Scholar
  45. 45.
    Gaver, W.W.: How do we hear in the world? Explorations in ecological acoustics. Ecol. Psychol. 5(4), 285–313 (1993)CrossRefMathSciNetGoogle Scholar
  46. 46.
    William, W.G.: Auditory icons: using sound in computer interfaces. Hum. Comput. Interact. 2(2), 167–177 (1986)CrossRefMathSciNetGoogle Scholar
  47. 47.
    Coleman, P.D.: An analysis of cues to auditory depth perception in free space. Psychological Bulletin 60(3), 302 (1963)CrossRefGoogle Scholar
  48. 48.
    Freed, D.: Auditory correlates of perceived mallet hardness for a set of recorded percussive sound events. J. Acoust. Soc. Am. 87(1), 311–322 (1990)CrossRefGoogle Scholar
  49. 49.
    Grey, J.M.: Multidimensional perceptual scaling of musical timbres. J. Acoust. Soc. Am. 61(5), 1270–1277 (1977)CrossRefGoogle Scholar
  50. 50.
    Lakatos, S., McAdams, S., Caussé, R.: The representation of auditory source characteristics: Simple geometric form. Percept. Psychophys. 59(8), 1180–1190 (1997)Google Scholar
  51. 51.
    Gygi, B., Kidd, G.R., Watson, C.S.: Similarity and Categorization of Environmental Sounds. Percept. Psychophys. 69(6), 839–855 (2007)Google Scholar
  52. 52.
    Aramaki, M., Kronland-Martinet, R.: Analysis-synthesis of impact sounds by real-time dynamic filtering. IEEE T Speech Audi P 14(2), 1–9 (2006)CrossRefGoogle Scholar
  53. 53.
    Avanzini, F.: Synthesis of Environmental Sounds in Interactive Multimodal Systems. In: Proceedings of the 13th International Conference on Auditory Display (ICAD 2007), Montreal, Canada, pp. 181–188 (2007)Google Scholar
  54. 54.
    Lee, J.-F., Shen, I.Y., Crouch, J., Aviles, W., Zeltzer, D., Durlach, N.: Using physically based models for collision-sound synthesis in virtual environments. J. Acoust. Soc. Am. 95(5), 2967 (1994)CrossRefGoogle Scholar
  55. 55.
    Cook, P.R.: Physically inspired sonic modeling (PhISM): Synthesis of percussive sounds. Comput. Music J. 21, 38–49 (1997)CrossRefGoogle Scholar
  56. 56.
    Rath, M.: An expressive real-time sound model of rolling. In: The 6th International Conference on Digital Audio Effects (DAFX 2003), pp. 165–168. University of London, Queen Mary (2003)Google Scholar
  57. 57.
    Stoelinga, C., Chaigne, A.: Time-Domain Modeling and Simulation of Rolling Objects. Acta Acust United Ac 93, 290–304 (2007)Google Scholar
  58. 58.
    van den Doel, K.: Physically-Based Models for Liquid Sounds. In: Tenth Meeting of the International Conference on Auditory Display, Sydney, Australia, pp. 1–8 (2004)Google Scholar
  59. 59.
    Avanzini, F., Serafin, S., Rocchesso, D.: Modeling Interactions Between Rubbed Dry Surfaces Using an Elasto-Plastic Friction Model. In: Proceedings of the COST-G6 Conf. Digital Audio Effects (DAFX 2002), Hamburg, pp. 111–116 (2002)Google Scholar
  60. 60.
    Lakatos, S., Cook, P.C., Scavone, G.P.: Selective attention to the parameters of a physically informed sonic model. J. Acoust. Soc. Am. 107(5,Pt1), L31-L36 (2000)CrossRefGoogle Scholar
  61. 61.
    Lutfi, R.A., Liu, C.-J.: Individual differences in source identification from synthesized impact sounds. J. Acoust. Soc. Am. 122(2), 1017–1028 (2007)CrossRefGoogle Scholar
  62. 62.
    Reed, R.K., Kidd, G.R.: Detection of Spectral Changes in Everyday Sounds. Indiana University, Unpublished data (2007)Google Scholar
  63. 63.
    Fernström, M., Brazil, E.: Human-Computer Interaction Design based on Interactive Sonification - Hearing Actions or Instruments/Agents. In: Proceedings of the 2004 International Workshop on Interactive Sonification, pp. 1–4. Bielefeld University, Germany (2004)Google Scholar
  64. 64.
    Heller, L.M.W.: When sound effects are better than the real thing. J. Acoust. Soc. Am. 111(5 pt.2), 2339 (2002)Google Scholar
  65. 65.
    Moore, B.C.J.: Cochlear hearing loss, pp. 47–88. Whurr Publishers, London (1998)Google Scholar
  66. 66.
    Badran, S., Osama, E.L.: Speech and environmental sound perception difficulties by patients with hearing loss requiring and using hearing aid. Indian J. Oto. 4(1), 13–16 (1998)Google Scholar
  67. 67.
    Shannon, R.V., Zeng, F.-G., Kamath, V., Wygonski, J., et al.: Speech recognition with primarily temporal cues. Science 270(5234), 303–304 (1995)CrossRefGoogle Scholar
  68. 68.
    Reed, C.M., Delhorne, L.A.: Reception of Environmental Sounds Through Cochlear Implants. Ear. Hear. 26(1), 48–61 (2005)CrossRefGoogle Scholar
  69. 69.
    Inverso, D.: Cochlear Implant-Mediated Perception of Nonlinguistic Sounds, Unpublished Thesis, Gallaudet University (2008)Google Scholar
  70. 70.
    Bronkhorst, A.W., Plomp, R.: Effect of multiple speechlike maskers on binaural speech recognition in normal and impaired hearing. J. Acoust. Soc. Am. 92(6), 3132 (1992)CrossRefGoogle Scholar
  71. 71.
    Festen, J.M., Plomp, R.: Effects of fluctuating noise and interfering speech on the speech-reception threshold for impaired and normal hearing. J. Acoust. Soc. Am. 88(4), 1725 (1990)CrossRefGoogle Scholar
  72. 72.
    Loizou, P.C., Hu, Y., Litovsky, R., Yu, G., Peters, R., Lake, J., Roland, P.: Speech recognition by bilateral cochlear implant users in a cocktail-party setting. J. Acoust. Soc. Am. 125(1), 372 (2009)CrossRefGoogle Scholar
  73. 73.
    Cherry, C.: Some experiments on the recognition of speech with one and with two ears. J. Acoust. Soc. Am. 26, 975–979 (1953)CrossRefGoogle Scholar
  74. 74.
    Salthouse, T.A.: The processing-speed theory of adult age differences in cognition. Psychol. Rev. 103(3), 403–428 (1996)CrossRefGoogle Scholar
  75. 75.
    Divenyi, P.L., Stark, P.B., Haupt, K.: Decline of Speech Understanding and Auditory Thresholds in the Elderly. J. Acoust. Soc. Am. 118, 1089–1100 (2005)CrossRefGoogle Scholar
  76. 76.
    Humes, L.E., Lee, J.H., Coughlin, M.P.: Auditory measures of selective and divided attention in young and older adults using single-talker competition. J. Acoust. Soc. Am. 120(5), 2926 (2006)CrossRefGoogle Scholar
  77. 77.
    Mynatt, E.D.: Transforming graphical interfaces into auditory interfaces for blind users. Hum. Comput. Interact. 12(1), 7–45 (1997)CrossRefGoogle Scholar
  78. 78.
    Roth, P., Petrucci, L., Pun, T., Assimacopoulos, A.: Auditory browser for blind and visually impaired users. In: CHI 1999 extended abstracts on Human factors in computing systems, pp. 1–2. ACM, Pittsburgh (1999)Google Scholar
  79. 79.
    Cobb, N.J., Lawrence, D.M., Nelson, N.D.: Report on blind subjects’ tactile and auditory recognition for environmental stimuli. Percept Mot Skills 48(2), 363–366 (1979)Google Scholar
  80. 80.
    Petrie, H., Morley, S.: The use of non-speech sounds in non-visual interfaces to the MS Windows GUI for blind computer users. In: Proceedings of the 5th International Conference on Auditory Display (ICAD 1998), pp. 1–5. British Computer Society, University of Glasgow, U.K (1998)Google Scholar
  81. 81.
    Wersényi, G.: Evaluation of auditory representations for selected applications of a graphical user interface. In: Proceedings of the 15th International Conference on Auditory Display (ICAD 2009), Re: New – Digital Arts Forum, Copenhagen, Denmark, pp. 41–48 (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Brian Gygi
    • 1
  • Valeriy Shafiro
    • 2
  1. 1.Veterans Affairs Northern California Health Care SystemMartinezUSA
  2. 2.Communications Disorders and SciencesRush University Medical Center 

Personalised recommendations