Abstract
The auditory perception of materials is a popular topic in the study of non-vocal sound-source perception. In this chapter, we review the empirical evidence on the mechanical and acoustical correlates of the perception of impacted stiff materials, and of the state of matter of sound-generating substances (solids, liquids, gases). As a whole, these studies suggest that recognition abilities are only highly accurate when differentiating between widely diverse materials (e.g. liquids vs. solids or plastics vs. metals) and that limitations in the auditory system, along with the possible internalization of biased statistics in the acoustical environment (e.g. clinking-glass sounds tend to be produced by small objects), might account for the less-than-perfect ability to differentiate between mechanically similar materials. This review is complemented by a summary of studies concerning the perception of deformable materials (fabrics and liquids) and the perceptual and motor-behaviour effects of auditory material-related information in audio-haptic contexts. The results of perceptual studies are the starting point for the development of interactive sound synthesis techniques for rendering the main auditory correlates of material properties, starting from physical models of the involved mechanical interactions. We review the recent literature dealing with contact sound synthesis in such fields as sonic interaction design and virtual reality. Special emphasis is given to softness/hardness correlates in impact sounds, associated with solid object resonances excited through impulsive contact. Synthesis methods for less studied sound-generating systems such as deformable objects (e.g. fabrics and liquids) and aggregate materials are also described.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adrien J-M (1991) The missing link: modal synthesis. In: De Poli G, Piccialli A, Roads C (eds) Representations of musical signals. MIT Press, Cambridge, pp 269–297
An SS, James DL, Marschner S (2012) Motion-driven concatenative synthesis of cloth sounds. ACM Trans Graph (TOG) 31(4) (Article No. 102)
Aramaki M, Marie C, Kronland-Martinet R, Ystad S, Besson M (2010) Sound categorization and conceptual priming for nonlinguistic and linguistic sounds. J Cognit Neurosci 22:2555–2569
Aramaki M, Besson M, Kronland-Martinet R, Ystad S (2011) Controlling the perceived material in an impact sound synthesizer. IEEE Trans Audio Speech Lang Process 19(2):301–314
Arnott SR, Cant JS, Dutton GN, Goodale MA (2008) Crinkling and crumpling: an auditory fmri study of material properties. Neuroimage 43:368–378
Ashby FG (1992) Multidimensional models of perception and cognition. Lawrence Erlbaum Associates, Hills-dale
Attias H, Schreiner CE (1997) Temporal low-order statistics of natural sounds. In: Mozer MC, Jordan MI, Petsche T (eds) Advances in neural information processing systems 9. MIT Press, Cambridge, pp 27–33
Avanzini F, Rath M, Rocchesso D, Ottaviani L (2003) Low-level sound models: resonators, interactions, surface textures. In: Rocchesso D, Fontana F (eds) The sounding object. Mondo Estremo, Firenze, pp 137–172
Avanzini F, Crosato P (2006b) Integrating physically-based sound models in a multimodal rendering architecture. Comput Anim Virtual Worlds 17(3–4):411–419. doi:10.1002/cav.v17:3/4
Avanzini F, Crosato P (2006a) Haptic-auditory rendering and perception of contact stiffness. In: McGookin D, Brewster S (eds) Haptic and audio interaction design. Lecture Notes in Computer Science 4129/2006, Springer, Berlin. pp 24–35
Avanzini F, Rocchesso D (2001) Controlling material properties in physical models of sounding objects. In: Proceedings of the international computer music conference (ICMC’01). La Habana, pp 91–94 (Available at http://www.soundobject.org)
Avanzini F, Rocchesso D (2004) Physical modeling of impacts: theory and experiments on contact time and spectral centroid. In: Proceedings of the sound and music computing conference (SMC2004), Paris, pp 287–293
Ballas JA (1993) Common factors in the identification of an assortment of brief everyday sounds. J Exp Psychol Hum Percept Perform 19:250–267
Belin P, Zatorre RJ, Lafaille P, Ahad P, Pike B (2000) Voice-selective areas in human auditory cortex. Nature 403:309–312
Bilbao S (2009) Numerical sound synthesis—finite difference schemes and simulation in musical acoustics. Wiley, Chichester
Bilbao S, Hamilton B, Torin A, Webb C, Graham P, Gray A, Perry J (2013) Large scale physical modeling sound synthesis. In: Proceedings of the Stockholm music acoustic conference (SMAC2013), Stockholm, pp 593–600
Bonneel N, Drettakis G, Tsingos N, Viaud-Delmon I, James D (2008) Fast modal sounds with scalable frequency-domain synthesis. ACM Trans Graph (TOG) 27(3) (Article no. 24)
Bonneel N, Suied C, Viaud-Delmon I, Drettakis G (2010) Bimodal perception of audio-visual material properties for virtual environments. ACM Trans Appl Percept 7(1) (Article No. 1)
Borg I, Groenen P (1997) Modern multidimensional scaling. Springer, New York
Cabe PA, Pittenger JB (2000) Human sensitivity to acoustic information from vessel filling. J Exp Psychol Hum Percept Perform 26:313–324
Carello C, Wagman JB, Turvey MT (2003) Acoustical specification of object properties. In: Anderson J, Anderson B (eds) Moving image theory: ecological considerations. Southern Illinois University Press, Carbondale
Castiello U, Giordano BL, Begliomini C, Ansuini C, Grassi M (2010) When ears drive hands: the influence of contact sound on reaching to grasp. PLoS ONE 5:e12240
Chadwick JN, Zheng C, James DL (2012) Precomputed acceleration noise for improved rigid-body sound. ACM Trans Graph (TOG)31(4) (Article No. 103)
Chatziioannou V, van Walstijn M (2013) An energy conserving finite difference scheme for simulation of collisions. In: Proceedings of the sound and music computing conference (SMC2013), Stockholm, pp 584–591
Cho G, Casali JG, Yi E (2001) December). Effect of fabric sound and touch on human subjective sensation. Fibers Polym 2(4):196–202
Cho G, Kim C, Cho J, Ha J (2005) March). Physiological signal analyses of frictional sound by structural parameters of warp knitted fabrics. Fibers Polym 6(1):89–94
Cirio G, Marchal M, Lécuyer A, Cooperstock J (2013) Vibrotactile rendering of splashing fluids. ACM Trans Haptics 6(1):117–122
Civille GV, Dus CA (1990) Development of terminology to describe the handfeel properties of paper and fabrics. J Sen Stud 5(1):19–32
Csapo AB, Baranyi P (2010) An interaction-based model for auditory substitution of tactile percepts. In: Proceedings of IEEE international conference on intelligent engineering systems (INES 2010), Las Palmas of Gran Canaria, pp 248–253
De Coensel B, Vanwetswinkel S, Botteldooren D (2011) Effects of natural sounds on the perception of road traffic noise. J Acoust Soc Am 129:EL148–EL153
Delle Monache, S., Polotti, P., & Rocchesso, D. (2010, September). A toolkit for explorations in sonic interaction design. In: Proceedings of audio mostly conference (AM’10). Piteå (Article no. 1)
DiFranco DE, Beauregard GL, Srinivasan MA (1997) The effect of auditory cues on the haptic perception of stiffness in virtual environments. In: Proceedings of the ASME dynamic systems and control division, DSC, vol 61, pp 17–22
Dombois F, Eckel G (2011) Audification. In: Hermann T, Hunt A, Neuhoff JG (eds) The sonification handbook. Logos Verlag, Berlin, pp 301–324
Doutaut V, Matignon D, Chaigne A (1998) Numerical simulations of xylophones. II. Time-domain modeling of the resonator and of the radiated sound pressure. J Acoust Soc Am 104(3):1633–1647
Drioli C, Rocchesso D (2012) Acoustic rendering of particle-based simulation of liquids in motion. J Multimodal User Interfaces 5(3–4):187–195
Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–433
Fletcher NH, Rossing TD (1991) The physics of musical instruments. Springer, New York
Flores P, Claro JP, Lankarani HM (2008) Kinematics and dynamics of multibody systems with imperfect joints: Models and case studies. Springer, Berlin
Fontana F, Bresin R (2003) Physics-based sound synthesis and control: crushing, walking and running by crumpling sounds. In: Proceedings of the colloquium on music informatics (CIM 2003), Firenze, pp 109–114
Franinović K, Serafin S (eds) (2013) Sonic interaction design. MIT Press, Cambridge
Freed DJ (1990) Auditory correlates of perceived mallet hardness for a set of recorded percussive events. J Acoust Soc Am 87:311–322
Gaver WW (1988) Everyday listening and auditory icons. Unpublished doctoral dissertation, University of California, San Diego
Gaver WW (1993) What in the world do we hear? an ecological approach to auditory event perception. Ecol Psychol 5:1–29
Geffen MN, Gervain J, Werker JF, Magnasco MO (2011) Auditory perception of self-similarity in water sounds. Front Integr Neurosci 5
Gibson JJ (1966) The senses considered as a perceptual system. Houghton Mifflin, Boston
Gibson JJ (1979) The ecological approach to visual perception. Houghton Mifflin, Boston
Giordano BL, Avanzini F, Wanderley M, McAdams S (2010) Multisensory integration in percussion performance. In: Actes du 10eme congrès français d’acoustique, Lyon (p. [CD-ROM]). Société Française d’Acoustique, Paris, France
Giordano BL (2005) Sound source perception in impact sounds. Unpublished doctoral dissertation, University of Padova, Italy
Giordano BL (2003) Material categorization and hardness scaling in real and synthetic impact sounds. In: Rocchesso D, Fontana F (eds) The sounding object. Mondo Estremo, Firenze, pp 73–93
Giordano BL, McAdams S (2006) Material identification of real impact sounds: effects of size variation in steel, glass, wood and plexiglass plates. J Acoust Soc Am 119:1171–1181
Giordano BL, McDonnell J, McAdams S (2010) Hearing living symbols and nonliving icons: category-specificities in the cognitive processing of environmental sounds. Brain Cogn 73:7–19
Giordano BL, Rocchesso D, McAdams S (2010) Integration of acoustical information in the perception of impacted sound sources: the role of information accuracy and exploitability. J Exp Psychol Hum Percept Perform 36:462–479
Giordano BL, Guastavino C, Murphy E, Ogg M, Smith BK, McAdams S (2011) Comparison of methods for collecting and modeling dissimilarity data: applications to complex sound stimuli. Multivar Behav Res 46:779–811
Giordano BL, Visell Y, Yao HY, Hayward V, Cooperstock J, McAdams S (2012) Identification of walked-upon materials in auditory, kinesthetic, haptic and audio-haptic conditions. J Acoust Soc Am 131:4002–4012
Grassi M (2005) Do we hear size or sound? Balls dropped on plates. Percept Psychophys 67:274–284
Gygi B, Kidd GR, Watson CS (2007) Similarity and categorization of environmental sounds. Percept Psychophys 69:839–855
Handel S, Erickson ML (2001) A rule of thumb: the bandwidth for timbre invariance is one octave. Music Percept 19:121–126
Houix O (2003) Categorisation auditive des sources sonores. Unpublished doctoral dissertation, Université du Maine, France
Houix H, Lemaitre G, Misdariis N, Susini P, Urdapilleta I (2012) A lexical analysis of environmental sound categories. J Exp Psychol Appl 18:52
Houle PA, Sethna JP (1996) Acoustic emission from crumpling paper. Phys Rev E 54(1):278–283
Huang G, Metaxas D, Govindaraj M (2003) Feel the "fabric": an audio-haptic interface. In: Proceedings of ACM siggraph/eurographics symposium on computer animation (SCA03). San Diego, pp 52–61
Hunt KH, Crossley FRE (1975) Coefficient of restitution interpreted as damping in vibroimpact. J Appl Mech 42:440–445
Jansson G (1993) Perception of the amount of fluid in a vessel shaken by hand. In: Valenti S, Pittinger J (eds) Studies in perception and action II. Posters presented at the VIIth international conference on event perception and action), pp 263–267
Jansson G, Juslin P, Poom L (2006) Liquid-specific stimulus properties can be used for haptic perception of the amount of liquid in a vessel put in motion. Perception 35:1421–1432
Jeon JY, Lee JL, You J, Kang J (2012) Acoustical characteristics of water sounds for soundscape enhancement in urban open spaces. J Acoust Soc Am 131:2101–2109
Kawabata S (1980) The standardization and analysis of hand evaluation. Textile Machinery Society of Japan, Osaka
Kidd GR, Watson CS (2003) The perceptual dimensionality of environmental sounds. Noise Control Eng J 51:216–231
Kitagawa M, Dokko D, Okamura AM, Yuh DD (2005) Effect of sensory substitution on suture-manipulation forces for robotic surgical systems. J Thorac Cardiovasc Surg 129(1):151–158
Klatzky RL, Pai DK, Krotkov EP (2000) Perception of material from contact sounds. Presence Teleoperators Virtual Environ 9(4):399–410
Kunkler-Peck AJ, Turvey MT (2000) Hearing shape. J Exp Psychol Hum Percept Perform 26:279–294
Lakatos S, McAdams S, Caussé R (1997) The representation of auditory source characteristics: simple geometric form. Percept Psychophys 59:1180–1190
Lambourg C, Chaigne A, Matignon D (2001) Time-domain simulation of damped impacted plates: II. Numerical Models and Results. J Acoust Soc Am 109(4):1433–1447
Lederman SJ (1979) Auditory texture perception. Perception 8(1):93–103
Lederman SJ, Klatzky RL (2004) Multisensory texture perception. In: Calvert G, Spence C, Stein B (eds) Handbook of multisensory processes. MIT Press, Cambridge, pp 107–122
Lemaitre G, Heller LM (2012) Auditory perception of material is fragile while action is strikingly robust. J Acoust Soc Am 131:1337–1348
Lemaitre G, Heller LM (2013) Evidence for a basic level in a taxonomy of everyday action sounds. Exp Brain Res 226(2):253–264
Lewis JW, Brefczynski JA, Phinney RE, Jannik JJ, DeYoe ED (2005) Distinct cortical pathways for processing tool versus animal sounds. J Neurosci 25:5148–5158
Lloyd B, Raghuvanshi N, Govindaraju NK (2011) Sound synthesis for impact sounds in video games. In: Proceedings of the ACM symposium on interactive 3D graphics and games (ACM I3D), San Francisco, pp 55–62
Lutfi RA, Oh EL (1997) Auditory discrimination of material changes in a struck-clamped bar. J Acoust Soc Am 102:3647–3656
Lutfi RA (2001) Auditory detection of hollowness. J Acoust Soc Am 110:1010–1019
Lutfi RA (2007) Human sound source identification. In: Yost WA, Fay RR, Popper AN (eds) Auditory Percept Sound Sources. Springer, New York, NY, pp 13–42
Lutfi RA, Liu CJ (2007) Individual differences in source identification from synthesized impact sounds. J Acoust Soc Am 122:1017–1028
Lutfi RA, Stoelinga CNJ (2010) Sensory constraints on auditory identification of the material and geometric properties of struck bars. J Acoust Soc Am 127:350–360
Marchal M, Cirio G, Visell Y, Fontana F, Serafin S, Cooperstock J, Lécuyer A (2013) Multimodal rendering of walking over virtual grounds. In: Steinicke F, Visell Y, Campos J, Lécuyer A (eds) Human walking in virtual environments. Springer, New York, pp 263–295
Marozeau J, de Cheveigné A, McAdams S, Winsberg S (2003) The dependency of timbre on fundamental frequency. J Acoust Soc Am 114:2946–2957
McAdams S, Chaigne A, Roussarie V (2004) The psychomecanics of simulated sound sources: material properties of impacted bars. J Acoust Soc Am 115:1306–1320
McAdams S, Roussarie V, Chaigne A, Giordano BL (2010) The psychomechanics of simulated sound sources: material properties of impacted thin plates. J Acoust Soc Am 128:1401–1413
Michaels CF, Carello C (1981) Direct perception. Prentice-Hall, Englewood Cliffs
Micoulaud-Franchi J-A, Aramaki M, Merer A, Cermolacce M, Ystad S, Kronland-Martinet R, Vion-Dury J (2011) Categorization and timbre perception of environmental sounds in schizophrenia. Psychiatry Res 189:149–152
Minnaert M (1933) On musical air-bubbles and the sounds of running water. Lond Edinb Dublin Philos Maga J Sci 16:235–248
Monaghan JJ (1992) Smoothed particle hydrodynamics. Ann Rev Astron Astrophys 30:543–574
Moss W, Yeh H, Hong J-M, Lin MC, Manocha D (2010) Sounding liquids: automatic sound synthesis from fluid simulation. ACM Trans Graph 29(3) (Article No. 21)
Nordahl R, Serafin S, Turchet L (2010) Sound synthesis and evaluation of interactive footsteps for virtual reality applications. In: Proceedings of the IEEE international conference on virtual reality (VR 2010), Waltham, pp 147–153
O’Brien JF, Shen C, Gatchalian CM (2002) Synthesizing sounds from rigid-body simulations. In: Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on computer animation. San Antonio, TX, pp 175–181
Papetti S, Avanzini F, Rocchesso D (2011) Numerical methods for a non-linear impact model: a comparative study with closed-form corrections. IEEE Trans Audio Speech Lang Process 19(7):2146–2158
Picard C, Frisson C, Faure F, Drettakis G, Kry P (2010) Advances in modal analysis using a robust and multiscale method. EURASIP J Adv Sig Process 2010 (Article ID 392782.)
Raghuvanshi N, Lin MC (2007) Physically based sound synthesis for large-scale virtual environments. Comput Graph Appl IEEE 27(1):14–18
Ren Z, Yeh H, Klatzky R, Lin MC (2013) Auditory perception of geometry-invariant material properties. IEEE Trans Vis Comput Graph 19(4):557–566
Ren Z, Yeh H, Lin MC (2013) Example-guided physically based modal sound synthesis. ACM Trans Graph (TOG) 32(1) (Article No. 1)
Rocchesso D, Ottaviani L, Fontana F, Avanzini F (2003) Size, shape, and material properties of sound models. In: Rocchesso D, Fontana F (eds) The sounding object. Mondo Estremo, Firenze, pp 95–110
Rocchesso D (2011) Explorations in sonic interaction design. Logos Verlag, Berlin
Smith JO III (2004) Virtual acoustic musical instruments: review and update. J New Music Res 33(3):283–304
Sreng J, Bergez F, Legarrec J, Lécuyer A, Andriot C (2007) Using an event-based approach to improve the multimodal rendering of 6dof virtual contact. In: Proceedings of the ACM symposium on virtual reality software and technology (VRST07), Newport Beach, CA, pp 165–173
Steele KM, Williams AK (2006) Is the bandwidth for timbre invariance only one octave? Music Percept 23:215–220
Steinicke F, Visell Y, Campos J, Lecuyer A (eds) (2013) Human walking in virtual environments: perception, technology, and applications. Springer, New York
Tucker S, Brown GJ (2003) Modelling the auditory perception of size, shape and material: applications to the classification of transient sonar sounds. In: Proceedings of the 114th convention of the Audio Engineering Society
Turchet L, Serafin S (2014) Semantic congruence in audio-haptic simulation of footsteps. Appl Acoust 75:59–66
Turchet L, Serafin S, Cesari P (2013) Walking pace affected by interactive sounds simulating stepping on different terrains. ACM Trans Appl Percept 10:23
Välimäki V, Pakarinen J, Erkut C, Karjalainen M (2006) Discrete-time modelling of musical instruments. Reports Prog Phys 69(1):1–78
van den Doel K (2004) Physically-based models for liquid sounds. In: Proceedings of the international conference on auditory display (ICAD04), Sydney
van den Doel K (2005) Physically based models for liquid sounds. ACM Trans Appl Percept 2:534–546
van den Doel K, Pai DK (1998) The sounds of physical shapes. Presence Teleoperators Virtual Environ 7(4):382–395
van den Doel K, Pai DK (2004) Modal synthesis for vibrating objects. In: Greenebaum K (ed) Audio anecdotes. AK Peters, Natick
van den Doel K, Kry PG, Pai DK (2001) FoleyAutomatic: physically-based sound effects for interactive simulation and animation. In: Proceedings of the international conference on computer graphics and interactive techniques (SIGGRAPH 01), Los Angeles, CA, pp 537–544
Vanderveer NJ (1979) Ecological acoustics: Human perception of environmental sounds. Unpublished doctoral dissertation, Cornell University. (Dissertation Abstracts International, 40, 4543B. (University Microfilms No. 80–04-002))
Velasco C, Jones R, King S, Spence C (2013) The sound of temperature: What information do pouring sounds convey concerning the temperature of a beverage. J Sens Stud 28:335–345
Visell Y, Fontana F, Giordano BL, Nordahl R, Serafin S, Bresin R (2009) Sound design and perception in walking interactions. Int J Hum Comput Stud 67(11):947–959
Voss RF, Clarke J (1975) 1/f noise in speech and music. Nature 258:317–318
Warren WH, Verbrugge RR (1984) Auditory perception of breaking and bouncing events: a case study in ecological acoustics. J Exp Psychol Hum Percept Perform 10:704–712
Waterman NA, Ashby MF (1997) The materials selector, 2nd edn. Chapman and Hall, London
Wildes R, Richards W (1988) Recovering material properties from sound. In: Richards W (ed) Nat Comput. MIT Press, Cambridge, MA, pp 356–363
Yao H-Y, Hayward V, Ellis RE (2005) A tactile enhancement instrument for minimally invasive surgery. Comput Aided Surg 10(4):233–239
Zheng C, James DL (2009) Harmonic fluids. ACM Trans Graph (TOG) 28(3) (Article No. 37)
Zheng C, James DL (2011) Toward high-quality modal contact sound. ACM Trans Graph (TOG) 30(4) (Article No. 38)
Acknowledgments
This work was partially supported by the Marie Curie Intra-European Fellowships program (FP7 PEOPLE-2011-IEF-30153, project BrainInNaturalSound to Bruno L. Giordano). The authors wish to thank Laurie Heller and Guillaume Lemaitre for sharing the sound stimuli used to prepare Figs. 4.1 and 4.3, and Laurie Heller, Federico Fontana and Stephen McAdams for providing helpful feedback about earlier versions of this chapter.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer-Verlag London
About this chapter
Cite this chapter
Giordano, B.L., Avanzini, F. (2014). Perception and Synthesis of Sound-Generating Materials. In: Di Luca, M. (eds) Multisensory Softness. Springer Series on Touch and Haptic Systems. Springer, London. https://doi.org/10.1007/978-1-4471-6533-0_4
Download citation
DOI: https://doi.org/10.1007/978-1-4471-6533-0_4
Published:
Publisher Name: Springer, London
Print ISBN: 978-1-4471-6532-3
Online ISBN: 978-1-4471-6533-0
eBook Packages: Computer ScienceComputer Science (R0)