Multisensory Recognition in Vertebrates (Especially Primates)

  • Ipek G. Kulahci
  • Asif A. Ghazanfar


A monkey wakes up next to her group mates as the sun rises. Throughout the day, she needs to make a number of decisions. Who should she forage with? Who should she cooperate with in order to chase away unfamiliar monkeys? Are there any particular individuals that she should avoid interacting with? When she is not foraging or defending her territory, she can usually be seen grooming another individual. However, choosing whom to groom presents yet another decision she needs to make. On this particular day, she may even end up deciding with whom she is going to mate. This is a complex but important decision, requiring the selection of a high quality male among many others based on a set of physical characteristics. All these myriad decisions require her to know the individuals in the group, recognize specific individuals among others, and remember past interactions with group members.


Auditory Cortex Vocal Tract Local Field Potential Macaque Monkey Multisensory Integration 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors gratefully acknowledge the scientific contributions and numerous discussions with the following people: Chand Chandrasekaran, Luis Lemus, Joost Maier, Darshana Narayanan, Stephen Shepherd, and Daniel Takahashi. This work was supported by NIH R01NS054898, NSF BCS-0547760 CAREER Award, and the James S. McDonnell Scholar Award.


  1. Barnes CL, Pandya DN (1992) Efferent cortical connections of multimodal cortex of the superior temporal sulcus in the rhesus-monkey. The Journal of Comparative Neurology 318:222–244PubMedCrossRefGoogle Scholar
  2. Barraclough NE, Xiao D, Baker CI, Oram MW, Perrett DI (2005) Integration of visual and auditory information by superior temporal sulcus neurons responsive to the sight of actions. Journal of Cognitive Neuroscience 17:377–391PubMedCrossRefGoogle Scholar
  3. Benevento LA, Fallon J, Davis BJ, Rezak M (1977) Auditory-visual interactions in single cells in the cortex of the superior temporal sulcus and the orbital frontal cortex of the macaque monkey. Experimental Neurology 57:849–872PubMedCrossRefGoogle Scholar
  4. Bernstein LE, Auer ET, Takayanagi S (2004) Auditory speech detection in noise enhanced by lipreading. Speech Communication 44:5–18CrossRefGoogle Scholar
  5. Besle J, Fort A, Delpuech C, Giard MH (2004) Bimodal speech: early suppressive visual effects in human auditory cortex. The European Journal of Neuroscience 20:2225–2234PubMedCrossRefGoogle Scholar
  6. Bizley JK, Nodal FR, Bajo VM, Nelken I, King AJ (2007) Physiological and anatomical evidence for multisensory interactions in auditory cortex. Cerebral Cortex 17:2172–2189PubMedCrossRefGoogle Scholar
  7. Bradbury JW (1981) The evolution of leks. In: Alexander RD, Tinkle DW (eds) Natural selection and social behavior. Chiron Press, New York, NYGoogle Scholar
  8. Bro-Jorgensen J (2010) Dynamics of multiple signalling systems: animal communication in a world in flux. Trends in Ecology & Evolution 25:292–300CrossRefGoogle Scholar
  9. Brown JL (1964) The evolution of diversity in avian territorial systems. The Wilson Bulletin 76:160–169Google Scholar
  10. Bruce C, Desimone R, Gross CG (1981) Visual properties of neurons in a polysensory area in superior temporal sulcus of the macaque. Journal of Neurophysiology 46:369–384PubMedGoogle Scholar
  11. Candolin U (2003) The use of multiple cues in mate choice. Biological Reviews 78:575–595PubMedCrossRefGoogle Scholar
  12. Cappe C, Thut G, Romei V, Murray MM (2009) Selective integration of auditory-visual looming cues by humans. Neuropsycholgia 47:1045–1052CrossRefGoogle Scholar
  13. Chandrasekaran, C., Lemus, L., & Ghazanfar, A. A. (2011) Dynamic faces speed up vocal processing in the auditory cortex of behaving monkeys. Washington, D.C: Society for Neuroscience.Google Scholar
  14. Chandrasekaran C, Ghazanfar AA (2009) Different neural frequency bands integrate faces and voices differently in the superior temporal sulcus. Journal of Neurophysiology 101:773–788PubMedCrossRefGoogle Scholar
  15. Chandrasekaran C, Lemus L, Trubanova A, Gondan M, Ghazanfar AA (2011b) Monkeys and humans share a common computation for face/voice integration. PLoS Computational Biology 7(9):e1002165PubMedCrossRefGoogle Scholar
  16. Chandrasekaran C, Trubanova A, Stillittano S, Caplier A, Ghazanfar AA (2009) The natural statistics of audiovisual speech. PLoS Computational Biology 5:e1000436PubMedCrossRefGoogle Scholar
  17. Cheney DL, Seyfarth RM (1982) How vervet monkeys perceive their grunts - field playback experiments. Animal Behaviour 30:739–751CrossRefGoogle Scholar
  18. Driver J, Noesselt T (2008) Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments. Neuron 57:11–23PubMedCrossRefGoogle Scholar
  19. Ettlinger G, Wilson WA (1990) Cross-modal performance: behavioural processes, phylogenetic considerations and neural mechanisms. Behavioural Brain Research 40:169–192PubMedCrossRefGoogle Scholar
  20. Evans TA, Howell S, Westergaard GC (2005) Auditory-visual cross-modal perception of communicative stimuli in tufted capuchin monkeys (Cebus apella). Journal of Experimental Psychology Animal Behavior Processes 31:399–406PubMedCrossRefGoogle Scholar
  21. Fitch WT (1997) Vocal tract length and formant frequency dispersion correlate with body size in rhesus macaques. The Journal of the Acoustical Society of America 102:1213–1222PubMedCrossRefGoogle Scholar
  22. Fu KMG, Shah AS, O’Connell MN, McGinnis T, Eckholdt H, Lakatos P et al (2004) Timing and laminar profile of eye-position effects on auditory responses in primate auditory cortex. Journal of Neurophysiology 92:3522–3531PubMedCrossRefGoogle Scholar
  23. Fusani L, Hutchison RE, Hutchison JB (1997) Vocal-postural Co-ordination of a sexually dimorphic display in a monomorphic species: The Barbary dove. Behaviour 134:321–335CrossRefGoogle Scholar
  24. Geschwind N (1965a) Disconnexion syndromes in animals and man, part I. Brain 88:237–294PubMedCrossRefGoogle Scholar
  25. Geschwind N (1965b) Disconnexion syndromes in animals and man, part II. Brain 88:585–644PubMedCrossRefGoogle Scholar
  26. Ghazanfar AA, Chandrasekaran CF (2007) Paving the way forward: integrating the senses through phase-resetting of cortical oscillations. Neuron 53:162–164PubMedCrossRefGoogle Scholar
  27. Ghazanfar AA, Chandrasekaran C, Logothetis NK (2008) Interactions between the superior temporal sulcus and auditory cortex mediate dynamic face/voice integration in rhesus monkeys. Journal of Neuroscience 28:4457–4469PubMedCrossRefGoogle Scholar
  28. Ghazanfar AA, Logothetis NK (2003) Facial expressions linked to monkey calls. Nature 423:937–938PubMedCrossRefGoogle Scholar
  29. Ghazanfar AA, Maier JX, Hoffman KL, Logothetis NK (2005) Multisensory integration of dynamic faces and voices in rhesus monkey auditory cortex. Journal of Neuroscience 25:5004–5012PubMedCrossRefGoogle Scholar
  30. Ghazanfar AA, Nielsen K, Logothetis NK (2006) Eye movements of monkeys viewing vocalizing conspecifics. Cognition 101:515–529PubMedCrossRefGoogle Scholar
  31. Ghazanfar AA, Rendall D (2008) Evolution of human vocal production. Current Biology 18:R457–R460PubMedCrossRefGoogle Scholar
  32. Ghazanfar AA, Schroeder CE (2006) Is neocortex essentially multisensory? Trends in Cognitive Sciences 10:278–285PubMedCrossRefGoogle Scholar
  33. Ghazanfar AA, Turesson HK, Maier JX, van Dinther R, Patterson RD, Logothetis NK (2007) Vocal tract resonances as indexical cues in rhesus monkeys. Current Biology 17:425–430PubMedCrossRefGoogle Scholar
  34. Gordon MS, Rosenblum LD (2005) Effects of intrastimulus modality change on audiovisual time-to-arrival judgments. Perception & Psychophysics 67:580–594CrossRefGoogle Scholar
  35. Gothard KM, Battaglia FP, Erickson CA, Spitler KM, Amaral DG (2007) Neural responses to facial expression and face identity in the monkey amygdala. Journal of Neurophysiology 97:1671–1683PubMedCrossRefGoogle Scholar
  36. Grafe TU, Wanger TC (2007) Multimodal signaling in male and female foot-flagging frogs Staurois guttatus (ranidae): An alerting function of calling. Ethology 113:772–781CrossRefGoogle Scholar
  37. Grant BR, Grant PR (1989) Evolutionary dynamics of a natural population: the large cactus finch of the Galapagos. Chicago University Press, Chicago, ILGoogle Scholar
  38. Grant PR, Grant BR (2002) Unpredictable evolution in a 30-year study of Darwin’s finches. Science 296:707–711PubMedCrossRefGoogle Scholar
  39. Guilford T, Dawkins MS (1991) Receiver psychology and the evolution of animal signals. Animal Behaviour 42:1–14CrossRefGoogle Scholar
  40. Hackett TA, Stepniewska I, Kaas JH (1999) Prefrontal connections of the parabelt auditory cortex in macaque monkeys. Brain Research 817:45–58PubMedCrossRefGoogle Scholar
  41. Harries MH, Perrett DI (1991) Visual processing of faces in temporal cortex - physiological evidence for a modular organization and possible anatomical correlates. Journal of Cognitive Neuroscience 3:9–24CrossRefGoogle Scholar
  42. Hauser MD, Chomsky N, Fitch W (2002) The faculty of language: What is it, who has it, and how did it evolve? Science 298:1569–1579PubMedCrossRefGoogle Scholar
  43. Hauser MD, Evans CS, Marler P (1993) The role of articulation in the production of rhesus-monkey, macaca-mulatta, vocalizations. Animal Behaviour 45:423–433CrossRefGoogle Scholar
  44. Hauser MD, Ybarra MS (1994) The role of lip configuration in monkey vocalizations - experiments using xylocaine as a nerve block. Brain and Language 46:232–244PubMedCrossRefGoogle Scholar
  45. Hebets EA (2005) Attention-altering signal interactions in the multimodal courtship display of the wolf spider Schizocosa uetzi. Behavioral Ecology 16:75–82CrossRefGoogle Scholar
  46. Hebets EA, Papaj DR (2005) Complex signal function: developing a framework of testable hypotheses. Behavioral Ecology and Sociobiology 57:197–214CrossRefGoogle Scholar
  47. Hibbitts T, Whiting M, Stuart-Fox D (2007) Shouting the odds: Vocalization signals status in a lizard. Behavioral Ecology and Sociobiology 61:1169–1176CrossRefGoogle Scholar
  48. Hoy R (2005) Animal awareness: The (un)binding of multisensory cues in decision making by animals. Proceedings of the National Academy of Sciences of the United States of America 102:2267–2268PubMedCrossRefGoogle Scholar
  49. Huber SK, Leon LFD, Hendry AP, Bermingham E, Podos J (2007) Reproductive isolation of sympatric morphs in a population of Darwin’s finches. Proceedings of the Royal Society B: Biological Sciences 274:1709–1714PubMedCrossRefGoogle Scholar
  50. Huber SK, Podos J (2006) Beak morphology and song features covary in a population of Darwin’s finches (Geospiza Fortis). Biological Journal of the Linnean Society 88:489–498CrossRefGoogle Scholar
  51. Izumi A, Kojima S (2004) Matching vocalizations to vocalizing faces in a chimpanzee (Pan troglodytes). Animal Cognition 7:179–184PubMedCrossRefGoogle Scholar
  52. Jiang JT, Alwan A, Keating PA, Auer ET, Bernstein LE (2002) On the relationship between face movements, tongue movements, and speech acoustics. Eurasip Journal on Applied Signal Processing 2002:1174–1188CrossRefGoogle Scholar
  53. Jordan KE, Brannon EM, Logothetis NK, Ghazanfar AA (2005) Monkeys match the number of voices they hear with the number of faces they see. Current Biology 15:1034–1038PubMedCrossRefGoogle Scholar
  54. Kayser C, Logothetis NK (2009) Directed interactions between auditory and superior temporal cortices and their role in sensory integration. Frontiers in Integrative Neuroscience 3:7PubMedCrossRefGoogle Scholar
  55. Kayser C, Petkov CI, Augath M, Logothetis NK (2007) Functional imaging reveals visual modulation of specific fields in auditory cortex. Journal of Neuroscience 27:1824–1835PubMedCrossRefGoogle Scholar
  56. Kayser C, Petkov CI, Logothetis NK (2008) Visual modulation of neurons in auditory cortex. Cerebral Cortex 18:1560–1574PubMedCrossRefGoogle Scholar
  57. Klin A, Jones W, Schultz R, Volkmar F, Cohen D (2002) Visual fixation patterns during viewing of naturalistic social situations as predictors of social competence in individuals with autism. Archives of General Psychiatry 59:809–816PubMedCrossRefGoogle Scholar
  58. Kuhl PK, Williams KA, Meltzoff AN (1991) Cross-modal speech perception in adults and infants using nonspeech auditory stimuli. Journal of Experimental Psychology Human Perception and Performance 17:829–840PubMedCrossRefGoogle Scholar
  59. Kuraoka K, Nakamura K (2007) Responses of single neurons in monkey amygdala to facial and vocal emotions. Journal of Neurophysiology 97:1379–1387PubMedCrossRefGoogle Scholar
  60. Lakatos P, Chen C-M, O’Connell MN, Mills A, Schroeder CE (2007) Neuronal oscillations and multisensory interaction in primary auditory cortex. Neuron 53:279–292PubMedCrossRefGoogle Scholar
  61. Lakatos P, Shah AS, Knuth KH, Ulbert I, Karmos G, Schroeder CE (2005) An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex. Journal of Neurophysiology 94:1904–1911PubMedCrossRefGoogle Scholar
  62. Lansing IR, McConkie GW (2003) Word identification and eye fixation locations in visual and visual-plus-auditory presentations of spoken sentences. Perception & Psychophysics 65:536–552CrossRefGoogle Scholar
  63. Lombardo S, Mackey E, Tang L, Smith B, Blumstein D (2008) Multimodal communication and spatial binding in pied currawongs (Strepera graculina). Animal Cognition 11:675–682PubMedCrossRefGoogle Scholar
  64. Maier JX, Chandrasekaran C, Ghazanfar AA (2008) Integration of bimodal looming signals through neuronal coherence in the temporal lobe. Current Biology 18:963–968PubMedCrossRefGoogle Scholar
  65. Maier JX, Neuhoff JG, Logothetis NK, Ghazanfar AA (2004) Multisensory integration of looming signals by rhesus monkeys. Neuron 43:177–181PubMedCrossRefGoogle Scholar
  66. Narins PM, Grabul DS, Soma KK, Gaucher P, Hodl W (2005) Cross-modal integration in a dart-poison frog. Proceedings of the National Academy of Sciences of the United States of America 102:2425–2429PubMedCrossRefGoogle Scholar
  67. Narins PM, Hodl W, Grabul DS (2003) Bimodal signal requisite for agonistic behavior in a dart-poison frog, epipedobatesfemoralis. Proceedings of the National Academy of Sciences of the United States of America 100:577–580PubMedCrossRefGoogle Scholar
  68. Noesselt T, Rieger JW, Schoenfeld MA, Kanowski M, Hinrichs H, Heinze H-J et al (2007) Audiovisual temporal correspondence modulates human multisensory superior temporal sulcus plus primary sensory cortices. Journal of Neuroscience 27:11431–11441PubMedCrossRefGoogle Scholar
  69. Oram MW, Perrett DI (1994) Responses of anterior superior temporal polysensory (Stpa) neurons to biological motion stimuli. Journal of Cognitive Neuroscience 6:99–116CrossRefGoogle Scholar
  70. Palombit RA, Cheney DL, Seyfarth RM (1999) Male grunts as mediators of social interaction with females in wild chacma baboons (papio cynocephalus ursinus). Behaviour 136:221–242CrossRefGoogle Scholar
  71. Parr LA (2004) Perceptual biases for multimodal cues in chimpanzee (pan troglodytes) affect recognition. Animal Cognition 7:171–178PubMedCrossRefGoogle Scholar
  72. Partan SR, Larco CP, Owens MJ (2009) Wild tree squirrels respond with multisensory enhancement to conspecific robot alarm behaviour. Animal Behaviour 77:1127–1135CrossRefGoogle Scholar
  73. Partan S, Marler P (1999) Communication goes multimodal. Science 283:1272–1273PubMedCrossRefGoogle Scholar
  74. Partan S, Yelda S, Price V, Shimizu T (2005) Female pigeons, Columba livia, respond to multisensory audio/video playbacks of male courtship behaviour. Animal Behaviour 70:957–966CrossRefGoogle Scholar
  75. Pevsner J (2002) Leonardo da Vinci’s contributions to neuroscience. Trends in Neurosciences 25:217–220PubMedCrossRefGoogle Scholar
  76. Podos J (2010) Acoustic discrimination of sympatric morphs in Darwin’s finches: a behavioural mechanism for assortative mating? Philosophical Transactions of the Royal Society B: Biological Sciences 365:1031–1039CrossRefGoogle Scholar
  77. Roberts JA, Taylor PW, Uetz GW (2007) Consequences of complex signaling: Predator detection of multimodal cues. Behavioral Ecology 18:236–240CrossRefGoogle Scholar
  78. Robinson DA, FUchs AF (1969) Eye movements evoked by stimulation of frontal eye fields. Journal of Neurophysiology 32:637–648PubMedGoogle Scholar
  79. Romanski LM, Averbeck BB, Diltz M (2005) Neural representation of vocalizations in the primate ventrolateral prefrontal cortex. Journal of Neurophysiology 93:734–747PubMedCrossRefGoogle Scholar
  80. Romanski LM, Bates JF, Goldman-Rakic PS (1999) Auditory belt and parabelt projections to the prefrontal cortex in the rhesus monkey. The Journal of Comparative Neurology 403:141–157PubMedCrossRefGoogle Scholar
  81. Rowe C (1999) Receiver psychology and the evolution of multicomponent signals. Animal Behaviour 58:921–931PubMedCrossRefGoogle Scholar
  82. Schall JD, Morel A, King DJ, Bullier J (1995) Topography of visual cortex connections with frontal eye field in macaque: Convergence and segregation of processing streams. Journal of Neuroscience 15:4464–4487PubMedGoogle Scholar
  83. Schroeder CE, Foxe JJ (2002) The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex. Cognitive Brain Research 14:187–198PubMedCrossRefGoogle Scholar
  84. Schroeder CE, Lakatos P, Kajikawa Y, Partan S, Puce A (2008) Neuronal oscillations and visual amplification of speech. Trends in Cognitive Sciences 12:106–113PubMedCrossRefGoogle Scholar
  85. Schwartz J-L, Berthommier F, Savariaux C (2004) Seeing to hear better: Evidence for early audio-visual interactions in speech identification. Cognition 93:B69–B78PubMedCrossRefGoogle Scholar
  86. Seltzer B, Pandya DN (1989) Frontal-lobe connections of the superior temporal sulcus in the rhesus-monkey. The Journal of Comparative Neurology 281:97–113PubMedCrossRefGoogle Scholar
  87. Seltzer B, Pandya DN (1994) Parietal, temporal, and occipital projections to cortex of the superior temporal sulcus in the rhesus monkey: A retrograde tracer study. The Journal of Comparative Neurology 343:445–463PubMedCrossRefGoogle Scholar
  88. Sherman PW, Reeve HK, Pfennig DW (1997) Recognition systems. In: Krebs JR, Davies NB (eds) Behavioural ecology: an evolutionary approach. Cambridge University Press, Cambridge, pp 69–96Google Scholar
  89. Sliwa J, Duhamel JR, Pascalis O, Wirth S (2011) Spontaneous voice-face identity matching by rhesus monkeys for familiar conspecifics and humans. Proceedings of the National Academy of Sciences of the United States of America 108:1735–1740PubMedCrossRefGoogle Scholar
  90. Sugihara T, Diltz MD, Averbeck BB, Romanski LM (2006) Integration of auditory and visual communication information in the primate ventrolateral prefrontal cortex. Journal of Neuroscience 26:11138–11147PubMedCrossRefGoogle Scholar
  91. Taylor RC, Buchanan BW, Doherty JL (2007) Sexual selection in the squirrel treefrog Hyla squirella: the role of multimodal cue assessment in female choice. Animal Behaviour 74:1753–1763CrossRefGoogle Scholar
  92. Taylor RC, Klein BA, Stein J, Ryan MJ (2008) Faux frogs: multimodal signalling and the value of robotics in animal behavior. Animal Behaviour 76:1089–1097CrossRefGoogle Scholar
  93. van Wassenhove V, Grant KW, Poeppel D (2005) Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences of the United States of America 102:1181–1186PubMedCrossRefGoogle Scholar
  94. Werner-Reiss U, Kelly KA, Trause AS, Underhill AM, Groh JM (2003) Eye position affects activity in primary auditory cortex of primates. Current Biology 13:554–562PubMedCrossRefGoogle Scholar
  95. Wilson EO (1975) Sociobiology the new synthesis. Harvard University Press, CambridgeGoogle Scholar
  96. Wollerman L (1999) Acoustic interference limits call detection in a neotropical frog, Hyla ebraccata. Animal Behaviour 57:529–536PubMedCrossRefGoogle Scholar
  97. Yehia HC, Kuratate T, Vatikiotis-Bateson E (2002) Linking facial animation, head motion and speech acoustics. Journal of Phonetics 30:555–568CrossRefGoogle Scholar
  98. Yehia H, Rubin P, Vatikiotis-Bateson E (1998) Quantitative association of vocal-tract and facial behavior. Speech Communication 26:23–43CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Department of Ecology & Evolutionary BiologyPrinceton UniversityPrincetonUSA
  2. 2.Department of PsychologyPrinceton UniversityPrincetonUSA
  3. 3.Neuroscience InstitutePrinceton UniversityPrincetonUSA

Personalised recommendations