Experimental Brain Research

, Volume 161, Issue 2, pp 233–242 | Cite as

Visual, haptic and crossmodal recognition of scenes

  • Fiona N. Newell
  • Andrew T. Woods
  • Marion Mernagh
  • Heinrich H. Bülthoff
Research Article


Real-world scene perception can often involve more than one sensory modality. Here we investigated the visual, haptic and crossmodal recognition of scenes of familiar objects. In three experiments participants first learned a scene of objects arranged in random positions on a platform. After learning, the experimenter swapped the position of two objects in the scene and the task for the participant was to identify the two swapped objects. In experiment 1, we found a cost in scene recognition performance when there was a change in sensory modality and scene orientation between learning and test. The cost in crossmodal performance was not due to the participants verbally encoding the objects (experiment 2) or by differences between serial and parallel encoding of the objects during haptic and visual learning, respectively (experiment 3). Instead, our findings suggest that differences between visual and haptic representations of space may affect the recognition of scenes of objects across these modalities.


Scene perception Crossmodal recognition Vision Haptics Orientation dependency 



This work was funded by a European Union IST programme grant (IST-2001-34712) awarded to the first author and by the Max-Planck Society, Germany. We thank Karl-Heinz Hofmann and Christina Baum of the Max-Planck Institute for Biological Cybernetics for building our experimental apparatus.


  1. Aginsky V, Tarr MJ (2000) How are different properties of a scene encoded in visual memory? Vis Cogn 7:147–162CrossRefGoogle Scholar
  2. Avery GC, Day RH (1969) Basis of horizontal-vertical illusion. J Exp Psychol 81:376–380PubMedGoogle Scholar
  3. Baddeley AD (1986) Working memory. Oxford University Press, Oxford, UKGoogle Scholar
  4. Biederman I, Rabinowitz JC, Glass AL, Stacey EWJ (1974) On the information extracted from a glance at a scene. J Exp Psychol 103:597–600PubMedGoogle Scholar
  5. Blumenfeld W (1937) The relationship between optical and haptic construction of space. Acta Psychol 2:125–175CrossRefGoogle Scholar
  6. Brockmole JR, Wang RF, Irwin DE (2002) Temporal integration between visual images and visual percepts. J Exp Psychol Hum Percept Perform 28:315–334CrossRefPubMedGoogle Scholar
  7. Christou CG, Bülthoff HH (1999) View dependence in scene recognition after active learning. Mem Cogn 27:996–1007Google Scholar
  8. Day RH, Avery GC (1970) Absence of the horizontal-vertical illusion in haptic space. J Exp Psychol Gen 83:172–173Google Scholar
  9. Diwadkar VA, McNamara TP (1997) Viewpoint dependence in scene recognition. Psychol Sci 8:302–307Google Scholar
  10. Faineteau H, Gentaz E, Viviani P (2003) The kinaesthetic perception of Euclidean distance: a study of the detour effect. Exp Brain Res 152:166–172CrossRefPubMedGoogle Scholar
  11. Gentaz E, Hatwell Y (1998) The haptic oblique effect in the perception of rod orientation by blind adults. Percept Psychophys 60:157–167PubMedGoogle Scholar
  12. Gentaz E, Hatwell Y (1999) Role of memorization conditions in the haptic processing of orientations and the ‘oblique effect’. Br J Psychol 90:373–388CrossRefPubMedGoogle Scholar
  13. Henderson JM, Hollingworth A (2003) Eye movements and visual memory: detecting changes to saccade targets in scenes. Percept Psychophys 65:58–71PubMedGoogle Scholar
  14. Hollingworth A (2003) Failures of retrieval and comparison constrain change detection in natural scenes. J Exp Psychol 29:388–403CrossRefGoogle Scholar
  15. Hollingworth A, Williams CC, Henderson JM (2001) To see and remember: visually specific information is retained in memory from previously attended objects in natural scenes. Psychonom Bull Rev 8:761–768Google Scholar
  16. Irwin DE, Zelinsky GJ (2002) Eye movements and scene perception: memory for things observed. Percept Psychophys 64:822–895Google Scholar
  17. Kappers AM (1999) Large systematic deviations in the haptic perception of parallelity. Perception 28:1001–1012PubMedGoogle Scholar
  18. Kappers AM, Koenderink JJ (1999) Haptic perception of spatial relations. Perception 28:781–795PubMedGoogle Scholar
  19. Klatzky RL (1999) Path completion after haptic exploration without vision: implications for haptic spatial representations. Percept Psychophys 61:220–235PubMedGoogle Scholar
  20. Kosslyn SM, Chabris CF, Marsolek CJ, Koenig O (1992) Categorical versus coordinate spatial relations: computational analyses and computer simulations. J Exp Psychol Hum Percept Perform 18:562–577CrossRefPubMedGoogle Scholar
  21. Lederman SJ, Klatzky RL (1987) Hand movements: a window into haptic object recognition. Cogn Psychol 19:342–368CrossRefGoogle Scholar
  22. Lederman SJ, Klatzky RL, Barber PO (1985) Spatial and movement-based heuristics for encoding pattern information through touch. J Exp Psychol Gen 114:33–49CrossRefPubMedGoogle Scholar
  23. Lederman SJ, Klatzky RL, Collins A, Wardell J (1987) Exploring environments by hand or foot: time-based heuristics for encoding distance in movement space. J Exp Psychol Learn Mem Cogn 13:606–614CrossRefPubMedGoogle Scholar
  24. Lederman SJ, Summers C, Klatzky RL (1996) Cognitive salience of haptic object properties: role of modality-encoding bias. Perception 25:983–998PubMedGoogle Scholar
  25. Loomis JM, Klatzky RL, Lederman SJ (1991) Similarity of tactual and visual picture recognition with limited field of view. Perception 20:167–177PubMedGoogle Scholar
  26. Mahrer P, Miles C (2002) Recognition memory for tactile sequences. Memory 10:7–20CrossRefPubMedGoogle Scholar
  27. Marchetti FM, Lederman SJ (1983) The haptic radial-tangential effect: two sets of Wong’s (1977) “moments-of-inertia” hypothesis. Bull Psychonom Soc 21:43–46Google Scholar
  28. Marks LE, Armstrong L (1996) Haptic and visual representations of space. In: Inui T, McClelland JL (eds) Attention and performance, XVI. Information integration in perception and communication. MIT Press, Cambridge, MA, pp 263–287Google Scholar
  29. McNamara TP (2003) How are locations of objects in the environment represented in memory? In: Freska C, Brauer W, Habel C, Wender K (eds) Spatial cognition, III. Routes and navigation, human memory and learning, spatial representation and spatial reasoning. Springer, Berlin Heidelberg New York, pp 174–191Google Scholar
  30. Millar S (1975) Effects of tactual and phonological similarity on the recall of Braille letters by blind children. Br J Psychol 66:193–201PubMedGoogle Scholar
  31. Nakatani C, Pollatsek A, Johnson SH (2002) Viewpoint-dependent recognition of scenes. Q J Exp Psychol 55A:115–139Google Scholar
  32. Newell FN, Ernst MO, Tjan BS, Bülthoff HH (2001) Viewpoint dependence in visual and haptic object recognition. Psychol Sci 12:37–42CrossRefPubMedGoogle Scholar
  33. Newport R, Rabb B, Jackson SR (2002) Noninformative vision improves haptic spatial perception. Curr Biol 12:1661–1664CrossRefPubMedGoogle Scholar
  34. Postma A, de Haan EHF (1996) What was where? Memory for object locations. Q J Exp Psychol 49A:187–199Google Scholar
  35. Potter MC (1976) Short-term conceptual memory for pictures. J Exp Psychol Hum Learn Mem 2:509–522CrossRefGoogle Scholar
  36. Rensink RA (2002) Change detection. Annu Rev Psychol 53:245–277CrossRefPubMedGoogle Scholar
  37. Rensink RA, O’Regan JK, Clark JJ (1997) To see or not to see: the need for attention to perceive changes in scenes. Psychol Sci 8:368–373Google Scholar
  38. Rensink RA, O’Regan JK, Clark JJ (2000) On the failure to detect changes in scenes across brief interruptions. Vis Cogn 7:127–145CrossRefGoogle Scholar
  39. Salamé P, Baddeley AD (1982) Disruption of short-term memory by irrelevant speech: implications for the structure of working memory. J Verbal Learn Verbal Behav 21:150–164Google Scholar
  40. Sanocki T (2003) Representation and perception of scenic layout. Cogn Psychol 47:43–86CrossRefGoogle Scholar
  41. Sanocki T, Epstein W (1997) Priming spatial layout of scenes. Psychol Sci 8:374–378Google Scholar
  42. Simons DJ (1996) In sight, out of mind: when object representations fail. Psychol Sci 7:301–305Google Scholar
  43. Simons DJ, Wang RF (1998) Perceiving real-world viewpoint changes. Psychol Sci 9:315–320CrossRefGoogle Scholar
  44. Simons DJ, Wang RF, Roddenberry D (2002) Object recognition is mediated by extraretinal information. Percept Psychophys 64:521–530PubMedGoogle Scholar
  45. Tarr MJ, Bülthoff HH (1998) Image-based object recognition in man, monkey and machine. Cognition 67:1–20CrossRefPubMedGoogle Scholar
  46. Thorpe S, Fize D, Marlot C (1996) Speed of processing in the human visual system. Nature 381:520–522CrossRefPubMedGoogle Scholar
  47. Wang RF, Simons DJ (1999) Active and passive scene recognition across views. Cognition 70:191–210CrossRefPubMedGoogle Scholar
  48. Woods AT, O’Modhrain S, Newell FN (2004) The effect of temporal delay and spatial differences on crossmodal object recognition. Cogn Affective Behav Neurosci (in press)Google Scholar
  49. Zuidhoek S, Kappers AML, van der Lubbe RHJ, Postma A (2003) Delay improves performance on a haptic spatial matching task. Exp Brain Res 149:320–330PubMedGoogle Scholar

Copyright information

© Springer-Verlag 2004

Authors and Affiliations

  • Fiona N. Newell
    • 1
  • Andrew T. Woods
    • 1
  • Marion Mernagh
    • 1
  • Heinrich H. Bülthoff
    • 2
  1. 1.Department of PsychologyUniversity of DublinDublinIreland
  2. 2.Max-Planck Institute for Biological CyberneticsTübingenGermany

Personalised recommendations