Skip to main content
SpringerLink
Log in

Visual, haptic and crossmodal recognition of scenes

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Real-world scene perception can often involve more than one sensory modality. Here we investigated the visual, haptic and crossmodal recognition of scenes of familiar objects. In three experiments participants first learned a scene of objects arranged in random positions on a platform. After learning, the experimenter swapped the position of two objects in the scene and the task for the participant was to identify the two swapped objects. In experiment 1, we found a cost in scene recognition performance when there was a change in sensory modality and scene orientation between learning and test. The cost in crossmodal performance was not due to the participants verbally encoding the objects (experiment 2) or by differences between serial and parallel encoding of the objects during haptic and visual learning, respectively (experiment 3). Instead, our findings suggest that differences between visual and haptic representations of space may affect the recognition of scenes of objects across these modalities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. All the studies reported were approved by the Trinity College Department of Psychology Ethics Committee, and thus conformed to the ethical standards laid down in the 1964 Declaration of Helsinki.

  2. An ISI of 20 s was necessary for the experimenter to move the objects. Although this ISI may contribute to a memory decay rate, a recent study suggests no difference in memory decay rates across vision and haptics (see Woods et al. 2004).

References

  • Aginsky V, Tarr MJ (2000) How are different properties of a scene encoded in visual memory? Vis Cogn 7:147–162

    Article  Google Scholar 

  • Avery GC, Day RH (1969) Basis of horizontal-vertical illusion. J Exp Psychol 81:376–380

    CAS  PubMed  Google Scholar 

  • Baddeley AD (1986) Working memory. Oxford University Press, Oxford, UK

  • Biederman I, Rabinowitz JC, Glass AL, Stacey EWJ (1974) On the information extracted from a glance at a scene. J Exp Psychol 103:597–600

    CAS  PubMed  Google Scholar 

  • Blumenfeld W (1937) The relationship between optical and haptic construction of space. Acta Psychol 2:125–175

    Article  Google Scholar 

  • Brockmole JR, Wang RF, Irwin DE (2002) Temporal integration between visual images and visual percepts. J Exp Psychol Hum Percept Perform 28:315–334

    Article  PubMed  Google Scholar 

  • Christou CG, Bülthoff HH (1999) View dependence in scene recognition after active learning. Mem Cogn 27:996–1007

    CAS  Google Scholar 

  • Day RH, Avery GC (1970) Absence of the horizontal-vertical illusion in haptic space. J Exp Psychol Gen 83:172–173

    CAS  Google Scholar 

  • Diwadkar VA, McNamara TP (1997) Viewpoint dependence in scene recognition. Psychol Sci 8:302–307

    Google Scholar 

  • Faineteau H, Gentaz E, Viviani P (2003) The kinaesthetic perception of Euclidean distance: a study of the detour effect. Exp Brain Res 152:166–172

    Article  PubMed  Google Scholar 

  • Gentaz E, Hatwell Y (1998) The haptic oblique effect in the perception of rod orientation by blind adults. Percept Psychophys 60:157–167

    CAS  PubMed  Google Scholar 

  • Gentaz E, Hatwell Y (1999) Role of memorization conditions in the haptic processing of orientations and the ‘oblique effect’. Br J Psychol 90:373–388

    Article  PubMed  Google Scholar 

  • Henderson JM, Hollingworth A (2003) Eye movements and visual memory: detecting changes to saccade targets in scenes. Percept Psychophys 65:58–71

    PubMed  Google Scholar 

  • Hollingworth A (2003) Failures of retrieval and comparison constrain change detection in natural scenes. J Exp Psychol 29:388–403

    Article  Google Scholar 

  • Hollingworth A, Williams CC, Henderson JM (2001) To see and remember: visually specific information is retained in memory from previously attended objects in natural scenes. Psychonom Bull Rev 8:761–768

    CAS  Google Scholar 

  • Irwin DE, Zelinsky GJ (2002) Eye movements and scene perception: memory for things observed. Percept Psychophys 64:822–895

    Google Scholar 

  • Kappers AM (1999) Large systematic deviations in the haptic perception of parallelity. Perception 28:1001–1012

    CAS  PubMed  Google Scholar 

  • Kappers AM, Koenderink JJ (1999) Haptic perception of spatial relations. Perception 28:781–795

    CAS  PubMed  Google Scholar 

  • Klatzky RL (1999) Path completion after haptic exploration without vision: implications for haptic spatial representations. Percept Psychophys 61:220–235

    CAS  PubMed  Google Scholar 

  • Kosslyn SM, Chabris CF, Marsolek CJ, Koenig O (1992) Categorical versus coordinate spatial relations: computational analyses and computer simulations. J Exp Psychol Hum Percept Perform 18:562–577

    Article  CAS  PubMed  Google Scholar 

  • Lederman SJ, Klatzky RL (1987) Hand movements: a window into haptic object recognition. Cogn Psychol 19:342–368

    Article  CAS  Google Scholar 

  • Lederman SJ, Klatzky RL, Barber PO (1985) Spatial and movement-based heuristics for encoding pattern information through touch. J Exp Psychol Gen 114:33–49

    Article  CAS  PubMed  Google Scholar 

  • Lederman SJ, Klatzky RL, Collins A, Wardell J (1987) Exploring environments by hand or foot: time-based heuristics for encoding distance in movement space. J Exp Psychol Learn Mem Cogn 13:606–614

    Article  CAS  PubMed  Google Scholar 

  • Lederman SJ, Summers C, Klatzky RL (1996) Cognitive salience of haptic object properties: role of modality-encoding bias. Perception 25:983–998

    CAS  PubMed  Google Scholar 

  • Loomis JM, Klatzky RL, Lederman SJ (1991) Similarity of tactual and visual picture recognition with limited field of view. Perception 20:167–177

    CAS  PubMed  Google Scholar 

  • Mahrer P, Miles C (2002) Recognition memory for tactile sequences. Memory 10:7–20

    Article  PubMed  Google Scholar 

  • Marchetti FM, Lederman SJ (1983) The haptic radial-tangential effect: two sets of Wong’s (1977) “moments-of-inertia” hypothesis. Bull Psychonom Soc 21:43–46

    Google Scholar 

  • Marks LE, Armstrong L (1996) Haptic and visual representations of space. In: Inui T, McClelland JL (eds) Attention and performance, XVI. Information integration in perception and communication. MIT Press, Cambridge, MA, pp 263–287

  • McNamara TP (2003) How are locations of objects in the environment represented in memory? In: Freska C, Brauer W, Habel C, Wender K (eds) Spatial cognition, III. Routes and navigation, human memory and learning, spatial representation and spatial reasoning. Springer, Berlin Heidelberg New York, pp 174–191

  • Millar S (1975) Effects of tactual and phonological similarity on the recall of Braille letters by blind children. Br J Psychol 66:193–201

    CAS  PubMed  Google Scholar 

  • Nakatani C, Pollatsek A, Johnson SH (2002) Viewpoint-dependent recognition of scenes. Q J Exp Psychol 55A:115–139

    Google Scholar 

  • Newell FN, Ernst MO, Tjan BS, Bülthoff HH (2001) Viewpoint dependence in visual and haptic object recognition. Psychol Sci 12:37–42

    Article  CAS  PubMed  Google Scholar 

  • Newport R, Rabb B, Jackson SR (2002) Noninformative vision improves haptic spatial perception. Curr Biol 12:1661–1664

    Article  CAS  PubMed  Google Scholar 

  • Postma A, de Haan EHF (1996) What was where? Memory for object locations. Q J Exp Psychol 49A:187–199

    Google Scholar 

  • Potter MC (1976) Short-term conceptual memory for pictures. J Exp Psychol Hum Learn Mem 2:509–522

    Article  CAS  Google Scholar 

  • Rensink RA (2002) Change detection. Annu Rev Psychol 53:245–277

    Article  PubMed  Google Scholar 

  • Rensink RA, O’Regan JK, Clark JJ (1997) To see or not to see: the need for attention to perceive changes in scenes. Psychol Sci 8:368–373

    Google Scholar 

  • Rensink RA, O’Regan JK, Clark JJ (2000) On the failure to detect changes in scenes across brief interruptions. Vis Cogn 7:127–145

    Article  Google Scholar 

  • Salamé P, Baddeley AD (1982) Disruption of short-term memory by irrelevant speech: implications for the structure of working memory. J Verbal Learn Verbal Behav 21:150–164

    Google Scholar 

  • Sanocki T (2003) Representation and perception of scenic layout. Cogn Psychol 47:43–86

    Article  Google Scholar 

  • Sanocki T, Epstein W (1997) Priming spatial layout of scenes. Psychol Sci 8:374–378

    Google Scholar 

  • Simons DJ (1996) In sight, out of mind: when object representations fail. Psychol Sci 7:301–305

    Google Scholar 

  • Simons DJ, Wang RF (1998) Perceiving real-world viewpoint changes. Psychol Sci 9:315–320

    Article  Google Scholar 

  • Simons DJ, Wang RF, Roddenberry D (2002) Object recognition is mediated by extraretinal information. Percept Psychophys 64:521–530

    PubMed  Google Scholar 

  • Tarr MJ, Bülthoff HH (1998) Image-based object recognition in man, monkey and machine. Cognition 67:1–20

    Article  CAS  PubMed  Google Scholar 

  • Thorpe S, Fize D, Marlot C (1996) Speed of processing in the human visual system. Nature 381:520–522

    Article  CAS  PubMed  Google Scholar 

  • Wang RF, Simons DJ (1999) Active and passive scene recognition across views. Cognition 70:191–210

    Article  CAS  PubMed  Google Scholar 

  • Woods AT, O’Modhrain S, Newell FN (2004) The effect of temporal delay and spatial differences on crossmodal object recognition. Cogn Affective Behav Neurosci (in press)

  • Zuidhoek S, Kappers AML, van der Lubbe RHJ, Postma A (2003) Delay improves performance on a haptic spatial matching task. Exp Brain Res 149:320–330

    PubMed  Google Scholar 

Download references

Acknowledgements

This work was funded by a European Union IST programme grant (IST-2001-34712) awarded to the first author and by the Max-Planck Society, Germany. We thank Karl-Heinz Hofmann and Christina Baum of the Max-Planck Institute for Biological Cybernetics for building our experimental apparatus.

Author information

Authors and Affiliations

  1. Department of Psychology, University of Dublin, Trinity College, 2, Dublin, Ireland

    Fiona N. Newell, Andrew T. Woods & Marion Mernagh

  2. Max-Planck Institute for Biological Cybernetics, Tübingen, Germany

    Heinrich H. Bülthoff

Authors
  1. Fiona N. Newell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew T. Woods
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marion Mernagh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Heinrich H. Bülthoff
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fiona N. Newell.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Newell, F.N., Woods, A.T., Mernagh, M. et al. Visual, haptic and crossmodal recognition of scenes. Exp Brain Res 161, 233–242 (2005). https://doi.org/10.1007/s00221-004-2067-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-004-2067-y

Keywords

Search

Navigation