Skip to main content
SpringerLink
Log in

Visual-haptic cue integration with spatial and temporal disparity during pointing movements

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Many perceptual cue combination studies have shown that humans can integrate sensory information across modalities as well as within a modality in a manner that is close to optimal. While the limits of sensory cue integration have been extensively studied in the context of perceptual decision tasks, the evidence obtained in the context of motor decisions provides a less consistent picture. Here, we studied the combination of visual and haptic information in the context of human arm movement control. We implemented a pointing task in which human subjects pointed at an invisible unknown target position whose vertical position varied randomly across trials. In each trial, we presented a haptic and a visual cue that provided noisy information about the target position half-way through the reach. We measured pointing accuracy as function of haptic and visual cue onset and compared pointing performance to the predictions of a multisensory decision model. Our model accounts for pointing performance by computing the maximum a posteriori estimate, assuming minimum variance combination of uncertain sensory cues. Synchronicity of cue onset has previously been demonstrated to facilitate the integration of sensory information. We tested this in trials in which visual and haptic information was presented with temporal disparity. We found that for our sensorimotor task temporal disparity between visual and haptic cue had no effect. Sensorimotor learning appears to use all available information and to apply the same near-optimal rules for cue combination that are used by perception.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Alais D, Burr D (2004) The ventriloquist effect results from near-optimal bimodal integration. Curr Biol 14:257–262

    PubMed  CAS  Google Scholar 

  • Baddeley RJ, Ingram HA, Miall RC (2003) System identification applied to a visuomotor task: near-optimal human performance in a noisy changing task. J Neurosci 23:3066–3075

    PubMed  CAS  Google Scholar 

  • Beck JM, Pouget A (2007) Exact inferences in a neural implementation of a hidden Markov model. Neural Comput 19:1344–1361

    Article  PubMed  Google Scholar 

  • Bertelson P, Aschersleben G (1998) Automatic visual bias of perceived auditory location. Psychon Bull Rev 5:482–489

    Article  Google Scholar 

  • Bresciani J, Ernst MO, Drewing K, Bouyer G, Maury V, Kheddar A (2005) Feeling what you hear: auditory signals can modulate tactile tap perception. Exp Brain Res 162:172–180

    Article  PubMed  Google Scholar 

  • Burge J, Ernst MO, Banks MS (2008) The statistical determinants of adaptation rate in human reaching. J Vis 8(4):1–19

    Article  PubMed  Google Scholar 

  • Calvert GA (1998) Crossmodal identification. Trends Cogn Sci 2:247–253

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Thesen T (2004) Multisensory integration: methodological approaches and emerging principles in the human brain. J Physiol Paris 98:191–205

    Article  PubMed  Google Scholar 

  • Deneve S (2008a) Bayesian spiking neurons I: inference. Neural Comput 20:91–117

    Article  PubMed  Google Scholar 

  • Deneve S (2008b) Bayesian spiking neurons II: learning. Neural Comput 20:118–145

    Article  PubMed  Google Scholar 

  • Deneve S, Latham PE, Pouget A (2001) Efficient computation and cue integration with noisy population codes. Nat Neurosci 4:826–831

    Article  PubMed  CAS  Google Scholar 

  • Ernst MO, Banks MS (2002) Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415:429–433

    Article  PubMed  CAS  Google Scholar 

  • Ernst MO, Bülthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8:162–169

    Article  PubMed  Google Scholar 

  • Gepshtein S, Burge J, Ernst MO, Banks MS (2005) The combination of vision and touch depends on spatial proximity. J Vis 5(11):1013–1023

    Article  PubMed  Google Scholar 

  • Gold JI, Shadlen MN (2007) The neural basis of decision making. Annu Rev Neurosci 30:535–574

    Article  PubMed  CAS  Google Scholar 

  • Griffiths TL, Tenenbaum JB (2006) Optimal predictions in everyday cognition. Psychol Sci 17:767–773

    Article  PubMed  Google Scholar 

  • Helbig HB, Ernst MO (2007) Knowledge about a common source can promote visual–haptic integration. Perception 36:1523–1533

    Article  PubMed  Google Scholar 

  • Hillis JM, Ernst MO, Banks MS, Landy MS (2002) Combining sensory information: mandatory fusion within, but not between, senses. Science 298:1627–1630

    Article  PubMed  CAS  Google Scholar 

  • Hillis JM, Watt SJ, Landy MS, Banks MS (2004) Slant from texture and disparity cues: optimal cue combination. J Vis 4(12):967–992

    Article  PubMed  Google Scholar 

  • Hospedales T, Cartwright J, Vijayakumar S. (2007) Structure inference for Bayesian multisensory perception and tracking. Proc Int Joint Conf Art Intell (IJCAI ‘07). 2122–2128

  • Knill DC (1998) Discrimination of planar surface slant from texture: human and ideal observers compared. Vis Res 38:1683–1711

    Article  PubMed  CAS  Google Scholar 

  • Knill DC, Saunders JA (2003) Do humans optimally integrate stereo and texture information for judgments of surface slant? Vis Res 43:2539–2558

    Article  PubMed  Google Scholar 

  • Körding KP, Wolpert DM (2004a) Advances in neural information processing systems 16. In: Thrun S, Saul L, Schölkopf B (eds) Advances in neural information processing systems 16. MIT Press, Cambridge, MA, pp 1327–1334

    Google Scholar 

  • Körding KP, Wolpert DM (2004b) Bayesian integration in sensorimotor learning. Nature 427:244–247

    Article  PubMed  Google Scholar 

  • Körding KP, Beierholm U, Ma WJ, Quartz S, Tenenbaum JB, Shams L (2007) Causal inference in multisensory perception. PLoS ONE 2(9):e943. doi:10.1371/journal.pone.0000943

    Article  PubMed  Google Scholar 

  • Landy MS, Maloney LT, Johnston EB, Young M (1995) Measurement and modeling of depth cue combination: in defense of weak fusion. Vis Res 35:389–412

    Article  PubMed  CAS  Google Scholar 

  • Lewald J, Ehrenstein WH, Guski R (2001) Spatio-temporal constraints for auditory-visual integration. Behav Brain Res 121:69–79

    Article  PubMed  CAS  Google Scholar 

  • Liu D, Todorov E (2007) Evidence for the flexible sensorimotor strategies predicted by optimal feedback control. J Neurosci 27:9354–9368

    Article  PubMed  CAS  Google Scholar 

  • Louw S, Smeets J, Brenner E (2007) Judging surface slant for placing objects: a role for motion parallax. Exp Brain Res 183:149–158

    Article  PubMed  Google Scholar 

  • Ma WJ, Pouget A (2008) Linking neurons to behavior in multisensory perception: a computational review. Brain Res 1242:4–12

    Article  PubMed  CAS  Google Scholar 

  • Ma WJ, Beck JM, Latham PE, Pouget A (2006) Bayesian inference with probabilistic population codes. Nat Neurosci 9:1432–1438

    Article  PubMed  CAS  Google Scholar 

  • Ma WJ, Beck JM, Pouget A (2008) Spiking networks for Bayesian inference and choice. Curr Opin Neurobiol 18:217–222

    Article  PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1983) Interactions among converging sensory inputs in the superior colliculus. Science 221:389–391

    Article  PubMed  CAS  Google Scholar 

  • Meredith MA, Stein BE (1996) Spatial determinants of multisensory integration in cat superior colliculus neurons. J Neurophysiol 75:1843–1857

    PubMed  CAS  Google Scholar 

  • Radeau M, Bertelson P (1987) Auditory-visual interaction and the timing of inputs. Psychol Res 49:17–22

    Article  PubMed  CAS  Google Scholar 

  • Rao RPN (2004) Bayesian computation in recurrent neural circuits. Neural Comput 16:1–38

    Article  PubMed  Google Scholar 

  • Saunders JA, Knill DC (2004) Visual feedback control of hand movements. J Neurosci 24:3223–3234

    Article  PubMed  CAS  Google Scholar 

  • Schmidt T (2002) The finger in flight: Real-time motor control by visually masked color stimuli. Psychol Sci 13:112–117

    Article  PubMed  Google Scholar 

  • Schwarz G (1978) Estimating the dimension of a model. Ann Stat 6:461–464

    Article  Google Scholar 

  • Serwe S, Drewing K, Trommershauser J (2009) Combination of noisy directional visual and proprioceptive information. J Vis 9:1–14

    Article  PubMed  Google Scholar 

  • Seydell A, McCann BC, Trommershäuser J, Knill DC (2008) Learning stochastic reward distributions in a speeded pointing task. J Neurosci 28:4356–4367

    Article  PubMed  CAS  Google Scholar 

  • Shadlen MN, Newsome WT (2001) Neural basis of a perceptual decision in the parietal cortex (Area LIP) of the Rhesus monkey. J Neurophysiol 86:1916–1936

    PubMed  CAS  Google Scholar 

  • Slutsky DA, Recanzone GH (2001) Temporal and spatial dependency of the ventriloquism effect. Neuroreport 12:7–10

    Article  PubMed  CAS  Google Scholar 

  • Song J, Nakayama K (2007) Automatic adjustment of visuomotor readiness. J Vis 7(5):1–9

    Article  Google Scholar 

  • Song J, Nakayama K (2008) Target selection in visual search as revealed by movement trajectories. Vis Res 48:853–861

    PubMed  Google Scholar 

  • Sousa R, Brenner E, Smeets JB (2009) Slant cue are combined early in visual processing: Evidence from visual search. Vis Res 49:257–261

    Article  PubMed  Google Scholar 

  • Spence C, Squire S (2003) Multisensory integration: maintaining the perception of synchrony. Curr Biol 13:R519–R521

    Article  PubMed  CAS  Google Scholar 

  • Stein BE, Meredith MA (1993) The merging of the senses. MIT Press, Cambridge, MA

    Google Scholar 

  • Stein BE, Stanford TR (2008) Multisensory integration: current issues from the perspective of the single neuron. Nat Rev Neurosci 9:255–266

    Article  PubMed  CAS  Google Scholar 

  • Takahashi C, Diedrichsen J, Watt SJ (2009) Integration of vision and haptics during tool use. J Vis 9(6):1–15

    Article  PubMed  Google Scholar 

  • Tassinari H, Hudson T, Landy MS (2006) Combining priors and noisy visual cues in a rapid pointing task. J Neurosci 26:10154–10163

    Article  PubMed  CAS  Google Scholar 

  • van Beers RJ, Sittig A, Denier van der Gon J (1999) Integration of proprioceptive and visual position-information: an experimentally supported model. J Neurophysiol 81:1355–1364

    PubMed  Google Scholar 

  • van Beers RJ, Wolpert DM, Haggard P (2002) When feeling is more important than seeing in sensorimotor adaptation. Curr Biol 12:834–837

    Article  PubMed  Google Scholar 

  • Vogels IMLC (2004) Detection of temporal delays in visual-haptic interfaces. Hum Factors 46:118–134

    Article  PubMed  Google Scholar 

  • Wallace MT, Meredith MA, Stein BE (1998) Multisensory integration in the superior colliculus of the alert cat. J Neurophysiol 80:1006–1010

    PubMed  CAS  Google Scholar 

  • Wallace MW, Roberson G, Hairston W, Stein BE, Vaughan J, Schirillo J (2004) Unifying multisensory signals across time and space. Exp Brain Res 158:252–258

    Article  PubMed  CAS  Google Scholar 

  • Wolpert DM, Ghahramani Z, Jordan M (1995) An internal model for sensorimotor integration. Science 269:1880–1882

    Article  PubMed  CAS  Google Scholar 

Download references

Acknowledgments

We thank Nathalie Wahl for help with data collection. Funded by the Deutsche Forschungsgemeinschaft (Emmy-Noether-Programm, TR, 528/1-2, 1-3).

Author information

Authors and Affiliations

  1. Department of Psychology, Giessen University, Otto-Behaghel-Str. 10F, 35394, Giessen, Germany

    Sascha Serwe & Julia Trommershäuser

  2. Department of Psychology, FernUniversität in Hagen, Universitätsstr. 11, 58084, Hagen, Germany

    Sascha Serwe

  3. Rehabilitation Institute of Chicago, Northwestern University, 345 E Superior Street, Chicago, IL, 60611, USA

    Konrad P. Körding

  4. Department of Psychology and Center for Neural Science, New York University, 4 Washington Place, Room 809, New York, NY, 10003, USA

    Julia Trommershäuser

Authors
  1. Sascha Serwe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Konrad P. Körding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Julia Trommershäuser
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sascha Serwe.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Serwe, S., Körding, K.P. & Trommershäuser, J. Visual-haptic cue integration with spatial and temporal disparity during pointing movements. Exp Brain Res 210, 67–80 (2011). https://doi.org/10.1007/s00221-011-2603-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-011-2603-5

Keywords

Search

Navigation