Experimental Brain Research

, Volume 227, Issue 4, pp 497–507 | Cite as

Force feedback facilitates multisensory integration during robotic tool use

  • Ali Sengül
  • Giulio Rognini
  • Michiel van Elk
  • Jane Elizabeth Aspell
  • Hannes Bleuler
  • Olaf Blanke
Research Article


The present study investigated the effects of force feedback in relation to tool use on the multisensory integration of visuo-tactile information. Participants learned to control a robotic tool through a surgical robotic interface. Following tool-use training, participants performed a crossmodal congruency task, by responding to tactile vibrations applied to their hands, while ignoring visual distractors superimposed on the robotic tools. In the first experiment it was found that tool-use training with force feedback facilitates multisensory integration of signals from the tool, as reflected in a stronger crossmodal congruency effect with the force feedback training compared to training without force feedback and to no training. The second experiment extends these findings by showing that training with realistic online force feedback resulted in a stronger crossmodal congruency effect compared to training in which force feedback was delayed. The present study highlights the importance of haptic information for multisensory integration and extends findings from classical tool-use studies to the domain of robotic tools. We argue that such crossmodal congruency effects are an objective measure of robotic tool integration and propose some potential applications in surgical robotics, robotic tools, and human–tool interaction.


Multisensory integration Crossmodal congruency effect Force feedback Robotic surgery 

Supplementary material

221_2013_3526_MOESM1_ESM.m4v (3.4 mb)
Supplementary material 1 (M4 V 3480 kb)
221_2013_3526_MOESM2_ESM.m4v (2.2 mb)
Supplementary material 2 (M4 V 2281 kb)


  1. Aspell JE, Lenggenhager B, Blanke O (2009) Keeping in touch with one’s self: multisensory mechanisms of self-consciousness. PLoS One 4(8):e6488. doi: 10.1371/journal.pone.0006488 PubMedCrossRefGoogle Scholar
  2. Ayav A, Bresler L, Brunaud L, Boissel P (2004) Early results of one-year robotic surgery using the Da Vinci system to perform advanced laparoscopic procedures. J Gastrointest Surg Off J Soc Surg Aliment Tract 8(6):720–726. doi: 10.1016/j.gassur.2004.06.002 CrossRefGoogle Scholar
  3. Bassolino M, Serino A, Ubaldi S, Ladavas E (2010) Everyday use of the computer mouse extends peripersonal space representation. Neuropsychologia 48(3):803–811. doi: 10.1016/J.Neuropsychologia.2009.11.009 PubMedCrossRefGoogle Scholar
  4. Blanchard C, Roll R, Roll JP, Kavounoudias A (2011) Combined contribution of tactile and proprioceptive feedback to hand movement perception. Brain Res 1382:219–229. doi: 10.1016/j.brainres.2011.01.066 PubMedCrossRefGoogle Scholar
  5. Conti F, Barbagli F, Morris D, Sewell C (2005) CHAI 3D: an open-source library for the rapid development of haptic scenes. Paper presented at the IEEE world haptics, Pisa, ItalyGoogle Scholar
  6. Ehrsson H, Rosén B, Stockselius A, Ragnö C, Köhler P, Lundborg G (2008) Upper limb amputees can be induced to experience a rubber hand as their own. Brain 131(12):3443–3452PubMedCrossRefGoogle Scholar
  7. Farne A, Iriki A, Ladavas E (2005) Shaping multisensory action-space with tools: evidence from patients with cross-modal extinction. Neuropsychologia 43(2):238–248. doi: 10.1016/j.neuropsychologia.2004.11.010 PubMedCrossRefGoogle Scholar
  8. Graziano MS (1999) Where is my arm? The relative role of vision and proprioception in the neuronal representation of limb position. Proc Natl Acad Sci USA 96(18):10418–10421PubMedCrossRefGoogle Scholar
  9. Graziano MS, Gandhi S (2000) Location of the polysensory zone in the precentral gyrus of anesthetized monkeys. Exp Brain Res 135(2):259–266PubMedCrossRefGoogle Scholar
  10. Graziano MS, Reiss LA, Gross CG (1999) A neuronal representation of the location of nearby sounds. Nature 397(6718):428–430. doi: 10.1038/17115 PubMedCrossRefGoogle Scholar
  11. Graziano MS, Cooke DF, Taylor CS (2000) Coding the location of the arm by sight. Science 290(5497):1782–1786PubMedCrossRefGoogle Scholar
  12. Heed T, Backhaus J, Röder B (2012) Integration of hand and finger location in external spatial coordinates for tactile localization. J Exp Psychol Hum Percept Perform 38(2):386PubMedCrossRefGoogle Scholar
  13. Holmes NP (2012) Does tool use extend peripersonal space? A review and re-analysis. Exp Brain Res 218(2):273–282. doi: 10.1007/S00221-012-3042-7 PubMedCrossRefGoogle Scholar
  14. Holmes NP, Calvert GA, Spence C (2004) Extending or projecting peripersonal space with tools? Multisensory interactions highlight only the distal and proximal ends of tools. Neurosci Lett 372(1–2):62–67. doi: 10.1016/J.Neulet.2004.09.024 PubMedCrossRefGoogle Scholar
  15. Holmes NP, Spence C (2006) Beyond the body schema. Human body perception from the inside out. Oxford University Press, Oxford, pp 15–64Google Scholar
  16. Horgan S, Vanuno D (2001) Robots in laparoscopic surgery. J Laparoendosc Adv Surg Tech Part A 11(6):415–419. doi: 10.1089/10926420152761950 CrossRefGoogle Scholar
  17. Iriki A, Tanaka M, Iwamura Y (1996) Coding of modified body schema during tool use by macaque postcentral neurones. NeuroReport 7(14):2325–2330PubMedCrossRefGoogle Scholar
  18. Lee EC, Rafiq A, Merrell R, Ackerman R, Dennerlein JT (2005) Ergonomics and human factors in endoscopic surgery: a comparison of manual vs telerobotic simulation systems. Surg Endosc 19(8):1064–1070. doi: 10.1007/S00464-004-8213-6 PubMedCrossRefGoogle Scholar
  19. Maeso S, Reza M, Mayol JA, Blasco JA, Guerra M, Andradas E, Plana MN (2010) Efficacy of the Da Vinci surgical system in abdominal surgery compared with that of laparoscopy: a systematic review and meta-analysis. Ann Surg 252(2):254–262. doi: 10.1097/SLA.0b013e3181e6239e PubMedCrossRefGoogle Scholar
  20. Mahvash M, Hayward V (2004) High-fidelity haptic synthesis of contact with deformable bodies. IEEE Comput Graph Appl 24(2):48–55PubMedCrossRefGoogle Scholar
  21. Marasco PD, Kim K, Colgate JE, Peshkin MA, Kuiken TA (2011) Robotic touch shifts perception of embodiment to a prosthesis in targeted reinnervation amputees. Brain 134(Pt 3):747–758. doi: 10.1093/brain/awq361 PubMedCrossRefGoogle Scholar
  22. Maravita A, Iriki A (2004) Tools for the body (schema). Trends Cognit Sci 8(2):79–86. doi: 10.1016/J.Tics.2003.12.008 CrossRefGoogle Scholar
  23. Maravita A, Spence C, Kennett S, Driver J (2002) Tool-use changes multimodal spatial interactions between vision and touch in normal humans. Cognition 83(2):B25–B34PubMedCrossRefGoogle Scholar
  24. Narazaki K, Oleynikov D, Stergiou N (2006) Robotic surgery training and performance: identifying objective variables for quantifying the extent of proficiency. Surg Endosc 20(1):96–103. doi: 10.1007/s00464-005-3011-3 PubMedCrossRefGoogle Scholar
  25. Okamura AM (2009) Haptic feedback in robot-assisted minimally invasive surgery. Curr Opin Urol 19(1):102–107. doi: 10.1097/MOU.0b013e32831a478c PubMedCrossRefGoogle Scholar
  26. Parikh SS, Chan S, Agrawal SK, Hwang PH, Salisbury CM, Rafii BY, Varma G, Salisbury KJ, Blevins NH (2009) Integration of patient-specific paranasal sinus computed tomographic data into a virtual surgical environment. Am J Rhinol Allerg 23(4):442–447. doi: 10.2500/ajra.2009.23.3335 CrossRefGoogle Scholar
  27. Pavani F, Spence C, Driver J (2000) Visual capture of touch: out-of-the-body experiences with rubber gloves. Psychol Sci 11(5):353–359PubMedCrossRefGoogle Scholar
  28. Prasad SM, Maniar HS, Soper NJ, Damiano RJ, Klingensmith ME (2002) The effect of robotic assistance on learning curves for basic laparoscopic skills. Am J Surg 183(6):702–707PubMedCrossRefGoogle Scholar
  29. Rognini G, Sengül A, Aspell JE, Bleuler H, Blanke O (2012) Visuo-tactile integration and body ownership during self-generated action. Eur J Neurosci (in revision)Google Scholar
  30. Rosen B, Ehrsson HH, Antfolk C, Cipriani C, Sebelius F, Lundborg G (2009) Referral of sensation to an advanced humanoid robotic hand prosthesis. Scand J Plastic Reconstr Surg Hand Surg /Nordisk plastikkirurgisk forening [and] Nordisk klubb for handkirurgi 43(5):260–266. doi: 10.3109/02844310903113107
  31. Santos-Carreras L, Hagen M, Gassert R, Bleuler H (2011) Survey on surgical instrument handle design: ergonomics and acceptance. Surg Innov. doi: 10.1177/1553350611413611
  32. Sengül A, van Elk M, Rognini G, Aspell JE, Bleuler H, Blanke O (2012) Extending the body to virtual tools using a robotic surgical interface: evidence from the crossmodal congruency task. PloS oneGoogle Scholar
  33. Serino A, Bassolino M, Farne A, Ladavas E (2007) Extended multisensory space in blind cane users. Psychol Sci 18(7):642–648. doi: 10.1111/j.1467-9280.2007.01952.x PubMedCrossRefGoogle Scholar
  34. Shore DI, Barnes ME, Spence C (2006) Temporal aspects of the visuotactile congruency effect. Neurosci Lett 392(1–2):96–100. doi: 10.1016/J.Neulet.2005.09.001 PubMedCrossRefGoogle Scholar
  35. Spence C, Pavani F, Driver J (2004) Spatial constraints on visual-tactile cross-modal distractor congruency effects Cognitive. Affect Behav Neurosci 4(2):148–169CrossRefGoogle Scholar
  36. Stanney KM, Mourant RR, Kennedy RS (1998) Human factors issues in virtual environments: a review of the literature. Presence 7(4):327–351CrossRefGoogle Scholar
  37. Tan H, Reed C, Durlach N, Haptics K, Lafayette W (2010) Optimum information- transfer rates for communication through haptic and other sensory modalities. IEEE Trans Haptics 3(2):98–108CrossRefGoogle Scholar
  38. van Elk M, Blanke O (2011) The relation between body semantics and spatial body representations. Acta Psychol 138(3):347–358CrossRefGoogle Scholar
  39. Zimmerli L, Krewer C, Gassert R, Muller F, Riener R, Lunenburger L (2012) Validation of a mechanism to balance exercise difficulty in robot-assisted upper-extremity rehabilitation after stroke. J Neuroeng Rehabil 9:6. doi: 10.1186/1743-0003-9-6 PubMedCrossRefGoogle Scholar
  40. Zopf R, Savage G, Williams MA (2010) Crossmodal congruency measures of lateral distance effects on the rubber hand illusion. Neuropsychologia 48(3):713–725. doi: 10.1016/j.neuropsychologia.2009.10.028 PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Ali Sengül
    • 1
    • 2
    • 3
  • Giulio Rognini
    • 1
    • 2
    • 3
  • Michiel van Elk
    • 1
    • 2
    • 4
  • Jane Elizabeth Aspell
    • 2
    • 5
  • Hannes Bleuler
    • 3
  • Olaf Blanke
    • 1
    • 2
    • 6
  1. 1.Center for Neuroprosthetics, School of Life SciencesEcole Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland
  2. 2.Laboratory of Cognitive Neuroscience, Brain Mind InstituteEcole Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland
  3. 3.Robotic Systems LaboratoryEcole Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland
  4. 4.Department of PsychologyUniversity of AmsterdamAmsterdamThe Netherlands
  5. 5.Department of PsychologyAnglia Ruskin UniversityCambridgeUK
  6. 6.Department of NeurologyUniversity Hospital of Geneva (HUG)GenevaSwitzerland

Personalised recommendations