Skip to main content

Control of aperture closure during reach-to-grasp movements in immersive haptic-free virtual reality

Abstract

Virtual reality (VR) has garnered much interest as a training environment for motor skill acquisition, including for neurological rehabilitation of upper extremities. While the focus has been on gross upper limb motion, VR applications that involve reaching for, and interacting with, virtual objects are growing. The absence of true haptics in VR when it comes to hand-object interactions raises a fundamentally important question: can haptic-free immersive virtual environments (hf-VEs) support naturalistic coordination of reach-to-grasp movements? This issue has been grossly understudied, and yet is of significant importance in the development and application of VR across a number of sectors. In a previous study (Furmanek et al., J Neuroeng Rehabil 16:78, 2019), we reported that reach-to-grasp movements are similarly coordinated in both the physical environment (PE) and hf-VE. The most noteworthy difference was that the closure phase—which begins at maximum aperture and lasts through the end of the movement—was longer in hf-VE than in PE, suggesting that different control laws might govern the initiation of closure between the two environments. To do so, we reanalyzed data from Furmanek et al. (J Neuroeng Rehabil 16:78, 2019), in which the participants reached to grasp three differently sized physical objects, and matching 3D virtual object renderings, placed at three different locations. Our analysis revealed two key findings pertaining to the initiation of closure in PE and hf-VE. First, the respective control laws governing the initiation of aperture closure in PE and hf-VE both included state estimates of transport velocity and acceleration, supporting a general unified control policy for implementing reach-to-grasp across physical and virtual environments. Second, the aperture was less informative to the control law in hf-VE. We suggest that the latter was likely because transport velocity at closure onset and aperture at closure onset were less independent in hf-VE than in PE, ultimately resulting in an aperture at closure onset having a weaker influence on the initiation of closure. In this way, the excess time and muscular effort needed to actively bring the fingers to a stop at the interface of a virtual object was factored into the control law governing the initiation of closure in hf-VE. Critically, this control law remained applicable, albeit with different weights in hf-VE, despite the absence of terminal haptic feedback and potential perceptual differences.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Adamovich SV, Fluet GG, Eugene T, Merians AS (2009) Sensorimotor training in virtual reality: a review. NeuroRehabilitation 25:25–44. https://doi.org/10.3233/NRE-2009-0497

    Article  Google Scholar 

  2. Armbrüster C, Wolter M, Kuhlen T et al (2008) Depth perception in virtual reality: distance estimations in peri- and extrapersonal space. Cyberpsychol Behav 11:9–15. https://doi.org/10.1089/cpb.2007.9935

    Article  PubMed  Google Scholar 

  3. Bingham G, Coats R, Mon-Williams M (2007) Natural prehension in trials without haptic feedback but only when calibration is allowed. Neuropsychologia 45:288–294. https://doi.org/10.1016/j.neuropsychologia.2006.07.011

    Article  PubMed  Google Scholar 

  4. Bootsma RJ, Marteniuk RG, MacKenzie CL, Zaal FTJM (1994) The speed-accuracy trade-off in manual prehension: effects of movement amplitude, object size and object width on kinematic characteristics. Exp Brain Res 98:535–541. https://doi.org/10.1007/BF00233990

    CAS  Article  PubMed  Google Scholar 

  5. Borst CW, Volz RA (2005) Evaluation of a haptic mixed reality system for interactions with a virtual control panel. Presence Teleoperators Virtual Environ 14:677–696. https://doi.org/10.1162/105474605775196562

    Article  Google Scholar 

  6. Burnham KP, Anderson DR (2002) Model selection and multimodel inference: a practical information-theoretic approach. Springer-Verlag, New York

    Google Scholar 

  7. Cavina-Pratesi C, Hesse C (2013) Why do the eyes prefer the index finger? Simultaneous recording of eye and hand movements during precision grasping. J Vis 13:15. https://doi.org/10.1167/13.5.15

    Article  PubMed  Google Scholar 

  8. Cheung KL, Tunik E, Adamovich SV, Boyd LA (2014) Neuroplasticity and virtual reality. In: Weiss (Tamar) PL, Keshner EA, Levin MF (eds) Virtual reality for physical and motor rehabilitation. Springer, New York, pp 5–24

    Google Scholar 

  9. Coats R, Bingham GP, Mon-Williams M (2008) Calibrating grasp size and reach distance: interactions reveal integral organization of reaching-to-grasp movements. Exp Brain Res 189:211–220. https://doi.org/10.1007/s00221-008-1418-5

    Article  PubMed  Google Scholar 

  10. Culbertson H, Schorr SB, Okamura AM (2018) Haptics: the present and future of artificial touch sensation. Annu Rev Control Robot Auton Syst 1:385–409. https://doi.org/10.1146/annurev-control-060117-105043

    Article  Google Scholar 

  11. Curtin F, Schulz P (1998) Multiple correlations and Bonferroni’s correction. Biol Psychiatry 44:775–777. https://doi.org/10.1016/S0006-3223(98)00043-2

    CAS  Article  PubMed  Google Scholar 

  12. Fitts PM (1954) The information capacity of the human motor system in controlling the amplitude of movement. J Exp Psychol 47:381–391. https://doi.org/10.1037/h0055392

    CAS  Article  PubMed  Google Scholar 

  13. Fukui T, Inui T (2013) How vision affects kinematic properties of pantomimed prehension movements. Front Psychol 4:44. https://doi.org/10.3389/fpsyg.2013.00044

    Article  PubMed  PubMed Central  Google Scholar 

  14. Furmanek MP, Schettino LF, Yarossi M et al (2019) Coordination of reach-to-grasp in physical and haptic-free virtual environments. J Neuroeng Rehabil 16:78. https://doi.org/10.1186/s12984-019-0525-9

    Article  PubMed  PubMed Central  Google Scholar 

  15. Geiger A, Bewersdorf I, Brandenburg E, Stark R (2018) Visual feedback for grasping in virtual reality environments for an interface to instruct digital human models. In: Ahram T, Falcão C (eds) Advances in usability and user experience. Springer International Publishing, Cham, pp 228–239

    Chapter  Google Scholar 

  16. Gentilucci M, Castiello U, Corradini ML et al (1991) Influence of different types of grasping on the transport component of prehension movements. Neuropsychologia 29:361–378. https://doi.org/10.1016/0028-3932(91)90025-4

    CAS  Article  PubMed  Google Scholar 

  17. Harris DJ, Buckingham G, Wilson MR, Vine SJ (2019) Virtually the same? How impaired sensory information in virtual reality may disrupt vision for action. Exp Brain Res 237:2761–2766. https://doi.org/10.1007/s00221-019-05642-8

    Article  PubMed  PubMed Central  Google Scholar 

  18. Holden MK (2005) Virtual environments for motor rehabilitation: review. Cyberpsychol Behav 8:187–211. https://doi.org/10.1089/cpb.2005.8.187

    Article  PubMed  Google Scholar 

  19. Iyengar V, Santos MJ, Aruin AS (2009) Role of movement velocity on the magnitude of grip force while lifting an object with touch from the contralateral finger. Mot Control 13:130–141. https://doi.org/10.1123/mcj.13.2.130

    Article  Google Scholar 

  20. Jeannerod M (1984) The timing of natural prehension movements. J Mot Behav 16:235–254. https://doi.org/10.1080/00222895.1984.10735319

    CAS  Article  PubMed  Google Scholar 

  21. MacKenzie CL, Graham ED (1997) Separating A and W effects: pointing to targets on computer displays. Behav Brain Sci 20:316–318. https://doi.org/10.1017/S0140525X97361445

    Article  Google Scholar 

  22. MacKenzie CL, Marteniuk RG, Dugas C et al (1987) Three-dimensional movement trajectories in Fitts’ task: implications for control. Q J Exp Psychol Sect A 39:629–647. https://doi.org/10.1080/14640748708401806

    Article  Google Scholar 

  23. Magdalon EC, Michaelsen SM, Quevedo AA, Levin MF (2011) Comparison of grasping movements made by healthy subjects in a 3-dimensional immersive virtual versus physical environment. Acta Psychol (Amst) 138:126–134. https://doi.org/10.1016/j.actpsy.2011.05.015

    Article  Google Scholar 

  24. McIntosh RD, Mon-Williams M, Tresilian JR (2018) Grasping at laws: speed-accuracy trade-offs in manual prehension. J Exp Psychol Hum Percept Perform 44:1022–1037. https://doi.org/10.1037/xhp0000512

    Article  PubMed  Google Scholar 

  25. Ogawa N, Narumi T, Hirose M (2018) Object size perception in immersive virtual reality: avatar realism affects the way we perceive. In: 2018 IEEE conf virtual real 3D user interfaces, pp 647–648. https://doi.org/10.1109/VR.2018.8446318

  26. Pacchierotti C, Sinclair S, Solazzi M et al (2017) Wearable haptic systems for the fingertip and the hand: taxonomy, review, and perspectives. IEEE Trans Haptics 10:580–600. https://doi.org/10.1109/TOH.2017.2689006

    Article  PubMed  Google Scholar 

  27. Paulignan Y, Jeannerod M, MacKenzie C, Marteniuk R (1991a) Selective perturbation of visual input during prehension movements. 2. The effects of changing object size. Exp Brain Res 87:407–420. https://doi.org/10.1007/BF00231858

    CAS  Article  PubMed  Google Scholar 

  28. Paulignan Y, MacKenzie C, Marteniuk R, Jeannerod M (1991b) Selective perturbation of visual input during prehension movements. 1. The effects of changing object position. Exp Brain Res 83:502–512. https://doi.org/10.1007/BF00229827

    CAS  Article  PubMed  Google Scholar 

  29. Prachyabrued M, Borst CW (2012) Visual interpenetration tradeoffs in whole-hand virtual grasping. In: 2012 IEEE symp 3D user interfaces, pp 39–42. https://doi.org/10.1109/3DUI.2012.6184182

  30. Prachyabrued M, Borst CW (2014) Visual feedback for virtual grasping. In: 2014 IEEE symp 3D user interfaces, pp 19–26. https://doi.org/10.1109/3DUI.2014.6798835

  31. Rand MK, Smiley-Oyen AL, Shimansky YP et al (2006a) Control of aperture closure during reach-to-grasp movements in Parkinson’s disease. Exp Brain Res 168:131–142. https://doi.org/10.1007/s00221-005-0073-3

    CAS  Article  PubMed  Google Scholar 

  32. Rand MK, Squire LM, Stelmach GE (2006b) Effect of speed manipulation on the control of aperture closure during reach-to-grasp movements. Exp Brain Res 174:74–85. https://doi.org/10.1007/s00221-006-0423-9

    Article  PubMed  PubMed Central  Google Scholar 

  33. Rand MK, Lemay M, Squire LM et al (2007) Role of vision in aperture closure control during reach-to-grasp movements. Exp Brain Res 181:447–460. https://doi.org/10.1007/s00221-007-0945-9

    Article  PubMed  PubMed Central  Google Scholar 

  34. Rand MK, Shimansky YP, Hossain ABMI, Stelmach GE (2008) Quantitative model of transport-aperture coordination during reach-to-grasp movements. Exp Brain Res 188:263–274. https://doi.org/10.1007/s00221-008-1361-5

    Article  PubMed  Google Scholar 

  35. Rand MK, Shimansky YP, Hossain ABMI, Stelmach GE (2010) Phase dependence of transport–aperture coordination variability reveals control strategy of reach-to-grasp movements. Exp Brain Res 207:49–63. https://doi.org/10.1007/s00221-010-2428-7

    Article  PubMed  Google Scholar 

  36. Renner RS, Velichkovsky BM, Helmert JR (2013) The perception of egocentric distances in virtual environments-a review. ACM Comput Surv 46:1–40. https://doi.org/10.1145/2543581.2543590

    Article  Google Scholar 

  37. Rizzo AS, Kim GJ (2005) A SWOT analysis of the field of virtual reality rehabilitation and therapy. Presence Teleoperators Virtual Environ 14:119–146. https://doi.org/10.1162/1054746053967094

    Article  Google Scholar 

  38. Rose FD, Brooks BM, Rizzo AA (2005) Virtual reality in brain damage rehabilitation: review. Cyberpsychol Behav 8:241–262. https://doi.org/10.1089/cpb.2005.8.241

    Article  PubMed  Google Scholar 

  39. Schultheis MT, Rizzo AA (2001) The application of virtual reality technology in rehabilitation. Rehabil Psychol 46:296–311. https://doi.org/10.1037/0090-5550.46.3.296

    Article  Google Scholar 

  40. Sveistrup H (2004) Motor rehabilitation using virtual reality. J Neuroeng Rehabil 1:10. https://doi.org/10.1186/1743-0003-1-10

    Article  PubMed  PubMed Central  Google Scholar 

  41. Viau A, Feldman AG, McFadyen BJ, Levin MF (2004) Reaching in reality and virtual reality: a comparison of movement kinematics in healthy subjects and in adults with hemiparesis. J Neuroeng Rehabil 1:11. https://doi.org/10.1186/1743-0003-1-11

    Article  PubMed  PubMed Central  Google Scholar 

  42. Welford AT (1968) The fundamentals of skill. Methuen, London

    Google Scholar 

  43. Zahariev MA, MacKenzie CL (2007) Grasping at ‘thin air’: multimodal contact cues for reaching and grasping. Exp Brain Res 180:69–84. https://doi.org/10.1007/s00221-006-0845-4

    Article  PubMed  Google Scholar 

  44. Zahariev MA, Mackenzie CL (2008) Auditory contact cues improve performance when grasping augmented and virtual objects with a tool. Exp Brain Res 186:619–627. https://doi.org/10.1007/s00221-008-1269-0

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported by NIH grants #R01NS085122 and #2R01HD058301, and NSF grants #CBET-1804550 and #CMMI-M3X-1935337, to Eugene Tunik. We thank Alex Hunton and Samuel Berin for developing the VR platform.

Author information

Affiliations

Authors

Contributions

MM, MY, MPF, and GT conceived and designed research; MPF performed experiments; MM, MPF, and MPF analyzed data; MM, MY, MPF, and GT interpreted results of experiments; MM prepared figures; MM drafted the manuscript; MM, MY, MPF, and GT edited and revised manuscript; MM, MY, MPF, and GT approved the final version of the manuscript.

Corresponding author

Correspondence to Madhur Mangalam.

Ethics declarations

Conflict of interest

The authors declare that no competing interests exist.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Communicated by Francesca Frassinetti.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mangalam, M., Yarossi, M., Furmanek, M.P. et al. Control of aperture closure during reach-to-grasp movements in immersive haptic-free virtual reality. Exp Brain Res 239, 1651–1665 (2021). https://doi.org/10.1007/s00221-021-06079-8

Download citation

Keywords

  • Collision detection
  • Coordination
  • Haptic feedback
  • Prehension
  • Rehabilitation
  • Virtual environment