Comparing Robot Grasping Teleoperation Across Desktop and Virtual Reality with ROS Reality

  • David WhitneyEmail author
  • Eric Rosen
  • Elizabeth Phillips
  • George Konidaris
  • Stefanie Tellex
Conference paper
Part of the Springer Proceedings in Advanced Robotics book series (SPAR, volume 10)


Teleoperation allows a human to remotely operate a robot to perform complex and potentially dangerous tasks such as defusing a bomb, repairing a nuclear reactor, or maintaining the exterior of a space station. Existing teleoperation approaches generally rely on computer monitors to display sensor data and joysticks or keyboards to actuate the robot. These approaches use 2D interfaces to view and interact with a 3D world, which can make using them difficult for complex or delicate tasks. To address this problem, we introduce a virtual reality interface that allows users to remotely teleoperate a physical robot in real-time. Our interface allows users to control their point of view in the scene using virtual reality, increasing situational awareness (especially of object contact), and to directly move the robot’s end effector by moving a hand controller in 3D space, enabling fine-grained dexterous control. We evaluated our interface on a cup-stacking manipulation task with 18 participants, comparing the relative effectiveness of a keyboard and mouse interface, virtual reality camera control, and positional hand tracking. Our system reduces task completion time from 153 s (\(\pm 44\)) to 53 s (\(\pm 37\)), a reduction of 66%, while improving subjective assessments of system usability and workload. Additionally, we have shown the effectiveness of our system over long distances, successfully completing a cup stacking task from over 40 miles away. Our paper contributes a quantitative assessment of robot grasping teleoperation across desktop and virtual reality interfaces.



This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Grants No. W911NF-15-1-0503, YFA: D15AP00104, YFA: GR5245014, and D15AP00102, as well as NASA under Grants No. GR5227035.

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA.


  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
    Bangor, A., Kortum, P.T., Miller, J.T.: An empirical evaluation of the system usability scale. Int. J. Hum.-Comput. Interact. 24(6), 574–594 (2008)CrossRefGoogle Scholar
  6. 6.
    Brooke, J., et al.: Sus-a quick and dirty usability scale. Usability Evaluation in Industry, vol. 189(194), pp. 4–7 (1996)Google Scholar
  7. 7.
    Calli, B., Singh, A., Walsman, A., Srinivasa, S., Abbeel, P., Dollar, A.M.: The ycb object and model set: towards common benchmarks for manipulation research. In: 2015 International Conference on Advanced Robotics (ICAR), pp. 510–517. IEEE (2015)Google Scholar
  8. 8.
    Chen, Y.-P., Lee, S.-Y., Howard, A.M.: Effect of virtual reality on upper extremity function in children with cerebral palsy: a meta-analysis. Pediatr. Phys. Ther. 26(3), 289–300 (2014)CrossRefGoogle Scholar
  9. 9.
    N. H. P. R. Group et al. Task load index (nasa-tlx) v1. 0 computerised version. NASA Ames Research Centre (1987)Google Scholar
  10. 10.
    Kam, H.R., Lee, S.-H., Park, T., Kim, C.-H.: Rviz: a toolkit for real domain data visualization. Telecommun. Syst. 60(2), 337–345 (2015)CrossRefGoogle Scholar
  11. 11.
    Koenig, N., Howard, A.: Design and use paradigms for gazebo, an open-source multi-robot simulator. In: 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2004. (IROS 2004). Proceedings, vol. 3, pp. 2149–2154. IEEE (2004)Google Scholar
  12. 12.
    Lohse, K.R., Hilderman, C.G., Cheung, K.L., Tatla, S., Van der Loos, H.M.: Virtual reality therapy for adults post-stroke: a systematic review and meta-analysis exploring virtual environments and commercial games in therapy. PloS one 9(3), e93318 (2014)CrossRefGoogle Scholar
  13. 13.
    Moroney, W.F., Biers, D.W., Eggemeier, F.T., Mitchell, J.A.: A comparison of two scoring procedures with the nasa task load index in a simulated flight task. In: Proceedings of the IEEE 1992 National Aerospace and Electronics Conference, 1992. NAECON 1992, pp. 734–740. IEEE (1992)Google Scholar
  14. 14.
    Pratt, G., Manzo, J.: The darpa robotics challenge [competitions]. IEEE Robot. Autom. Mag. 20(2), 10–12 (2013)CrossRefGoogle Scholar
  15. 15.
    Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3, p. 5. Kobe (2009)Google Scholar
  16. 16.
    Serrano, D.: Introduction to ros–robot operating systemGoogle Scholar
  17. 17.
    Zucker, M., Joo, S., Grey, M.X., Rasmussen, C., Huang, E., Stilman, M., Bobick, A.: A general-purpose system for teleoperation of the drc-hubo humanoid robot. J. Field Robot. 32(3), 336–351 (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • David Whitney
    • 1
    Email author
  • Eric Rosen
    • 1
  • Elizabeth Phillips
    • 1
  • George Konidaris
    • 1
  • Stefanie Tellex
    • 1
  1. 1.Department of Computer ScienceBrown UniversityProvidenceUSA

Personalised recommendations