Autonomous Robots

, Volume 41, Issue 6, pp 1401–1422 | Cite as

Autonomy infused teleoperation with application to brain computer interface controlled manipulation

  • Katharina MuellingEmail author
  • Arun Venkatraman
  • Jean-Sebastien Valois
  • John E. Downey
  • Jeffrey Weiss
  • Shervin Javdani
  • Martial Hebert
  • Andrew B. Schwartz
  • Jennifer L. Collinger
  • J. Andrew Bagnell
Part of the following topical collections:
  1. Special Issue on "Robotics: Science and Systems"


Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain–Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operators capabilities and their perception of control authority. Additionally, a custom servo controller design allow for safe interactions of the robotic arm with the environment. We present experimental results demonstrating significant performance improvement using our shared-control assistance framework on adapted rehabilitation benchmarks with two subjects at various timepoints relative to their implantation with intracortical BCIs. Our results indicate that shared assistance mitigates perceived user difficulty in using a seven-degree of freedom robotic arm as a prosthetic and enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with objects previously unknown to the system in densely cluttered environments.


Brain computer interface Shared control telerobotics Neuroprosthetics Assistive robotics 



The authors gratefully acknowledge funding under the Defense Advanced Research Projects Agencys Autonomous Robotic Manipulation Software Track (ARM-S) program and the Revolutionizing Prosthetics program (contract number N66001-10-C-4056). The material presented in this paper is based upon work supported by the National Science Foundation’s NRI Purposeful Prediction program (Award No. 1227495) and the GRF program (Award No. DGE-1252522). This study was completed under an investigational device exemption granted by the US Food and Drug Administration. We thank the study participants for their dedication and insightful discussions with the study team. The views expressed herein are those of the authors and do not represent the official policy or position of the Department of Veterans Affairs, Department of Defense, National Science Foundation, or the US Government. We thank Sidd Srinivasa for helpful conversations and Pedro Mediano for his work on the “Dragonfly” software bridge that enabled this effort.


  1. Aarno, D., Ekvall, S., & Kragic, D. (2005). Adaptive virtual fixtures for machine-assisted teleoperation tasks. In IEEE international conference on robotics and automation.Google Scholar
  2. Aarno, D., & Kragic, D. (2008). Motion intention recognition in robot assisted applications. Robotics and Autonomous Systems, 56, 692–705.CrossRefGoogle Scholar
  3. Aigner, P., & McCarragher, B. (1997). Human integration into robot control utilising potential fields. In IEEE intemational conference on robotics and automation Albuquerque, (pp. 291–296).Google Scholar
  4. Ambrose, R., Aldridge, H., Askew, R. S., Burridge, R., Bluethmann, W., Diftler, M., et al. (2000). Robonaut: Nasas space humanoid. IEEE Intelligent Systems, 15(4), 57–63.CrossRefGoogle Scholar
  5. Anderson, S., Peters, S., & Iagnemma, K. (2010). Semi-autonomous stability control and hazard avoidance for manned and unmanned ground vehicles. In 27th army science conference Google Scholar
  6. Bagnell, J. A., Cavalcanti, F., Cui, L., Galluzzo, T., Hebert, M., Kazemi, M., Klingensmith, M., Libby, J., Liu, T. Y., Pollard, N., Pivtoraiko, M., Valois, J. -S., & Zhu, R. (2012). An integrated system for autonomous robotics manipulation. In IEEE/RSJ international conference on intelligent robots and systems (pp. 2955–2962).Google Scholar
  7. Bicchi, A., & Kumar, V. (2000). Robotic grasping and contact: A review. In IEEE international conference of robotics and automation, (pp. 348–353).Google Scholar
  8. Boularias, A., Andrew Bagnell, J., & Stentz, A. (2014). Efficient optimization for autonomous robotic manipulation of natural objects. In Proceedings of the twenty-eighth AAAI conference on artificial intelligence (pp. 2520–2526).Google Scholar
  9. Carlson, T., & Demiris, Y. (2012). Collaborative control for a robotic wheelchair: Evaluation of performance, attention, and workload. , IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 42(3), 876–888.CrossRefGoogle Scholar
  10. Collinger, J. L., Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., et al. (2013). High-performance neuroprosthetic control by an individual with tetraplegia. The Lancet, 381, 557–564.CrossRefGoogle Scholar
  11. Crandall, J. W., & Goodrich, M. A. (2002). Characterizing efficiency of human robot interaction: A case study of shared-control teleoperation. In . IEEE/RSJ International Conference on Intelligent robots and systems, 2002, vol. 2, (pp. 1290–1295).Google Scholar
  12. Desai, M., & Yanco, H. (2005). Blending human and robot inputs for sliding scale autonomy. In International workshop on robot and human interactive communication (pp. 537–542).Google Scholar
  13. Dragan, A., Lee, K., & Srinivasa, S. (2013). Legibility and predictability of robot motion. In Human–robot interaction.Google Scholar
  14. Dragan, A., & Srinivasa, S. (2013). A policy-blending formalism for shared control. The International Journal of Robotics Research, 32, 790–805.CrossRefGoogle Scholar
  15. Fagg, A. H., Rosenstein, M., Platt, R., & Grupen, R. A. (2004). Extracting user intent in mixed initiative teleoperator control. In Proceedings of the American institute of aeronautics and astronautics intelligent systems technical conference.Google Scholar
  16. Goodrich, M. A., & Olsen, D. R. (2003). Jr. Seven principles of efficient human robot interaction. In IEEE transactions on systems, man, and cybernetics, part a.Google Scholar
  17. Green, S., Billinghurst, M., Chen, X., & Chase, J. G. (2008). Human–robot collaboration: A literature review and augmented reality approach in design. International Journal of Advanced Robotic Systems, 5, 1–18.Google Scholar
  18. Grest, D., Woetzel, J., & Koch, R. (2005). Nonlinear body pose estimation from depth images. In Pattern Recognition (pp. 285–292), Springer.Google Scholar
  19. Hauser, K. (2013). Recognition, prediction, and planning for assisted teleoperation of freeform tasks. Autonomous Robots, 35, 241–254.CrossRefGoogle Scholar
  20. Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., et al. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature, 485(7398), 372–375.CrossRefGoogle Scholar
  21. Javdani, S., Srinivasa, S., Bagnell, J., & Andrew, D. (2015). Shared autonomy via hindsight optimization. In Robotics: Science and Systems.Google Scholar
  22. Katyal, K. D., Johannes, M. S., Kellis, S., Aflalo, T., Klaes, C., McGee, T. G., Para, M. P., Shi, Y., Lee, B., Pejsa, K., Liu, C., Wester, B. A., Tenore, F., Beaty, J. D., Ravitz, A. D., Andersen, R. A., & McLoughlin, M. P. (2014). A collaborative bci approach to autonomous control of a prosthetic limb system. In IEEE International Conference on Systems, Man and Cybernetics (SMC), 2014 (pp. 1479–1482), IEEE.Google Scholar
  23. Katz, D., Venkatraman, A., Kazemi, M., Bagnell, J. A., & Stentz, A. (2013). Perceiving, learning, and exploiting object affordances for autonomous pile manipulation. In Robotics: Science and Systems Conference.Google Scholar
  24. Khatib, O., & Burdick, J. (1986). Motion and force control of robot manipulators. In Proceedings of IEEE international conference on robotics and automation , volume 3, (pp. 1381–1386), IEEE.Google Scholar
  25. Kim, H. K., Biggs, J., Schloerb, D. W., Carmena, J. M., Lebedev, M. A., Nicolelis, M., et al. (2006). Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Transactions on Biomedical Engineering, 53(6), 1164–1173.CrossRefGoogle Scholar
  26. Kim, D., Hazlett-Knudsen, R., Culver-Godfrey, H., Rucks, G., Cunningham, T., Portee, D., et al. (2012). How autonomy impacts performance and satisfaction: Results from a study with spinal cord injured subjects using an assistive robot. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 42(1), 2–14.CrossRefGoogle Scholar
  27. Klingensmith, M., Galluzzo, T., Dellin, C., Kazemi, M., Bagnell, J. A., & Pollard, N. (2013). Closed-loop servoing using real-time markerless arm tracking. In International conference on robotics and automation (humanoids workshop).Google Scholar
  28. Kofman, J., Wu, X., & Luu, T. (2005). Teleoperation of a robot manipulator using a vision-based human–robot interface. IEEE Transactions on Industrial Electronics, 5, 1206–1219.CrossRefGoogle Scholar
  29. Koppula, H., & Saxena, A. (2013). Anticipating human activities using object affordances for reactive robotic response. In Robotics: Science and systems.Google Scholar
  30. Kortenkamp, D., Keim-Schreckenghost, D., & Bonasso, R. P. (2000). “Adjustable control autonomy for manned space flight”. “IEEE aerospace conference proceedings” (pp. 629–640).Google Scholar
  31. Kragic, D., Marayong, P., Li, M., Okamura, A. M., & Hager, G. D. (2005). Human–machine collaborative systems for microsurgical applications. The International Journal of Robotics Research, 24, 162–171.Google Scholar
  32. Lampe, T., Fiederer, L. D. J., Voelker, M., Knorr, A., Riedmiller, M., & Ball, T. (2014). A brain–computer interface for high-level remote control of an autonomous, reinforcement-learning-based robotic system for reaching and grasping. In Proceedings of the 19th international conference on intelligent user interfaces (pp. 83–88).Google Scholar
  33. Le, Q. V., Kamm, D., Kara, A. F., & Ng, A. (2010). Learning to grasp objects with multiple contact points. In IEEE international conference on robotics and automation (ICRA) (pp. 5062–5069), IEEE.Google Scholar
  34. Leeb, R., Perdikis, S., Tonin, L., Biasiucci, A., Tavella, M., Creatura, M., Molina, A., Al-Khodairy, A., Carlson, T., & Millán, J. R. (2013). Transferring braincomputer interfaces beyond the laboratory: Successful application control for motor-disabled users. Artificial Intelligence in Medicine, 59, 121–132.Google Scholar
  35. Leeper, A. E., Hsiao, K., Ciocarlie, M., Takayama, L., & Gossow, D. (2012). Strategies for human-in-the-loop robotic grasping. In Proceedings of the seventh annual ACM/IEEE international conference on human–robot interaction, HRI ’12, (pp. 1–8), (New York, NY, USA, ACM). ISBN 978-1-4503-1063-5.Google Scholar
  36. Lewis, J. P. (1995). Fast normalized cross-correlation. In Vision Interface, 10, 120–123.Google Scholar
  37. Li, M., & Okamura, A. M. (2003). Recognition of operator motions for real-time assistance using virtual fixtures. In International symposium on Haptic interfaces for virtual environment and teleoperator systems.Google Scholar
  38. Li, M., Ishii, M., & Taylor, R. H. (2007). Spatial motion constraints using virtual fixtures generated by anatomy. IEEE Transactions on Robotics, 23, 4–19.Google Scholar
  39. Lyle, R. C. (1981). A performance test for assessment of upper limb function in physical rehabilitation treatment and research. International Journal of Rehabilitation Research, 4, 483–492.CrossRefGoogle Scholar
  40. Marayong, P., Li, M., Okamura, A., & Hager, G. (2003). Spatial motion constraints: Theory and demonstrations for robot guidance using virtual fixtures. In 2003 IEEE international conference of robotics and automation (pp. 1954–1959).Google Scholar
  41. Mathiowetz, V., Volland, G., Kashman, N., & Weber, K. (1985). Adult norms for the box and block test of manual dexterity. The American Journal of Occupational Therapy, 39, 386–391.CrossRefGoogle Scholar
  42. McMullen, D. P., Hotson, G., Katyal, K. D., Wester, B. A., Fifer, M. S., McGee, T. G., et al. (2014). Demonstration of a semi-autonomous hybrid brain–machine interface using human intracranial eeg, eye tracking, and computer vision to control a robotic upper limb prosthetic. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(4), 784–796.CrossRefGoogle Scholar
  43. Murray, R. M., Li, Z., & Sastry, S. S. (1994). A mathematical introduction to robotic manipulation. Boca Raton: CRC Press.zbMATHGoogle Scholar
  44. Nakanishi, J., Cory, R., Mistry, M., Peters, J., & Schaal, S. (2008). Operational space control: A theoretical and empirical comparison. The International Journal of Robotics Research, 27(6), 737–757.CrossRefGoogle Scholar
  45. Palankar, M., De Laurentis, K. J., Alqasemi, R., Veras, E., Dubey, R., Arbel, Y., & Donchin, E. (2009). Control of a 9-dof wheelchair-mounted robotic arm system using a p300 brain computer interface: Initial experiments. In IEEE International Conference on Robotics and Biomimetics, ROBIO 2008 (pp. 348–353), IEEE.Google Scholar
  46. Park, S., Howe, R. D., & Torchiana, D. F. (2001). Virtual fixtures for robotic cardiac surgery. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2001 (pp. 1419–1420), Springer.Google Scholar
  47. Ratliff, N. D., Bagnell, J. A., & Zinkevich, M. A. (2006). Maximum margin planning. In Proceedings of the 23rd international conference on machine learning (pp. 729–736), ACM.Google Scholar
  48. Saxena, A., Driemeyer, J., & Ng, A. (2008). Robotic grasping of novel objects using vision. The International Journal of Robotics Research, 27(2), 157–173.CrossRefGoogle Scholar
  49. Schrempf, O. C., Albrecht, D., & Hanebeck, U. D. (2007). Tractable probabilistic models for intention recognition based on expert knowledge. In IEEE/RSJ international conference on intelligent robots and systems.Google Scholar
  50. Schröer, S., Killmann, I., Frank, B., Voelker, M., Fiederer, L. D. J., Ball, T., & Burgard, W. (2015). An autonomous robotic assistant for drinking. In IEEE International Conference on Robotics and Automation.Google Scholar
  51. Schwartz, A. B., Weber, D. J., & Moran, D. W. (2006). Brain-controlled interfaces: Movement restoration with neural prosthetics. Neuron, 52(1), 205–220.CrossRefGoogle Scholar
  52. Shen, J., Ibanez-Guzman, J., Ng, T. C., & Chew, B. S., (2004). A collaborative-shared control system with safe obstacle avoidance capability. In ieee international conference on robotics, automation, and mechatronics.Google Scholar
  53. Sheridan, Thomas  B. (1992). Telerobotics, automation, and human supervisory control. Cambridge: MIT Press.Google Scholar
  54. Siciliano, B., & Khatib, O. (2008). Springer handbook of robotics. Springer.Google Scholar
  55. Siciliano, B., Sciavicco, L., Villani, L., & Oriolo, G. (2008). Robotics: modelling, planning and control. Springer Publishing Company, Incorporated, 1st edition, ISBN 1846286417, 9781846286414.Google Scholar
  56. Simeral, J. D., Kim, S. P., Black, M. J., Donoghue, J. P., & Hochberg, L. R. (2011). Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. Journal of Neural Engineering, 8(2), 025027.CrossRefGoogle Scholar
  57. Sutherland, I. E. (1964). Sketchpad: A man-machine graphical communication system. In American Federation of Information Processing Societies (Vol. 23, pp. 323–328).Google Scholar
  58. Toshev, A., Makadia, A., & Daniilidis, K. (2009). Shape-based object recognition in videos using 3d synthetic object models. In ieee conference on computer vision and pattern recognition, CVPR 2009 (pp. 288–295), IEEE.Google Scholar
  59. Trautman, P., (2015). Assistive planning in complex, dynamic environments: A probabilistic approach. In HRI Workshop on human–robot teaming.Google Scholar
  60. Vanhooydonck, D., Demeester, E., Nuttin, M., & Brussel, H. V. (2003). Shared control for intelligent wheelchairs: an implicit estimation of the user intention. In Proceedings of the ASER international workshop on advances in service robotics.Google Scholar
  61. Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453(7198), 1098–1101.CrossRefGoogle Scholar
  62. Vogel, J., Haddadin, S., Simeral, J. D., Stavisky, S. D., Bacher, D., Hochberg, L. R., Donoghue, J. P., & van der Smagt, P. (2014). Continuous control of the dlr light-weight robot iii by a human with tetraplegia using the braingate2 neural interface system. In Experimental Robotics (pp. 125–136), Springer.Google Scholar
  63. Vogel, J., Haddadin, S., Jarosiewicz, B., Simeral, J. D., Bacher, D., Hochberg, L. R., et al. (2014). An assistive decision-and-control architecture for force-sensitive hand-arm systems driven by human-machine interfaces. The International Journal of Robotics Research, 34(6), 763–780.CrossRefGoogle Scholar
  64. Volpe, R., & Khosla, P. (1993). A theoretical and experimental investigation of explicit force control strategies for manipulators. IEEE Transactions onAutomatic Control, 38(11), 1634–1650.MathSciNetCrossRefGoogle Scholar
  65. Wang, W., Chan, S. S., Heldman, D. A., & Moran, D. W. (2007). Motor cortical representation of position and velocity during reaching. Journal of Neurophysiology, 97(6), 4258–4270.Google Scholar
  66. Wang, Z., Mülling, K., Deisenroth, M. P., Amor, H. B., Vogt, D., Schölkopf, B., & Peters, J. (2013). Probabilistic movement modeling for intention inference in human–robot interaction. The International Journal of Robotics Research, 8, 841–858.Google Scholar
  67. Weber, C., Nitsch, V., Unterhinninghofen, U., Faerber, B., & Buss, M. (2009). Position and force augmentation in a telepresence system and their effects on perceived realism. In Symposium on Haptic interfaces for virtual environment and teleoperator systems (pp. 226–231).Google Scholar
  68. Wodlinger, B., Downey, J. E., Tyler-Kabara, E. C., Schwartz, A. B., Boninger, M. L., & Collinger, J. L. (2015). 10 dimensional anthropomorphic arm control in a human brain–machine interface: Difficulties, solutions, and limitations. Journal of Neural Engineering, 12, 01611.Google Scholar
  69. You, E., & Hauser, K. (2011). Assisted teleoperation strategies for aggressively controlling a robot arm with 2d input. In Proc. Robotics: Science and Systems (Vol. 7, pp. 354–361).Google Scholar
  70. Yozbatiran, N., Der-Yeghiaian, L., & Cramer, S. C. (2008). A standardized approach to performing the action research arm test. Neurorehabil Neural Repair, 22, 78–90.Google Scholar
  71. Yu, W., Alqasemi, R., Dubey, R., & Pernalete, N. (2005). Telemanipulation assistance based on motion intention recognition. In ieee international conference on robotics and automation (pp. 1121 – 1126).Google Scholar
  72. Ziebart, B. A., Dey, A. K., & Bagnell, J. A. (2012). Probabilistic pointing target prediction via inverse optimal control.Google Scholar
  73. Ziebart, B. D., Maas, A., Bagnell, J. A., & Dey, A. (2008). Maximum entropy inverse reinforcement learning. In AAAI (Vol. NA, pp. 1433–1438).Google Scholar
  74. Ziebart, B. D., Ratliff, N., Gallagher, G., Mertz, C., Peterson, K., Bagnell, J. A., Hebert, M., Dey, A., & Srinivasa, S. (2009). Planning-based prediction for pedestrians. In IEEE/RSJ IROS.Google Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Katharina Muelling
    • 1
    Email author
  • Arun Venkatraman
    • 1
  • Jean-Sebastien Valois
    • 1
  • John E. Downey
    • 2
  • Jeffrey Weiss
    • 3
  • Shervin Javdani
    • 1
  • Martial Hebert
    • 1
  • Andrew B. Schwartz
    • 1
    • 5
  • Jennifer L. Collinger
    • 2
    • 3
    • 4
  • J. Andrew Bagnell
    • 1
  1. 1.Robotics InstituteCarnegie Mellon UniversityPittsburghUSA
  2. 2.Department of BioengineeringUniversity of PittsburghPittsburghUSA
  3. 3.Department of Physical Medicine and RehabilitationUniveristy of PittsburghPittsburghUSA
  4. 4.VA Pittsburgh Healthcare SystemPittsburghUSA
  5. 5.Center for the Neural Basis of CognitionPittsburghUSA

Personalised recommendations