Skip to main content

Autonomy infused teleoperation with application to brain computer interface controlled manipulation

Autonomous Robots volume 41, pages 1401–1422 (2017)Cite this article

Abstract

Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain–Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operators capabilities and their perception of control authority. Additionally, a custom servo controller design allow for safe interactions of the robotic arm with the environment. We present experimental results demonstrating significant performance improvement using our shared-control assistance framework on adapted rehabilitation benchmarks with two subjects at various timepoints relative to their implantation with intracortical BCIs. Our results indicate that shared assistance mitigates perceived user difficulty in using a seven-degree of freedom robotic arm as a prosthetic and enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with objects previously unknown to the system in densely cluttered environments.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Notes

  1. http://clinicaltrials.gov/ct2/show/NCT01364480.

References

  • Aarno, D., Ekvall, S., & Kragic, D. (2005). Adaptive virtual fixtures for machine-assisted teleoperation tasks. In IEEE international conference on robotics and automation.

  • Aarno, D., & Kragic, D. (2008). Motion intention recognition in robot assisted applications. Robotics and Autonomous Systems, 56, 692–705.

    Article  Google Scholar 

  • Aigner, P., & McCarragher, B. (1997). Human integration into robot control utilising potential fields. In IEEE intemational conference on robotics and automation Albuquerque, (pp. 291–296).

  • Ambrose, R., Aldridge, H., Askew, R. S., Burridge, R., Bluethmann, W., Diftler, M., et al. (2000). Robonaut: Nasas space humanoid. IEEE Intelligent Systems, 15(4), 57–63.

    Article  Google Scholar 

  • Anderson, S., Peters, S., & Iagnemma, K. (2010). Semi-autonomous stability control and hazard avoidance for manned and unmanned ground vehicles. In 27th army science conference

  • Bagnell, J. A., Cavalcanti, F., Cui, L., Galluzzo, T., Hebert, M., Kazemi, M., Klingensmith, M., Libby, J., Liu, T. Y., Pollard, N., Pivtoraiko, M., Valois, J. -S., & Zhu, R. (2012). An integrated system for autonomous robotics manipulation. In IEEE/RSJ international conference on intelligent robots and systems (pp. 2955–2962).

  • Bicchi, A., & Kumar, V. (2000). Robotic grasping and contact: A review. In IEEE international conference of robotics and automation, (pp. 348–353).

  • Boularias, A., Andrew Bagnell, J., & Stentz, A. (2014). Efficient optimization for autonomous robotic manipulation of natural objects. In Proceedings of the twenty-eighth AAAI conference on artificial intelligence (pp. 2520–2526).

  • Carlson, T., & Demiris, Y. (2012). Collaborative control for a robotic wheelchair: Evaluation of performance, attention, and workload. , IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 42(3), 876–888.

    Article  Google Scholar 

  • Collinger, J. L., Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., et al. (2013). High-performance neuroprosthetic control by an individual with tetraplegia. The Lancet, 381, 557–564.

    Article  Google Scholar 

  • Crandall, J. W., & Goodrich, M. A. (2002). Characterizing efficiency of human robot interaction: A case study of shared-control teleoperation. In . IEEE/RSJ International Conference on Intelligent robots and systems, 2002, vol. 2, (pp. 1290–1295).

  • Desai, M., & Yanco, H. (2005). Blending human and robot inputs for sliding scale autonomy. In International workshop on robot and human interactive communication (pp. 537–542).

  • Dragan, A., Lee, K., & Srinivasa, S. (2013). Legibility and predictability of robot motion. In Human–robot interaction.

  • Dragan, A., & Srinivasa, S. (2013). A policy-blending formalism for shared control. The International Journal of Robotics Research, 32, 790–805.

    Article  Google Scholar 

  • Fagg, A. H., Rosenstein, M., Platt, R., & Grupen, R. A. (2004). Extracting user intent in mixed initiative teleoperator control. In Proceedings of the American institute of aeronautics and astronautics intelligent systems technical conference.

  • Goodrich, M. A., & Olsen, D. R. (2003). Jr. Seven principles of efficient human robot interaction. In IEEE transactions on systems, man, and cybernetics, part a.

  • Green, S., Billinghurst, M., Chen, X., & Chase, J. G. (2008). Human–robot collaboration: A literature review and augmented reality approach in design. International Journal of Advanced Robotic Systems, 5, 1–18.

  • Grest, D., Woetzel, J., & Koch, R. (2005). Nonlinear body pose estimation from depth images. In Pattern Recognition (pp. 285–292), Springer.

  • Hauser, K. (2013). Recognition, prediction, and planning for assisted teleoperation of freeform tasks. Autonomous Robots, 35, 241–254.

    Article  Google Scholar 

  • Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., et al. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature, 485(7398), 372–375.

    Article  Google Scholar 

  • Javdani, S., Srinivasa, S., Bagnell, J., & Andrew, D. (2015). Shared autonomy via hindsight optimization. In Robotics: Science and Systems.

  • Katyal, K. D., Johannes, M. S., Kellis, S., Aflalo, T., Klaes, C., McGee, T. G., Para, M. P., Shi, Y., Lee, B., Pejsa, K., Liu, C., Wester, B. A., Tenore, F., Beaty, J. D., Ravitz, A. D., Andersen, R. A., & McLoughlin, M. P. (2014). A collaborative bci approach to autonomous control of a prosthetic limb system. In IEEE International Conference on Systems, Man and Cybernetics (SMC), 2014 (pp. 1479–1482), IEEE.

  • Katz, D., Venkatraman, A., Kazemi, M., Bagnell, J. A., & Stentz, A. (2013). Perceiving, learning, and exploiting object affordances for autonomous pile manipulation. In Robotics: Science and Systems Conference.

  • Khatib, O., & Burdick, J. (1986). Motion and force control of robot manipulators. In Proceedings of IEEE international conference on robotics and automation , volume 3, (pp. 1381–1386), IEEE.

  • Kim, H. K., Biggs, J., Schloerb, D. W., Carmena, J. M., Lebedev, M. A., Nicolelis, M., et al. (2006). Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Transactions on Biomedical Engineering, 53(6), 1164–1173.

    Article  Google Scholar 

  • Kim, D., Hazlett-Knudsen, R., Culver-Godfrey, H., Rucks, G., Cunningham, T., Portee, D., et al. (2012). How autonomy impacts performance and satisfaction: Results from a study with spinal cord injured subjects using an assistive robot. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans, 42(1), 2–14.

    Article  Google Scholar 

  • Klingensmith, M., Galluzzo, T., Dellin, C., Kazemi, M., Bagnell, J. A., & Pollard, N. (2013). Closed-loop servoing using real-time markerless arm tracking. In International conference on robotics and automation (humanoids workshop).

  • Kofman, J., Wu, X., & Luu, T. (2005). Teleoperation of a robot manipulator using a vision-based human–robot interface. IEEE Transactions on Industrial Electronics, 5, 1206–1219.

    Article  Google Scholar 

  • Koppula, H., & Saxena, A. (2013). Anticipating human activities using object affordances for reactive robotic response. In Robotics: Science and systems.

  • Kortenkamp, D., Keim-Schreckenghost, D., & Bonasso, R. P. (2000). “Adjustable control autonomy for manned space flight”. “IEEE aerospace conference proceedings” (pp. 629–640).

  • Kragic, D., Marayong, P., Li, M., Okamura, A. M., & Hager, G. D. (2005). Human–machine collaborative systems for microsurgical applications. The International Journal of Robotics Research, 24, 162–171.

  • Lampe, T., Fiederer, L. D. J., Voelker, M., Knorr, A., Riedmiller, M., & Ball, T. (2014). A brain–computer interface for high-level remote control of an autonomous, reinforcement-learning-based robotic system for reaching and grasping. In Proceedings of the 19th international conference on intelligent user interfaces (pp. 83–88).

  • Le, Q. V., Kamm, D., Kara, A. F., & Ng, A. (2010). Learning to grasp objects with multiple contact points. In IEEE international conference on robotics and automation (ICRA) (pp. 5062–5069), IEEE.

  • Leeb, R., Perdikis, S., Tonin, L., Biasiucci, A., Tavella, M., Creatura, M., Molina, A., Al-Khodairy, A., Carlson, T., & Millán, J. R. (2013). Transferring braincomputer interfaces beyond the laboratory: Successful application control for motor-disabled users. Artificial Intelligence in Medicine, 59, 121–132.

  • Leeper, A. E., Hsiao, K., Ciocarlie, M., Takayama, L., & Gossow, D. (2012). Strategies for human-in-the-loop robotic grasping. In Proceedings of the seventh annual ACM/IEEE international conference on human–robot interaction, HRI ’12, (pp. 1–8), (New York, NY, USA, ACM). ISBN 978-1-4503-1063-5.

  • Lewis, J. P. (1995). Fast normalized cross-correlation. In Vision Interface, 10, 120–123.

    Google Scholar 

  • Li, M., & Okamura, A. M. (2003). Recognition of operator motions for real-time assistance using virtual fixtures. In International symposium on Haptic interfaces for virtual environment and teleoperator systems.

  • Li, M., Ishii, M., & Taylor, R. H. (2007). Spatial motion constraints using virtual fixtures generated by anatomy. IEEE Transactions on Robotics, 23, 4–19.

  • Lyle, R. C. (1981). A performance test for assessment of upper limb function in physical rehabilitation treatment and research. International Journal of Rehabilitation Research, 4, 483–492.

    Article  Google Scholar 

  • Marayong, P., Li, M., Okamura, A., & Hager, G. (2003). Spatial motion constraints: Theory and demonstrations for robot guidance using virtual fixtures. In 2003 IEEE international conference of robotics and automation (pp. 1954–1959).

  • Mathiowetz, V., Volland, G., Kashman, N., & Weber, K. (1985). Adult norms for the box and block test of manual dexterity. The American Journal of Occupational Therapy, 39, 386–391.

    Article  Google Scholar 

  • McMullen, D. P., Hotson, G., Katyal, K. D., Wester, B. A., Fifer, M. S., McGee, T. G., et al. (2014). Demonstration of a semi-autonomous hybrid brain–machine interface using human intracranial eeg, eye tracking, and computer vision to control a robotic upper limb prosthetic. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22(4), 784–796.

    Article  Google Scholar 

  • Murray, R. M., Li, Z., & Sastry, S. S. (1994). A mathematical introduction to robotic manipulation. Boca Raton: CRC Press.

    MATH  Google Scholar 

  • Nakanishi, J., Cory, R., Mistry, M., Peters, J., & Schaal, S. (2008). Operational space control: A theoretical and empirical comparison. The International Journal of Robotics Research, 27(6), 737–757.

    Article  Google Scholar 

  • Palankar, M., De Laurentis, K. J., Alqasemi, R., Veras, E., Dubey, R., Arbel, Y., & Donchin, E. (2009). Control of a 9-dof wheelchair-mounted robotic arm system using a p300 brain computer interface: Initial experiments. In IEEE International Conference on Robotics and Biomimetics, ROBIO 2008 (pp. 348–353), IEEE.

  • Park, S., Howe, R. D., & Torchiana, D. F. (2001). Virtual fixtures for robotic cardiac surgery. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2001 (pp. 1419–1420), Springer.

  • Ratliff, N. D., Bagnell, J. A., & Zinkevich, M. A. (2006). Maximum margin planning. In Proceedings of the 23rd international conference on machine learning (pp. 729–736), ACM.

  • Saxena, A., Driemeyer, J., & Ng, A. (2008). Robotic grasping of novel objects using vision. The International Journal of Robotics Research, 27(2), 157–173.

    Article  Google Scholar 

  • Schrempf, O. C., Albrecht, D., & Hanebeck, U. D. (2007). Tractable probabilistic models for intention recognition based on expert knowledge. In IEEE/RSJ international conference on intelligent robots and systems.

  • Schröer, S., Killmann, I., Frank, B., Voelker, M., Fiederer, L. D. J., Ball, T., & Burgard, W. (2015). An autonomous robotic assistant for drinking. In IEEE International Conference on Robotics and Automation.

  • Schwartz, A. B., Weber, D. J., & Moran, D. W. (2006). Brain-controlled interfaces: Movement restoration with neural prosthetics. Neuron, 52(1), 205–220.

    Article  Google Scholar 

  • Shen, J., Ibanez-Guzman, J., Ng, T. C., & Chew, B. S., (2004). A collaborative-shared control system with safe obstacle avoidance capability. In ieee international conference on robotics, automation, and mechatronics.

  • Sheridan, Thomas  B. (1992). Telerobotics, automation, and human supervisory control. Cambridge: MIT Press.

    Google Scholar 

  • Siciliano, B., & Khatib, O. (2008). Springer handbook of robotics. Springer.

  • Siciliano, B., Sciavicco, L., Villani, L., & Oriolo, G. (2008). Robotics: modelling, planning and control. Springer Publishing Company, Incorporated, 1st edition, ISBN 1846286417, 9781846286414.

  • Simeral, J. D., Kim, S. P., Black, M. J., Donoghue, J. P., & Hochberg, L. R. (2011). Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. Journal of Neural Engineering, 8(2), 025027.

    Article  Google Scholar 

  • Sutherland, I. E. (1964). Sketchpad: A man-machine graphical communication system. In American Federation of Information Processing Societies (Vol. 23, pp. 323–328).

  • Toshev, A., Makadia, A., & Daniilidis, K. (2009). Shape-based object recognition in videos using 3d synthetic object models. In ieee conference on computer vision and pattern recognition, CVPR 2009 (pp. 288–295), IEEE.

  • Trautman, P., (2015). Assistive planning in complex, dynamic environments: A probabilistic approach. In HRI Workshop on human–robot teaming.

  • Vanhooydonck, D., Demeester, E., Nuttin, M., & Brussel, H. V. (2003). Shared control for intelligent wheelchairs: an implicit estimation of the user intention. In Proceedings of the ASER international workshop on advances in service robotics.

  • Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453(7198), 1098–1101.

    Article  Google Scholar 

  • Vogel, J., Haddadin, S., Simeral, J. D., Stavisky, S. D., Bacher, D., Hochberg, L. R., Donoghue, J. P., & van der Smagt, P. (2014). Continuous control of the dlr light-weight robot iii by a human with tetraplegia using the braingate2 neural interface system. In Experimental Robotics (pp. 125–136), Springer.

  • Vogel, J., Haddadin, S., Jarosiewicz, B., Simeral, J. D., Bacher, D., Hochberg, L. R., et al. (2014). An assistive decision-and-control architecture for force-sensitive hand-arm systems driven by human-machine interfaces. The International Journal of Robotics Research, 34(6), 763–780.

    Article  Google Scholar 

  • Volpe, R., & Khosla, P. (1993). A theoretical and experimental investigation of explicit force control strategies for manipulators. IEEE Transactions onAutomatic Control, 38(11), 1634–1650.

    MathSciNet  Article  Google Scholar 

  • Wang, W., Chan, S. S., Heldman, D. A., & Moran, D. W. (2007). Motor cortical representation of position and velocity during reaching. Journal of Neurophysiology, 97(6), 4258–4270.

  • Wang, Z., Mülling, K., Deisenroth, M. P., Amor, H. B., Vogt, D., Schölkopf, B., & Peters, J. (2013). Probabilistic movement modeling for intention inference in human–robot interaction. The International Journal of Robotics Research, 8, 841–858.

  • Weber, C., Nitsch, V., Unterhinninghofen, U., Faerber, B., & Buss, M. (2009). Position and force augmentation in a telepresence system and their effects on perceived realism. In Symposium on Haptic interfaces for virtual environment and teleoperator systems (pp. 226–231).

  • Wodlinger, B., Downey, J. E., Tyler-Kabara, E. C., Schwartz, A. B., Boninger, M. L., & Collinger, J. L. (2015). 10 dimensional anthropomorphic arm control in a human brain–machine interface: Difficulties, solutions, and limitations. Journal of Neural Engineering, 12, 01611.

  • You, E., & Hauser, K. (2011). Assisted teleoperation strategies for aggressively controlling a robot arm with 2d input. In Proc. Robotics: Science and Systems (Vol. 7, pp. 354–361).

  • Yozbatiran, N., Der-Yeghiaian, L., & Cramer, S. C. (2008). A standardized approach to performing the action research arm test. Neurorehabil Neural Repair, 22, 78–90.

  • Yu, W., Alqasemi, R., Dubey, R., & Pernalete, N. (2005). Telemanipulation assistance based on motion intention recognition. In ieee international conference on robotics and automation (pp. 1121 – 1126).

  • Ziebart, B. A., Dey, A. K., & Bagnell, J. A. (2012). Probabilistic pointing target prediction via inverse optimal control.

  • Ziebart, B. D., Maas, A., Bagnell, J. A., & Dey, A. (2008). Maximum entropy inverse reinforcement learning. In AAAI (Vol. NA, pp. 1433–1438).

  • Ziebart, B. D., Ratliff, N., Gallagher, G., Mertz, C., Peterson, K., Bagnell, J. A., Hebert, M., Dey, A., & Srinivasa, S. (2009). Planning-based prediction for pedestrians. In IEEE/RSJ IROS.

Download references

Acknowledgements

The authors gratefully acknowledge funding under the Defense Advanced Research Projects Agencys Autonomous Robotic Manipulation Software Track (ARM-S) program and the Revolutionizing Prosthetics program (contract number N66001-10-C-4056). The material presented in this paper is based upon work supported by the National Science Foundation’s NRI Purposeful Prediction program (Award No. 1227495) and the GRF program (Award No. DGE-1252522). This study was completed under an investigational device exemption granted by the US Food and Drug Administration. We thank the study participants for their dedication and insightful discussions with the study team. The views expressed herein are those of the authors and do not represent the official policy or position of the Department of Veterans Affairs, Department of Defense, National Science Foundation, or the US Government. We thank Sidd Srinivasa for helpful conversations and Pedro Mediano for his work on the “Dragonfly” software bridge that enabled this effort.

Author information

Authors and Affiliations

  1. Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA

    Katharina Muelling, Arun Venkatraman, Jean-Sebastien Valois, Shervin Javdani, Martial Hebert, Andrew B. Schwartz & J. Andrew Bagnell

  2. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, USA

    John E. Downey & Jennifer L. Collinger

  3. Department of Physical Medicine and Rehabilitation, Univeristy of Pittsburgh, Pittsburgh, PA, USA

    Jeffrey Weiss & Jennifer L. Collinger

  4. VA Pittsburgh Healthcare System, Pittsburgh, PA, USA

    Jennifer L. Collinger

  5. Center for the Neural Basis of Cognition, Pittsburgh, PA, USA

    Andrew B. Schwartz

Authors
  1. Katharina Muelling
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arun Venkatraman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean-Sebastien Valois
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John E. Downey
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jeffrey Weiss
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shervin Javdani
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Martial Hebert
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Andrew B. Schwartz
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Jennifer L. Collinger
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. J. Andrew Bagnell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Katharina Muelling.

Additional information

This is one of several papers published in Autonomous Robots comprising the “Special Issue on Robotics Science and Systems”.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Muelling, K., Venkatraman, A., Valois, JS. et al. Autonomy infused teleoperation with application to brain computer interface controlled manipulation. Auton Robot 41, 1401–1422 (2017). https://doi.org/10.1007/s10514-017-9622-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-017-9622-4

Keywords

  • Brain computer interface
  • Shared control telerobotics
  • Neuroprosthetics
  • Assistive robotics