Autonomous Robots

, Volume 43, Issue 6, pp 1357–1373 | Cite as

Online adaptive teleoperation via motion primitives for mobile robots

  • Xuning YangEmail author
  • Ayush Agrawal
  • Koushil Sreenath
  • Nathan Michael
Part of the following topical collections:
  1. Special Issue: Learning for Human-Robot Collaboration


Assistive teleoperation aims to help operators control robotic systems with ease. In this work, we present a novel adaptive teleoperation approach that is amenable to mobile systems using motion primitives for long-duration teleoperation, such as exploration using mobile vehicles or walking for humanoid systems. We first describe teleoperation using motion primitives, which are dynamically feasible and safe local trajectories based on a kinematic or dynamic model. We take a predict-and-adapt approach to assistive teleoperation, whereby adaptation is based on the predicted user intent. By representing the operator as an optimizing controller, a probabilistic distribution can be constructed for the available future actions based on some reward function. Adaptation is provided in the form of subsampling, which tailors the set of available actions based on the likelihood of action selection. We describe the framework for general systems and delineate the extrapolation to ground, air, and legged mobile robots, and demonstrate generalizability of this framework on two systems via simulation and experimentation; namely, a quadrotor micro air vehicle, and a simulated 3D humanoid system. Both systems show provably better performance in teleoperation by measures of behavioral entropy.


Teleoperation Adaptive teleoperation User intent prediction Quadrotor control Humanoid control Human-in-the-loop Intent prediction 


  1. Admoni, H., & Srinivasa, S. (2016). Predicting user intent through eye gaze for shared autonomy. In The 2016 AIII fall symposium series: Shared autonomy in research and practice. Technical Report FS-16-05.Google Scholar
  2. Aigner, P., & McCarragher, B. (1997). Human integration into robot control utilising potential fields. In Robotics and automation, 1997. Proceedings., 1997 IEEE international conference on (Vol. 1, pp. 291–296). IEEE.Google Scholar
  3. Anderson, S. J., Walker, J. M., & Iagnemma, K. (2014). Experimental performance analysis of a homotopy-based shared autonomy framework. IEEE Transactions on Human-Machine Systems, 44, 190–199.CrossRefGoogle Scholar
  4. Bachrach, A. G. (2013). Trajectory bundle estimation for perception-driven planning. Ph.D. Thesis, Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge.Google Scholar
  5. Borghini, G., Astolfi, L., Vecchiato, G., Mattia, D., & Babiloni, F. (2014). Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neuroscience and Biobehavioral Reviews, 44, 58–75.CrossRefGoogle Scholar
  6. Carlson, T., & Demiris, Y. (2008). Human-wheelchair collaboration through prediction of intention and adaptive assistance. In Proceedings of the IEEE international conference on robot and automation, Pasadena, CA, pp. 3926–3931.Google Scholar
  7. Chevallereau, C., Grizzle, J. W., & Shih, C. L. (2010). Steering of a 3D bipedal robot with an underactuated ankle. In Intelligent robots and systems (IROS), 2010 IEEE/RSJ international conference on (pp. 1242–1247). IEEE.Google Scholar
  8. Cohen, B. J., Subramania, G., Chitta, S., & Likhachev, M. (2011). Planning for manipulation with adaptive motion primitives. In Proceedings of the IEEE international conference on robot and automation (pp. 5478–5485). IEEE.Google Scholar
  9. Crandall, J. W., & Goodrich, M. A. (2002). Characterizing efficiency of human robot interaction: A case study of shared-control teleoperation. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, Lausanne, Switzerland, pp. 1–6.Google Scholar
  10. Delson, N., & West H. (1994). Robot programming by human demonstration: The use of human inconsistency in improving 3D robot trajectories. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems. (pp. 1248–1255).Google Scholar
  11. Demeester, E., Hüntemann, A., Vanhooydonck, D., Vanacker, G., Brussel, H. V., & Nuttin, M. (2008). User-adapted plan recognition and user-adapted shared control: A bayesian approach to semi-autonomous wheelchair driving. In Autonomous robots (pp. 193–211).Google Scholar
  12. Demeester, E., Hüntemann, A., Poorten, E. V., & Schutter, J. D. (2012a). ML, MAP and greedy POMDP shared control: Comparison of wheelchair navigation assistance for switch interfaces. In Proceedings of the international symposium on robotics, Taipei, Taiwan, pp. 1106–1111.Google Scholar
  13. Demeester, E., Poorten, E. V., Hüntemann, A., De Schutter, J., Lau, B., Kuderer, M., et al. (2012b). Robotic adaptation to humans adapting to robots. In 1st International Conference on Systems and Computer Science (ICSCS 2012).Google Scholar
  14. Derry, M., & Argall, B. (2014). A probabilistic representation of user intent for assistive robots. In International conference on intelligent robots and systems workshop on rehabilitation and assistive robotics, Chicago, IL, USA.Google Scholar
  15. Dragan, A. D., & Srinivasa, S. S. (2013). A policy-blending formalism for shared control. The International Journal of Robotics Research, 32(7), 790–805.CrossRefGoogle Scholar
  16. Gao, M., Oberl, J., Schamm, T., & Marius, J. Z. (2014). Contextual task-aware shared autonomy for assistive mobile robot teleoperation. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, Chicago, IL, USA, pp. 3311–3318.Google Scholar
  17. Glinton, R., Owens, S., Giampapa, J., Sycara, K., Lewis, M., & Grindle, C. (2005). Intent inference using a potential field model of environmental influences intent inference data flow. In 2005 7th international conference on information fusion. IEEE.Google Scholar
  18. Gnatzig, S., Schuller, F., Lienkamp, M. (2012). Human–machine interaction as key technology for driverless driving: A trajectory-based shared autonomy control approach. In Proceedings of the IEEE international symposium on robot and human interactive communication (pp. 913–918).Google Scholar
  19. Goil, A., Derry, M., & Argall, B. D. (2013). Using machine learning to blend human and robot controls for assisted wheelchair navigation. In: IEEE international conference on rehabilitation robotics, Seattle, WA, USA.Google Scholar
  20. Goodrich, M. A., Boer, E. R., Crandall, J. W., Ricks, R. W., & Quigley, M. L. (2004). Behavioral entropy in human–robot interaction. Technical reports on Brigham Young University.Google Scholar
  21. Grizzle, J. W., Abba, G., & Plestan, F. (2001). Asymptotically stable walking for biped robots: Analysis via systems with impulse effects. IEEE Transactions on Automatic Control, 46(1), 51–64.MathSciNetCrossRefzbMATHGoogle Scholar
  22. Grizzle, J. W., Chevallereau, C., Ames, A. D., & Sinnet, R. W. (2010). 3D Bipedal robotic walking: Models, feedback control, and open problems. IFAC Proceedings Volumes, 43(14), 505–532.CrossRefzbMATHGoogle Scholar
  23. Hauser, K. (2013). Recognition, prediction, and planning for assisted teleoperation of freeform tasks. Autonomous Robots, 35(4), 241–254.CrossRefGoogle Scholar
  24. Hauser, K., Bretl, T., Harada, K., & Latombe, J. C. (2008). Using motion primitives in probabilistic sample-based planning for humanoid robots. In Algorithmic foundation of robotics VII (pp. 507–522).Google Scholar
  25. Hereid, A., Cousineau, E. A., Hubicki, C. M., & Ames, A. D. (2016). 3D Dynamic walking with underactuated humanoid robots: A direct collocation framework for optimizing hybrid zero dynamics. In IEEE international conference on intelligent robotics and automation.Google Scholar
  26. Huntemann, A., Demeester, E., Poorten, E. V., & Brussel, H. V. (2013). Probabilistic approach to recognize local navigation plans by fusing past driving information with a personalized user model. In Proceedings of the IEEE International Conference on Intelligent Robotics and Automation, Karlsruhe, Germany, pp. 4376–4383.Google Scholar
  27. Jain, S., & Argall, B. (2016). An approach for online user customization of shared autonomy for intelligent assistive devices. In Proceedings of the IEEE international conference on intelligent robotics and automation.Google Scholar
  28. Javdani, S., Srinivasa, S. S., & Bagnell, J. A. (2015). Shared autonomy via hindsight optimization. In Proceedings of robotics: Science and system, Rome, Italy.Google Scholar
  29. Javdani, S., Bagnell, J. A., & Srinivasa, S. S. (2016). Minimizing user cost for shared autonomy. In 2016 11th ACM/IEEE international conference on human–robot interaction (HRI) (pp. 621–622).Google Scholar
  30. Jimenez-Fabian, R., & Verlinden, O. (2012). Review of control algorithms for robotic ankle systems in lower-limb orthoses, prostheses, and exoskeletons. Medical Engineering & Physics, 34(4), 397–408.CrossRefGoogle Scholar
  31. Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998). Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101(1–2), 99–134.MathSciNetCrossRefzbMATHGoogle Scholar
  32. Kawamoto, H., Kanbe, S., & Sankai, Y. (2003a). Power assist method for HAL-3 estimating operator’s intention based on motion information. In Robot and human interactive communication, 2003. Proceedings. ROMAN 2003. The 12th IEEE international workshop on (pp. 67–72). IEEE.Google Scholar
  33. Kawamoto, H., Lee, S., Kanbe, S., & Sankai, Y. (2003b). Power assist method for HAL-3 using EMG-based feedback controller. In Systems, man and cybernetics, 2003. IEEE international conference on (Vol. 2, pp. 1648–1653). IEEE.Google Scholar
  34. Kofman, J., Wu, X., Luu, T. J., & Verma, S. (2005). Teleoperation of a robot manipulator using a vision-based human–robot interface. IEEE Transactions on Industrial Electronics, 52(5), 1206–1219.CrossRefGoogle Scholar
  35. Kretzschmar, H., Kuderer, M., & Burgard, W. (2014). Learning to predict trajectories of cooperatively navigating agents. In Proceedings of the IEEE international conference on robotics and automation.Google Scholar
  36. Kulic, D., & Croft, E. A. (2003). Estimating intent for human–robot interaction. In IEEE international conference on advanced robotics.Google Scholar
  37. Loeb, G. E. (2012). Optimal isn’t good enough. Biological Cybernetics, 106(11), 757–765.CrossRefGoogle Scholar
  38. MacAdam, C. C. (2003). Understanding and modeling the human driver. Vehicle System Dynamics, 40(1–3), 101–134.CrossRefGoogle Scholar
  39. McLachlan, S., Arblaster, J., Liu, D., Miro, J. V., & Chenoweth, L. (2005). A multi-stage shared control method for an intelligent mobility assistant. In IEEE international conference on rehabilitation robotics, Chicago, IL, USA.Google Scholar
  40. Medina, J. R., Lorenz, T., & Hirche, S. (2015). Synthesizing anticipatory haptic assistance considering human behavior uncertainty. IEEE Transactions on Robotics, 31(1), 180–190.CrossRefGoogle Scholar
  41. Mellinger, D., & Kumar, V. (2011). Minimum snap trajectory generation and control for quadrotors. In Proceedings of the IEEE international conference on robotics and automation (pp. 2520–2525). IEEE.Google Scholar
  42. Milliken, L., & Hollinger, G. A. (2016). Modeling user expertise for choosing levels of shared autonomy. In Proceedings of robotics: Science and system workshop on planning for human–robot interaction, Ann Arbor, MI, USA.Google Scholar
  43. Mombaur, K., Jp, Laumond, & Yoshida, E. (2010). An optimal control-based formulation to determine natural locomotor paths for humanoid robots. Advanced Robotics, 24, 515–535.CrossRefGoogle Scholar
  44. Motahar, M. S., Veer, S., & Poulakakis, I. (2016). Composing limit cycles for motion planning of 3D bipedal walkers. In IEEE conference on decision and control.Google Scholar
  45. Muelling , K., Venkatraman, A., Valois , JS., Downey , JE., Weiss , J., Javdani , S., Hebert , M., Schwartz , AB., Collinger, JL., & Bagnell , JA . (2015) . Autonomy infused teleoperation with application to BCI manipulation. In Proceedings of robotics: Science and systems, Ann Arbor, MI, USA.Google Scholar
  46. Nakayama, O., Futami, T., & Nakamura, T. (1999). SAE Technical development of a steering entropy method for evaluating driver workload. Technical Report (p. 724).Google Scholar
  47. Nelson, E. A., & Michael, N. (2015). Environment model adaptation for autonomous exploration. Master’s Thesis, The Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA.Google Scholar
  48. Pivtoraiko, M., & Kelly, A. (2011). Kinodynamic motion planning with state lattice motion primitives. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 2172–2179). IEEE.Google Scholar
  49. Ralston, H. J. (1958). Energy-speed relation and optimal speed during level walking. European Journal of Applied Physiology and Occupational Physiology, 17(4), 277–283.CrossRefGoogle Scholar
  50. Sa, I., & Corke, P. (2014). Vertical infrastructure inspection using a quadcopter and shared autonomy control. In Field and service robotics (pp. 219–232). Springer.Google Scholar
  51. Sheridan, T. B., & Parasuraman, R. (2005). Human-automation interaction. Reviews of Human Factors and Ergonomics, 1(1), 89–129.CrossRefGoogle Scholar
  52. Shih, C. L., Grizzle, J., & Chevallereau, C. (2012). From stable walking to steering of a 3d bipedal robot with passive point feet. Robotica, 30(07), 1119–1130.CrossRefGoogle Scholar
  53. Vanhooydonck, D., Demeester, E., Hüntemann, A., Philips, J., Vanacker, G., Brussel, H. V., et al. (2010). Adaptable navigational assistance for intelligent wheelchairs by means of an implicit personalized user model. Robotics and Autonomous Systems, 58(8), 963–977.CrossRefGoogle Scholar
  54. Varol, H. A., & Goldfarb, M. (2007). Real-time intent recognition for a powered knee and ankle transfemoral prosthesis. In Rehabilitation robotics, 2007. ICORR 2007. IEEE 10th international conference on (pp. 16–23). IEEE.Google Scholar
  55. Varol, H. A., Sup, F., & Goldfarb, M. (2010). Multiclass real-time intent recognition of a powered lower limb prosthesis. IEEE Transactions on Biomedical Engineering, 57(3), 542–551.CrossRefGoogle Scholar
  56. Vijayakumar, S., D’Souza, A., & Schaal, S. (2005). Incremental online learning in high dimensions. Neural Computation, 17(12), 2602–2634.MathSciNetCrossRefGoogle Scholar
  57. Wang, Z., Mülling, K., Deisenroth, M. P., Amor, H. B., Vogt, D., Schölkopf, B., et al. (2013). Probabilistic movement modeling for intention inference in human–robot interaction. The International Journal of Robotics Research, 32(7), 841–858.CrossRefGoogle Scholar
  58. Wasson, G., Sheth, P., Huang, C., & Ledoux, A. (2004). A physics-based model for predicting user intent in shared-control pedestrian mobility aids. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems, Sendai, Japan, pp. 1914–1919.Google Scholar
  59. Westervelt, E. R., Grizzle, J. W., Chevallereau, C., Choi, J. H., & Morris, B. (2007). Feedback control of dynamic bipedal robot locomotion (Vol. 28). Boca Raton: CRC Press.Google Scholar
  60. Yang, X., Sreenath, K., Michael, N. (2017). A framework for efficient teleoperation via online adaptation. In International conference on robotics and automation (ICRA), Singapore.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.The Robotics InstituteCarnegie Mellon UniversityPittsburghUSA
  2. 2.Department of Mechanical EngineeringUniversity of CaliforniaBerkeleyUSA

Personalised recommendations