Autonomous Robots

, Volume 35, Issue 4, pp 241–254 | Cite as

Recognition, prediction, and planning for assisted teleoperation of freeform tasks

  • Kris Hauser


The approach of inferring user’s intended task and optimizing low-level robot motions has promise for making robot teleoperation interfaces more intuitive and responsive. But most existing methods assume a finite set of candidate tasks, which limits a robot’s functionality. This paper proposes the notion of freeform tasks that encode an infinite number of possible goals (e.g., desired target positions) within a finite set of types (e.g., reach, orient, pick up). It also presents two technical contributions to help make freeform UIs possible. First, an intent predictor estimates the user’s desired task, and accepts freeform tasks that include both discrete types and continuous parameters. Second, a cooperative motion planner continuously updates the robot’s trajectories to achieve the inferred tasks by repeatedly solving optimal control problems. The planner is designed to respond interactively to changes in the indicated task, avoid collisions in cluttered environments, handle time-varying objective functions, and achieve high-quality motions using a hybrid of numerical and sampling-based techniques. The system is applied to the problem of controlling a 6D robot manipulator using 2D mouse input in the context of two tasks: static target reaching and dynamic trajectory tracking. Simulations suggest that it enables the robot to reach intended targets faster and to track intended trajectories more closely than comparable techniques.


Human-robot interaction Intent recognition Motion planning Estimation Optimal control 


  1. Aarno, D., Ekvall, S., & Kragic, D. (2005). Adaptive virtual fixtures for machine-assisted teleoperation tasks. IEEE International Conference on Robotics and Automation (pp. 897–903).Google Scholar
  2. Aarno, D., & Kragic, D. (2008). Motion intention recognition in robot assisted applications. Robotics and Autonomous Systems, 56, 692–705.Google Scholar
  3. Anderson, S. J., Peters, S. C., Iagnemma, K. D., & Pilutti, T. E. (2009). A unified approach to semi-autonomous control of passenger vehicles in hazard avoidance scenarios. Proceedings of IEEE International Conference on Systems, Man and Cybernetics, (pp. 2032–2037) San Antonio, TX, USA.Google Scholar
  4. Asano, T., Sharlin, E., Kitamura, Y., Takashima, K. & Kishino, F. (2005). Predictive interaction using the delphian desktop. Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology (pp. 133-141).Google Scholar
  5. Betts, J. T. (1998). Survey of numerical methods for trajectory optimization. Journal of Guidance, Control, and Dynamics, 21(2), 193–207.zbMATHCrossRefGoogle Scholar
  6. Blom, H. A. P., & Bar-Shalom, Y. (1988). The interacting multiple model algorithm for systems with markovian switching coefficients. IEEE Transaction on Automatic Control, 33, 780–783.zbMATHCrossRefGoogle Scholar
  7. Bobrow, J., Martin, B., Sohl, G., Wang, E., Park, F., & Kim, J. (2001). Optimal robot motions for physical criteria. Journal of Robotic Systems, 18(12), 785–795.zbMATHCrossRefGoogle Scholar
  8. Burridge, R. R. & Hambuchen, K. A. (2009). Using prediction to enhance remote robot supervision across time delay. IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 5628–5634).Google Scholar
  9. Dragan, A., & Srinivasa, S. (2012). Formalizing assistive teleoperation. In Robotics: Science and Systems.Google Scholar
  10. Frazzoli, E., Dahleh, M. A., & Feron, E. (2002). Real-time motion planning for agile autonomous vehicles. AIAA Journal of Guidance, Control, and Dynamics, 25(1), 116–129.CrossRefGoogle Scholar
  11. Gauvain, J.-L., & Lee, C.-H. (1994). Maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains. IEEE Transactions on Speech and Audio Processing, 2(2), 291– 298.Google Scholar
  12. Goodrich, M. A. & Olsen, J. D. R. (2003) Seven principles of efficient interaction. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (pp. 3943–3948).Google Scholar
  13. Green, S. A., Billinghurst, M., Chen, X., & Chase, J. G. (2008). Human-robot collaboration: A literature review and augmented reality approach in design. International Journal of Advanced Robotic Systems, 5(1), 1–18.Google Scholar
  14. Hauser, K. (2011). On responsiveness, safety, and completeness in real-time motion planning. Autonomous Robots, 32(1), 35–48.CrossRefGoogle Scholar
  15. Hauser, K., & Ng-Thow-Hing, V. (2010). Fast smoothing of manipulator trajectories using optimal bounded-acceleration shortcuts. Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Anchorage, USAGoogle Scholar
  16. Karaman, S., & Frazzoli, E. (2010). Incremental sampling-based algorithms for optimal motion planning. In Robotics: Science and Systems (RSS), Zaragoza, Spain.Google Scholar
  17. Kim, C.-J. (1994). Dynamic linear models with markov-switching. Journal of Econometrics, 60, 1–22.MathSciNetzbMATHCrossRefGoogle Scholar
  18. Kragic, D., Marayong, P., Li, M., Okamura, A. M., & Hager, G. D. (2005). Human-machine collaborative systems for microsurgical applications. The International Journal of Robotics Research, 24(9), 731–741.CrossRefGoogle Scholar
  19. Lane, D., Peres, S., Sándor, A., & Napier, H. (2005). A process for anticipating and executing icon selection in graphical user interfaces. International Journal of Human-Computer Interaction, 19(2), 241–252.Google Scholar
  20. Lank, E., Cheng, Y., Ruiz, J. (2007). Endpoint prediction using motion kinematics. Proceedings of the SIGCHI Conference on Human Factors in Computing System, (pp. 637-646).Google Scholar
  21. LaValle, S. M. (2006). Planning algorithms. Cambridge, UK: Cambridge University Press.zbMATHCrossRefGoogle Scholar
  22. Leeper,A., Hsiao, K., Ciocarlie, M., Takayama, L., & Gossow D. (2012). Strategies for human-in-the-loop robotic grasping. Proceedings of Human-Robot Interaction (HRI), (pp. 1–8) Boston, MA.Google Scholar
  23. Petti, S., & Fraichard, T. (2005) Safe motion planning in dynamic environments. IEEE International Conference on Intelligent Robots and Systems (IROS), pp. (3726–3731).Google Scholar
  24. Schrempf, O., Albrecht, D. & Hanebeck, U. (2007). Tractable probabilistic models for intention recognition based on expert knowledge. Proceedings of the International Conference on Informatics in Control, Automation and Robotics, (pp 1429–1434). San Diego, CA, USA.Google Scholar
  25. Shen, J., Ibanez-Guzman, J., Ng, T. C., & Chew, B.-S. (2004). A collaborative-shared control system with safe obstacle avoidance capability. IEEE Conference on Robotics, Automation and Mechatronics, Vol. 1, (pp. 119–123).Google Scholar
  26. Yu, W., Alqasemi, R., Dubey, R., & Pernalete, N. (2005). Telemanipulation assistance based on motion intention recognition. IEEE International Conference on Robotics and Automation, (pp. 1121– 1126).Google Scholar
  27. Ziebart, B., Dey, A. K., Bagnell, J. A. (2012) Probabilistic pointing target prediction via inverse optimal control. Proceedings of the International Conference on Intelligent User Interfaces.Google Scholar
  28. Zollner, R., Rogalla, O., Dillmann, R., & Zollner, M. (2002). Understanding users intention: programming fine manipulation tasks by demonstration. International Conference of Intelligent Robots and Systems, Vol 2, (pp. 1114–1119).Google Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.School of Informatics and ComputingIndiana UniversityBloomingtonUSA

Personalised recommendations