Abstract
Robust motion control algorithms are fundamental to the successful, autonomous operation of mobile robots. Motion control is known to be a difficult problem, and is often dictated by a policy, or state-action mapping. In this chapter, we present an approach for the refinement of mobile robot motion control policies, that incorporates corrective feedback from a human teacher. The target application domain of this work is the low-level motion control of a mobile robot. Within such domains, the rapid sampling rate and continuous action space of policies are both key challenges to providing policy corrections. To address these challenges, we contribute advice-operators as a corrective feedback form suitable for providing continuous-valued corrections, and Focused Feedback For Mobile Robot Policies (F3MRP) as a framework suitable for providing feedback on policies sampled at a high frequency. Under our approach, policies refined through teacher feedback are initially derived using Learning from Demonstration (LfD) techniques, which generalize a policy from example task executions by a teacher. We apply our techniques within the Advice-Operator Policy Improvement (A-OPI) algorithm, validated on a Segway RMP robot within a motion control domain. A-OPI refines LfD policies by correcting policy performance via advice-operators and F3MRP. Within our validation domain, policy performance is found to improve with corrective teacher feedback, and moreover to be similar or superior to that of policies provided with more teacher demonstrations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Abbeel, P., Coates, A., Quigley, M., Ng, A.Y.: An application of reinforcement learning to aerobatic helicopter flight. In: Proceedings of Advances in Neural Information Proccessing, NIPS 2007 (2007)
Abbeel, P., Ng, A.Y.: Exploration and apprenticeship learning in reinforcement learning. In: Proceedings of the 22nd International Conference on Machine Learning, ICML 2005 (2005)
Argall, B., Browning, B., Veloso, M.: Learning robot motion control with demonstration and advice-operators. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2008 (2008)
Argall, B., Chernova, S., Browning, B., Veloso, M.: A survey of robot learning from demonstration. Robotics and Autonomous Systems 57(5), 469–483 (2009)
Argall, B., Gu, Y., Browning, B., Veloso, M.: The first segway soccer experience: Towards peer-to-peer human-robot teams. In: First Annual Conference on Human-Robot Interactions, HRI 2006 (2006)
Argall, B.D.: Learning Mobile Robot Motion Control from Demonstration and Corrective Feedback. PhD thesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, Technical report CMU-RI-TR-09-13 (2009)
Atkeson, C.G., Moore, A.W., Schaal, S.: Locally weighted learning. Artificial Intelligence Review 11, 11–73 (1997)
Atkeson, C.G., Schaal, S.: Robot learning from demonstration. In: Proceedings of the Fourteenth International Conference on Machine Learning, ICML 1997 (1997)
Bagnell, J.A., Schneider, J.G.: Autonomous helicopter control using reinforcement learning policy search methods. In: Proceedings of the IEEE International Conference on Robotics and Automation, ICRA 2001 (2001)
Bentivegna, D.C.: Learning from Observation Using Primitives. PhD thesis, College of Computing, Georgia Institute of Technology, Atlanta, GA (2004)
Billard, A., Callinon, S., Dillmann, R., Schaal, S.: Robot programming by demonstration. In: Siciliano, B., Khatib, O. (eds.) Handbook of Robotics, ch. 59. Springer, New York (2008)
Breazeal, C., Scassellati, B.: Robots that imitate humans. Trends in Cognitive Sciences 6(11), 481–487 (2002)
Browning, B., Xu, L., Veloso, M.: Skill acquisition and use for a dynamically-balancing soccer robot. In: Proceedings of 19th National Conference on Artificial Intelligence, AAAI 2004 (2004)
Calinon, S., Billard, A.: Incremental learning of gestures by imitation in a humanoid robot. In: Proceedings of the 2nd ACM/IEEE International Conference on Human-Robot Interactions, HRI 2007 (2007)
Chernova, S., Veloso, M.: Learning equivalent action choices from demonstration. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2008 (2008)
Chernova, S., Veloso, M.: Multi-thresholded approach to demonstration selection for interactive robot learning. In: Proceedings of the 3rd ACM/IEEE International Conference on Human-Robot Interaction, HRI 2008 (2008)
Dautenhahn, K., Nehaniv, C.L. (eds.): Imitation in animals and artifacts. MIT Press, Cambridge (2002)
Demiris, Y., Hayes, G.: Imitation as a dual-route process featuring predictive and learning components: a biologically-plausible computational model. In: Dautenhahn and Nehaniv [17], ch. 13
Grollman, D.H., Jenkins, O.C.: Dogged learning for robots. In: Proceedings of the IEEE International Conference on Robotics and Automation, ICRA 2007 (2007)
Ijspeert, A.J., Nakanishi, J., Schaal, S.: Learning rhythmic movements by demonstration using nonlinear oscillators. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2002 (2002)
Ijspeert, A.J., Nakanishi, J., Schaal, S.: Movement imitation with nonlinear dynamical systems in humanoid robots. In: Proceedings of the IEEE International Conference on Robotics and Automation, ICRA 2002 (2002)
Kolter, J.Z., Abbeel, P., Ng, A.Y.: Hierarchical apprenticeship learning with application to quadruped locomotion. In: Proceedings of Advances in Neural Information Proccessing, NIPS 2008 (2008)
Kulic, D., Nakamura, Y.: Incremental learning of full body motion primitives. In: Peters, Sigaud [31], pp. 381–404
Lopes, M., Melo, F., Montesano, L., Santos-Victor, J.: Abstraction levels for robotic imitation: Overview and computational approaches. In: Peters, Sigaud [31], pp. 313–355
Matarić, M.J.: Sensory-motor primitives as a basis for learning by imitation: Linking perception to action and biology to robotics. In: Dautenhahn, Nehaniv [17], ch. 15
Nehaniv, C.L., Dautenhahn, K.: The correspondence problem. In: Dautenhahn, Nehaniv [17], ch. 2
Ng, A.Y., Coates, A., Diel, M., Ganapathi, V., Schulte, J., Tse, B., Berger, E., Liang, E.: Inverted autonomous helicopter flight via reinforcement learning. In: International Symposium on Experimental Robotics (2004)
Nguyen, H.G., Morrell, J., Mullens, K., Burmeister, A., Miles, S., Thomas, K., Gage, D.W.: Segway robotic mobility platform. In: SPIE Mobile Robots XVII (2004)
Nicolescu, M.N., Matarić, M.J.: Methods for robot task learning: Demonstrations, generalization and practice. In: Proceedings of the Second International Joint Conference on Autonomous Agents and Multi-Agent Systems, AAMAS 2003 (2003)
Peters, J., Schaal, S.: Natural actor-critic. Neurocomputing 71(7-9), 1180–1190 (2008)
Peters, J., Sigaud, O. (eds.): From Motor Learning to Interaction Learning in Robots. SCI, vol. 264. Springer, Heidelberg (2010)
Ratliff, N., Bradley, D., Bagnell, J.A., Chestnutt, J.: Boosting structured prediction for imitation learning. In: Proceedings of Advances in Neural Information Processing Systems, NIPS 2007 (2007)
Rybski, P.E., Yoon, K., Stolarz, J., Veloso, M.M.: Interactive robot task training through dialog and demonstration. In: Proceedings of the 2nd ACM/IEEE International Conference on Human-Robot Interactions, HRI 2007 (2007)
Smart, W.D.: Making Reinforcement Learning Work on Real Robots. PhD thesis, Department of Computer Science, Brown University, Providence, RI (2002)
Stefani, R., Shahian, B., Savant, C., Hostetter, G.: Design of Feedback Control Systems. Oxford University Press, Oxford (2001)
Stolle, M., Atkeson, C.G.: Knowledge transfer using local features. In: Proceedings of IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, ADPRL 2007 (2007)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Argall, B.D., Browning, B., Veloso, M.M. (2010). Mobile Robot Motion Control from Demonstration and Corrective Feedback. In: Sigaud, O., Peters, J. (eds) From Motor Learning to Interaction Learning in Robots. Studies in Computational Intelligence, vol 264. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-05181-4_18
Download citation
DOI: https://doi.org/10.1007/978-3-642-05181-4_18
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-05180-7
Online ISBN: 978-3-642-05181-4
eBook Packages: EngineeringEngineering (R0)