Abstract
Research on robot Learning from Demonstration has seen significant growth in recent years, but the field has had only limited evaluation of existing algorithms with respect to algorithm usability by naïve users. In this article we present findings from a user study in which we asked non-expert users to use and evaluate three different robot Learning from Demonstration algorithms. The three algorithms selected—Behavior Networks, Interactive Reinforcement Learning, and Confidence Based Autonomy—utilize distinctly different policy learning and demonstration approaches, enabling us to examine a broad spectrum of the field. Participants in the study were asked to teach a simple task to a small humanoid robot in a real world domain. they controlled the robot directly (teleoperation and guidance) instead of providing retroactive feedback for past actions (reward and correction). We present our quantitative findings about: (a) the correlation between the number of user-agent interactions and the performance of the agent and (b) the correlation between agent’s final performance and its perceived accuracy by the participant. Comparatively, the strongest correlation was found in CBA data. We also discuss the possible reasons of our qualitative results. Additionally, we identify common trends and misconceptions that arise when non-experts are asked to use these algorithms, with the aim of informing future Learning from Demonstration approaches. Our results show that users achieved better performance in teaching the task using the CBA algorithm, whereas the Interactive Reinforcement Learning algorithm modeled user behavior most accurately.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abbeel P, Ng AY (2004) Apprenticeship learning via inverse reinforcement learning. In: Proceedings of the 21st international conference on machine learning (ICML ’04), Baniff, Canada
Akgun B, Cakmak M, Yoo JW, Thomaz AL (2012) Trajectories and keyframes for kinesthetic teaching: a human-robot interaction perspective. In: Proceedings of the international conference on human-robot interaction (HRI)
Aleotti J, Caselli S (2005) Trajectory clustering and stochastic approximation for robot programming by demonstration. In: 2005 IEEE/RSJ international conference on intelligent robots and systems (IROS 2005)
Argall BD, Chernova S, Veloso M, Browning B (2009) A survey of robot learning from demonstration. Robot Auton Syst 57:469–483
Atkeson CG, Schaal S (1997) Robot learning from demonstration. In: Fisher DH Jr (ed) Machine learning: proceedings of the fourteenth international conference (ICML ’97), San Francisco, CA, pp 12–20
Bentivegna DC, Ude A, Atkeson CG, Cheng G (2002) Humanoid robot learning and game playing using PC-based vision. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS ’02), Swiss Federal Institute of Technology Lausanne (EPFL), Switzerland, October
Billing E, Hellström T (2010) A formalism for learning from demonstration. Paladyn J Behav Robot 1:1–13. doi:10.2478/s13230-010-0001-5
Bradski G, Kaehler A (2008) Learning OpenCV. O’Reilly Media
Cakmak M, Chao C, Thomaz AL (2010) Designing interactions for robot active learners. IEEE Trans Auton Ment Dev 2(2):108–118
Calinon S, Billard A (2007) Incremental learning of gestures by imitation in a humanoid robot. In: Second annual conference on human-robot interactions (HRI ’07), Arlington, VA, March
Chella A, Dindo H, Infantino I (2006) A cognitive framework for imitation learning. Robot Auton Syst (Special issue on the social mechanisms of robot programming by demonstration) 54(5):403–408
Chernova S, Veloso M (2007) Confidence-based learning from demonstration using Gaussian mixture models. In: Proceedings of the international conference on autonomous agents and multiagent systems (AMMAS ’07)
Chernova S, Veloso M (2009) Interactive policy learning through confidence-based autonomy. J Artificial Intelligence Research 1–25
Grollman DH, Jenkins OC (2007) Dogged learning for robots. In: International conference on robotics and automation, Rome, Italy, April, pp 2483–2488
Ijspeert AJ, Nakanishi J, Schaal S (2002) Movement imitation with nonlinear dynamical systems in humanoid robots. In: Proceedings of the IEEE international conference on robotics and automation (ICRA ’02) (Received the ICRA ’02 best paper award)
Kim ES, Leyzberg D, Tsui KM, Scassellati B (2009) How people talk when teaching a robot. In: Proceedings of the international conference on human robot interaction (HRI ’09)
Bradley Knox W, Stone P (2010) Combining manual feedback with subsequent mdp reward signals for reinforcement learning. In: Proceedings of the international conference on autonomous agents and multiagent systems
Mataric MJ (2002) Sensory-motor primitives as a basis for learning by imitation: linking perception to action and biology to robotics. In: Dautenhahn K, Nehaniv C (eds) Imitation in animals and artifacts. MIT Press, Cambridge, pp 392–422
Nehmzow U, Akanyeti O, Weinrich C, Kyriacou T, Billings SA (2007) Robot programming by demonstration through system identification. In: IEEE/RSJ international conference on intelligent robots and systems (IROS 2007)
Nicolescu MN, Mataric MJ (2001) Learning and interacting in human-robot domains. IEEE Trans Syst Man Cybern, Part A, Syst Hum 31(5):419–430
Nicolescu MN, Matarić MJ (2002) A hierarchical architecture for behavior-based robots. In: Proceedings of the first international joint conference on autonomous agents and multiagent systems, Part 1 (AAMAS ’02), New York, NY, USA. ACM, New York, pp 227–233
Nicolescu MN, Mataric MJ (2003) Natural methods for robot task learning: instructive demonstrations, generalization and practice. In: Proceedings of the second international joint conference on autonomous agents and multiagent systems (AAMAS ’03), New York, NY, USA. ACM, New York, pp 241–248
Smart WD (2002) Making reinforcement learning work on real robots. PhD thesis, Department of Computer Science, Brown University, Providence, RI
Suay HB, Chernova S (2011) A comparison of two algorithms for robot learning from demonstration. In: IEEE international conference on systems, man, and cybernetics
Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge
Thomaz AL, Breazeal C (2007) Asymmetric interpretations of positive and negative human feedback for a social learning agent. In: Proceedings of the 16th IEEE international symposium on robot and human interactive communication (RO-MAN)
Thomaz AL, Breazeal C (2008) Experiments in socially guided exploration: lessons learned in building robots that learn with and without human teachers. Connect Sci 20:91–110
Thomaz AL, Breazeal C (2006) Reinforcement learning with human teachers: evidence of feedback and guidance with implications for learning performance. In: Proceedings of the 21st national conference on artificial intelligence (AAAI)
van Lent M, Laird JE (2001) Learning procedural knowledge through observation. In: Proceedings of the 1st international conference on knowledge capture (K-CAP ’01), New York, NY, USA. ACM, New York, pp 179–186
Voyles RM, Khosla PK (2001) A multi-agent system for programming robots by human demonstration. Integr Comput-Aided Eng 8(1):59–67
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Suay, H.B., Toris, R. & Chernova, S. A Practical Comparison of Three Robot Learning from Demonstration Algorithm. Int J of Soc Robotics 4, 319–330 (2012). https://doi.org/10.1007/s12369-012-0158-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12369-012-0158-7