Advertisement

Autonomous Robots

, Volume 42, Issue 3, pp 529–551 | Cite as

Using probabilistic movement primitives in robotics

  • Alexandros Paraschos
  • Christian Daniel
  • Jan Peters
  • Gerhard Neumann
Article
  • 987 Downloads

Abstract

Movement Primitives are a well-established paradigm for modular movement representation and generation. They provide a data-driven representation of movements and support generalization to novel situations, temporal modulation, sequencing of primitives and controllers for executing the primitive on physical systems. However, while many MP frameworks exhibit some of these properties, there is a need for a unified framework that implements all of them in a principled way. In this paper, we show that this goal can be achieved by using a probabilistic representation. Our approach models trajectory distributions learned from stochastic movements. Probabilistic operations, such as conditioning can be used to achieve generalization to novel situations or to combine and blend movements in a principled way. We derive a stochastic feedback controller that reproduces the encoded variability of the movement and the coupling of the degrees of freedom of the robot. We evaluate and compare our approach on several simulated and real robot scenarios.

Keywords

Imitation learning Movement primitives Trajectory representation Control Robotics 

Notes

Acknowledgements

The research leading to these results has received funding from the European Community’s Framework Programme CoDyCo (FP7-ICT-2011-9 Grant No. 600716), CompLACS (FP7-ICT-2009-6 Grant No. 270327), GeRT (FP7-ICT-2009-4 Grant No. 248273), and ERC StG SKILLS4ROBOTS.

References

  1. Bruno, D., Calinon, S., Malekzadeh, M. S., & Caldwell, D. G. (2015). Learning the stiffness of a continuous soft manipulator from multiple demonstrations. In Intelligent robotics and applications (pp. 185–195).Google Scholar
  2. Buchli, J., Stulp, F., Theodorou, E., & Schaal, S. (2011). Learning variable impedance control. International Journal of Robotics Research, 30(7), 820–833.CrossRefGoogle Scholar
  3. Calinon, S. (2016). A tutorial on task-parameterized movement learning and retrieval. Intelligent Service Robotics, 9(1), 1–29.CrossRefGoogle Scholar
  4. Calinon, S., D’Halluin, F., Sauser, E. L., Caldwell, D. G., & Billard, A. G. (2010). Learning and reproduction of gestures by imitation. IEEE Robotics and Automation Magazine, 17, 44–54.CrossRefGoogle Scholar
  5. Calinon, S., Sardellitti, I., & Caldwell, D. G. (2010b). Learning-based control strategy for safe human–robot interaction exploiting task and robot redundancies. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 249–254).Google Scholar
  6. Daniel, C., Neumann, G., & Peters, J. (2012). Learning concurrent motor skills in versatile solution spaces. In IEEE/RSJ international conference on intelligent robots and systems (IROS), (pp. 3591–3597).Google Scholar
  7. da Silva, B., Konidaris, G., & Barto, A. (2012). Learning parameterized skills. In International conference on machine learning (pp. 1679–1686).Google Scholar
  8. dAvella, A., & Bizzi, E. (2005). Shared and specific muscle synergies in natural motor behaviors. Proceedings of the National Academy of Sciences (PNAS), 102(3), 3076–3081.CrossRefGoogle Scholar
  9. Degallier, S., Righetti, L., Gay, S., & Ijspeert, A. (2011). Toward simple control for complex, autonomous robotic applications: Combining discrete and rhythmic motor primitives. Autonomous Robots, 31, 155–181.CrossRefGoogle Scholar
  10. Dominici, N., Ivanenko, Y. P., Cappellini, G., dAvella, A., Mondì, V., Cicchese, M., et al. (2011). Locomotor primitives in newborn babies and their development. Science, 334(6058), 997–999.CrossRefGoogle Scholar
  11. Ernesti, J., Righetti, L., Do, M., Asfour, T., & Schaal, S. (2012). Encoding of periodic and their transient motions by a single dynamic movement primitive. In IEEE-RAS international conference on humanoid robots (humanoids) (pp. 57–64).Google Scholar
  12. Ewerton, M., Maeda, G., Peters, J., & Neumann, G. (2015). Learning motor skills from partially observed movements executed at different speeds. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 456–463).Google Scholar
  13. Forte, D., Gams, A., Morimoto, J., & Ude, A. (2012). On-line motion synthesis and adaptation using a trajectory database. Robotics and Autonomous Systems, 60, 1327–1339.CrossRefGoogle Scholar
  14. Gams, A., Nemec, B., Ijspeert, A. J., & Ude, A. (2014). Coupling movement primitives: Interaction with the environment and bimanual tasks. IEEE Transactions on Robotics, 30(4), 816–830.CrossRefGoogle Scholar
  15. Higham, N. J. (1988). Computing a nearest symmetric positive semidefinite matrix. Linear Algebra and its Applications, 103, 103–118.MathSciNetCrossRefzbMATHGoogle Scholar
  16. Ijspeert, A. J. (2008). Central pattern generators for locomotion control in animals and robots: A review. Neural Networks, 21(4), 642–653.CrossRefGoogle Scholar
  17. Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., & Schaal, S. (2013). Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25(2), 328–373.MathSciNetCrossRefzbMATHGoogle Scholar
  18. Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2003). Learning attractor landscapes for learning motor primitives. In Advances in neural information processing systems (NIPS) (pp. 1547–1554).Google Scholar
  19. Khansari-Zadeh, S. M., & Billard, A. (2011). Learning stable nonlinear dynamical systems with Gaussian mixture models. IEEE Transactions on Robotics, 27(5), 943–957.CrossRefGoogle Scholar
  20. Khansari-Zadeh, S. M., Kronander, K., & Billard, A. (2014). Modeling robot discrete movements with state-varying stiffness and damping: A framework for integrated motion generation and impedance control. In Robotics science and systems (R:SS).Google Scholar
  21. Klug, S., Lens, T., von Stryk, O., Möhl, B., & Karguth, A. (2008). Biologically inspired robot manipulator for new applications in automation engineering. In Proceedings of robotik.Google Scholar
  22. Kober, J., Muelling, K., Kroemer, O., Lampert, C. H., Scholkopf, B., & Peters, J. (2010). Movement templates for learning of hitting and batting. In International conference on robotics and automation (ICRA) (pp. 853–858).Google Scholar
  23. Konidaris, G., Kuindersma, S., Grupen, R., & Barto, A. (2012). Robot learning from demonstration by constructing skill trees. International Journal of Robotics Research (IJRR), 31(3), 360–375.CrossRefGoogle Scholar
  24. Kormushev, P., Calinon, S., & Caldwell, D. G. (2010). Robot motor skill coordination with EM-based reinforcement learning. In International conference on intelligent robots and systems (IROS) (pp. 3232–3237).Google Scholar
  25. Kulvicius, T., Ning, K., Tamosiunaite, M., & Worgotter, F. (2012). Joining movement sequences: Modified dynamic movement primitives for robotics applications exemplified on handwriting. IEEE Transactions on Robotics, 28(1), 145–157.CrossRefGoogle Scholar
  26. Lazaric, A., & Ghavamzadeh, M. (2010). Bayesian multi-task reinforcement learning. In International conference on machine learning (ICML) (pp. 599–606).Google Scholar
  27. Li, W., & Todorov, E. (2010). Iterative linear quadratic regulator design for nonlinear biological movement systems. In International conference on informatics in control, automation and robotics (ICINCO) (pp. 222–229).Google Scholar
  28. Maeda, G., Ewerton, M., Lioutikov, R., Amor, H., Peters, J., & Neumann, G. (2014). Learning interaction for collaborative tasks with probabilistic movement primitives. In International conference on humanoid robots (Humanoids) (pp. 527–534).Google Scholar
  29. Matsubara, T., Hyon, S. H., & Morimoto, J. (2011). Learning parametric dynamic movement primitives from multiple demonstrations. Neural Networks, 24(5), 493–500.CrossRefGoogle Scholar
  30. Moro, F. L., Tsagarakis, N. G., & Caldwell, D. G. (2012). On the kinematic motion primitives (kMPs)—Theory and application. Frontiers in Neurorobotics, 6(10), 1–18.Google Scholar
  31. Muelling, K., Kober, J., & Peters, J. (2011). A biomimetic approach to robot table tennis. Adaptive Behavior Journal, 19(5), 359–376.CrossRefGoogle Scholar
  32. Mülling, K., Kober, J., Kroemer, O., & Peters, J. (2013). Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3), 263–279.CrossRefGoogle Scholar
  33. Nakanishi, J., Morimoto, J., Endo, G., Cheng, G., Schaal, S., & Kawato, M. (2004). Learning from demonstration and adaptation of biped locomotion. Robotics and Autonomous Systems, 47, 79–91.CrossRefGoogle Scholar
  34. Neumann, G., Daniel, C., Paraschos, A., Kupcsik, A., & Peters, J. (2014). Learning modular policies for robotics. Frontiers in Computational Neuroscience, 8(62), 1.Google Scholar
  35. Neumann, G., Maass, W., & Peters, J. (2009). Learning complex motions by sequencing simpler motion templates. In International conference on machine learning (ICML) (pp. 753–760)Google Scholar
  36. OHagan, A., & Forster, J. (2004). Kendalls advanced theory of statistics: Bayesian inference (2nd ed.). Arnold, New York. Technical report, ISBN 0-340-80752-0.Google Scholar
  37. Paraschos, A., Daniel, C., Peters, J., & Neumann, G. (2013a). Probabilistic movement primitives. In Advances in neural information processing systems (NIPS) (pp. 2616–2624).Google Scholar
  38. Paraschos, A., Neumann, G., & Peters, J. (2013b). A probabilistic approach to robot trajectory generation. In International conference on humanoid robots (humanoids) (pp. 477–483)Google Scholar
  39. Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009). Learning and generalization of motor skills by learning from demonstration. In International conference on robotics and automation (ICRA) (pp. 763–768)Google Scholar
  40. Pastor, P., Righetti, L., Kalakrishnan, M., & Schaal, S. (2011). Online movement adaptation based on previous sensor experiences. In International conference on intelligent robots and systems (IROS) (pp. 365–371)Google Scholar
  41. Peters, J., Mistry, M., Udwadia, F. E., Nakanishi, J., & Schaal, S. (2008). A unifying methodology for robot control with redundant DOFs. Autonomous Robots, 24(1), 1–12.CrossRefGoogle Scholar
  42. Righetti, L., & Ijspeert, A. J. (2006). Programmable central pattern generators: An application to biped locomotion control. In International conference on robotics and automation, (ICRA) (pp. 1585–1590).Google Scholar
  43. Rozo, L., Calinon, S., Caldwell, D., Jiménez, P., & Torras, C. (2013). Learning collaborative impedance-based robot behaviors. In AAAI conference on artificial intelligence (pp. 1422–1428).Google Scholar
  44. Rückert, E. A., Neumann, G., Toussaint, M., & Maass, W. (2012). Learned graphical models for probabilistic planning provide a new class of movement primitives. Frontiers in Computational Neuroscience, 6(97), 1.Google Scholar
  45. Rueckert, E., Mundo, J., Paraschos, A., Peters, J., & Neumann, G. (2015). Extracting low-dimensional control variables for movement primitives. In International conference on robotics and automation (ICRA) (pp. 1511–1518).Google Scholar
  46. Schaal, S., Mohajerian, P., & Ijspeert, A. (2007). Dynamics systems vs. optimal control—A unifying view. Computational Neuroscience: Theoretical Insights into Brain Function, 165, 425–445.Google Scholar
  47. Schaal, S., Peters, J., Nakanishi, J., & Ijspeert, A. (2005). Learning movement primitives. In International symposium on robotics research (pp. 561–572).Google Scholar
  48. Stark, H., & Woods, J. (2001). Probability and random processes with applications to signal processing (3rd ed.). Upper Saddle River: Prentice-Hall.Google Scholar
  49. Stengel, R. F. (2012). Optimal control and estimation. North Chelmsford, MA: Courier Corporation.zbMATHGoogle Scholar
  50. Todorov, E. (2008). General duality between optimal control and estimation. Conference on Decision and Control, 5, 4286–4292.Google Scholar
  51. Todorov, E., & Jordan, M. (2002). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5, 1226–1235.CrossRefGoogle Scholar
  52. Toussaint, M. (2009). Robot trajectory optimization using approximate inference. In International conference on machine learning (ICML) (pp. 1049–1056).Google Scholar
  53. Ude, A., Gams, A., Asfour, T., & Morimoto, J. (2010). Task-specific generalization of discrete and periodic dynamic movement primitives. Transactions in Robotics, 5, 800–815.CrossRefGoogle Scholar
  54. Williams B., Toussaint, M., & Storkey, A. (2007). Modelling motion primitives and their timing in biologically executed movements. In Advances in neural information processing systems (NIPS) (pp. 1609–1616).Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2017

Authors and Affiliations

  • Alexandros Paraschos
    • 1
  • Christian Daniel
    • 2
  • Jan Peters
    • 1
    • 3
  • Gerhard Neumann
    • 4
  1. 1.Technische Universität DarmstadtDarmstadtGermany
  2. 2.Bosch Center for Artificial IntelligenceRenningenGermany
  3. 3.Max-Planck-Institut für Intelligente SystemeTübingenGermany
  4. 4.Computational Learning for Autonomous SystemsSchool of Computer Science, University of LincolnLincolnUK

Personalised recommendations