Curriculum Learning for Motor Skills

  • Andrej Karpathy
  • Michiel van de Panne
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7310)


Humans and animals acquire their wide repertoire of motor skills through an incremental learning process, during which progressively more complex skills are acquired and subsequently integrated with prior abilities. Inspired by this general idea, we develop an approach for learning motor skills based on a two-level curriculum. At the high level, the curriculum specifies an order in which different skills should be learned. At the low level, the curriculum defines a process for learning within a skill. We develop a set of integrated motor skills for a planar articulated figure capable of doing parameterized hops, flips, rolls, and acrobatic sequences. The same curriculum can be applied to yield individualized motor skill sets for articulated figures of varying proportions.


Motor Skill Reinforcement Learning Task Parameter Learn Motor Skill Curriculum Learn 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Asada, M., Noda, S., Tawaratsumida, S., Hosoda, K.: Purposive behavior acquisition for a real robot by vision-based reinforcement learning. Machine Learning 23(2), 279–303 (1996)Google Scholar
  2. 2.
    Bengio, Y., Louradour, J., Collobert, R., Weston, J.: Curriculum learning. In: Proc. Intl. Conf. on Machine Learning, pp. 41–48. ACM (2009)Google Scholar
  3. 3.
    Boone, G.: Minimum-time control of the acrobot. In: IEEE Intl. Conf. on Robotics and Automation, pp. 3281–3287 (1997)Google Scholar
  4. 4.
    Hart, S.W.: The development of hierarchical knowledge in robot systems. PhD thesis, University of Massachusetts Amherst (2009)Google Scholar
  5. 5.
    Hauser, J., Murray, R.M.: Nonlinear controllers for non-integrable systems: the acrobot example. In: American Control Conf., pp. 669–671 (1990)Google Scholar
  6. 6.
    Konidaris, G., Barto, A.G.: Skill discovery in continuous reinforcement learning domains using skill chaining. In: Proc. NIPS, vol. 22, pp. 1015–1023 (2009)Google Scholar
  7. 7.
    Neumann, G., Maass, W., Peters, J.: Learning complex motions by sequencing simpler motion templates. In: Proceedings of the 26th Annual International Conference on Machine Learning, pp. 753–760. ACM (2009)Google Scholar
  8. 8.
    Ng, A.Y.: Shaping and policy search in reinforcement learning. PhD thesis, University of California, Berkeley (2003)Google Scholar
  9. 9.
    Pickett, M., Barto, A.G.: Policyblocks: An algorithm for creating useful macro-actions in reinforcement learning. In: Proc. ICML, pp. 506–513 (2002)Google Scholar
  10. 10.
    Ponsen, M., Taylor, M.E., Tuyls, K.: Abstraction and Generalization in Reinforcement Learning: A Summary and Framework. In: Taylor, M.E., Tuyls, K. (eds.) ALA 2009. LNCS, vol. 5924, pp. 1–32. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Schmidt, R.A., Lee, T.D.: Motor control and learning: A behavioral emphasis. Human Kinetics Publishers (2005)Google Scholar
  12. 12.
    Skinner, B.F.: Reinforcement today. American Psychologist 13(3), 94 (1958)CrossRefGoogle Scholar
  13. 13.
    Spong, M.W.: The swing up control problem for the acrobot. IEEE Control Systems 15(1), 49–55 (1995)CrossRefGoogle Scholar
  14. 14.
    Stout, A., Barto, A.G.: Competence progress intrinsic motivation. In: IEEE Intl. Conf. on Development and Learning, pp. 257–262. IEEE (2010)Google Scholar
  15. 15.
    Yin, K.K., Coros, S., Beaudoin, P., van de Panne, M.: Continuation methods for adapting simulated skills. In: ACM SIGGRAPH 2008 Papers, pp. 1–7. ACM (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Andrej Karpathy
    • 1
  • Michiel van de Panne
    • 1
  1. 1.University of British ColumbiaCanada

Personalised recommendations