Simulation for Control

  • KangKang YinEmail author
  • Libin Liu
  • Michiel van de Panne
Reference work entry


Computer simulation is a powerful tool for developing motion controllers in humanoid robotics and physics-based character animation. In this chapter, we first motivate simulation-based methods for the control of physical robots and digital avatars and discuss the major differences between developing controls in simulation and on real robots. We then detail a series of model-free algorithms developed by our group over the years. These algorithms are based on sampling time-indexed control actions for short-duration motion fragments, in order to track desired reference motion trajectories. We term the family of algorithms SAMCON (sampling-based motion control). The basic SAMCON and the improved SAMCON methods reconstruct open-loop controls; the reduced-order feedback policy learning and the guided SAMCON then build robust linear feedback strategies around the open-loop controls. We further introduce a general framework – control graphs – that learns and organizes multiple physics-based motion skills and their transitions from a set of example motion capture clips. This offers a potential solution for the development of rich motion repertoires for humanoid robots as well as encourages widespread adoption of physics-based methods for character animation. Finally, we discuss research opportunities for future research on simulation-based control methods and transferring such controls to physical robots.


Feedback Policy Reference Motion Control Graph Reduced Order Feedback Physics-based Character 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    M. Al Borno, M. de Lasa, A. Hertzmann, Trajectory optimization for full-body movements with complex contacts. IEEE Trans. Vis. Comput. Graph. 19(8), 1405–1414 (2013)CrossRefGoogle Scholar
  2. 2.
    S. Chenney, D.A. Forsyth, Sampling plausible solutions to multi-body constraint problems, in SIGGRAPH’00, 2000, pp. 219–228Google Scholar
  3. 3.
    S. Coros, P. Beaudoin, M. van de Panne, Robust task-based control policies for physics-based characters. ACM Trans. Graph. 28(5), 170:1–170:9 (2009)CrossRefGoogle Scholar
  4. 4.
    S. Coros, P. Beaudoin, M. van de Panne, Generalized biped walking control. ACM Trans. Graph. 29(4), 130:1–130:9 (2010)CrossRefGoogle Scholar
  5. 5.
    M. de Lasa, I. Mordatch, A. Hertzmann, Feature-based locomotion controllers. ACM Trans. Graph. 29(4), 131:1–131:10 (2010)Google Scholar
  6. 6.
    K. Ding, L. Liu, M. van de Panne, K. Yin, Learning reduced-order feedback policies for motion skills, in ACM SIGGRAPH/Eurographics Symposium on Computer Animation, SCA’15 (ACM, 2015), pp. 83–92. 10.1145/2786784.2786802Google Scholar
  7. 7.
    A. Doucet, N. de Freitas, N. Gordon, An introduction to sequential monte carlo methods, in Sequential Monte Carlo Methods in Practice (Springer, New York, 2001)CrossRefGoogle Scholar
  8. 8.
    A. Doucet, A.M. Johansen, A tutorial on particle filtering and smoothing: fifteen years later, in Handbook of Nonlinear Filtering (Oxford University Press, Oxford, 2011)Google Scholar
  9. 9.
    T. Erez, Y. Tassa, E. Todorov, Simulation tools for model-based robotics: comparison of Bullet, Havok, MuJoCo, ODE and PhysX, in ICRA’15, 2015, pp. 4397–4404Google Scholar
  10. 10.
    S. Giovanni, K. Yin, Locotest: deploying and evaluating physics-based locomotion on multiple simulation platforms. Lect. Notes Comput. Sci. 7060, 227–241 (2011)Google Scholar
  11. 11.
    A. Goswami, S.K. Yun, U. Nagarajan, S.-H. Lee, K. Yin, S. Kalyanakrishnan, Direction-changing fall control of humanoid robots: theory and experiments. Auton. Robot. 36(3), 199–223 (2014)CrossRefGoogle Scholar
  12. 12.
    S. Ha, K. Yamane, Reducing hardware experiments for model learning and policy optimization, in ICRA’15, 2015, pp. 2620–2626Google Scholar
  13. 13.
    S. Ha, K. Yamane, Reducing hardware experiments for model learning and policy optimization, in 2015 IEEE International Conference on Robotics and Automation (ICRA), (IEEE, 2015), pp. 2620–2626Google Scholar
  14. 14.
    N. Hansen, The CMA evolution strategy: a comparing review, in Towards a New Evolutionary Computation. Studies in Fuzziness and Soft Computing, Vol. 192 (Springer, Berlin/Heidelberg, 2006), pp. 75–102Google Scholar
  15. 15.
    J.K. Hodgins, W.L. Wooten, D.C. Brogan, J.F. O’Brien, Animating human athletics, in SIGGRAPH’95, 1995, pp. 71–78Google Scholar
  16. 16.
    M. Isard, A. Blake, Condensation—conditional density propagation for visual tracking. Int. J. Comput. Vision 29(1), 5–28 (1998)Google Scholar
  17. 17.
    L. Kovar, M. Gleicher, F. Pighin, Motion graphs, in SIGGRAPH’02 (ACM, New York, 2002), pp. 473–482Google Scholar
  18. 18.
    J.J. Kuffner Jr., S. Kagami, K. Nishiwaki, M. Inaba, H. Inoue, Dynamically-stable motion planning for humanoid robots. Auton. Robot. 12(1), 105–118 (2002)Google Scholar
  19. 19.
    S.M. LaValle, J.J. Kuffner, Rapidly-exploring random trees: Progress and prospects, in Workshop on the Algorithmic Foundations of Robotics, 2000Google Scholar
  20. 20.
    Y. Lee, S. Kim, J. Lee, Data-driven biped control. ACM Trans. Graph. 29(4), 129:1–129:8 (2010)Google Scholar
  21. 21.
    S. Levine, V. Koltun, Guided policy search, in ICML’13, 2013Google Scholar
  22. 22.
    S. Levine, V. Koltun, Learning complex neural network policies with trajectory optimization, in ICML’14, 2014Google Scholar
  23. 23.
    T.P. Lillicrap, J.J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous control with deep reinforcement learning. CoRR, abs/1509.02971, 2015Google Scholar
  24. 24.
    L. Liu, M. van de Panne, K. Yin, Guided learning of control graphs for physics-based characters. ACM Trans. Graph. 35(3), Article 29 (2016). 10.1145/2893476Google Scholar
  25. 25.
    L. Liu, K. Yin, B. Guo, Improving sampling-based motion control. Comput. Graph. Forum 34(2), 415–423 (2015). Scholar
  26. 26.
    L. Liu, K. Yin, M. van de Panne, B. Guo, Terrain runner: control, parameterization, composition, and planning for highly dynamic motions. ACM Trans. Graph. 31(6), Article 154 (2012). 10.1145/2366145.2366173CrossRefGoogle Scholar
  27. 27.
    L. Liu, K. Yin, M. van de Panne, T. Shao, W. Xu, Sampling-based contact-rich motion control. ACM Trans. Graph. 29(4), Article 128 (2010). 10.1145/1778765.1778865CrossRefGoogle Scholar
  28. 28.
    L. Liu, K. Yin, B. Wang, B. Guo, Simulation and control of skeleton-driven soft body characters. ACM Trans. Graph. 32(6), Article 215 (2013). 10.1145/2508363.2508427Google Scholar
  29. 29.
    A. Macchietto, V. Zordan, C.R. Shelton, Momentum control for balance. ACM Trans. Graph. 28(3), Article 80 (2009)Google Scholar
  30. 30.
    I. Mordatch, K. Lowrey, E. Todorov, Ensemble-CIO: full-body dynamic motion planning that transfers to physical humanoids, in IROS (IEEE, 2015), pp. 5307–5314Google Scholar
  31. 31.
    I. Mordatch, E. Todorov, Combining the benefits of function approximation and trajectory optimization, in Robotics: Science and Systems, Berkeley, 2014Google Scholar
  32. 32.
    U. Muico, Y. Lee, J. Popović, Z. Popović, Contact-aware nonlinear control of dynamic characters. ACM Trans. Graph. 28(3), Article 81 (2009)CrossRefGoogle Scholar
  33. 33.
    X.B. Peng, G. Berseth, M. van de Panne, Dynamic terrain traversal skills using reinforcement learning. ACM Trans. Graph. 34(4), 80:1–80:11 (2015)CrossRefGoogle Scholar
  34. 34.
    J. Peters, J. Kober, Using reward-weighted imitation for robot reinforcement learning, in Adaptive Dynamic Programming and Reinforcement Learning, 2009, pp. 226–232Google Scholar
  35. 35.
    J. Peters, S. Schaal, Reinforcement learning by reward-weighted regression for operational space control, in ICML’07, 2007, pp. 745–750Google Scholar
  36. 36.
    J. Peters, S. Schaal, Reinforcement learning of motor skills with policy gradients. Neural Netw. 21(4), 682–697 (2008)CrossRefGoogle Scholar
  37. 37.
    N.S. Pollard, J.K. Hodgins, M.J. Riley, C.G. Atkeson, Adapting human motion for the control of a humanoid robot, in IEEE International Conference on Robotics and Automation, 2002. Proceedings. ICRA’02, Vol. 2 (IEEE, 2002), pp. 1390–1397Google Scholar
  38. 38.
    J. Popović, S.M. Seitz, M. Erdmann, Z. Popović, A. Witkin, Interactive manipulation of rigid body simulations, in SIGGRAPH’00, 2000, pp. 209–217Google Scholar
  39. 39.
    A. Punjani, P. Abbeel, Deep learning helicopter dynamics models, in ICRA’15, 2015, pp. 3223–3230Google Scholar
  40. 40.
    K. Sims, Evolving virtual creatures, in SIGGRAPH’94, 1994, pp. 15–22Google Scholar
  41. 41.
    J. Tan, Y. Gu, C. Karen Liu, G. Turk, Learning bicycle stunts, ACM Trans. Graph. 33(4), 50:1–50:12 (2014)CrossRefGoogle Scholar
  42. 42.
    J. Tan, C. Karen Liu, G. Turk, Stable proportional-derivative controllers. IEEE Comput. Graph. Appl. 31(4), 34–44 (2011)CrossRefGoogle Scholar
  43. 43.
    K.I. Tsianos, I.A. Sucan, L.E. Kavraki, Sampling-based robot motion planning: towards realistic applications. Comput. Sci. Rev. 1, 2–11 (2007)CrossRefGoogle Scholar
  44. 44.
    C.D. Twigg, D.L. James, Many-worlds browsing for control of multibody dynamics. ACM Trans. Graph. 26(3), 14 (2007)CrossRefGoogle Scholar
  45. 45.
    K. Wampler, Z. Popović, Optimal gait and form for animal locomotion. ACM Trans. Graph. 28(3), Article 60 (2009)CrossRefGoogle Scholar
  46. 46.
    J.M. Wang, D.J. Fleet, A. Hertzmann, Optimizing walking controllers. ACM Trans. Graph. 28(5), Article 168 (2009)Google Scholar
  47. 47.
    A. Witkin, M. Kass, Spacetime constraints, in SIGGRAPH’88, 1988, pp. 159–168CrossRefGoogle Scholar
  48. 48.
    K. Yamane, J.J. Kuffner, J.K. Hodgins, Synthesizing animations of human manipulation tasks. ACM Trans. Graph. 23(3), 532–539 (2004)CrossRefGoogle Scholar
  49. 49.
    K. Yin, M.B. Cline, D.K. Pai, Motion perturbation based on simple neuromotor control models, in PG’03 (Pacific Conference on Computer Graphics and Applications), 2003, pp. 445–449Google Scholar
  50. 50.
    K. Yin, S. Coros, P. Beaudoin, M. van de Panne, Continuation methods for adapting simulated skills. ACM Trans. Graph. 27(3), Article 81 (2008)CrossRefGoogle Scholar
  51. 51.
    K. Yin, K. Loken, M. van de Panne, SIMBICON: simple biped locomotion control. ACM Trans. Graph. 26(3), Article 105 (2007)Google Scholar
  52. 52.
    V.B. Zordan, J.K. Hodgins, Motion capture-driven simulations that hit and react, in SCA (ACM SIGGRAPH/Eurographics Symposium on Computer Animation), 2002, pp. 89–96Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  • KangKang Yin
    • 1
    Email author
  • Libin Liu
    • 2
  • Michiel van de Panne
    • 2
  1. 1.Department of Computer ScienceNational University of SingaporeSingaporeSingapore
  2. 2.University of British ColumbiaVancouverCanada

Personalised recommendations