Bayesian optimization for learning gaits under uncertainty

An experimental comparison on a dynamic bipedal walker
  • Roberto CalandraEmail author
  • André Seyfarth
  • Jan Peters
  • Marc Peter Deisenroth


Designing gaits and corresponding control policies is a key challenge in robot locomotion. Even with a viable controller parametrization, finding near-optimal parameters can be daunting. Typically, this kind of parameter optimization requires specific expert knowledge and extensive robot experiments. Automatic black-box gait optimization methods greatly reduce the need for human expertise and time-consuming design processes. Many different approaches for automatic gait optimization have been suggested to date. However, no extensive comparison among them has yet been performed. In this article, we thoroughly discuss multiple automatic optimization methods in the context of gait optimization. We extensively evaluate Bayesian optimization, a model-based approach to black-box optimization under uncertainty, on both simulated problems and real robots. This evaluation demonstrates that Bayesian optimization is particularly suited for robotic applications, where it is crucial to find a good set of gait parameters in a small number of experiments.


Gait optimization Bayesian optimization Robotics Locomotion 

Mathematics Subject Classification (2010)

68T05 90C26 49Mxx 68T40 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Chernova, S., Veloso, M.: An evolutionary approach to gait learning for four-legged robots. In: Intelligent Robots and Systems (IROS), vol. 3, pp. 2562–2567. IEEE (2004)Google Scholar
  2. 2.
    Gibbons, P., Mason, M., Vicente, A., Bugmann, G., Culverhouse, P.: Optimisation of dynamic gait for small bipedal robots. In: Proc. 4th Workshop on Humanoid Soccer Robots (Humanoids 2009), pp. 9–14 (2009)Google Scholar
  3. 3.
    Kulk, J., Welsh, J.: Evaluation of walk optimisation techniques for the NAO robot. In: Humanoids 2011, pp. 306–311 (2011)Google Scholar
  4. 4.
    Kushner, H.J.: A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Basic Eng. 86, 97 (1964)CrossRefGoogle Scholar
  5. 5.
    Jones, D.R.: A taxonomy of global optimization methods based on response surfaces. J. Global Optim. 21, 345–383 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Osborne, M.A., Garnett, R., Roberts, S.J.: Gaussian processes for global optimization. In: Learning and Intelligent Optimization (LION), pp. 1–15 (2009)Google Scholar
  7. 7.
    Brochu, E., Cora, V.M., De Freitas, N.: A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 (2010)
  8. 8.
    Garnett, R., Osborne, M.A., Roberts, S.J.: Bayesian optimization for sensor set selection. In: Proceedings of the 9th ACM/IEEE International Conference on Information Processing in Sensor Networks. IPSN ’10, pp 209–219. ACM, New York (2010)Google Scholar
  9. 9.
    Lizotte, D.J., Wang, T., Bowling, M., Schuurmans, D.: Automatic gait optimization with Gaussian process regression. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 944–949 (2007)Google Scholar
  10. 10.
    Tesch, M., Schneider, J., Choset, H.: Using response surfaces and expected improvement to optimize snake robot gait parameters. In: International Conference on Intelligent Robots and Systems, pp 1069–1074. IEEE, IROS (2011)Google Scholar
  11. 11.
    Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Learning and Intelligent Optimization (LION), pp. 507–523. Springer (2011)Google Scholar
  12. 12.
    Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems (NIPS) (2012)Google Scholar
  13. 13.
    Calandra, R., Seyfarth, A., Peters, J., Deisenroth, M.P.: An experimental comparison of Bayesian optimization for bipedal locomotion. In: International Conference on Robotics and Automation (ICRA) (2014)Google Scholar
  14. 14.
    Calandra, R., Gopalan, N., Seyfarth, A., Peters, J., Deisenroth, M.P.: Bayesian gait optimization for bipedal locomotion. In: Learning and Intelligent Optimization (LION), pp. 274–290 (2014)Google Scholar
  15. 15.
    Brooks, S.H.: A discussion of random methods for seeking maxima. Oper. Res. 6, 244–251 (1958)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. (JMLR) 13, 281–305 (2012)MathSciNetzbMATHGoogle Scholar
  17. 17.
    Yamane, K.: Geometry and biomechanics for locomotion synthesis and control. In: Modeling, Simulation and Optimization of Bipedal Walking. Volume 18 of Cognitive Systems Monographs, pp. 273–287. Springer (2013)Google Scholar
  18. 18.
    Tedrake, R., Zhang, T., Seung, H.: Stochastic policy gradient reinforcement learning on a simple 3D biped. In: International Conference on Intelligent Robots and Systems (IROS), pp. 2849–2854 (2004)Google Scholar
  19. 19.
    Tang, Z., Zhou, C., Sun, Z.: Humanoid walking gait optimization using ga-based neural network. In: Advances in Natural Computation. Volume 3611 of Lecture Notes in Computer Science, pp. 252–261 . Springer (2005)Google Scholar
  20. 20.
    Niehaus, C., Röfer, T., Laue, T.: Gait optimization on a humanoid robot using particle swarm optimization. In: Proceedings of the Second Workshop on Humanoid Soccer Robots in conjunction with the (2007)Google Scholar
  21. 21.
    Hemker, T., Stelzer, M., von Stryk, O., Sakamoto, H.: Efficient walking speed optimization of a humanoid robot. Int. J. Robot. Res. (IJRR) 28, 303–314 (2009)CrossRefGoogle Scholar
  22. 22.
    Geng, T., Porr, B., Wörgötter, F.: Fast biped walking with a sensor-driven neuronal controller and real-time online learning. Int. J. Robot. Res. (IJRR) 25, 243–259 (2006)CrossRefGoogle Scholar
  23. 23.
    Lizotte, D.J., Greiner, R., Schuurmans, D.: An experimental methodology for response surface optimization methods. J. Global Optim. 53, 699–736 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  24. 24.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13, 455–492 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Huang, D., Allen, T.T., Notz, W.I., Zeng, N.: Global optimization of stochastic black-box systems via sequential kriging meta-models. J. Global Optim. 34, 441–466 (2006)MathSciNetCrossRefzbMATHGoogle Scholar
  26. 26.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press (2006)Google Scholar
  27. 27.
    Mockus, J., Tiesis, V., Zilinskas, A.: The application of Bayesian methods for seeking the extremum. Towards Global Optim 2, 117–129 (1978)zbMATHGoogle Scholar
  28. 28.
    Cox, D.D., John, S.: SDO: A statistical method for global optimization. Multidisciplinary Design Optimization: State of the Art, pp. 315–329 (1997)Google Scholar
  29. 29.
    Hennig, P., Schuler, C.J.: Entropy search for information-efficient global optimization. J. Mach. Learn. Res. (JMLR) 13, 1809–1837 (2012)MathSciNetzbMATHGoogle Scholar
  30. 30.
    Srinivas, N., Krause, A., Kakade, S., Seeger, M.: Gaussian process optimization in the bandit setting: No regret and experimental design. In: International Conference on Machine Learning (ICML), Omnipress, pp. 1015–1022 (2010)Google Scholar
  31. 31.
    Jones, D.R., Perttunen, C.D., Stuckman, B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl 79, 157–181 (1993)MathSciNetCrossRefzbMATHGoogle Scholar
  32. 32.
    Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput 16, 1190–1208 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput 9, 159–195 (2001)CrossRefGoogle Scholar
  34. 34.
    Bertsekas, D.P.: Dynamic Programming and Optimal Control. 3rd edn. Athena Scientific (2007)Google Scholar
  35. 35.
    Renjewski, D.: An engineering contribution to human gait biomechanics. PhD thesis, TU Ilmenau (2012)Google Scholar
  36. 36.
    Renjewski, D., Seyfarth, A.: Robots in human biomechanics - a study on ankle push-off in walking. Bioinspiration Biomimetics 7, 036005 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Intelligent Autonomous SystemsTU DarmstadtDarmstadtGermany
  2. 2.Lauflabor Locomotion LaboratoryTU DarmstadtDarmstadtGermany
  3. 3.Max Planck Institute for Intelligent SystemsTübingenGermany
  4. 4.Department of ComputingImperial College LondonLondonUK

Personalised recommendations