Skip to main content
SpringerLink
Log in

Bayesian optimization for learning gaits under uncertainty

An experimental comparison on a dynamic bipedal walker

  • Published:
Annals of Mathematics and Artificial Intelligence Aims and scope Submit manuscript

Abstract

Designing gaits and corresponding control policies is a key challenge in robot locomotion. Even with a viable controller parametrization, finding near-optimal parameters can be daunting. Typically, this kind of parameter optimization requires specific expert knowledge and extensive robot experiments. Automatic black-box gait optimization methods greatly reduce the need for human expertise and time-consuming design processes. Many different approaches for automatic gait optimization have been suggested to date. However, no extensive comparison among them has yet been performed. In this article, we thoroughly discuss multiple automatic optimization methods in the context of gait optimization. We extensively evaluate Bayesian optimization, a model-based approach to black-box optimization under uncertainty, on both simulated problems and real robots. This evaluation demonstrates that Bayesian optimization is particularly suited for robotic applications, where it is crucial to find a good set of gait parameters in a small number of experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Chernova, S., Veloso, M.: An evolutionary approach to gait learning for four-legged robots. In: Intelligent Robots and Systems (IROS), vol. 3, pp. 2562–2567. IEEE (2004)

  2. Gibbons, P., Mason, M., Vicente, A., Bugmann, G., Culverhouse, P.: Optimisation of dynamic gait for small bipedal robots. In: Proc. 4th Workshop on Humanoid Soccer Robots (Humanoids 2009), pp. 9–14 (2009)

  3. Kulk, J., Welsh, J.: Evaluation of walk optimisation techniques for the NAO robot. In: Humanoids 2011, pp. 306–311 (2011)

  4. Kushner, H.J.: A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. J. Basic Eng. 86, 97 (1964)

    Article  Google Scholar 

  5. Jones, D.R.: A taxonomy of global optimization methods based on response surfaces. J. Global Optim. 21, 345–383 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  6. Osborne, M.A., Garnett, R., Roberts, S.J.: Gaussian processes for global optimization. In: Learning and Intelligent Optimization (LION), pp. 1–15 (2009)

  7. Brochu, E., Cora, V.M., De Freitas, N.: A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599 (2010)

  8. Garnett, R., Osborne, M.A., Roberts, S.J.: Bayesian optimization for sensor set selection. In: Proceedings of the 9th ACM/IEEE International Conference on Information Processing in Sensor Networks. IPSN ’10, pp 209–219. ACM, New York (2010)

    Google Scholar 

  9. Lizotte, D.J., Wang, T., Bowling, M., Schuurmans, D.: Automatic gait optimization with Gaussian process regression. In: International Joint Conference on Artificial Intelligence (IJCAI), pp. 944–949 (2007)

  10. Tesch, M., Schneider, J., Choset, H.: Using response surfaces and expected improvement to optimize snake robot gait parameters. In: International Conference on Intelligent Robots and Systems, pp 1069–1074. IEEE, IROS (2011)

    Google Scholar 

  11. Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Learning and Intelligent Optimization (LION), pp. 507–523. Springer (2011)

  12. Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems (NIPS) (2012)

  13. Calandra, R., Seyfarth, A., Peters, J., Deisenroth, M.P.: An experimental comparison of Bayesian optimization for bipedal locomotion. In: International Conference on Robotics and Automation (ICRA) (2014)

  14. Calandra, R., Gopalan, N., Seyfarth, A., Peters, J., Deisenroth, M.P.: Bayesian gait optimization for bipedal locomotion. In: Learning and Intelligent Optimization (LION), pp. 274–290 (2014)

  15. Brooks, S.H.: A discussion of random methods for seeking maxima. Oper. Res. 6, 244–251 (1958)

    Article  MathSciNet  Google Scholar 

  16. Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. (JMLR) 13, 281–305 (2012)

    MathSciNet  MATH  Google Scholar 

  17. Yamane, K.: Geometry and biomechanics for locomotion synthesis and control. In: Modeling, Simulation and Optimization of Bipedal Walking. Volume 18 of Cognitive Systems Monographs, pp. 273–287. Springer (2013)

  18. Tedrake, R., Zhang, T., Seung, H.: Stochastic policy gradient reinforcement learning on a simple 3D biped. In: International Conference on Intelligent Robots and Systems (IROS), pp. 2849–2854 (2004)

  19. Tang, Z., Zhou, C., Sun, Z.: Humanoid walking gait optimization using ga-based neural network. In: Advances in Natural Computation. Volume 3611 of Lecture Notes in Computer Science, pp. 252–261 . Springer (2005)

  20. Niehaus, C., Röfer, T., Laue, T.: Gait optimization on a humanoid robot using particle swarm optimization. In: Proceedings of the Second Workshop on Humanoid Soccer Robots in conjunction with the (2007)

  21. Hemker, T., Stelzer, M., von Stryk, O., Sakamoto, H.: Efficient walking speed optimization of a humanoid robot. Int. J. Robot. Res. (IJRR) 28, 303–314 (2009)

    Article  Google Scholar 

  22. Geng, T., Porr, B., Wörgötter, F.: Fast biped walking with a sensor-driven neuronal controller and real-time online learning. Int. J. Robot. Res. (IJRR) 25, 243–259 (2006)

    Article  Google Scholar 

  23. Lizotte, D.J., Greiner, R., Schuurmans, D.: An experimental methodology for response surface optimization methods. J. Global Optim. 53, 699–736 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  24. Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13, 455–492 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  25. Huang, D., Allen, T.T., Notz, W.I., Zeng, N.: Global optimization of stochastic black-box systems via sequential kriging meta-models. J. Global Optim. 34, 441–466 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  26. Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press (2006)

  27. Mockus, J., Tiesis, V., Zilinskas, A.: The application of Bayesian methods for seeking the extremum. Towards Global Optim 2, 117–129 (1978)

    MATH  Google Scholar 

  28. Cox, D.D., John, S.: SDO: A statistical method for global optimization. Multidisciplinary Design Optimization: State of the Art, pp. 315–329 (1997)

  29. Hennig, P., Schuler, C.J.: Entropy search for information-efficient global optimization. J. Mach. Learn. Res. (JMLR) 13, 1809–1837 (2012)

    MathSciNet  MATH  Google Scholar 

  30. Srinivas, N., Krause, A., Kakade, S., Seeger, M.: Gaussian process optimization in the bandit setting: No regret and experimental design. In: International Conference on Machine Learning (ICML), Omnipress, pp. 1015–1022 (2010)

  31. Jones, D.R., Perttunen, C.D., Stuckman, B.E.: Lipschitzian optimization without the Lipschitz constant. J. Optim. Theory Appl 79, 157–181 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  32. Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput 16, 1190–1208 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  33. Hansen, N., Ostermeier, A.: Completely derandomized self-adaptation in evolution strategies. Evol. Comput 9, 159–195 (2001)

    Article  Google Scholar 

  34. Bertsekas, D.P.: Dynamic Programming and Optimal Control. 3rd edn. Athena Scientific (2007)

  35. Renjewski, D.: An engineering contribution to human gait biomechanics. PhD thesis, TU Ilmenau (2012)

  36. Renjewski, D., Seyfarth, A.: Robots in human biomechanics - a study on ankle push-off in walking. Bioinspiration Biomimetics 7, 036005 (2012)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Intelligent Autonomous Systems, TU Darmstadt, Darmstadt, Germany

    Roberto Calandra & Jan Peters

  2. Lauflabor Locomotion Laboratory, TU Darmstadt, Darmstadt, Germany

    André Seyfarth

  3. Max Planck Institute for Intelligent Systems, Tübingen, Germany

    Jan Peters

  4. Department of Computing, Imperial College London, London, UK

    Marc Peter Deisenroth

Authors
  1. Roberto Calandra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. André Seyfarth
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jan Peters
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marc Peter Deisenroth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roberto Calandra.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Calandra, R., Seyfarth, A., Peters, J. et al. Bayesian optimization for learning gaits under uncertainty. Ann Math Artif Intell 76, 5–23 (2016). https://doi.org/10.1007/s10472-015-9463-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10472-015-9463-9

Keywords

Mathematics Subject Classification (2010)

Search

Navigation