Advertisement

Paladyn

, Volume 3, Issue 2, pp 75–91 | Cite as

From humans to humanoids: The optimal control framework

  • Serena IvaldiEmail author
  • Olivier Sigaud
  • Bastien Berret
  • Francesco Nori
Review Article
  • 127 Downloads

Abstract

In the last years of research in cognitive control, neuroscience and humanoid robotics have converged to different frameworks which aim, on one side, at modeling and analyzing human motion, and, on the other side, at enhancing motor abilities of humanoids. In this paper we try to cover the gap between the two areas, giving an overview of the literature in the two fields which concerns the production of movements. First, we survey computational motor control models based on optimality principles; then, we review available implementations and techniques to transfer these principles to humanoid robots, with a focus on the limitations and possible improvements of the current implementations. Moreover, we propose Stochastic Optimal Control as a framework to take into account delays and noise, thus catching the unpredictability aspects typical of both humans and humanoids systems. Optimal Control in general can also easily be integrated with Machine Learning frameworks, thus resulting in a computational implementation of human motor learning. This survey is mainly addressed to roboticists attempting to implement human-inspired controllers on robots, but can also be of interest for researchers in other fields, such as computational motor control.

Keywords

humanoids human motor control optimality stochastic optimal control 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    Adams, B., Breazeal, C., Brooks, R. A., and Scassellati, B. (2000). Humanoid robots: a new kind of tool. IEEE Intelligent Systems, 15(4):25–31.CrossRefGoogle Scholar
  2. [2]
    Alexander, R. M. (1997). A minimum energy cost hypothesis for human arm trajectories. Biological cybernetics, 76(2):97–105.zbMATHCrossRefGoogle Scholar
  3. [3]
    Andersen, R. A., Snyder, L. H., Bradley, D. C., and Xing, J. (1997). Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annual review of Neuroscience, 20(1):303–330.CrossRefGoogle Scholar
  4. [4]
    Arechavaleta, G., Laumond, J.-P., Hicheur, H., and Berthoz, A. (2008). An optimality principle governing human walking. IEEE Transactions on Robotics, 24:5–14.CrossRefGoogle Scholar
  5. [5]
    Argall, B. D., Chernova, S., Veloso, M., and Browning, B. (2009). A survey of robot learning from demonstration. Robotics and Autonomous Systems, 57(5):469–483.CrossRefGoogle Scholar
  6. [6]
    Atkeson, C. G., Hale, J. G., Pollick, F., Riley, M., Kotosaka, S., Schaul, S., Shibata, T., Tevatia, G., Ude, A., Vijayakumar, S., Kawato, E., and Kawato, M. (2000). Using humanoid robots to study human behavior. IEEE Intelligent Systems and Their Applications, 15(4):46–56.CrossRefGoogle Scholar
  7. [7]
    Atkeson, C. G. and Stephens, B. (2007). Multiple balance strategies from one optimization criterion. 2007 7th IEEE-RAS Int. Conf. on Humanoid Robots}, pages 57–6Google Scholar
  8. [8]
    Barambones, O. and Etxebarria, V. (2002). Robust neural control for robotic manipulators. Automatica, 38:235–242.zbMATHCrossRefGoogle Scholar
  9. [9]
    Bauml, B., Wimbock, T., and Hirzinger, G. (2010). Kinematically optimal catching a flying ball with a hand-arm-system. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 2592–2599, Taipei, Taiwan.Google Scholar
  10. [10]
    Bellman, R. (1957). Dynamic Programming. Princeton University Press, Princeton, NJ.zbMATHGoogle Scholar
  11. [11]
    Ben-Itzhak, S. and Karniel, A. (2008). Minimum acceleration criterion with constraints implies bang-bang control as an underlying principle for optimal trajectories of arm reaching movements. Neural Computation, 20(3):779–812.MathSciNetzbMATHCrossRefGoogle Scholar
  12. [12]
    Bennequin, D., Fuchs, R., Berthoz, A., and Flash, T. (2009). Movement timing and invariance arise from several geometries. PLoS Computational Biology, 5(7):e1000426.MathSciNetCrossRefGoogle Scholar
  13. [13]
    Bernstein, N. (1967). The Co-ordination and Regulation of Movements. Oxford, UK: Pergamo.Google Scholar
  14. [14]
    Berret, B., Chiovetto, E., Nori, F., and Pozzo, T. (2011a). Evidence for composite cost functions in arm movement planning: An inverse optimal control approach. PLoS Computational Biology, 7(10):e1002183.MathSciNetCrossRefGoogle Scholar
  15. [15]
    Berret, B., Darlot, C., Jean, F., Pozzo, T., Papaxanthis, C., and Gauthier, J.-P. (2008). The inactivation principle: mathematical solutions minimizing the absolute work and biological implications for the planning of arm movements. PLoS Computational Biology, 4(10):e1000194.MathSciNetCrossRefGoogle Scholar
  16. [16]
    Berret, B., Ivaldi, S., Nori, F., and Sandini, G. (2011b). Stochastic optimal control with variable impedance manipulators in presence of uncertainties and delayed feedback. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 4354–4359.Google Scholar
  17. [17]
    Bertsekas, D. P. (1995). Dynamic Programming and Optimal Control. Athena Scientific.Google Scholar
  18. [18]
    Bertsekas, D. P. and Tsitsiklis, J. N. (1996). Neuro-dynamic programming. Athena Scientific.Google Scholar
  19. [19]
    Biess, A., Liebermann, D. G., and Flash, T. (2007). A computational model for redundant human three-dimensional pointing movements: integration of independent spatial and temporal motor plans simplifies movement dynamics. The Journal of Neuroscience, 27(48):13045–13064.CrossRefGoogle Scholar
  20. [20]
    Billard, A., Calinon, S., Dillmann, R., and Schaal, S. (2007). Handbook of Robotics (Siciliano, B. and Khatib, O. Eds), Robot Programming by Demonstration, pages 1371–1394. Springer.Google Scholar
  21. [21]
    Blair, J. and Iwasaki, T. (2011). Optimal Gaits for Mechanical Rectifier Systems. IEEE Transactions on Automatic Control, 56(1):59–71.MathSciNetCrossRefGoogle Scholar
  22. [22]
    Braganza, D., Dixon, W. E., Dawson, D. M., and Xian, B. (2005). Tracking control for robot manipulators with kinematic and dynamic uncertainty. In 44th IEEE Conf. on Decision and Control.Google Scholar
  23. [23]
    Braun, D. A., Aertsen, A., Wolpert, D. M., and Mehring, C. (2009). Learning optimal adaptation strategies in unpredictable motor tasks. The Journal of Neuroscience, 29(20):6472–6478.CrossRefGoogle Scholar
  24. [24]
    Bryson, A. E. and Ho, Y.-C. (1975). Applied Optimal Control: Optimization, Estimation, and Control. John Wiley & Sons Inc.Google Scholar
  25. [25]
    Buchli, J., Stulp, F., Theodorou, E., and Schaal, S. (2011). Learning variable impedance control. The Int. Journal of Robotics Research, 30(7):820–833.CrossRefGoogle Scholar
  26. [26]
    Buchli, J., Theodorou, E., Stulp, F., and Schaal, S. (2010). Variable impedance control — a reinforcement learning approach. In Robotics Science and Systems.Google Scholar
  27. [27]
    Butz, M., Pedersen, G., and Stalph, P. (2009). Learning sensorimotor control structures with XCSF: redundancy exploitation and dynamic control. In 11th Annual Conf. on Genetic and Evolutionary Computation, pages 1171–1178.Google Scholar
  28. [28]
    Caccavale, F., Chiaverini, S., and Siciliano, B. (1997). Secondorder kinematic control of robot manipulators with jacobian damped least-squares inverse: theory and experiments. IEEE/ASME Transactions on Mechatronics, 2(3):188–194.CrossRefGoogle Scholar
  29. [29]
    Cardinali, L., Frassinetti, F., Brozzoli, C., Urquizar, C., Roy, A. C., and Farnè, A. (2009). Tool-use induces morphological updating of the body schema. Current Biology, 19(12):R478–9.CrossRefGoogle Scholar
  30. [30]
    Cesqui, B., d’Avella, A., Portone, A., and Lacquaniti, F. (2012). Catching a ball at the right time and place: individual factors matter. PLoS one, 7(2):e31770.CrossRefGoogle Scholar
  31. [31]
    Chevallereau, C. and Aoustin, Y. (2001). Optimal reference trajectories for walking and running of a biped robot. Robotica, 19:557–569.CrossRefGoogle Scholar
  32. [32]
    Chiaverini, S., Egeland, O., and Kanestrom, R. K. (1991). Achieving user-defined accuracy with damped least-squares inverse kinematics. In 5th Int. Conf. on Advanced Robotics, pages 672–677.Google Scholar
  33. [33]
    Crevecoeur, F., Thonnard, J.-L., and Lefèvre, P. (2009). Optimal integration of gravity in trajectory planning of vertical pointing movements. Journal of neurophysiology, 102(2):786–796.CrossRefGoogle Scholar
  34. [34]
    da Silva, M., Durand, F., and Popovic, J. (2009). Linear Bellman combination for control of character animation. ACM Transactions on Graphics, 28(3):1.CrossRefGoogle Scholar
  35. [35]
    Dahiya, R. S., Metta, G., Valle, M., and Sandini, G. (2010). Tactile sensing: From humans to humanoids. IEEE Transactions on Robotics, 26(1):1–20.CrossRefGoogle Scholar
  36. [36]
    Dapena, J. (1980a). Mechanics of rotation in the fosbury-flop. Medicine and Science in Sports and Exercise, 12(1):45–53.Google Scholar
  37. [37]
    Dapena, J. (1980b). Mechanics of translation in the fosbury-flop. Medicine and Science in Sports and Exercise, 12(1):37–44.Google Scholar
  38. [38]
    Dapena, J. (2002). The evolution of high jumping technique: Biomechanical analysis. In of 20th Internat. Symp. Biomech. Sports, C ceres, Spain.Google Scholar
  39. [39]
    De Santis, A., Siciliano, B., De Luca, A., and Bicchi, A. (2008). An atlas of physical human-robot interaction. Mechanism and Machine Theory, 43(3):253–270.zbMATHCrossRefGoogle Scholar
  40. [40]
    Desmurget, M. and Grafton, S. (2000). Forward modeling allows feedback control for fast reaching movements. Trends in Cognitive Sciences, 4:423–431.CrossRefGoogle Scholar
  41. [41]
    Diedrichsen, J., Shadmehr, R., and Ivry, R. B. (2010a). The coordination of movement: optimal feedback control and beyond. Trends in Cognitive Sciences, 14(1):31–39.CrossRefGoogle Scholar
  42. [42]
    Diedrichsen, J., White, O., Newman, D., and Lally, N. (2010b). Use-dependent and error-based learning of motor behaviors. Journal of Neuroscience, 30(15):5159–5166.CrossRefGoogle Scholar
  43. [43]
    Diehl, M., Bock, H. G., Diedam, H., and Wieber, P. B. (2006). Fast Motions in Biomechanics and Robotics (Diehl, M. and Mombaur, K. Eds), vol. 340, Fast Direct Multiple Shooting algorithms for optimal robot control, pages 65–93. LNCIS, Springer.Google Scholar
  44. [44]
    Diehl, M., Ferreau, H. J., and Haverbeke, N. (2009). Nonlinear Model Predictive Control (Magni, L. et al. Eds), vol. 384, Effcient numerical methods for nonlinear MPC and Moving Horizon estimation, pages 541–550. LNCIS, Springer.Google Scholar
  45. [45]
    Dominici, N., Ivanenko, Y. P., Cappellini, G., d’Avella, A., Mond, V., Cicchese, M., Fabiano, A., Silei, T., Di Paolo, A., Giannini, C., Poppele, R. E., and Lacquaniti, F. (2011). Locomotor primitives in newborn babies and their development. Science, 334(6058):997–999.CrossRefGoogle Scholar
  46. [46]
    Dupree, K., Patre, P., Johnson, M., and Dixon, W. (2009). Inverse optimal adaptive control of a nonlinear euler-lagrange system, part i: Full state feedback. In 48th IEEE Conf. on Decision and Control, pages 321–326.Google Scholar
  47. [47]
    Feldman, A. G. and Levin, M. F. (1995). The origin and use of positional frames of reference in motor control. Behavioral and Brain Sciences, 18(4):723–744.CrossRefGoogle Scholar
  48. [48]
    Fitts, P. (1954). The information capacity of the human motor system in controlling the amplitude of movement. J. Exp. Psychol., 47(6):381–391.CrossRefGoogle Scholar
  49. [49]
    Flash, T. and Hogan, N. (1985). The coordination of arm movements: an experimentally confirmed mathematical model. The Journal of Neuroscience, 5(7):1688–1703.Google Scholar
  50. [50]
    Franklin, D. W., Burdet, E., Tee, K. P., Osu, R., Chew, C.-M., Milner, T. E., and Kawato, M. (2008). CNS learns stable, accurate, and effcient movements using a simple algorithm. The Journal of Neuroscience, 28(44):11165–11173.CrossRefGoogle Scholar
  51. [51]
    Franklin, D. W., So, U., Burdet, E., and Kawato, M. (2007). Visual feedback is not necessary for the learning of novel dynamics. PloS one, 2(12):e1336.CrossRefGoogle Scholar
  52. [52]
    Friston, K. (2010). The free-energy principle: a unified brain theory? Nature Reviews, 11:127–138.CrossRefGoogle Scholar
  53. [53]
    Ganesh, G., Albu-Schaffer, A., Haruno, M., Kawato, M., and Burdet, E. (2010). Biomimetic motor behavior for simultaneous adaptation of force, impedance and trajectory in interaction tasks. In IEEE Int. Conf. on Robotics and Automation, pages 2705–2711.Google Scholar
  54. [54]
    Gepshtein, S., Seydell, A., and Trommershäuser, J. (2007). Optimality of human movement under natural variations of visualmotor uncertainty. Journal of Vision, 7(5):1–18.CrossRefGoogle Scholar
  55. [55]
    Gienger, M., Janssen, H., and Goerick, C. (2005). Task-oriented whole body motion for humanoid robots. In 5th IEEE-RAS Int. Conf. on Humanoid Robots, pages 238–244.Google Scholar
  56. [56]
    Guigon, E. (2011). Motor Control (Danion, F. and Latash, M.L. Eds), Models and Architectures for motor control: Simple or complex?, pages 478–502. Oxford University Press.Google Scholar
  57. [57]
    Guigon, E., Baraduc, P., and Desmurget, M. (2008a). Computational motor control: feedback and accuracy. European Journal of Neuroscience, 27(4):1003–1016.CrossRefGoogle Scholar
  58. [58]
    Guigon, E., Baraduc, P., and Desmurget, M. (2008b). Optimality, stochasticity and variability in motor behavior. Journal of Computational Neuroscience, 24(1):57–68.CrossRefGoogle Scholar
  59. [59]
    Haddadin, S., Albu-Schäffer, A., and Hirzinger, G. (2010). Safety analysis for a human-friendly manipulator. Int. Journal of Social Robotics, 2:235–252.CrossRefGoogle Scholar
  60. [60]
    Hansen, N., Muller, S. D., and Koumoutsakos, P. (2003). Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, 11(1):1–18.CrossRefGoogle Scholar
  61. [61]
    Harris, C. M. and Wolpert, D. M. (1998). Signal-dependent noise determines motor planning. Nature, 394(6695):780–784.CrossRefGoogle Scholar
  62. [62]
    Harris, C. M. and Wolpert, D. M. (2006). The main sequence of saccades optimizes speed-accuracy trade-off. Biological Cybernetics, 95(1):21–29.MathSciNetzbMATHCrossRefGoogle Scholar
  63. [63]
    Hauser, H., Neumann, G., Ijspeert, A. J., and Maass, W. (2011). Biologically inspired kinematic synergies enable linear balance control of a humanoid robot. Biological cybernetics, 104(4-5):235–249.MathSciNetzbMATHCrossRefGoogle Scholar
  64. [64]
    He, G.-P. and Geng, Z.-Y. (2007). Optimal motion planning of a one-legged hopping robot. In IEEE Int. Conf. on Robotics and Biomimetics, pages 1178–1183, Sanya, China.Google Scholar
  65. [65]
    Heidrich-Meisner, V. and Igel, C. (2008). Similarities and differences between policy gradient methods and evolution strategies. In 16th Europ. Symp. on Artificial Neural Networks (ESANN), pages 149–154.Google Scholar
  66. [66]
    Herbort, O. and Butz, M. (2011). The continuous end-state comfort effect: weighted integration of multiple biases. Psychological Research, pages 1–19.Google Scholar
  67. [67]
    Ijspeert, A. J., Nakanishi, J., and Schaal, S. (2003). Learning attractor landscapes for learning motor primitives. In Advances in Neural Information Processing Systems 15, volume 15, pages 1547–1554.Google Scholar
  68. [68]
    Ivaldi, S., Baglietto, M., Metta, G., and Zoppoli, R. (2009). Nonlinear Model Predictive Control (Magni, L. et al. Eds), vol. 384, An application of receding-horizon neural control in humanoid robotics, pages 541–550. LNCIS, Springer.Google Scholar
  69. [69]
    Ivaldi, S., Fumagalli, M., Nori, F., Baglietto, M., and Metta, G. (2010). Approximate optimal control for reaching and trajectory planning in a humanoid robot. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 1290–1296, Taipei, Taiwan.Google Scholar
  70. [70]
    Ivaldi, S., Fumagalli, M., Randazzo, M., Nori, F., Metta, G., and Sandini, G. (2011). Computing robot internal/external wrenches by means of inertial, tactile and F/T sensors: theory and implementation on the iCub. In 11th IEEE-RAS Int.Conf. on Humanoid Robots, pages 521–528.Google Scholar
  71. [71]
    Izawa, J., Rane, T., Donchin, O., and Shadmehr, R. (2008). Motor adaptation as a process of reoptimization. The Journal of Neuroscience, 28(11):2883–2891.CrossRefGoogle Scholar
  72. [72]
    Kadiallah, A., Liaw, G., Burdet, E., Kawato, M., and Franklin, D. W. (2008). Impedance control is tuned to multiple directions of movement. In IEEE Int. Eng. Med. Biol. Soc. Conf., pages 5358-5361.Google Scholar
  73. [73]
    Kajita, S., Kanehiro, F., Kaneko, K., Fujiwara, K., Harada, K., Yokoi, K., and Hirukawa, H. (2003). Resolved momentum control: Humanoid motion planning based on the linear and angular momentum. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, volume 2, pages 1644–1650.Google Scholar
  74. [74]
    Kaneko, Y., Nakano, E., Osu, R., Wada, Y., and Kawato, M. (2005). Trajectory formation based on the minimum commanded torque change model using euler-poisson equation. Systems and Computers in Japan, 36:92–103.CrossRefGoogle Scholar
  75. [75]
    Kanoun, O., Yoshida, E., and Laumond, J.-P. (2009). An optimization formulation for footsteps planning. In IEEE-RAS Int. Conf. on Humanoid Robots, pages 202–207, Paris, France.Google Scholar
  76. [76]
    Kappen, H. J. (2005). A linear theory for control of non-linear stochastic systems. Physical Review Letters, 95:200–201.MathSciNetCrossRefGoogle Scholar
  77. [77]
    Kim, Y. H., Lewis, F. L., and Dawson, D. M. (2000). Intelligent optimal control of robotic manipulators using neural networks. Automatica, 36:1355–1364.MathSciNetzbMATHCrossRefGoogle Scholar
  78. [78]
    Kirk, D. E. (1970). Optimal control theory: An Introduction. Prentice-Hall, New Jersey.Google Scholar
  79. [79]
    Kober, J. and Peters, J. (2008). Policy search for motor primitives in robotics. Advances in Neural Information Processing Systems (NIPS), pages 1–8.Google Scholar
  80. [80]
    Kodl, J., Ganesh, G., and Burdet, E. (2011). The CNS Stochastically Selects Motor Plan Utilizing Extrinsic and Intrinsic Representations. PLoS one, 6(9):e24229.CrossRefGoogle Scholar
  81. [81]
    Konczak, J. and Dichgans, J. (1997). The development toward stereotypic arm kinematics during reaching in the first 3 years of life. Experimental Brain Research, 117:346–354.CrossRefGoogle Scholar
  82. [82]
    Kormushev, P., Calinon, S., and Caldwell, D. (2010). Robot motor skill coordination with em-based reinforcement learning. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 3232–3237.Google Scholar
  83. [83]
    Krstic, M. (2009). Inverse optimal adaptive control: the interplay between update laws, control laws, and lyapunov functions. In American Control Conf., pages 1250–1255.Google Scholar
  84. [84]
    Kuo, A. (2005). An optimal state estimation model of sensory integration in human postural balance. Journal of Neural Engineering, 2:S235–S249.CrossRefGoogle Scholar
  85. [85]
    Lacquaniti, F., Terzuolo, C., and Viviani, P. (1983). The law relating kinematic and figural aspects of drawing movements. Acta Psychologica, 54:115–130.CrossRefGoogle Scholar
  86. [86]
    Lan, N. and Crago, P. E. (1994). Optimal control of antagonistic muscle stiffness during voluntary movements. Biological cybernetics, 71(2):123–135.zbMATHCrossRefGoogle Scholar
  87. [87]
    Lengagne, S., Ramdani, N., and Fraisse, P. (2009). Planning and fast re-planning of safe motions for humanoid robots: Application to a kicking motion. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, pages 441–446.Google Scholar
  88. [88]
    Lenzi, T., Vitiello, N., McIntyre, J., Roccella, S., and Carrozza, M. C. (2011). A robotic model to investigate human motor control. Biological Cybernetics, 105(1):1–19.zbMATHCrossRefGoogle Scholar
  89. [89]
    Li, W. and Todorov, E. (2004). Iterative linear quadratic regulator applied to nonlinear biological movement systems. In 1st Int. Conf. on Informatics in Control, Automation and Robotics, pages 222–229.Google Scholar
  90. [90]
    Lockhart, D. and Ting, L. (2007). Optimal sensorimotor transformations for balance. Nature Neuroscience, 10(10):1329–1336.CrossRefGoogle Scholar
  91. [91]
    MacKenzie, I. S. (1992). Fitts’ law as a research and design tool in human-computer interaction. Human-Computer Interaction, 7:91–139.CrossRefGoogle Scholar
  92. [92]
    Marin, D. and Sigaud, O. (2012). Towards fast and adaptive optimal control policies for robots: A direct policy search approach. In Proceedings Robotica, pages 21–26.Google Scholar
  93. [93]
    Matsui, T. (2008). A new optimal control model for reproducing two-point reaching movements of human three-joint arm with wrist joint’s freezing mechanism. In IEEE Int. Conf. on Robotics and Biomimetics, pages 383–388.Google Scholar
  94. [94]
    Matsui, T., Honda, M., and Nakazawa, N. (2006). A new optimal control model for reproducing human arm’s two-point reaching movements: a modified minimum torque change model. In IEEE Int. Conf. on Robotics and Biomimetics, pages 1541–1546.Google Scholar
  95. [95]
    Matsui, T., Takeshita, K., and Shibusawa, T. (2009). Effectiveness of human three-joint arm’s optimal control model characterized by hand-joint’s freezing mechanism in reproducing constrained reaching movement characteristics. In ICROS-SICE Int. Joint Conf., pages 1206–1211.Google Scholar
  96. [96]
    Mettin, U., Shiriaev, A. S., Freidovich, L. B., and Sampei, M. (2010). Optimal ball pitching with an underactuated model of a human arm. In IEEE Int. Conf. on Robotics and Automation, pages 5009–5014.Google Scholar
  97. [97]
    Mistry, M., Theodorou, E., Liaw, G., Yoshioka, T., Schaal, S., and Kawato, M. (2008). Adaptation to a sub-optimal desired trajectory. In Society for Neuroscience — Symp. on Advances in Computational Motor Control, Washington DC, USA.Google Scholar
  98. [98]
    Mitrovic, D., Klanke, S., and Vijayakumar, S. (2010). From Motor to Interaction Learning in Robotics (Sigaud, O. and Peters, J. Eds), vol. 264, Adaptive Optimal Feedback Control with Learned Internal Dynamics Models, pages 65–84. Springer-Verlag.CrossRefGoogle Scholar
  99. [99]
    Mombaur, K., Laumond, J.-P., and Yoshida, E. (2008). An optimal control model unifying holonomic and nonholonomic walking. In 8th IEEE-RAS Int. Conf. on Humanoid Robots, Daejeon, Korea.Google Scholar
  100. [100]
    Mombaur, K., Truong, A., and Laumond, J.-P. (2010). From human to humanoid locomotion: an inverse optimal control approach. Autonomous Robots, 28:369–383.CrossRefGoogle Scholar
  101. [101]
    Morasso, P. (1983). Three dimensional arm trajectories. Biological Cybernetics, 48:187–194.CrossRefGoogle Scholar
  102. [102]
    Mugan, J. and Kuipers, B. (2009). Autonomously learning an action hierarchy using a learned qualitative state representation. In Int. Joint Conf. on Artificial Intelligence.Google Scholar
  103. [103]
    Mussa-Ivaldi, F. A., Giszter, S. F., and Bizzi, E. (1994). Linear combinations of primitives in vertebrate motor control. Proc. National Academy of Sciences USA, 91(16):7534–7538.CrossRefGoogle Scholar
  104. [104]
    Nagengast, A. J., Braun, D. A., and Wolpert, D. M. (2011). Risksensitivity and the mean-variance trade-off: decision making in sensorimotor control. Proceedings Biological Sciences / The Royal Society, 278(1716):2325–2332.CrossRefGoogle Scholar
  105. [105]
    Nakamura, Y. (1991). Advanced Robotics: redundancy and optimization. Addison Wesley.Google Scholar
  106. [106]
    Nakano, E., Imamizu, H., Osu, R., Uno, Y., Gomi, H., Yoshioka, T., and Kawato, M. (1999). Quantitative examinations of internal representations for arm trajectory planning: Minimum commanded torque change model. Journal of Neurophysiology, 81:2140–2155.Google Scholar
  107. [107]
    Nakaoka, S., Nakazawa, A., Yokoi, K., Hirukawa, H., and Ikeuchi, K. (2003). Generating whole body motions for a biped humanoid robot from captured human dances. In IEEE Int. Conf. on Robotics and Automation, volume 3, pages 3905–3910.Google Scholar
  108. [108]
    Nelson, W. L. (1983). Physical principles for economies of skilled movements. Biological Cybernetics, 46:135–147.zbMATHCrossRefGoogle Scholar
  109. [109]
    Nishii, J. and Murakami, T. (2002). Energetic optimality of arm trajectory. In Int. Conf. on Biomechanics of Man, pages 30-33.Google Scholar
  110. [110]
    Nori, F. and Frezza, R. (2005). A control theory approach to the analysis and synthesis of the experimentally observed motion primitives. Biological Cybernetics, 93(5):323–342.MathSciNetzbMATHCrossRefGoogle Scholar
  111. [111]
    Oksendal, B. (1995). Stochastic Differential Equations. Springer Berlin, 4th edition.Google Scholar
  112. [112]
    Parker, G. A. and Smith, J. M. (1990). Optimality theory in evolutionary biology. Nature, 348(6296):27–33.CrossRefGoogle Scholar
  113. [113]
    Pattacini, U., Nori, F., Natale, L., Metta, G., and Sandini, G. (2010). An experimental evaluation of a novel minimum-jerk cartesian controller for humanoid robots. In IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Taipei, Taiwan.Google Scholar
  114. [114]
    Peters, J. and Schaal, S. (2007). Natural actor-critic. Neurocomputing, 71:1180–1190.CrossRefGoogle Scholar
  115. [115]
    Peters, J. and Schaal, S. (2008). Reinforcement learning of motor skills with policy gradients. Neural networks, 21:682–97.CrossRefGoogle Scholar
  116. [116]
    Pollard, N. S., Hodgins, J. K., Riley, M. J., and Atkeson, C. G. (2002). Adapting human motion for the control of a humanoid robot. In IEEE Int. Conf. on Robotics and Automation, volume 2, pages 1390–1397.Google Scholar
  117. [117]
    Pontryagin, L. S., Boltyanskii, V. G., Gamkrelidze, R. V., and Mishchenko, E. F. (1964). The Mathematical Theory of Optimal Processes. Pergamon Press.Google Scholar
  118. [118]
    Pouget, A. and Snyder, L. (2000). Computational approaches to sensorimotor transformations. Nature Neuroscience, 3:1192–1198.CrossRefGoogle Scholar
  119. [119]
    Pozzo, T., Berthoz, A., and Lefort, L. (1990). Head stabilisation during various locomotor tasks in humans. i. normal subjects. Experimental Brain Research, 82:97–106.CrossRefGoogle Scholar
  120. [120]
    Ribas-Fernandes, J. J. F., Solway, A., Diuk, C., McGuire, J. T., Barto, A. G., Niv, Y., and Botvinick, M. M. (2011). A neural signature of hierarchical reinforcement learning. Neuron, 71(2):370–379.CrossRefGoogle Scholar
  121. [121]
    Richardson, M. J. E. and Flash, T. (2000). On the emulation of natural movements by humanoid robots. In IEEE-RAS Int. Conf. on Humanoids Robots.Google Scholar
  122. [122]
    Richardson, M. J. E. and Flash, T. (2002). Comparing smooth arm movements with the two-thirds power law and the related segmented-control hypothesis. Journal of Neuroscience, 22(18):8201–8211.Google Scholar
  123. [123]
    Rigoux, L., Sigaud, O., Terekhov, A., and Guigon, E. (2010). Movement duration as an emergent property of reward directed motor control. In Annual Symp. Advances in Computational Motor Control.Google Scholar
  124. [124]
    Rubinstein, R. Y. (1997). Optimization of computer simulation models with rare events. European Journal of Operational Research, 99(1):89–112.CrossRefGoogle Scholar
  125. [125]
    Salini, J., Padois, V., and Bidaud, P. (2011). Synthesis of complex humanoid whole-body behavior: a focus on sequencing and tasks transitions. In IEEE Int. Conf. on Robotics and Automation, pages 1283–1290.Google Scholar
  126. [126]
    Sastry, S. and Bodson, M. (1994). Adaptive Control: Stability, Convergence, and Robustness. Advanced Reference Series (Engineering). Prentice-Hall.Google Scholar
  127. [127]
    Schaal, S. (1997). Advances in Neural Information Processing Systems (Mozer, M.C. et al. Eds), Learning from demonstration, pages 1040–1046. MIT Press.Google Scholar
  128. [128]
    Schaal, S., Peters, J., Nakanishi, J., and Ijspeert, A. J. (2003). Learning movement primitives. In Int. Symp. on Robotics Research (ISRR), pages 561-572.Google Scholar
  129. [129]
    Schaal, S. and Schweighofer, N. (2005). Computational motor control in humans and robots. Current Opinion in Neurobiology, 15:675–682.CrossRefGoogle Scholar
  130. [130]
    Scheidt, R. A., Reinkensmeyer, D. J., Conditt, M. A., Rymer, W. Z., and Mussa-Ivaldi, F. A. (2000). Persistence of motor adaptation during constrained, multi-joint, arm movements. Journal of Neurophysiology, 84(2):853–862.Google Scholar
  131. [131]
    Schmidt, R. A. (1975). A schema theory of discrete motor skill learning. Psychological review, 82(4):225.CrossRefGoogle Scholar
  132. [132]
    Schmidt, R. A., Zelaznik, H., Hawkins, B., Frank, J. S., and Quinn, J. T. (1979). Motor output variability: a theory for the accuracy of rapid motor acts. Psychol. Rev., 47:415–51.CrossRefGoogle Scholar
  133. [133]
    Schöner, G. and Kelso, J. A. (1988). Dynamic pattern generation in behavioral and neural systems. Science, 239(4847):1513.CrossRefGoogle Scholar
  134. [134]
    Schultz, G. and Mombaur, K. (2010). Modeling and optimal control of human-like running. IEEE/ASME Trans. on Mechatronics, 15(5):783–792.CrossRefGoogle Scholar
  135. [135]
    Scott, S. (2004). Optimal feedback control and the neural basis of volitional motor control. Nature Reviews Neuroscience, 5:532–546.CrossRefGoogle Scholar
  136. [136]
    Scott Kelso, J. A. (1982). Human motor behavior: an introduction. Lawrence Erlbaum Associates.Google Scholar
  137. [137]
    Seki, H. and Tadakuma, S. (2004). Minimum jerk control of power assisting robot based on human arm behavior characteristics. In IEEE Int. Conf. on System, Man and Cybernetics, volume 1, pages 722–727.Google Scholar
  138. [138]
    Sentis, L. and Khatib, O. (2005). Synthesis of whole-body behaviors through hierarchical control of behavioral primitives. The Int. Journal of Humanoid Robotics, 2(4):505–518.CrossRefGoogle Scholar
  139. [139]
    Shadmehr, R. and Krakauer, J. W. (2008). A computational neuroanatomy for motor control. Experimental Brain Research, 185:359–381.CrossRefGoogle Scholar
  140. [140]
    Shadmehr, R., Orban de Xivry, J.-J., Xu-Wilson, M., and Shih, T.-Y. (2010). Temporal discounting of reward and the cost of time in motor control. The Journal of Neuroscience, 30(31):10507–10516.CrossRefGoogle Scholar
  141. [141]
    Shadmehr, R. and Wise, S. (2005). The Computational Neurobiology of Reaching and Pointing: a foundation for Motor Learning. MIT Press.Google Scholar
  142. [142]
    Shiller, Z. and Dubowsky, S. (1991). On computing the global time-optimal motions of robotic manipulators in the presence of obstacles. IEEE Transactions on Robotics and Automation, 7(6):785–797.CrossRefGoogle Scholar
  143. [143]
    Sigaud, O., Salaün, C., and Padois, V. (2011). On-line regression algorithms for learning mechanical models of robots: a survey. Robotics and Autonomous Systems, 51:1117–1125.Google Scholar
  144. [144]
    Simmons, G. and Demiris, Y. (2005). Optimal robot arm control using the minimum variance model. Journal of Robotic Systems, 22(11):677–690.zbMATHCrossRefGoogle Scholar
  145. [145]
    Stengel, R. F. (1994). Optimal Control and Estimation. Dover Publications.Google Scholar
  146. [146]
    Stulp, F., Buchli, J., Ellmer, A., Mistry, M., Theodorou, E., and Schaal, S. (2011). Reinforcement learning of impedance control in stochastic force fields. In IEEE Int. Conf. on Development and Learning, volume 2,pages 1–6.CrossRefGoogle Scholar
  147. [147]
    Stulp, F. and Sigaud, O. (2012). Path integral policy improvement with covariance matrix adaptation. In 29th Int. Conf. on Machine Learning.Google Scholar
  148. [148]
    Sun, F. C., Li, H. X., and Li, L. (2002). Robot discrete adaptive control based on dynamic inversion using dynamical neural networks. Automatica, 38:1977–1983.zbMATHCrossRefGoogle Scholar
  149. [149]
    Tanaka, H., Krakauer, J. W., and Qian, N. (2006). An optimization principle for determining movement duration. Journal of Neurophysiology, 95:38750–3886.CrossRefGoogle Scholar
  150. [150]
    Terekhov, A. V. and Zatsiorsky, V. M. (2011). Analytical and numerical analysis of inverse optimization problems: conditions of uniqueness and computational methods. Biological Cybernetics, 104:75–93.MathSciNetzbMATHCrossRefGoogle Scholar
  151. [151]
    Theodorou, E., Buchli, J., and Schaal, S. (2010). Reinforcement learning of motor skills in high dimensions: a path integral approach. In Int. Conf. on Robotics and Automation, pages 2397–2403. IEEE.Google Scholar
  152. [152]
    Thrommershäuser, J., Maloney, L. T., and Landy, M. S. (2008). Decision making, movement planning, and statistical decision theory. Trends in Cognitive Sciences, 12(8):291–297.CrossRefGoogle Scholar
  153. [153]
    Tlalolini, D., Chevallereau, C., and Aoustin, Y. (2011). Humanlike walking: Optimal motion of a bipedal robot with toerotation motion. IEEE/ASME Transactions on Mechatronics, 16(2):310–320.CrossRefGoogle Scholar
  154. [154]
    Todorov, E. (2004). Optimality principles in sensorimotor control. Nature Neuroscience, 7(9):907–915.CrossRefGoogle Scholar
  155. [155]
    Todorov, E. (2005). Stochastic optimal control and estimation methods adapted to the noise characteristics of the sensorimotor system. Neural computation, 17(5):1084–1108.MathSciNetzbMATHCrossRefGoogle Scholar
  156. [156]
    Todorov, E. (2009a). Compositionality of optimal control laws. Advances in Neural Information Processing Systems, 3:2–6.Google Scholar
  157. [157]
    Todorov, E. (2009b). Effcient computation of optimal actions. Proc. Natl. Acad. Sci. USA, 106(28):11478–11483.zbMATHCrossRefGoogle Scholar
  158. [158]
    Todorov, E. and Jordan, M. I. (2002). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5(11):1226–1235.CrossRefGoogle Scholar
  159. [159]
    Todorov, E. and Jordan, M. I. (2003). A minimal intervention principle for coordinated movement. In Advances in Neural Information Processing Systems, volume 15, pages 27–34.Google Scholar
  160. [160]
    Todorov, E. and Li, W. (2005). A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In American Control Conf., pages 300-306.Google Scholar
  161. [161]
    Toussaint, M., Gienger, M., and Goerick, C. (2007). Optimization of sequential attractor-based movement for compact behaviour generation. In IEEE-RAS Int. Conf. on Humanoid Robots, pages 122–129.Google Scholar
  162. [162]
    Tuan, T., Souères, P., Taïx, M., and Guigon, E. (2008). A principled approach to biological motor control for generating humanoid robot reaching movements. In IEEE Int. Conf. Biomedical Robotics and Biomechatronics, pages 783–788.Google Scholar
  163. [163]
    Uno, Y., Kawato, M., and Suzuki, R. (1989). Formation and control of optimal trajectory in human multijoint arm movement. Biological Cybernetics, 61:89–101.CrossRefGoogle Scholar
  164. [164]
    Vidyasagar, M. (1987). Control System Synthesis: A factorization approach. MIT Press.Google Scholar
  165. [165]
    Vijayakumar, S. and Schaal, S. (2000). Locally Weighted Projection Regression: An O(n) Algorithm for Incremental Real Time Learning in High Dimensional Space. In 7th Int. Conf. on Machine Learning, pages 1079–1086.Google Scholar
  166. [166]
    Viviani, P. (1986). Generation and modulation of action patterns (Heuer, H. and Fromm, C. Eds), Do units of motor action really exist?, pages 201–216. Springer-Verlag.Google Scholar
  167. [167]
    Viviani, P. and Flash, T. (1995). Minimum-jerk, two-thirds power law, and isochrony: converging approaches to movement planning. Journal of Experimental Psychology: Human Perception and Performance, 21:32–53.CrossRefGoogle Scholar
  168. [168]
    Viviani, P. and Stucchi, N. (1992). Biological movements look constant: Evidence of motor perceptual interactions. Journal of Experimental Psychology: Human Perception and Performance, 18:603–623.CrossRefGoogle Scholar
  169. [169]
    Volkinshtein, D. and Meir, R. (2011). Delayed feedback control requires an internal forward model. Biological cybernetics, 105(1):41–53.MathSciNetzbMATHCrossRefGoogle Scholar
  170. [170]
    Wächter, A. and Biegler, L. (2006). On the implementation of a primal-dual interior point filter line search algorithm for largescale nonlinear programming. Mathematical Programming, 106:25–57.MathSciNetzbMATHCrossRefGoogle Scholar
  171. [171]
    Wada, Y., Kaneko, Y., Nakano, E., Osu, R., and Kawato, M. (2001). Quantitative examinations for multi joint arm trajectory planning-using a robust calculation algorithm of the minimum commanded torque change trajectory. Neural Networks, 14(4–5):381–393.CrossRefGoogle Scholar
  172. [172]
    Wada, Y., Yamanaka, K., Soga, Y., Tsuyuki, K., and Kawato, M. (2006). Can a kinetic optimization criterion predict both arm trajectory and final arm posture? Annual Int. Conf. of the IEEE Eng. Med. Biol. Society, 1:1197–200.Google Scholar
  173. [173]
    Whitman, E. C. and Atkeson, C. G. (2009). Control of a walking biped using a combination of simple policies. In IEEE-RAS Int. Conf. on Humanoid Robots, pages 520–527, Paris, France.Google Scholar
  174. [174]
    Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256.zbMATHGoogle Scholar
  175. [175]
    Wolpert, D. M. and Flanagan, J. R. (2001). Motor prediction. Current Biology, 18(18):729–732.CrossRefGoogle Scholar
  176. [176]
    Wolpert, D. M., Ghahramani, Z., and Jordan, M. I. (1995). Are arm trajectories planned in kinematic or dynamic coordinates? an adaptation study. Experimental Brain Research, 103:460–470.CrossRefGoogle Scholar
  177. [177]
    Wolpert, D. M., Miall, R. C., and Kawato, M. (1998). Internal models in the cerebellum. Trends in Cognitive Sciences, 2(9):338–347.CrossRefGoogle Scholar
  178. [178]
    Yoshida, E., Esteves, C., Kanoun, O., Poirier, M., Mallet, A., Laumond, J.-P., and Yokoi, K. (2010). Motion Planning for Humanoid Robots (Harada, K. et al. Eds), Planning Whole-body Humanoid Locomotion, Reaching, and Manipulation, pages 99–128. Springer.Google Scholar
  179. [179]
    Zhao, H. and Chen, D. (1996). Optimal motion planning for flexible space robots. In IEEE Int. Conf. on Robotics and Automation, pages 393–398.Google Scholar

Copyright information

© Versita Warsaw and Springer-Verlag Wien 2012

Authors and Affiliations

  • Serena Ivaldi
    • 1
    Email author
  • Olivier Sigaud
    • 1
  • Bastien Berret
    • 2
  • Francesco Nori
    • 2
  1. 1.Institut des Systèmes Intelligents et de Robotique — CNRS UMR 7222Université Pierre et Marie CurieParis CEDEX 05France
  2. 2.Robotics, Brain and Cognitive Sciences Dept. Istituto Italiano di TecnologiaGenovaItaly

Personalised recommendations