A Framework for Learning in Humanoid Simulated Robots

  • Esther Luna Colombini
  • Alexandre da Silva Simöes
  • Antônio Cesar Germano Martins
  • Jackson Paul Matsuura
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5001)


One of the most important characteristics of intelligent activity is the ability to change behaviour according to many forms of feedback. Through learning an agent can interact with its environment to improve its performance over time. However, most of the techniques known that involves learning are time expensive, i.e., once the agent is supposed to learn over time by experimentation, the task has to be executed many times. Hence, high fidelity simulators can save a lot of time. In this context, this paper describes the framework designed to allow a team of real RoboNova-I humanoids robots to be simulated under USARSim environment. Details about the complete process of modeling and programming the robot are given, as well as the learning methodology proposed to improve robot’s performance. Due to the use of a high fidelity model, the learning algorithms can be widely explored in simulation before adapted to real robots.


Real Robot Reinforcement Learn Algorithm Static Mesh Legged Robot Forward Walk 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Wang, J.: USARSim V2.0.2 Manual: A Game-based Simulation of the NIST Reference Arenas (2006)Google Scholar
  2. 2.
    Zaratti, M., Fratarcangeli, M., Iocchi, L.: A 3d simulator of multiple legged robots based on usarsim. In: Lakemeyer, G., Sklar, E., Sorrenti, D.G., Takahashi, T. (eds.) RoboCup 2006: Robot Soccer World Cup X. LNCS (LNAI), vol. 4434, pp. 13–24. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  3. 3.
    Greggio, N., Silvestri, G., Antonello, S., Menegatti, E., Pagello, E.: A 3d model of a humanoid for usarsim simulator. In: First Workshop on Humanoid Soccer Robots, Genova, pp. 17–24 (2006)Google Scholar
  4. 4.
    Sutton, R.S., Barto, A.G.: Reinforcement learning: an introduction. MIT Press, USA (1998)Google Scholar
  5. 5.
    Watkins, C.: Learning from delayed rewards. PhD thesis, King’s College (1998)Google Scholar
  6. 6.
    Howden, N., Renquist, R.: Jack intelligent agents - summary of an agent infrastructure. In: 5th Int. Conf. on Autonomous Agents, Montreal, Canada (2001)Google Scholar
  7. 7.
    McCallum, A.K.: Reinforcement Learning with Selective Perception and Hidden State. PhD thesis, University of Rochester, New York (1996)Google Scholar
  8. 8.
    Reis, L.P., Lau, N.: Fc portugal team description: Robocup 2000 simulation league champion. In: Stone, P., Balch, T., Kraetzschmar, G.K. (eds.) RoboCup 2000. LNCS (LNAI), vol. 2019, pp. 29–40. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  9. 9.
    The Unreal Engine Site (2007),

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Esther Luna Colombini
    • 1
  • Alexandre da Silva Simöes
    • 2
  • Antônio Cesar Germano Martins
    • 2
  • Jackson Paul Matsuura
    • 1
  1. 1.Technological Institute of Aeronautics (ITA)Itandroids Research GroupBrazil
  2. 2.Automation and Integrated Systems Group (GASI)São Paulo State University (UNESP)Brazil

Personalised recommendations