Emergency-Response Locomotion of Hexapod Robot with Heuristic Reinforcement Learning Using Q-Learning

  • Ming-Chieh Yang
  • Hooman SamaniEmail author
  • Kening Zhu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11659)


The locomotion of legged robot is often controlled by predefined gaits, and this approach works well when all joints and motors are operating normally. However, walking legged robots usually have high risk of being damaged during operation, causing the breakdown of the robotic joints. In this paper, we introduce a reinforcement learning based approach for the legged robot to generate real-time locomotion response to the emergence of locomotion breakdown. Our approach detects the functionality of the available joints, substitutes the pre-defined gaits with proper gait function accordingly, and upgrades the gait-generation function by Q-Learning for the proper locomotion.


Reinforcement learning Q-Learning Hexapod robot Emergency response 


  1. 1.
    Eason, G., Noble, B., Sneddon, I.N.: On certain integrals of Lipschitz-Hankel type involving products of Bessel functions. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Sci. 247(935), 529–551 (1955)MathSciNetCrossRefGoogle Scholar
  2. 2.
    Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning, vol. 135. MIT Press, Cambridge (1998)zbMATHGoogle Scholar
  3. 3.
    Watkins, C.J., Dayan, P.: Q-learning. Mach. Learn. 8(3–4), 279–292 (1992)zbMATHGoogle Scholar
  4. 4.
    Borkar, V.S., Meyn, S.P.: The ODE method for convergence of stochastic approximation and reinforcement learning. SIAM J. Control Optim. 38(2), 447–469 (2000)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Auslander, B., Lee-Urban, S., Hogg, C., Muñoz-Avila, H.: Recognizing the enemy: combining reinforcement learning with strategy selection using case-based reasoning. In: Althoff, K.-D., Bergmann, R., Minor, M., Hanft, A. (eds.) ECCBR 2008. LNCS (LNAI), vol. 5239, pp. 59–73. Springer, Heidelberg (2008). Scholar
  6. 6.
    Bianchi, R.A., Ribeiro, C.H., Costa, A.H.: Accelerating autonomous learning by using heuristic selection of actions. J. Heuristics 14(2), 135–168 (2008)CrossRefGoogle Scholar
  7. 7.
    Bianchi, R.A.C., Ros, R., Lopez de Mantaras, R.: Improving reinforcement learning by using case based heuristics. In: McGinty, L., Wilson, David C. (eds.) ICCBR 2009. LNCS (LNAI), vol. 5650, pp. 75–89. Springer, Heidelberg (2009). Scholar
  8. 8.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, Cambridge (2018)zbMATHGoogle Scholar
  9. 9.
    Bianchi, R.A.C.: Using heuristics to accelerate reinforcement learning algorithms. Dissertation Ph.D. thesis, University of São Paulo (2004)Google Scholar
  10. 10.
    Yu, T.K., Yang, M.C., Samani, H.: Reinforcement learning and convolutional neural network system for firefighting rescue robot. In: MATEC Web of Conferences, vol. 161, p. 03028 (2018)CrossRefGoogle Scholar
  11. 11.
    Samani, H., Zhu, R.: Robotic automated external defibrillator ambulance for emergency medical service in smart cities. IEEE Access 4, 268–283 (2016)CrossRefGoogle Scholar
  12. 12.
    Samani, H.: Cognitive Robotics. CRC Press, Boca Raton (2015)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Electrical EngineeringNational Taipei UniversityNew Taipei CityTaiwan
  2. 2.School of Creative MediaCity University of Hong KongKowloon TongHong Kong

Personalised recommendations