Skip to main content

Advertisement

SpringerLink
Log in

Human locomotion with reinforcement learning using bioinspired reward reshaping strategies

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Recent learning strategies such as reinforcement learning (RL) have favored the transition from applied artificial intelligence to general artificial intelligence. One of the current challenges of RL in healthcare relates to the development of a controller to teach a musculoskeletal model to perform dynamic movements. Several solutions have been proposed. However, there is still a lack of investigations exploring the muscle control problem from a biomechanical point of view. Moreover, no studies using biological knowledge to develop plausible motor control models for pathophysiological conditions make use of reward reshaping. Consequently, the objective of the present work was to design and evaluate specific bioinspired reward function strategies for human locomotion learning within an RL framework. The deep deterministic policy gradient (DDPG) method for a single-agent RL problem was applied. A 3D musculoskeletal model (8 DoF and 22 muscles) of a healthy adult was used. A virtual interactive environment was developed and simulated using opensim-rl library. Three reward functions were defined for walking, forward, and side falls. The training process was performed with Google Cloud Compute Engine. The obtained outcomes were compared to the NIPS 2017 challenge outcomes, experimental observations, and literature data. Regarding learning to walk, simulated musculoskeletal models were able to walk from 18 to 20.5 m for the best solutions. A compensation strategy of muscle activations was revealed. Soleus, tibia anterior, and vastii muscles are main actors of the simple forward fall. A higher intensity of muscle activations was also noted after the fall. All kinematics and muscle patterns were consistent with experimental observations and literature data. Regarding the side fall, an intensive level of muscle activation on the expected fall side to unbalance the body was noted. The obtained outcomes suggest that computational and human resources as well as biomechanical knowledge are needed together to develop and evaluate an efficient and robust RL solution. As perspectives, current solutions will be extended to a larger parameter space in 3D. Furthermore, a stochastic reinforcement learning model will be investigated in the future in scope with the uncertainties of the musculoskeletal model and associated environment to provide a general artificial intelligence solution for human locomotion learning.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Notes

  1. https://www.crowdai.org/challenges/nips-2017-learning-to-run

  2. https://www.crowdai.org/challenges/nips-2018-ai-for-prosthetics-challenge

  3. https://www.aicrowd.com/challenges/neurips-2019-learning-to-move-walk-around

  4. http://osim-rl.stanford.edu/

References

  1. Holmes SJ (1911) The beginnings of intelligence. Science 33(848):473–480. https://doi.org/10.1126/science.33.848.473

    Article  CAS  PubMed  Google Scholar 

  2. Sternberg RJ (1985) Human Intelligence: The Model Is the Message. Science 230(4730):1111–1118. https://doi.org/10.1126/science.230.4730.1111

    Article  CAS  PubMed  Google Scholar 

  3. Wang W, Pedretti G, Milo V, Carboni R, Calderoni A, Ramaswamy N, Spinelli AS, Ielmini D (2018) Learning of spatiotemporal patterns in a spiking neural network with resistive switching synapses. Sci Adv 4(9):eaat4752. https://doi.org/10.1126/sciadv.aat4752

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  4. McCulloch WS, Pitts W (1943) A logical calculus of the ideas immanent in nervous activity. Bull Math Biophys 5:115–133

    Article  Google Scholar 

  5. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521:436–444. https://doi.org/10.1038/nature14539

    Article  CAS  PubMed  Google Scholar 

  6. Zhang Q, Liu Y, Gong C, Chen Y, Yu H (2020) Applications of deep learning for dense scenes analysis in agriculture: a review. Sensors (Basel) 20(5)):E1520. https://doi.org/10.3390/s20051520

    Article  Google Scholar 

  7. Kruger N, Janssen P, Kalkan S, Lappe M, Leonardis A, Piater J, Rodriguez-Sanchez AJ, Wiskott L (2013) Deep hierarchies in the primate visual cortex: what can we learn for computer vision? IEEE Trans Pattern Anal Mach Intell 35(8):1847–1871

    Article  Google Scholar 

  8. Vinyals O, Babuschkin I, Czarnecki WM, Mathieu M, Dudzik A, Chung J, Choi DH, Powell R, Ewalds T, Georgiev P, Oh J, Horgan D, Kroiss M, Danihelka I, Huang A, Sifre L, Cai T, Agapiou JP, Jaderberg M, Vezhnevets AS, Leblond R, Pohlen T, Dalibard V, Budden D, Sulsky Y, Molloy J, Paine TL, Gulcehre C, Wang Z, Pfaff T, Wu Y, Ring R, Yogatama D, Wünsch D, McKinney K, Smith O, Schaul T, Lillicrap T, Kavukcuoglu K, Hassabis D, Apps C, Silver D (2019) Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575:350–354. https://doi.org/10.1038/s41586-019-1724-z

    Article  CAS  PubMed  Google Scholar 

  9. Jin W, Fatehi M, Abhishek K, Mallya M, Toyota B, Hamarneh G (2020) Artificial intelligence in glioma imaging: challenges and advances. J Neural Eng 17:021002. https://doi.org/10.1088/1741-2552/ab8131

    Article  PubMed  Google Scholar 

  10. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  CAS  PubMed  Google Scholar 

  11. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  12. I Goodfellow, J Pouget-Abadie, M Mirza, B Xu, D Warde-Farley, S Ozair, A Courville, Y Bengio. Generative adversarial networks. Proceedings of the International Conference on Neural Information Processing Systems (NIPS 2014). pp. 2672–2680, 2014.

  13. Pratt LY (1993) Discriminability-based transfer between neural networks. NIPS Conference: Advances in Neural Information Processing Systems 5. Morgan Kaufmann Publishers. pp. 204–211

  14. Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–285

    Article  Google Scholar 

  15. Feher da Silva C, Victorino CG, Caticha N et al (2017) Exploration and recency as the main proximate causes of probability matching: a reinforcement learning analysis. Sci Rep 7:15326. https://doi.org/10.1038/s41598-017-15587-z

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  16. Li J, Dong D, Wei Z et al (2020) Quantum reinforcement learning during human decision-making. Nat Hum Behav 4:294–307. https://doi.org/10.1038/s41562-019-0804-2

    Article  PubMed  Google Scholar 

  17. Pesce E, Montana G (2020) Improving coordination in small-scale multi-agent deep reinforcement learning through memory-driven communication. Mach Learn 109:1727–1747. https://doi.org/10.1007/s10994-019-05864-5

    Article  Google Scholar 

  18. Gottesman O, Johansson F, Komorowski M, Faisal A, Sontag D, Doshi-Velez F, Celi LA (2019 Jan) Guidelines for reinforcement learning in healthcare. Nat Med 25(1):16–18. https://doi.org/10.1038/s41591-018-0310-5

    Article  CAS  PubMed  Google Scholar 

  19. Maia T, Frank M (2011) From reinforcement learning models to psychiatric and neurological disorders. Nat Neurosci 14:154–162. https://doi.org/10.1038/nn.2723

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Jonsson A (2019 Feb) Deep reinforcement learning in medicine. Kidney Dis (Basel) 5(1):18–22. https://doi.org/10.1159/000492670

    Article  Google Scholar 

  21. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015 Feb 26) Human-level control through deep reinforcement learning. Nature. 518(7540):529–533. https://doi.org/10.1038/nature14236

    Article  CAS  PubMed  Google Scholar 

  22. Ł Kidziński, SP Mohanty, C Ong, Z Huang, S Zhou, A Pechenko, A Stelmaszczyk, P Jarosik, M Pavlov, S Kolesnikov, S Plis, Z Chen, Z Zhang, J Chen, J Shi, Z Zheng, C Yuan, Z Lin, H Michalewski, P Miłoś, B Osiński, A Melnik, M Schilling, H Ritter, S Carroll, J Hicks, S Levine, M Salathé, S Delp. Learning to Run challenge solutions: adapting reinforcement learning methods for neuromusculoskeletal environments. Escalera S., Weimer M. (eds) The NIPS ‘17 Competition: Building Intelligent Systems. The Springer Series on Challenges in Machine Learning. Springer, Cham, arXiv:1804.00361, https://doi.org/10.1007/978-3-319-94042-7_7

  23. Kidziński Ł, Ong C, Mohanty SP, Hicks J, Carroll S, Zhou B, Zeng H, Wang F, Lian R, Tian H, Jaśkowski W, Andersen G, Lykkebø OR, Toklu NE, Shyam P, Srivastava RK, Kolesnikov S, Hrinchuk O, Pechenko A, Ljungström M, Wang Z, Hu X, Hu Z, Qiu M, Huang J, Shpilman A, Sosin I, Svidchenko O, Malysheva A, Kudenko D, Rane L, Bhatt A, Wang Z, Qi P, Yu Z, Peng P, Yuan Q, Li W, Tian Y, Yang R, Ma P, Khadka S, Majumdar S, Dwiel Z, Liu Y, Tumer E, Watson J, Salathé M, Levine S, Delp S (2020) Artificial intelligence for prosthetics: challenge solutions. In: Escalera S, Herbrich R (eds) The NeurIPS ‘18 competition. The Springer Series on Challenges in Machine Learning. Springer, Cham. https://doi.org/10.1007/978-3-030-29135-8_4

    Chapter  Google Scholar 

  24. B Zhou, H Zeng, F Wang, Y Li, H Tian. Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion. arXiv:1912.05328

  25. Kuehne H, Jhuang H, Garrote E, Poggio T, Serre T (2011) HMDB: a large video database for human motion recognition, 2011 International Conference on Computer Vision, Barcelona, pp. 2556-2563

  26. Kwolek B, Kepski M (2014) Human fall detection on embedded platform using depth maps and wireless accelerometer. Comput Methods Prog Biomed 117(3):489–501

    Article  Google Scholar 

  27. Bobick A, Davis J (2001) The recognition of human movement using temporal templates. IEEE Trans PAMI 23(3):257–267

    Article  Google Scholar 

  28. Heintz S, Gutierrez-Farewik EM (2007) Static optimization of muscle forces during gait in comparison to EMG-to-force processing approach. Gait Posture 26(2):279–288

    Article  Google Scholar 

  29. Meng L, Ceccarelli M, Yu Z, Chen X, Huang Q (2017) An experimental characterization of human falling down. Mech Sci 8:79–89. https://doi.org/10.5194/ms-8-79-2017

    Article  Google Scholar 

  30. Dao TT, Marin F, Pouletaut P, Aufaure P, Charleux F, Tho MCHB (2012) Estimation of accuracy of patient specific musculoskeletal modeling: case study on a post polio residual paralysis subject. Computer Method Biomech Biomed Eng 15(7):745–751

    Article  CAS  Google Scholar 

  31. Delp SL, Anderson FC, Arnold AS, Loan P, Habib A, John CT, Guendelman E, Thelen DG (2007) Opensim: opensource software to create and analyze dynamic simulations of movement. IEEE Trans Biomed Eng 54(11):1940–1950

    Article  Google Scholar 

  32. Pavol MJ, Owings TM, Foley KT, Grabiner MD (2001) Mechanisms leading to a fall from an induced trip in healthy older adults. J Gerontol Ser A Biol Med Sci 56:M428{M437. https://doi.org/10.1093/gerona/56.7.M428

    Article  Google Scholar 

  33. Erdemir A, McLean S, Herzog W, van den Bogert AJ (2007) Model-based estimation of muscle forces exerted during movements. Clin Biomech (Bristol, Avon) 22(2):131–154

    Article  Google Scholar 

  34. Thelen DG, Anderson FC (2006) Using computed muscle control to generate forward dynamic simulations of human walking from experimental data. J Biomech 39(6):1107–1115

    Article  Google Scholar 

  35. Esrafilian A, Stenroth L, Mononen ME, Tanska P, Avela J, Korhonen RK (2020) EMG-Assisted Muscle Force Driven Finite Element Model of the Knee Joint with Fibril-Reinforced Poroelastic Cartilages and Menisci. Sci Rep 10(1):3026. https://doi.org/10.1038/s41598-020-59602-2

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  36. Samadi S, Arjmand N (2018) A novel stability-based EMG-assisted optimization method for the spine. Med Eng Phys

Download references

Acknowledgments

The authors would like to thank Nicolò Salvatico, Dao Zhou, Yunfei Zhao, and Jayson Galante for their support and contribution in different simulation tasks.

Author information

Authors and Affiliations

  1. Université de technologie de Compiègne, CNRS, Biomechanics and Bioengineering, Centre de recherche Royallieu, CS 60 319 - 60 203, Compiègne Cedex, France

    Katharine Nowakowski, Philippe Carvalho, Jean-Baptiste Six, Yann Maillet, Anh Tu Nguyen, Ismail Seghiri, Loick M’Pemba, Theo Marcille, Sy Toan Ngo & Tien-Tuan Dao

  2. Univ. Lille, CNRS, Centrale Lille, UMR 9013 LaMcube - Laboratoire de Mécanique, Multiphysique, Multiéchelle, F-59000, Lille, France

    Tien-Tuan Dao

Authors
  1. Katharine Nowakowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Philippe Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jean-Baptiste Six
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yann Maillet
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anh Tu Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ismail Seghiri
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Loick M’Pemba
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Theo Marcille
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sy Toan Ngo
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Tien-Tuan Dao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tien-Tuan Dao.

Ethics declarations

Conflict of interest

The authors declare no potential conflict of interests related to the present work.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nowakowski, K., Carvalho, P., Six, JB. et al. Human locomotion with reinforcement learning using bioinspired reward reshaping strategies. Med Biol Eng Comput 59, 243–256 (2021). https://doi.org/10.1007/s11517-020-02309-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-020-02309-3

Keywords

Search

Navigation