Skip to main content
SpringerLink
Log in

Deep reinforcement learning coupled with musculoskeletal modelling for a better understanding of elderly falls

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

Reinforcement learning (RL) has been used to study human locomotion learning. One of the current challenges in healthcare is our understanding of and ability to slow the decline due to muscle ageing and its effect on human falls. The purpose of this study was to investigate reinforcement learning for human movement strategies when modifying muscle parameters to account for age-related changes. In particular, human falls with modified physiological factors were modelled and simulated to determine the effect of muscle descriptors for ageing on kinematic behaviour and muscle force control. A 3D musculoskeletal model (8 DoF and 22 muscles) of the human body was used. The deep deterministic policy gradient (DDPG) method was implemented. Different muscle descriptors for ageing were integrated, including changes in maximum isometric force, contraction velocity, the deactivation time constant and passive muscle strain. Additionally, the effects of isometric force reductions of 10, 20 and 30% were also considered independently. An environment for the simulation was developed using the opensim-rl package for Python with the training process completed on Google Compute Engine. The simulation outcomes for healthy young adult and elderly falls under modified muscle behaviours were compared to experimental observations for validation. The result of our elderly simulation for multiple ageing-related factors (M_all) produced a walking speed of 0.26 m/s for the two steps taken prior to the fall. The over activation of the hip extensors and inactivation of knee extensors led to a backward fall for this elderly simulation. The inactivated rectus femoris and right tibialis are main actors of the forward fall. Our simulation outcomes are consistent with experimental observations through the comparison of kinematic features and motion history evolution. We showed in the present study, for the first time, that RL can be used as a strategy to explore the effect of ageing muscle physiological factors on kinematics and muscle control during falls. Our findings show that the elderly fall model for the M_all condition more closely resembles experimental elderly fall data than our simulations which considered age-related reductions of force alone. As future perspectives, the behaviour preceding a fall will be studied to establish the strategies used to avoid falls or fall with minimal consequence, leading to the identification of patient-specific rehabilitation programmes for elderly people.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. TEF (2020) Insée Références - Population, Tableau de l'économie française, p 266

  2. Torres M, Pédrono G, Lasbeur L, Carcaillon-Bentata L, Rigou A, Beltzer N (2018) Chutes des personnes âgées à domicille: caractériqtiques des chuteurs et des circonstances de la chute.," Volet "Hospitalisation" de l'enquête ChuPADom. Saint-Maurice : Santé publique France, pp. 1–138

  3. van den Kroonenberg AJ, Hayes WC, McMahon TA (1996) Hip impact velocities and body configurations for voluntary falls from standing height. J Biomechanics 29(6):807–811

    Article  Google Scholar 

  4. James S, Lucchesi L, Bisignano C et al (2020) The global burden of falls: global, regional and national estimates of morbidity and mortality from the Global Burden of Disease Study 2017. Inj Prev 26:i3–i11

    Article  Google Scholar 

  5. Meyer M, Constancias F, Vogel T, Kaltenbach G, Schmitt E (2021) Gait Disorder among Elderly People, Psychomotor Disadaptation Syndrome: Post-Fall Syndrome, Risk Factors and Follow-Up- A Cohort Study of 70 Patients. Gerontology 67:17–24

    Article  Google Scholar 

  6. Aversa Z, Zhang X, Fielding RA, Lanza I, LeBrasseur NK (2019) The clinical impact and biological mechanisms of skeletal muscle aging. Bone 127:26–36

    Article  CAS  Google Scholar 

  7. Goodpaster BH, Park SW, Harris TB, Kritchevsky SB, Nevitt M, Schwartz AV, Simonsick EM, Tylavsky FA, Visser M, Newman AB (2006) The loss of skeletak muscle strength, mass, and quality in older adults: the health, aging and body composition study. J Gerontol A Biol Sci Med Sci 61:1059–1064

    Article  Google Scholar 

  8. Pion CH, Barbat-Artigas S, St-Jean-Pelletier F, Chevalier S, Gaudreau P, Gouspillou G, Morais JA, Bélanger M, Aubertin-Leheudre M (2017) Muscle strength and force development in high- and low- functioning elderly men: Influence of muscular and neural factors. Exp Gerontol 96:19–28

  9. Roach KE, Miles TP (1991) Normal hip and knee active range of motion: the relationship to age. Phys Ther 71:656–665

    Article  CAS  Google Scholar 

  10. Sheth KA, Iyer CC, Wier CG, Crum AE, Bratasz A, Kolb SJ, Clark BC, Burghes AHM, Arnold WD (2018) Muscle strength and size are associated with motor unit connectivity in aged mice.Neurobiol Aging 67(128–136)

  11. Straight CR, Ades PA, Toth MJ, Miller MS (2018) Age-related reduction in single muscle fiber caclium sensitivity is associated with decreased muscle power in men and women. Exp Gerontol 102:84–92

    Article  CAS  Google Scholar 

  12. Narici M, McPhee J, Conte M, Franchi MV, Mitchell K, Tagliaferri S, Monti E, Marcolin G, Atherton PJ, Smith K, Phillips B, Lund J, Franceschi C, Maggio M, Butler-Browne GS (2021) Age-related alterations in muiscle architecture are a signature of sarcopenia: the ultrasound sarcopenia index. J Cachexia, Sarcopenia and Muscle 12(4):973–982

    Article  Google Scholar 

  13. Xu J, Fu SN, Hug F (2021) Age-related increase in muscle stiffness is muscle length dependent and associated with muscle force in senior females. BMC Musculoskelet Disord 22(1):829

    Article  Google Scholar 

  14. Caceres Santana E, Bermundez Moreno C, Ramirez Suarez J, Bahamonde Roman C, Murie-Fernandez M (2019) Incidencia de caídas en el hospital de larga-media estancia. Factores de riesgo y estrategias para la prevención.Neurologia. https://doi.org/10.1016/j.nrl.2019.03.002

  15. Sucerquia A, Lopez JD, Vargas-Bonilla JF (2017) SisFall: a fall and movement dataset. Sensors 17(1):198

    Article  Google Scholar 

  16. Vacher M, Bouakaz S, Bobillier-Chaumon ME, Aman F, Khan RA, Bekkadja S, Portet F, Guillou E, Rossato S, Lecouteux B (2016) The CIRDO Corpus: Comprehensive Audio/Video Database of Domestic Falls of Elderly People," 10th International Conference on Language Resources and Evaluation (LREC 2016), ELRA, pp. 1389–1396

  17. Robinovitch SN, Feldman F, Yang Y, Schonnop R, Leung PM, Sarraf T, Sims-Gould J, Loughlin M (2013) Video capture of the circumstances of falls in elderly people residing in long-term care: an observational study. The Lancet 381(9860):47–54

    Article  Google Scholar 

  18. Roelker SA, Caruthers EJ, Baker RK, Pelz NC, Chaudhari AMW, Siston RA (2017) Interpreting Musculoskeletal Models and Dynamic Simuations: Causes and Effects of Differences Between Models. Annals of Biomed Eng 45:2635–2647

    Article  Google Scholar 

  19. Song S, Geyer H (2018) Predictive neuromechanical simulatins indicate why walking performance declines with ageing. J Physiol 596(7):1199–1210

    Article  CAS  Google Scholar 

  20. Nowakowski K, Carvalho P, Six J-B, Maillet Y, Nguyen AT, Seghiri I, M’Pemba L, Marcille T, Ngo ST, Dao T-T (2021) Human locomotion with reinforcement learning using bioinspired reward reshaping strategies. Med Biol Eng Comput 59(1):243–256

    Article  Google Scholar 

  21. Dao TT (2019) From Deep Learning to Transfer Learning for the Prediction of Skeletal Muscle Forces. Med Biol Eng Compu 57(5):1049–1058

    Article  Google Scholar 

  22. Kidzinski L, Mohanty SP, Ong C, Hicks J, Francis S, Levine S, Salathe M, Delp S (2018) Learning to Run Challenge: Synthesizing Physiologically Accurate Motion Using Deep Reinforcement Learning. In: Escalera, S., Weimer, M. (eds) The NIPS '17 Competition: Building Intelligent Systems. The Springer Series on Challenges in Machine Learning. Springer, Cham. https://doi.org/10.1007/978-3-319-94042-7_6

  23. Lillicrap TP, Hunt JJ, Pritzel A, Heess NM, Erez T, Tassa Y, SilverD, Wierstra D (2016) Continuous control with deep reinforcement learning.CoRR, vol. abs/1509.02971

  24. Mastorakis G, Ellis T, Makris D (2018) Fall detection without people: A simulation approach tackling video data scarcity. Expert Syst Appl 112:125–137

    Article  Google Scholar 

  25. Choi W, Wakeling J, Robinovitch S (2015) Kinematic analysis of video-captured falls experienced by older adults in long-term care. J Biomech 48:911–920

    Article  CAS  Google Scholar 

  26. Okubo Y, Brodie MA, Sturnieks DL, Hicks C, Lord SR (2019) A pilot study of reactive balance training using trips and slips with increasing unpredictability in young and older adults: Biomechanical mechanisms, falls and clinical feasibility. Clin Biomech 67:171–179

    Article  Google Scholar 

  27. Martins Ribeiro D, Santos Bueno GA, Martins Gervasio F, Losada de Menezes R (2019) Foot-ground clearance characteristics in women: A comparison across different ages.Gait Posture 69:121–125

  28. Arakawa T, Otani T, Kobayashi Y, Tanaka M (2021) 2-D forward dynamics simulation of gait adaptation to muscle weakness in elderly gait. Gait Posture 85:71–77

    Article  Google Scholar 

  29. Kerrigan D, Lee L, Collins J, Riley P, Lipsitz L (2001) Reduced hip extension during walking: Healthy elderly and fallers versus young adults. Arch Phys Med Rehabil 82:26–30

    Article  CAS  Google Scholar 

  30. Akyol A (2007) Falls in the elderly: what can be done? Wiley Online Library 54(2):191–196

    CAS  Google Scholar 

  31. Thompson JD, Plummer P, Franz JR (2018) Age and falls history effects on antagonist leg muscle coavtivation during walking with balance perturbations. Clin Biomech 59:94–100

    Article  Google Scholar 

  32. Kidzinski L, Mohanty SP, Ong C, Hicks J, Francis S, Levine S, Salathé M, Delp S (2017) NIPS 2017 Competition Book: Learning to Run challenge: Synthesizing physiologically accurate motion using deep reinforcement learning, arXiv:1804.00198

  33. Creative Commons, "Creative Commons," [Online]. Available: https://creativecommons.org/licenses/by/3.0/. Accessed 31 August 2021

  34. Fernandez NB, Hars M, Trombetti A, Vuilleumier P (2019) Age-related changes in attention control and their relationship with gait performance in older adults with high risk of falls. Neuroimage 189:551–559

    Article  Google Scholar 

  35. Gerstle EE, O'Connor K, Keenan KG, Slavens BA, Cobb SC (2021) The influence of age and fall history on single transition step kinematics. Clinical Biomechanics 89:105456

  36. Kobayashi Y, Hobara H, Matsushita S, Mochimaru M (2014) Key joint kinematic characteristics of the gait of fallers identified by principal component analysis. J Biomech 47:2424–2429

    Article  Google Scholar 

  37. Waterval N, Veerkamp K, Geijtenbeek T, Harlaar J, Nollet F, Brehm MA, van der Krogt MM (2021) Validation of forward simulations to predict the effects of bilateral plantarflexor weakness on gait. Gait Posture 87:33–42

    Article  CAS  Google Scholar 

Download references

Funding

The authors would like to thank the Hauts-de-France Region and Labex MS2T (Maîtrise des systèmes de systèmes technologiques) for the funding of this work.

Author information

Authors and Affiliations

  1. Université de technologie de Compiègne, CNRS, Biomechanics and Bioengineering, Centre de Recherche Royallieu, CS 60 319-60 203, Compiègne, France

    Katharine Nowakowski & Karim El Kirat

  2. Univ. Lille, CNRS, Centrale Lille, UMR 9013, LaMcube, Laboratoire de Mécanique, Multiphysique, Multiéchelle, 59655, Villeneuve d’Ascq Cedex, F-59000, Lille, France

    Tien-Tuan Dao

Authors
  1. Katharine Nowakowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Karim El Kirat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tien-Tuan Dao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tien-Tuan Dao.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Below is the link to the electronic supplementary material.

Supplementary file1 (JPG 67 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nowakowski, K., El Kirat, K. & Dao, TT. Deep reinforcement learning coupled with musculoskeletal modelling for a better understanding of elderly falls. Med Biol Eng Comput 60, 1745–1761 (2022). https://doi.org/10.1007/s11517-022-02567-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-022-02567-3

Keywords

Search

Navigation