Skip to main content

Advertisement

SpringerLink
Log in

Machine Learning for 3D Kinematic Analysis of Movements in Neurorehabilitation

  • Neurorehabilitation and Recovery (J. Krakauer and T. Kitago, Section Editors)
  • Published:
Current Neurology and Neuroscience Reports Aims and scope Submit manuscript

Abstract

Purpose of Review

Recent advances in the machine learning field, especially in deep learning, provide the opportunity for automated, detailed, and unbiased analysis of motor behavior. Although there has not yet been wide use of these techniques in the motor rehabilitation field, they have great potential. In this review, I describe how the current state of machine learning can be applied to 3D kinematic analysis, and how this will have an impact on neurorehabilitation.

Recent Findings

Applications of deep learning methods, in the form of convolutional neural networks, have been revolutionary for image analysis such as face recognition and object detection in images, exceeding human level performance. Recent studies have shown applicability of these deep learning approaches to human posture and movement classification. It is to be expected that portable stereo-camera systems will bring 3D pose estimation into the clinical setting and allow the assessment of movement quality in response to interventions.

Summary

Advances in machine learning can help automate the process of obtaining 3D kinematics of human movements and to identify/classify patterns of movement.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance

  1. Camacho DM, Collins KM, Powers RK, Costello JC, Collins JJ. Next-generation machine learning for biological networks. Cell. 2018;173(7):1581–92.

    Article  CAS  Google Scholar 

  2. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015;521(7553):436–44.

    Article  CAS  Google Scholar 

  3. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. eprint arXiv:151203385. 2015:arXiv:1512.03385.

  4. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1; Lake Tahoe, Nevada. 2999257: Curran Associates Inc.; 2012. p. 1097–105.

  5. Szegedy C, Wei L, Yangqing J, Sermanet P, Reed S, Anguelov D, et al., editors. Going deeper with convolutions. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2015 7–12 June 2015.

  6. Hinton G, Deng L, Yu D, Dahl GE, Mohamed AR, Jaitly N, et al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Proc Mag. 2012;29(6):82–97.

  7. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017;542(7639):115–8.

  8. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316(22):2402–10.

  9. Saba L, Biswas M, Kuppili V, Cuadrado Godia E, Suri HS, Edla DR, et al. The present and future of deep learning in radiology. Eur J Radiol. 2019;114:14–24.

  10. Demers M, Levin MF. Do activity level outcome measures commonly used in neurological practice assess upper-limb movement quality? Neurorehabil Neural Repair. 2017;31(7):623–37.

    Article  Google Scholar 

  11. Alt Murphy M, Willen C, Sunnerhagen KS. Responsiveness of upper extremity kinematic measures and clinical improvement during the first three months after stroke. Neurorehabil Neural Repair. 2013;27(9):844–53.

    Article  Google Scholar 

  12. Bernhardt J, Hayward KS, Kwakkel G, Ward NS, Wolf SL, Borschmann K, et al. Agreed definitions and a shared vision for new standards in stroke recovery research: The Stroke Recovery and Rehabilitation Roundtable taskforce. Int J Stroke. 2017;12(5):444–50.

  13. Kwakkel G, Lannin NA, Borschmann K, English C, Ali M, Churilov L, et al. Standardized measurement of sensorimotor recovery in stroke trials: consensus-based core recommendations from the stroke recovery and rehabilitation roundtable. Int J Stroke. 2017;12(5):451–61.

  14. Fugl-Meyer AR, Jaasko L, Leyman I, Olsson S, Steglind S. The post-stroke hemiplegic patient. 1. A method for evaluation of physical performance. Scand J Rehabil Med. 1975;7(1):13–31.

    CAS  PubMed  Google Scholar 

  15. Yozbatiran N, Der-Yeghiaian L, Cramer SC. A standardized approach to performing the action research arm test. Neurorehabil Neural Repair. 2008;22(1):78–90.

    Article  Google Scholar 

  16. •• Krakauer JW, Carmichael ST. Broken movement: the neurobiology of motor recovery after stroke, vol. xiv. Cambridge: The MIT Press; 2017. p. 269. This book discusses the current state of motor recovery in stroke, and provides perspectives for future studies.

    Book  Google Scholar 

  17. •• Kwakkel G, Wegen EV, Burridge JH, Winstein CJ, Dokkum LV, Murphy MA, et al. Standardized measurement of quality of upper limb movement after stroke: Consensus-based core recommendations from the Second Stroke Recovery and Rehabilitation Roundtable. Int J Stroke. 2019;14(8):783–791 The authors provide expert opinion why kinematic data should be obtained for measurement of movement quality and provide recommendations on the types of tasks that should be used for obtaining kinematic data.

  18. Twitchell TE. The restoration of motor function following hemiplegia in man. Brain. 1951;74(4):443–80.

    Article  CAS  Google Scholar 

  19. Brunnstrom S. Movement therapy in hemiplegia. New York: Harper & Row; 1970.

    Google Scholar 

  20. Bernhardt J, Hayward KS, Kwakkel G, Ward NS, Wolf SL, Borschmann K, et al. Agreed definitions and a shared vision for new standards in stroke recovery research: The Stroke Recovery and Rehabilitation Roundtable Taskforce. Neurorehabil Neural Repair. 2017;31(9):793–9.

  21. Mitchell TM. Machine Learning (1st. ed.). USA: McGraw-Hill, Inc; 1997.

  22. •• Goodfellow I, Bengio Y, Courville A. Deep Learning. The MIT Press. 2016. This is an exceptional book for those who would like to learn the details about deep learning.

  23. Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998;86(11):2278–324.

    Article  Google Scholar 

  24. Zhao Z-Q, Zheng P, Xu S-t, Wu X. Object Detection with deep learning: a review. arXiv e-prints [Internet]. 2018 July 01, 2018. Available from: https://ui.adsabs.harvard.edu/abs/2018arXiv180705511Z.

  25. Cao Z, Simon T, Wei S-E, Sheikh Y. Realtime multi-person 2D pose estimation using part affinity fields. arXiv e-prints [Internet]. 2016 November 01, 2016. Available from: https://ui.adsabs.harvard.edu/abs/2016arXiv161108050C.

  26. Carreira J, Agrawal P, Fragkiadaki K, Human MJ Pose estimation with iterative error feedback. arXiv e-prints [Internet]. 2015 July 01, 2015. Available from: https://ui.adsabs.harvard.edu/abs/2015arXiv150706550C.

  27. Chen Y, Wang Z, Peng Y, Zhang Z, Yu G, Cascaded SJ. Pyramid network for multi-person pose estimation. arXiv e-prints [Internet]. 2017 November 01, 2017. Available from: https://ui.adsabs.harvard.edu/abs/2017arXiv171107319C.

  28. Chu X, Yang W, Ouyang W, Ma C, Yuille AL, Wang X. Multi-context attention for human pose estimation. arXiv e-prints [Internet]. 2017 February 01, 2017. Available from: https://ui.adsabs.harvard.edu/abs/2017arXiv170207432C.

  29. Insafutdinov E, Pishchulin L, Andres B, Andriluka M, Schiele B. DeeperCut: a deeper, stronger, and faster multi-person pose estimation model. arXiv e-prints [Internet]. 2016 May 01, 2016. Available from: https://ui.adsabs.harvard.edu/abs/2016arXiv160503170I.

  30. Newell A, Yang K, Stacked DJ. Hourglass networks for human pose estimation. arXiv e-prints [Internet]. 2016 March 01, 2016. Available from: https://ui.adsabs.harvard.edu/abs/2016arXiv160306937N.

  31. Pfister T, Charles J, Zisserman A. Flowing ConvNets for human pose estimation in videos. arXiv e-prints [Internet]. 2015 June 01, 2015. Available from: https://ui.adsabs.harvard.edu/abs/2015arXiv150602897P.

  32. Sun K, Xiao B, Liu D, Wang J. Deep high-resolution representation learning for human pose estimation. arXiv e-prints [Internet]. 2019 February 01, 2019. Available from: https://ui.adsabs.harvard.edu/abs/2019arXiv190209212S.

  33. Tompson J, Goroshin R, Jain A, LeCun Y, Bregler C. Efficient object localization using convolutional networks. arXiv e-prints [Internet]. 2014 November 01, 2014. Available from: https://ui.adsabs.harvard.edu/abs/2014arXiv1411.4280T.

  34. Toshev A, Szegedy C. DeepPose: Human pose estimation via deep neural networks. arXiv e-prints [Internet]. 2013 December 01, 2013. Available from: https://ui.adsabs.harvard.edu/abs/2013arXiv1312.4659T.

  35. Wei S-E, Ramakrishna V, Kanade T, Sheikh Y. Convolutional pose machines. arXiv e-prints [Internet]. 2016 January 01, 2016. Available from: https://ui.adsabs.harvard.edu/abs/2016arXiv160200134W.

  36. Xiao B, Wu H, Wei Y. Simple baselines for human pose estimation and tracking. arXiv e-prints [Internet]. 2018 April 01, 2018. Available from: https://ui.adsabs.harvard.edu/abs/2018arXiv180406208X.

  37. •• Arac A, Zhao P, Dobkin BH, Carmichael ST, Golshani P. DeepBehavior: a deep learning toolbox for automated analysis of animal and human behavior imaging data. Front Syst Neurosci. 2019;13:20. The authors provide open-source methods for obtaining 3D kinematic data in humans with potential clinical applicability.

  38. Shukla S, Arac A. A step-by-step implementation of DeepBehavior, deep learning toolbox for automated behavior analysis. J Vis Exp. 2020;156:e60763.

    Google Scholar 

  39. Pavllo D, Feichtenhofer C, Grangier D, Auli M. 3D human pose estimation in video with temporal convolutions and semi-supervised training. arXiv e-prints [Internet]. 2018 November 01, 2018. Available from: https://ui.adsabs.harvard.edu/abs/2018arXiv181111742P.

  40. Kitago T, Liang J, Huang VS, Hayes S, Simon P, Tenteromano L, et al. Improvement after constraint-induced movement therapy: recovery of normal motor control or task-specific compensation? Neurorehabil Neural Repair. 2013;27(2):99–109.

  41. Cappozzo A, Catani F, Croce UD, Leardini A. Position and orientation in space of bones during movement: anatomical frame definition and determination. Clin Biomech (Bristol, Avon). 1995;10(4):171–8.

    Article  CAS  Google Scholar 

  42. Dobkin BH. Wearable motion sensors to continuously measure real-world physical activities. Curr Opin Neurol. 2013;26(6):602–8.

    Article  Google Scholar 

  43. • Schwarz A, Kanzler CM, Lambercy O, Luft AR, Veerbeek JM. Systematic review on kinematic assessments of upper limb movements after stroke. Stroke. 2019;50(3):718–27. This is a systematic review on the types of most commonly used kinematic parameters.

  44. • Halilaj E, Rajagopal A, Fiterau M, Hicks JL, Hastie TJ, Delp SL. Machine learning in human movement biomechanics: best practices, common pitfalls, and new opportunities. J Biomech. 2018;81:1–11 The authors provide best practices for applications of machine learning techniques in human movement biomechanics. It is an essential paper for those who would like to learn about practical applications of machine learning techniques.

    Article  Google Scholar 

  45. • Backenroth D, Goldsmith J, Harran MD, Cortes JC, Krakauer JW, Kitago T. Modeling motor learning using heteroskedastic functional principal components analysis. J Am Stat Assoc. 2018;113(523):1003–15. These three papers demonstrate 2D kinematic analysis using holistic approach.

  46. • Goldsmith J, Kitago T. Assessing systematic effects of stroke on motorcontrol by using hierarchical function-on-scalar regression. J R Stat Soc Ser C Appl Stat. 2016;65(2):215–36. These three papers demonstrate 2D kinematic analysis using holistic approach.

  47. • Cortes JC, Goldsmith J, Harran MD, Xu J, Kim N, Schambra HM, et al. A short and distinct time window for recovery of arm motor control early after stroke revealed with a global measure of trajectory kinematics. Neurorehabil Neural Repair. 2017;31(6):552–60. These three papers demonstrate 2D kinematic analysis using holistic approach.

  48. Doya K. What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural Netw. 1999;12(7–8):961–74.

    Article  CAS  Google Scholar 

  49. Wolpert DM, Ghahramani Z. Computational principles of movement neuroscience. Nat Neurosci. 2000;3(Suppl):1212–7.

    Article  CAS  Google Scholar 

  50. Todorov E, Jordan MI. Optimal feedback control as a theory of motor coordination. Nat Neurosci. 2002;5(11):1226–35.

    Article  CAS  Google Scholar 

  51. Wolpert DM, Ghahramani Z, Jordan MI. An internal model for sensorimotor integration. Science. 1995;269(5232):1880–2.

    Article  CAS  Google Scholar 

  52. Wolpert DM, Miall RC. Forward models for physiological motor control. Neural Netw. 1996;9(8):1265–79.

    Article  Google Scholar 

  53. Shadmehr R, Krakauer JW. A computational neuroanatomy for motor control. Exp Brain Res. 2008;185(3):359–81.

    Article  Google Scholar 

  54. Kargo WJ, Nitz DA. Improvements in the signal-to-noise ratio of motor cortex cells distinguish early versus late phases of motor skill learning. J Neurosci. 2004;24(24):5560–9.

    Article  CAS  Google Scholar 

  55. Krakauer JW, Ghazanfar AA, Gomez-Marin A, MacIver MA, Poeppel D. Neuroscience needs behavior: correcting a reductionist bias. Neuron. 2017;93(3):480–90.

    Article  CAS  Google Scholar 

Download references

Acknowledgments

I would like to thank John W Krakauer and Bruce H Dobkin for stimulating discussions and critical reading of the manuscript.

Author information

Authors and Affiliations

  1. Department of Neurology, David Geffen School of Medicine, University of California, Los Angeles, 710 Westwood Plaza, Rm 3-232, Los Angeles, CA, 90095, USA

    Ahmet Arac

Authors
  1. Ahmet Arac
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ahmet Arac.

Ethics declarations

Conflict of Interest

Ahmet Arac was supported by grants from NIH/NINDS (NS109315) and NVIDIA (GPU grant).

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the Topical Collection on Neurorehabilitation and Recovery

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Arac, A. Machine Learning for 3D Kinematic Analysis of Movements in Neurorehabilitation. Curr Neurol Neurosci Rep 20, 29 (2020). https://doi.org/10.1007/s11910-020-01049-z

Download citation

  • Published:

  • DOI: https://doi.org/10.1007/s11910-020-01049-z

Keywords

Search

Navigation