Abstract
This paper proposes a weakly-supervised learning framework for dynamics estimation from human motion. Although there are many solutions to capture pure human motion readily available, their data is not sufficient to analyze quality and efficiency of movements. Instead, the forces and moments driving human motion (the dynamics) need to be considered. Since recording dynamics is a laborious task that requires expensive sensors and complex, time-consuming optimization, dynamics data sets are small compared to human motion data sets and are rarely made public. The proposed approach takes advantage of easily obtainable motion data which enables weakly-supervised learning on small dynamics sets and weakly-supervised domain transfer. Our method includes novel neural network (NN) layers for forward and inverse dynamics during end-to-end training. On this basis, a cyclic loss between pure motion data can be minimized, i.e. no ground truth forces and moments are required during training. The proposed method achieves state-of-the-art results in terms of ground reaction force, ground reaction moment and joint torque regression and is able to maintain good performance on substantially reduced sets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arnab, A., Doersch, C., Zisserman, A.: Exploiting temporal context for 3D human pose estimation in the wild. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
de Avila Belbute-Peres, F., Smith, K., Allen, K., Tenenbaum, J., Kolter, J.Z.: End-to-end differentiable physics for learning and control. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31, pp. 7178–7189. Curran Associates, Inc. (2018)
Bastien, G.J., Gosseye, T.P., Penta, M.: A robust machine learning enabled decomposition of shear ground reaction forces during the double contact phase of walking. Gait Posture 73, 221–227 (2019)
Brubaker, M.A., Fleet, D.J.: The kneed walker for human pose tracking. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8, June 2008
Choi, A., Lee, J.M., Mun, J.H.: Ground reaction forces predicted by using artificial neural network during asymmetric movements. Int. J. Precis. Eng. Manuf. 14(3), 475–483 (2013)
CMU: Human motion capture database (2014). http://mocap.cs.cmu.edu/
Degrave, J., Hermans, M., Dambre, J., Wyffels, F.: A differentiable physics engine for deep learning in robotics. Front. Neurorobot. 13, 6 (2019)
Devin, C., Gupta, A., Darrell, T., Abbeel, P., Levine, S.: Learning modular neural network policies for multi-task and multi-robot transfer. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2169–2176, May 2017
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., Zisserman, A.: Temporal cycle-consistency learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
Federolf, P., Boyer, K., Andriacchi, T.: Application of principal component analysis in clinical gait research: identification of systematic differences between healthy and medial knee-osteoarthritic gait. J. Biomech. 46(13), 2173–2178 (2013)
Finn, C., Tan, X.Y., Duan, Y., Darrell, T., Levine, S., Abbeel, P.: Deep spatial autoencoders for visuomotor learning. In: 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 512–519, May 2016
Fregly, B.J., Reinbolt, J.A., Rooney, K.L., Mitchell, K.H., Chmielewski, T.L.: Design of patient-specific gait modifications for knee osteoarthritis rehabilitation. IEEE Trans. Biomed. Eng. 54(9), 1687–1695 (2007)
Hoffman, J., et al.: CyCADA: cycle-consistent adversarial domain adaptation (2017)
Johnson, L., Ballard, D.H.: Efficient codes for inverse dynamics during walking. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pp. 343–349. AAAI Press (2014)
Johnson, W., Alderson, J., Lloyd, D., Mian, A.: Predicting athlete ground reaction forces and moments from spatio-temporal driven CNN models. IEEE Trans. Biomed. Eng. (2018)
Leporace, G., Batista, L.A., Metsavaht, L., Nadal, J.: Residual analysis of ground reaction forces simulation during gait using neural networks with different configurations. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2812–2815, August 2015
Leporace, G., Batista, L., Nadal, J.: Prediction of 3d ground reaction forces during gait based on accelerometer data. Res. Biomed. Eng. 34 (2018)
Lv, X., Chai, J., Xia, S.: Data-driven inverse dynamics for human motion. ACM Trans. Graph. 35(6), 163:1–163:12 (2016)
Maas, A.L.: Rectifier nonlinearities improve neural network acoustic models (2013)
von Marcard, T., Henschel, R., Black, M.J., Rosenhahn, B., Pons-Moll, G.: Recovering accurate 3D human pose in the wild using IMUs and a moving camera. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11214, pp. 614–631. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01249-6_37
Mukhopadhyay, R., Chaki, R., Sutradhar, A., Chattopadhyay, P.: Model learning for robotic manipulators using recurrent neural networks. In: TENCON 2019–2019 IEEE Region 10 Conference (TENCON), pp. 2251–2256, October 2019
Oh, S.E., Choi, A., Mun, J.H.: Prediction of ground reaction forces during gait based on kinematics and a neural network model. J. Biomech. 46(14), 2372–2380 (2013)
Reinbolt, J.A., Fox, M.D., Schwartz, M.H., Delp, S.L.: Predicting outcomes of rectus femoris transfer surgery. Gait Posture 30(1), 100–105 (2009)
Schwab, A.L., Delhaes, G.M.J.: Lecture Notes Multibody Dynamics B, wb1413 (2009)
Seon Choi, H., Hee Lee, C., Shim, M., In Han, J., Su Baek, Y.: Design of an artificial neural network algorithm for a low-cost insole sensor to estimate the ground reaction force (GRF) and calibrate the center of pressure (COP). Sensors 18, 4349 (2018)
Shah, M., Chen, X., Rohrbach, M., Parikh, D.: Cycle-consistency for robust visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
Sharma, S., Varigonda, P.T., Bindal, P., Sharma, A., Jain, A.: Monocular 3D human pose estimation by generation and ordinal ranking. In: The IEEE International Conference on Computer Vision (ICCV), October 2019
Sim, T., et al.: Predicting complete ground reaction forces and moments during gait with insole plantar pressure information using a wavelet neural network. J. Biomech. Eng. 137 (2015). 9 pages
Takahashi, K., Ogata, T., Nakanishi, J., Cheng, G., Sugano, S.: Dynamic motion learning for multi-DoF flexible-joint robots using active-passive motor babbling through deep learning. Adv. Robot. 31(18), 1002–1015 (2017)
Wandt, B., Rosenhahn, B.: RepNet: weakly supervised training of an adversarial reprojection network for 3D human pose estimation. In: Computer Vision and Pattern Recognition (CVPR), June 2019
Wang, X., Jabri, A., Efros, A.A.: Learning correspondence from the cycle-consistency of time. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019
Winter, D.: Biomechanics and Motor Control of Human Movement. Wiley, New York (2009)
Xiong, B., et al.: Intelligent prediction of human lower extremity joint moment: an artificial neural network approach. IEEE Access 7, 29973–29980 (2019)
Zell, P., Rosenhahn, B.: Learning inverse dynamics for human locomotion analysis. Neural Comput. Appl. 32(15), 11729–11743 (2019). https://doi.org/10.1007/s00521-019-04658-z
Zell, P., Wandt, B., Rosenhahn, B.: Physics-Based Models for Human Gait Analysis. In: Müller, B., Wolf, S. (eds.) Handbook of Human Motion. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-14418-4_164
Zhu, J., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251 (2017)
Acknowledgement
Research supported by the European Research Council (ERC-2013-PoC). The authors would like to thank all subjects who participated in data acquisition.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mpeg 134134 KB)
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zell, P., Rosenhahn, B., Wandt, B. (2020). Weakly-Supervised Learning of Human Dynamics. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12371. Springer, Cham. https://doi.org/10.1007/978-3-030-58574-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-58574-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58573-0
Online ISBN: 978-3-030-58574-7
eBook Packages: Computer ScienceComputer Science (R0)