Understanding and Transfer of Human Skills in Robotics Using Deep Learning and Musculoskeletal Modeling

Conference paper
Part of the Springer Proceedings in Advanced Robotics book series (SPAR, volume 11)


With the application of deep learning, prosthetic rehabilitation can be carried out in a manner that not only emulates human manipulation skills and performance, but can also work more efficiently. In this study, we introduced computer vision capability for a rehabilitation robot using a convolutional neural network (CNN). The human skill of scooping was studied by dividing it into four motion primitives or sub-tasks. For each primitive, optimum human posture was identified in terms of muscular effort. Human motion skills were analyzed in terms of physiological parameters, including wrist pronation-supination angle, elbow flexion angle, shoulder rotation/abduction/flexion angles, and hand accelerations by three dimensional musculoskeletal modeling. This analysis identified how humans execute the same activity for eight different materials. Optimum human motion for each material was mapped to a robotic arm with six degrees-of-freedom (DOFs), which was equipped with a camera. The success ratio while examining the scooping motion over all trials was found to be 85%. Consequently, the activity can be performed efficiently based on human intuition in a dynamic environment.

Supplementary material

489953_1_En_5_MOESM1_ESM.mp4 (2.2 mb)
Supplementary material 1 (mp4 2262 KB)


  1. 1.
    Orin, D.E.: Application of robotics to prosthetic control. Ann. Biomed. Eng. 8, 305 (1980). Scholar
  2. 2.
    Demircan, E., Wheeler, J., Anderson, F.C., Besier, T., Delp, S.: EMG-informed computed muscle control for dynamic simulations of movement. In: Proceedings of the XXII Congress of the International Society of Biomechanics (ISB), Cape Town, South Africa (2009)Google Scholar
  3. 3.
    Riener, R., Frey, M., Bernhardt, M., Nef, T., Colombo, G.: Human-centered rehabilitation robotics. In: 9th International Conference on Rehabilitation Robotics 2005. ICORR 2005, pp. 319-322 (2005)Google Scholar
  4. 4.
    Zhang, X.Y., Wang, K.X.: Robot assisted rehabilitation technology robotics and intelligent devices. Chin. J. Rehabil. 28(4), 246–248 (2013)Google Scholar
  5. 5.
    Gonzalez, R.V., Buchanan, T.S., Delp, S.L.: How muscle architecture and moment arms affect wrist flexion-extension moments. J. Biomech. 30(7), 705–712 (1997)CrossRefGoogle Scholar
  6. 6.
    Grigore, E.C., Scassellati, B.: Discovering action primitive granularity from human motion for human-robot collaboration. In: Proceedings of Robotics: Science and Systems, July 2017Google Scholar
  7. 7.
    Lim, B., Ra, S., Park, F.: Movement primitives principal component analysis and the efficient generation of natural motions. In: Proceedings IEEE International Conference on Robotics Automation, pp. 4630-4635 (2005)Google Scholar
  8. 8.
    Optitrack system.
  9. 9.
    Khatib, O., Demircan, E., De Sapio, V., Sentis, L., Besier, T., Delp, S.: Robotics-based synthesis of human motion. J. Physiol. Paris 103, 211–219 (2009)CrossRefGoogle Scholar
  10. 10.
    Seth, A., Sherman, M., Reinbolt, J.A., Delp, S.L.: OpenSim: a musculoskeletal modeling and simulation framework for in silico investigations and exchange. PMC, April 2015Google Scholar
  11. 11.
    Thelen, D.G., Anderson, F.C., Delp, S.L.: Generating dynamic simulations of movement using computed muscle control. J. Biomech. 36(3), 321–328 (2003)CrossRefGoogle Scholar
  12. 12.
    Park, J.S., Park, C., Manocha, D.: Intention-aware motion planning using learning based human motion prediction. In: Proceedings of Robotics: Science and Systems, July 2017Google Scholar
  13. 13.
    Dantam, N.T., Kingston, Z.K., Chaudhuri, S., Kavraki, L.E.: Incremental task and motion planning: a constraint-based approach. In: Robotics: Science and Systems (2016)Google Scholar
  14. 14.
    Albrecht, S., Ramırez-Amaro, K., Ruiz-Ugalde, F., Weikersdorfer, D., Leibold, M., Ulbrich, M., Beetz, M.: Imitating human reaching motions using physically inspired optimization principles. In: IEEE-RAS International Conference on Humanoid Robots, Bled, Slovenia (2011)Google Scholar
  15. 15.
    Demircan, E., Besier, T., Menon, S., Khatib, O.: Human motion reconstruction and synthesis of human skills. In: Lenarcic, J., Stanisic, M.M. (Eds.) Advances in Robot Kinematics, pp. 283-292. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  16. 16.
    Holzbaur, K.R., Murray, W.M., Delp, S.L.: A model of the upper extremity for simulating musculoskeletal surgery and analyzing neuromuscular control. Ann. Biomed. Eng. 33(6), 829–840 (2005)CrossRefGoogle Scholar
  17. 17.
    Demircan, E., Sentis, L., De Sapio, V., Khatib, O.: Human motion reconstruction by direct control of marker trajectories. In: Lenarcic, J., Stanisic, M.M. (Eds.) Advances in Robot Kinematics, pp. 263-272. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  18. 18.
  19. 19.
    Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22, 1345–1359 (2010)CrossRefGoogle Scholar
  20. 20.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, pp. 1717-1724, 23–28 June 2014Google Scholar
  21. 21.
    Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur, M., Levenberg, J., Monga, R., Moore, S., Murray, D.G., Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke, M., Yu, Y., Zheng, X.: TensorFlow: a system for large-scale machine learning. Google Brain. In: 12th USENIX Symposium on OSDI, (2016)Google Scholar
  22. 22.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556v6 [cs.CV], 10 April 2015
  23. 23.
  24. 24.
  25. 25.
    Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (NIPS) (2012)Google Scholar
  26. 26.

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Department of Computer ScienceCSULBLong BeachUSA
  2. 2.Department of Mechanical and Aerospace EngineeringCSULBLong BeachUSA
  3. 3.Department of Biomedical EngineeringCSULBLong BeachUSA

Personalised recommendations