Abstract
In this paper, we proposed a method of real-time human motion detection and skeleton control based on the depth image. For the human body part, we use Kinect to detect the original three-dimensional coordinates of the human joint; then smooth the joint movement; calculate the amount of rotation based on the smoothed data. Finally, the FBX file is used to drive the virtual mannequin. For the human face part, we use 3D modeling software to modify the FBX model and replace the head of the human model. With the help of the Kinect device, we combined the idea of linear blend skinning (LBS) to make the human body action effect more realistic. The experimental results show that the motion capture system can better recover the 3D skeleton motion of the captured real human body. Compared with other motion capture system, the motion control time of the human body is greatly reduced, and the 3D mesh and skeleton can be obtained immediately according to the detected human body motion. The modified FBX model can apply specific expression effects.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Yin, J., Yang, Q., Pan, J.J.: Sensor-based abnormal human-activity detection. IEEE Trans. Knowl. Data Eng. 20(8), 1082–1090 (2008)
Moeslund, T.B., Hilton, A., KrÃijger, V.: A survey of advances in vision-based human motion capture and analysis. CVIU 104(2–3), 90–126 (2006)
Wu, D., Pigou, L., Kindermans, P.J., et al.: Deep dynamic neural networks for multimodal gesture segmentation and recognition. IEEE Trans. Pattern Anal. Mach. Intell. 38(8), 1583–1597 (2016)
Mehta, D., Sridhar, S., Sotnychenko, O., et al.: VNect: real-time 3D human pose estimation with a single RGB camera. ACM Trans. Graph. 36(4), 44 (2017)
Pavlakos, G., Zhu, L., Zhou, X., et al.: Learning to estimate 3D human pose and shape from a single color image. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 459–468 (2018)
Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, Michael J.: Keep it SMPL: automatic estimation of 3D human pose and shape from a single image. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9909, pp. 561–578. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46454-1_34
Loper, M., Mahmood, N., Romero, J., et al.: SMPL: a skinned multi-person linear model. ACM Trans. Graph. 34(6), 248 (2015)
Holden, D., Saitoy, J., Komuraz, T.: A deep learning framework for character motion synthesis and editing. ACM Trans. Graph. (TOG) 35(4) (2016)
Tong, X., Xu, P., Yan, X.: Research on skeleton animation motion data based on kinect. In: Proceedings of the 2012 Fifth International Symposium on Computational Intelligence and Design, ISCID 2012, vol. 2, pp. 347–350 (2012)
Meyer, G.P., Do, M.N.: Real-time 3D face modeling with a commodity depth camera. In: 2013 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1–4. IEEE (2013)
de Farias, M.M.C., Apolinário Jr., A.L., dos Santos, S.A.C.: Kinect fusion for faces: real-time 3D face tracking and modeling using a kinect camera for a markerless AR system. SBC J. Interact. Syst. 4(2), 2–7 (2013)
Blanz, V., Vetter, T.: Face recognition based on fitting a 3D morphable model. IEEE Trans. Pattern Anal. Mach. Intell. 25(9), 1063–1074 (2003)
Zhou, S.: The research and implementation of virtual human face modeling and 3D emotion interaction of facial expression. CNKI, ZJU (2019)
Wang, C.: Adaptive 3D virtual human modeling and animation system in VR scene. CNKI, ZJU (2018)
Acknowledgment
We would like to acknowledge the support of the National Key Research and Development Project (Grant No. 2017YFB1002803) and the Guangzhou Innovation and Entrepreneurship Leading Team Project under grant CXLJTD-201609.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer-Verlag GmbH Germany, part of Springer Nature
About this chapter
Cite this chapter
Jing, Y., Wang, C., Pan, Z., Yao, Z., Zhang, M. (2020). Human Skeleton Control with the Face Expression Changed Synchronously. In: Pan, Z., Cheok, A., Müller, W., Zhang, M. (eds) Transactions on Edutainment XVI. Lecture Notes in Computer Science(), vol 11782. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-61510-2_15
Download citation
DOI: https://doi.org/10.1007/978-3-662-61510-2_15
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-662-61509-6
Online ISBN: 978-3-662-61510-2
eBook Packages: Computer ScienceComputer Science (R0)