Abstract
Real-time full-body motion capture (MoCap) is becoming necessary for enabling natural interactions and creating deeper immersion in virtual reality (VR). To reduce the cost and complexity of MoCap systems, some studies attempt to track only the joint data of root and end effectors and reconstruct full-body motion by solving inverse kinematics (IK) problems. However, ensuring the accuracy of full-body motion reconstruction in real-time is challenging because the problem is inherently under-constrained. In this paper, we propose PE-DLS, a novel method to perform full-body motion reconstruction in two stages: pose estimation (PE) and damped least squares (DLS) optimization. First, we use analytical IK solvers to estimate the spine and limbs in sequence. To further improve model accuracy, we use the DLS method to optimize the results obtained from PE. To evaluate the model performance, we compare it with other methods in terms of the reconstruction error and computational time of full-body reconstruction via testing on publicly available datasets. These results indicate that PE-DLS outperforms other methods in terms of the mean per joint position error (2.11 cm) and mean per joint rotation error (10.75°) with low time cost (1.65 ms per frame). Furthermore, we implement a full-body MoCap system based on an HTC Vive headset and five Vive trackers. Live demos and qualitative comparisons show that our system achieves comparable quality to the commercial MoCap system. With high accuracy and low time cost, PE-DLS contributes to construct a real-time MoCap system in VR.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and material
All data generated or analysed during this study are included in this published article.
Code availability
The source code is available at GitHub: https://github.com/A-Qiang/VRMoCap_PEDLS.
Notes
Optoelectronic motion capture system: https://www.vicon.com, last visited on Jul 26th, 2021.
Bio-IK: https://assetstore.unity.com/packages/tools/animation/bio-ik-67819, last visited on Jul 26th, 2021.
Final IK: https://assetstore.unity.com/packages/tools/animation/final-ik-14290, last visited on Jul 26th, 2021.
PEDLS for MoCap Demo: https://github.com/A-Qiang/VRMoCap_PEDLS, last visited on Jul 26th, 2021.
Inertial sensor motion capture system: https://www.noitom.com.cn, last visited on Jul 26th, 2021.
Basic Motions: https://assetstore.unity.com/packages/3d/animations/basic-motions-free-154271, last visited on Jul 26th, 2021.
CMU MoCap database: http://mocap.cs.cmu.edu, last visited on Jul 26th, 2021.
References
Aristidou A, Lasenby J (2011) FABRIK: A fast, iterative solver for the Inverse Kinematics problem. Graph Models 73:243–260. https://doi.org/10.1016/j.gmod.2011.05.003
Aristidou A, Lasenby J, Chrysanthou Y, Shamir A (2018) Inverse kinematics techniques in computer graphics: a survey. Comput Graph Forum 37:35–58
Baerlocher P, Boulic R (2004) An inverse kinematics architecture enforcing an arbitrary number of strict priority levels. Vis Comput 20:402–417. https://doi.org/10.1007/s00371-004-0244-4
Boulic R, Varona J, Unzueta L et al (2006) Evaluation of on-line analytic and numeric inverse kinematics approaches driven by partial vision input. Virtual Real 10:48–61. https://doi.org/10.1007/s10055-006-0024-8
Buss SR, Kim J-S (2005) Selectively damped least squares for inverse kinematics. J Graph Tools 10:37–49. https://doi.org/10.1080/2151237x.2005.10129202
Buss SR (2004) Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods. IEEE J Robot Autom 17:16
Caserman P, Achenbach P, Gobel S (2019a) Analysis of inverse kinematics solutions for full-body reconstruction in virtual reality. In: 2019a IEEE 7th international conference on serious games and applications for health SeGAH 2019a. https://doi.org/10.1109/SeGAH.2019a.8882429
Caserman P, Garcia-Agundez A, Konrad R et al (2019b) Real-time body tracking in virtual reality using a Vive tracker. Virtual Real 23:155–168. https://doi.org/10.1007/s10055-018-0374-z
Caserman P, Garcia-Agundez A, Gobel S (2020) A survey of full-body motion reconstruction in immersive virtual reality applications. IEEE Trans vis Comput Graph 26:3089–3108. https://doi.org/10.1109/TVCG.2019.2912607
Chai J, Hodgins JK (2005) Performance animation from low-dimensional control signals. ACM Trans Graph 24:686–696. https://doi.org/10.1145/1073204.1073248
Genova C, Biffi E, Arlati S et al (2021) A simulator for both manual and powered wheelchairs in immersive virtual reality CAVE. Virtual Real. https://doi.org/10.1007/s10055-021-00547-w
Giuberti M (2014) Inertial sensing for human motion analysis: processing, technologies, and applications. 1–180
Gonçalves G, Melo M, Barbosa L et al (2021) Evaluation of the impact of different levels of self-representation and body tracking on the sense of presence and embodiment in immersive VR. Virtual Real. https://doi.org/10.1007/s10055-021-00530-5
Grochow K, Martin SL, Hertzmann A, Popović Z (2004) Style-based inverse kinematics. ACM Trans Graph 23:522. https://doi.org/10.1145/1015706.1015755
Harish P, Mahmudi M, Le Callennec B, Boulic R (2016) Parallel inverse kinematics for multithreaded architectures. ACM Trans Graph. https://doi.org/10.1145/2887740
Ho ESL, Shum HPH, Cheung Y, Yuen PC (2013) Topology aware data-driven inverse kinematics. Comput Graph Forum 32:61–70. https://doi.org/10.1111/cgf.12212
Huang J, Wang Q, Fratarcangeli M et al (2017) Multi-variate Gaussian-based inverse kinematics. Comput Graph Forum 36:418–428. https://doi.org/10.1111/cgf.13089
Jiang F, Yang X, Feng L (2016) Real-time full-body motion reconstruction and recognition for off-the-shelf VR devices. In: Proceedings of the VRCAI 2016 15th ACM SIGGRAPH conference on virtual-reality continuum and its applications in industry, vol 1, pp 309–318. https://doi.org/10.1145/3013971.3013987
Kallmann M (2008) Analytical inverse kinematics with body posture control. Comput Animat Virtual Worlds 19:79–91. https://doi.org/10.1002/cav.176
Kenwright B (2012) Inverse Kinematics – cyclic coordinate descent (CCD). J Graph Tools 16:177–217. https://doi.org/10.1080/2165347x.2013.823362
Kim J, Seol Y, Lee J (2012) Realtime performance animation using sparse 3D motion sensors. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 7660 LNCS:31–42. https://doi.org/10.1007/978-3-642-34710-8_4
Korein JU (1985) Geometric Investigation of Reach. MIT Press
Kulpa R, Multon F, Arnaldi B (2005) Morphology-independent representation of motions for interactive human-like animation. Comput Graph Forum 24:343–351. https://doi.org/10.1111/j.1467-8659.2005.00859.x
Leoncini P, Sikorski B, Baraniello V, et al (2017) Multiple NUI device approach to full body tracking for collaborative virtual environments. Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics) 10324 LNCS:131–147. https://doi.org/10.1007/978-3-319-60922-5_10
Liu H, Wei X, Chai J, et al (2011) Realtime human motion control with a small number of inertial sensors. In: Proceedings of the symposium on interactive 3D graph, pp 133–140. https://doi.org/10.1145/1944745.1944768
Molla E, Boulic R (2013) Singularity free parametrization of human limbs. Proc - Motion Games 2013. MIG 2013:165–174. https://doi.org/10.1145/2522628.2522649
Muller-Cajar R, Mukundan R (2007) Triangualation—a new algorithm for inverse kinematics. Image vis Comput New Zeal 2007:181–186
Parger M, Schmalstieg D, Mueller JH, Steinberger M (2018) Human upper-body inverse kinematics for increased embodiment in consumer-grade virtual reality. Proc ACM Symp Virtual Real Softw Technol VRST. https://doi.org/10.1145/3281505.3281529
Seele S, Misztal S, Buhler H, et al (2017) Here’s Looking At You Anyway! 531–540. https://doi.org/10.1145/3116595.3116619
Shin HJ, Lee J, Shin SY, Gleicher M (2001) Computer puppetry: an importance-based approach. ACM Trans Graph 20:67–94. https://doi.org/10.1145/502122.502123
Starke S, Hendrich N, Zhang J (2019) Memetic evolution for generic full-body inverse kinematics in robotics and animation. IEEE Trans Evol Comput 23:406–420. https://doi.org/10.1109/TEVC.2018.2867601
Thomas JS, France CR, Leitkam ST et al (2016) Effects of real-world versus virtual environments on joint excursions in full-body reaching tasks. IEEE J Transl Eng Heal Med 4:1–8. https://doi.org/10.1109/JTEHM.2016.2623787
Tolani D, Badler NI (1996) Real-time inverse kinematics of the human arm. Presence Teleoperators Virtual Environ 5:393–401. https://doi.org/10.1162/pres.1996.5.4.393
Tolani D, Goswami A, Badler NI (2000) Real-time inverse kinematics techniques for anthropomorphic limbs. Graph Models 62:353–388. https://doi.org/10.1006/gmod.2000.0528
Tong L, Liu R, Peng L (2020) LSTM-based lower limbs motion reconstruction using low-dimensional input of inertial motion capture system. IEEE Sens J 20:3667–3677. https://doi.org/10.1109/JSEN.2019.2959639
Unzueta L, Peinado M, Boulic R, Suescun Á (2008) Full-body performance animation with Sequential Inverse Kinematics. Graph Models 70:87–104. https://doi.org/10.1016/j.gmod.2008.03.002
Von Marcard T, Pons-Moll G, Rosenhahn B (2016) Human pose estimation from video and IMUs. IEEE Trans Pattern Anal Mach Intell 38:1533–1547. https://doi.org/10.1109/TPAMI.2016.2522398
Wang L-CT, Chen CC (1991) A combined optimization method for solving the inverse kinematics problems of mechanical manipulators. IEEE Trans Robot Autom 7:489–499. https://doi.org/10.1109/70.86079
Xia S, Gao L, Lai YK et al (2017) A survey on human performance capture and animation. J Comput Sci Technol 32:536–554. https://doi.org/10.1007/s11390-017-1742-y
Yamane K, Nakamura Y (2003) Natural motion animation through constraining and deconstraining at will. IEEE Trans vis Comput Graph 9:352–360. https://doi.org/10.1109/TVCG.2003.1207443
Yi X, Zhou Y, Xu F (2021) TransPose: real-time 3D human translation and pose estimation with six inertial sensors. ACM Trans Graph 40:1–13. https://doi.org/10.1145/3450626.3459786
Acknowledgements
We appreciated Sebastian Starke for sharing the Unity3D implementation of the Bio-IK online. The data used in this project was obtained from http://mocap.cs.cmu.edu. The database was created with funding from NSF EIA-0196217.
Funding
No funding was received for conducting this study.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by [Qiang Zeng]. The first draft of the manuscript was written by [Qiang Zeng] and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflicts of interest
The authors have no relevant financial or non-financial interests to disclose.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Consent to participate
Not applicable.
Consent for publication
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file1 (MP4 328064 kb)
Rights and permissions
About this article
Cite this article
Zeng, Q., Zheng, G. & Liu, Q. PE-DLS: a novel method for performing real-time full-body motion reconstruction in VR based on Vive trackers. Virtual Reality 26, 1391–1407 (2022). https://doi.org/10.1007/s10055-022-00635-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-022-00635-5