Advertisement

Real-time body tracking in virtual reality using a Vive tracker

Abstract

Due to recent improvements in virtual reality (VR) technology, the number of novel applications for entertainment, education, and rehabilitation has increased. The primary goal of these applications is to enhance the sense of belief that the user is “present” in the virtual environment. By tracking the user’s skeleton in real-time, it is possible to synchronize the avatar’s motions with the user’s motions. Although current common devices implement body tracking to a certain degree, most approaches are limited by either high latency or insufficient accuracy. Due to the lack of positional and rotation data, the current VR applications typically do not represent the user’s motions. In this paper, we present an accurate, low-latency body tracking approach for VR-based applications using Vive Trackers. Using a HTC Vive headset and Vive Trackers, we have been able to create an immersive VR experience, by animating the motions of the avatar as smoothly, rapidly and as accurately as possible. An evaluation showed our solution is capable of tracking both joint rotation and position with reasonable accuracy and a very low end-to-latency of \(6.71 \pm 0.80\hbox { ms}\). Due to this merely imperceptible delay and precise tracking, our solution can show the movements of the user in real-time in order to create deeper immersion.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Notes

  1. 1.

    Vive Tracker: https://www.vive.com/us/vive-tracker/, last visited on April 3rd, 2018.

  2. 2.

    IKinema Orion: https://ikinema.com/orion, last visited on April 3rd, 2018.

  3. 3.

    Vive IK Demo: https://github.com/JamesBear/vive_ik_demo, last visited on April 3rd, 2018.

  4. 4.

    Final IK: https://assetstore.unity.com/packages/tools/animation/final-ik-14290, last visited on April 3rd, 2018.

  5. 5.

    Body Tracking Demo: https://github.com/CatCuddler/BodyTracking, last visited on April 4th, 2018.

  6. 6.

    Microsoft Kinect: https://developer.microsoft.com/en-us/windows/kinect, last visited on January 28th, 2018.

  7. 7.

    Nintendo Wii: https://www.nintendo.co.uk/Wii/Wii-94559.html, last visited on January 28th, 2018.

  8. 8.

    OptiTrack system: http://www.optitrack.com, last visited on January 17th, 2018.

  9. 9.

    PrioVR: https://yostlabs.com/priovr/, last visited on July 31st, 2018.

  10. 10.

    Perception Neuron: https://neuronmocap.com, last visited on July 31st, 2018.

  11. 11.

    Xsense: https://www.xsens.com/, last visited on 31st July, 2018.

  12. 12.

    MakeHuman: http://www.makehuman.org, last visited on February 3rd, 2018.

  13. 13.

    Kore: https://github.com/Kode/Kore, last visited on April 3rd, 2018.

  14. 14.

    OpenGEX: http://opengex.org, last visited on February 21st, 2018.

  15. 15.

    KCF Tracker: http://docs.opencv.org/trunk/d2/dff/classcv_1_1TrackerKCF.html, last visited on February 17th, 2017.

  16. 16.

    Minimum requirements: https://www.vive.com/us/ready/, last visited on February 5th, 2018.

  17. 17.

    Question: “I find the VR in general exciting”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=4.92\), \({\hbox {SD}}=\pm \,0.27\), question: “I like the idea of body tracking in VR”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=4.92\), \({\hbox {SD}}=\pm \,0.27\).

  18. 18.

    Question: “I felt like I was a part of the VR”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=4.3\), \({\hbox {SD}}=\pm \,0.48\), question: “I could identify myself with the avatar”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=4.07\), \({\hbox {SD}}=\pm \,0.49\).

  19. 19.

    Question: “The movements in the VR have corresponded to the real movements”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=4.23\), \({\hbox {SD}}=\pm \,0.59\), question: “The tracking was accurate”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=4.07\), \({\hbox {SD}}=\pm \,0.64\).

  20. 20.

    Question: “The movements of the avatar were delayed”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=1.23\), \({\hbox {SD}}=\pm \,0.43\).

  21. 21.

    Question: “The tracking had some jitter problems”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=2.37\), \({\hbox {SD}}=\pm \,1.25\).

  22. 22.

    Question: “I would like body tracking also in other VR games”, five-level Likert scale, \(N=13\), \({\hbox {AVR}}=4.92\), \({\hbox {SD}}=\pm \,0.27\).

  23. 23.

    Leap Motion: https://www.leapmotion.com, last visited on January 19th, 2018.

  24. 24.

    Hi5 VR Glove: https://hi5vrglove.com, last visited on January 19th, 2018.

  25. 25.

    VRgluv: https://vrgluv.com, last visited on January 19th, 2018.

  26. 26.

    HaptX: https://haptx.com, last visited on January 19th, 2018.

  27. 27.

    VRtouch: https://www.gotouchvr.com/order_vrtouch/, last visited on January 19th, 2018.

References

  1. Aristidou A, Lasenby J (2011) FABRIK: a fast, iterative solver for the inverse kinematics problem. Graph Models 73(5):243–260

  2. Banakou D, Groten R, Slater M (2013) Illusory ownership of a virtual child body causes overestimation of object sizes and implicit attitude changes. Proc Natl Acad Sci 110(31):12846–12851

  3. Bolton J, Lambert M, Lirette D, Unsworth B (2014) PaperDude: a virtual reality cycling exergame. CHI’14 Extended Abstracts on Human Factors in Computing Systems. CHI EA’14. ACM, New York, NY, USA, pp 475–478

  4. Botev J, Rothkugel S (2017) High-precision gestural input for immersive large-scale distributed virtual environments. In: Proceedings of the 9th workshop on massively multiuser virtual environments, MMVE’17. ACM, New York, NY, USA, pp 7–11

  5. Caserman P, Krabbe P, Wojtusch J, von Stryk O (2016) Real-time step detection using the integrated sensors of a head-mounted display. In: 2016 IEEE international conference on systems, man, and cybernetics (SMC), pp 3510–3515

  6. Chan JCP, Leung H, Tang JKT, Komura T (2011) A virtual reality dance training system using motion capture technology. IEEE Trans Learn Technol 4(2):187–195

  7. Choi SW, Seo MW, Lee SL, Park JH, Oh EY, Baek JS, Kang SJ (2016) Head position model-based latency measurement system for virtual reality head mounted display. SID Symp Dig Tech Papers 47(1):1381–1384

  8. Collingwoode-Williams T, Gillies M, McCall C, Pan X (2017) The effect of lip and arm synchronization on embodiment: a pilot study. In: 2017 IEEE virtual reality (VR). IEEE, pp 253–254

  9. Dempsey P (2016) The teardown: HTC Vive VR headset. Eng Technol 11(7–8):80–81

  10. Desai PR, Desai PN, Ajmera KD, Mehta K (2014) A review paper on oculus rift—a virtual reality headset. Int J Eng Trends Technol (IJETT) 13(4):175–179

  11. Desai K, Raghuraman S, Jin R, Prabhakaran B (2017) QoE studies on interactive 3D tele-immersion. In: 2017 IEEE international symposium on multimedia (ISM), pp 130–137

  12. Farahani N, Post R, Duboy J, Ahmed I, Kolowitz BJ, Krinchai T, Monaco SE, Fine JL, Hartman DJ, Pantanowitz L (2016) Exploring virtual reality technology and the oculus rift for the examination of digital pathology slides. J Pathol Inform 7:22

  13. Friðriksson FA, Kristjánsson HS, Sigurðsson DA, Thue D, Vilhjálmsson HH (2016) Become your avatar: fast skeletal reconstruction from sparse data for fully-tracked VR. In: Proceedings of the 26th international conference on artificial reality and telexistence and the 21st Eurographics symposium on virtual environments: posters and demos, pp 19–20

  14. Friston S, Steed A (2014) Measuring latency in virtual environments. IEEE Trans Vis Comput Graph 20(4):616–625

  15. Galna B, Barry G, Jackson D, Mhiripiri D, Olivier P, Rochester L (2014) Accuracy of the microsoft kinect sensor for measuring movement in people with Parkinson’s disease. Gait Posture 39(4):1062–1068

  16. Goradia I, Doshi J, Kurup L (2014) A review paper on oculus rift & project morpheus. Int J Curr Eng Technol 4(5):3196–3200

  17. Grochow K, Martin SL, Hertzmann A, Popović Z (2004) Style-based inverse kinematics. ACM Trans Graph 23(3):522–531

  18. Huang J, Wang Q, Fratarcangeli M, Yan K, Pelachaud C (2017) Multi-variate gaussian-based inverse kinematics. Comput Graph Forum 36(8):418–428

  19. Jain D, Sra M, Guo J, Marques R, Wu R, Chiu J, Schmandt C (2016) Immersive terrestrial scuba diving using virtual reality. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. ACM, New York, USA, pp 1563–1569

  20. Jiang F, Yang X, Feng L (2016) Real-time full-body motion reconstruction and recognition for off-the-shelf VR devices. In: Proceedings of the 15th ACM SIGGRAPH conference on virtual-reality continuum and its applications in industry—Volume 1, VRCAI’16. ACM, pp 309–318

  21. Johnson M, Humer I, Zimmerman B, Shallow J, Tahai L, Pietroszek K (2016) Low-cost latency compensation in motion tracking for smartphone-based head mounted display. In: Proceedings of the international working conference on advanced visual interfaces, AVI’16. ACM, New York, NY, USA, pp 316–317

  22. Kasahara S, Konno K, Owaki R, Nishi T, Takeshita A, Ito T, Kasuga S, Ushiba J (2017) Malleable embodiment: changing sense of embodiment by spatial-temporal deformation of virtual human body. In: Proceedings of the 2017 CHI conference on human factors in computing systems, CHI’17. ACM, New York, NY, USA, pp 6438–6448

  23. Kavan L, Sloan PP, O’Sullivan C (2010) Fast and efficient skinning of animated meshes. Comput Graph Forum 29(2):327–336

  24. Kenwright B (2012) Real-time character inverse kinematics using the Gauss–Seidel iterative approximation method. Int Conf Creat Content Technol 4:63–68

  25. Lange B, Rizzo S, Chang CY, Suma EA, Bolas M (2011) Markerless full body tracking: depth-sensing technology within virtual environments. In: Interservice/industry training, simulation, and education conference (I/ITSEC)

  26. Latoschik ME, Lugrin JL, Habel M, Roth D, Seufert C, Grafe S (2016) Breaking bad behavior: immersive training of class room management. In: Proceedings of the 22nd ACM conference on virtual reality software and technology, VRST’16. ACM, New York, NY, USA, pp 317–318

  27. Latoschik ME, Roth D, Gall D, Achenbach J, Waltemate T, Botsch M (2017) The effect of avatar realism in immersive social virtual realities. In: Proceedings of the 23rd ACM symposium on virtual reality software and technology, VRST’17. ACM, New York, NY, USA, pp 39:1–39:10

  28. Martindale J (2018) Oculus Rift vs. HTC Vive. https://www.digitaltrends.com/virtual-reality/oculus-rift-vs-htc-vive/. Accessed 4 May 2017​

  29. Melo M, Rocha T, Barbosa L, Bessa M (2016) The impact of body position on the usability of multisensory virtual environments: case study of a virtual bicycle. In: Proceedings of the 7th international conference on software development and technologies for enhancing accessibility and fighting info-exclusion, DSAI 2016. ACM, New York, NY, USA, pp 20–24

  30. Nakamura Y, Hanafusa H (1986) Inverse kinematic solutions with singularity robustness for robot manipulator control. J Dyn Syst Meas Control 108(3):163–171

  31. Orin DE, Schrader WW (1984) Efficient computation of the Jacobian for robot manipulators. Int J Robot Res 3(4):66–75

  32. Peck TC, Seinfeld S, Aglioti SM, Slater M (2013) Putting yourself in the skin of a black avatar reduces implicit racial bias. Conscious Cognit 22(3):779–787

  33. Raaen K, Kjellmo I (2015) Measuring latency in virtual reality systems. In: Chorianopoulos K, Divitini M, Baalsrud Hauge J, Jaccheri L, Malaka R (eds) Entertainment computing—ICEC 2015. Springer, Cham, pp 457–462

  34. Roberts D, Duckworth T, Moore C, Wolff R, O’Hare J (2009) Comparing the end to end latency of an immersive collaborative environment and a video conference. In: Proceedings of the 2009 13th IEEE/ACM international symposium on distributed simulation and real time applications, DS-RT’09. IEEE Computer Society, Washington, DC, USA, pp 89–94

  35. Schmidt D, Kovacs R, Mehta V, Umapathi U, Köhler S, Cheng LP, Baudisch P (2015) Level-ups: motorized stilts that simulate stair steps in virtual reality. In: Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems, CHI EA’15. ACM, New York, NY, USA, pp 359–362

  36. Seele S, Misztal S, Buhler H, Herpers R, Schild J (2017) Here’s looking at you anyway!: how important is realistic gaze behavior in co-located social virtual reality games? In: Proceedings of the annual symposium on computer-human interaction in play, CHI PLAY’17. ACM, New York, NY, USA, pp 531–540

  37. Shoemake K (1985) Animating rotation with quaternion curves. In: Proceedings of the 12th annual conference on computer graphics and interactive techniques, SIGGRAPH’85. ACM, New York, NY, USA, pp 245–254

  38. Shum H, Ho ES (2012) Real-time physical modelling of character movements with microsoft kinect. In: Proceedings of the 18th ACM symposium on virtual reality software and technology, VRST’12. ACM, pp 17–24

  39. Sra M, Schmandt C (2015) MetaSpace II: object and full-body tracking for interaction and navigation in social VR. CoRR abs/1512.02922

  40. Steed A (2008) A simple method for estimating the latency of interactive, real-time graphics simulations. In: Proceedings of the 2008 ACM symposium on virtual reality software and technology, VRST’08. ACM, New York, NY, USA, pp 123–129

  41. Tao G, Archambault PS, Levin MF (2013) Evaluation of kinect skeletal tracking in a virtual reality rehabilitation system for upper limb hemiparesis. In: 2013 international conference on virtual rehabilitation (ICVR), pp 164–165

  42. Thomas JS, France CR, Leitkam ST, Applegate ME, Pidcoe PE, Walkowski S (2016) Effects of real-world versus virtual environments on joint excursions in full-body reaching tasks. IEEE J Transl Eng Health Med 4:1–8

  43. Tsai TC, Chen CY, Su GJ (2015) U-art: your art and ubiquitous art. In: Adjunct proceedings of the 2015 ACM international joint conference on pervasive and ubiquitous computing and proceedings of the 2015 ACM international symposium on wearable computers, UbiComp/ISWC’15 Adjunct. ACM, New York, NY, USA, pp 1295–1302

Download references

Author information

Correspondence to Polona Caserman.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 31171 KB)

Supplementary material 1 (mp4 31171 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Caserman, P., Garcia-Agundez, A., Konrad, R. et al. Real-time body tracking in virtual reality using a Vive tracker. Virtual Reality 23, 155–168 (2019). https://doi.org/10.1007/s10055-018-0374-z

Download citation

Keywords

  • Virtual reality
  • Real-time tracking
  • Full-body avatar
  • Low-latency
  • HTC Vive tracker
  • Inverse kinematics