Multimedia Tools and Applications

, Volume 76, Issue 1, pp 1291–1312 | Cite as

Increasing pose comprehension through augmented reality reenactment

  • Fabian Lorenzo DayritEmail author
  • Yuta Nakashima
  • Tomokazu Sato
  • Naokazu Yokoya


Standard video does not capture the 3D aspect of human motion, which is important for comprehension of motion that may be ambiguous. In this paper, we apply augmented reality (AR) techniques to give viewers insight into 3D motion by allowing them to manipulate the viewpoint of a motion sequence of a human actor using a handheld mobile device. The motion sequence is captured using a single RGB-D sensor, which is easier for a general user, but presents the unique challenge of synthesizing novel views using images captured from a single viewpoint. To address this challenge, our proposed system reconstructs a 3D model of the actor, then uses a combination of the actor’s pose and viewpoint similarity to find appropriate images to texture it. The system then renders the 3D model on the mobile device using visual SLAM to create a map in order to use it to estimate the mobile device’s camera pose relative to the original capturing environment. We call this novel view of a moving human actor a reenactment, and evaluate its usefulness and quality with an experiment and a survey.


Augmented reality Mobile Novel view synthesis Reenactment 



This work was partially supported by JSPS Grant-in-Aid for Scientific Research Nos. 23240024 and 25540086.


  1. 1.
    Alexiadis DS, Zarpalas D, Daras P (2013) Real-time, full 3-D reconstruction of moving foreground objects from multiple consumer depth cameras. IEEE Trans Multimed 15(2):339–358CrossRefGoogle Scholar
  2. 2.
    Anderson F, Grossman T, Matejka J, Fitzmaurice G (2013) YouMove: Enhancing movement training with an augmented reality mirror. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp 311–320Google Scholar
  3. 3.
    Azuma RT (1997) A survey of augmented reality. Presence 6(4):355–385CrossRefGoogle Scholar
  4. 4.
    Beck S, Kunert A, Kulik A, Froehlich B (2013) Immersive group-to-group telepresence. IEEE Trans Vis Comput Graph 19(4):616–625CrossRefGoogle Scholar
  5. 5.
    Carranza J, Theobalt C, Magnor M, Seidel H (2003) Free-viewpoint video of human actors. ACM Trans Graph 22(3):569–577CrossRefGoogle Scholar
  6. 6.
    Castle R, Klein G, Murray D (2008) Video-rate localization in multiple maps for wearable augmented reality. In: Proceedings of the IEEE International Symposium on Wearable Computers, pp 15–22Google Scholar
  7. 7.
    Dai B, Yang X (2013) A low-latency 3D teleconferencing system with image based approach. In: Proceedings of the ACM SIGGRAPH International Conference on Virtual-Reality Continuum and Its Applications in Industry, pp 243–248Google Scholar
  8. 8.
    Dayrit FL, Nakashima Y, Sato T, Yokoya N (2014) Free-viewpoint AR human-motion reenactment based on a single RGB-D video stream. In: Proceedings of the IEEE International Conference on Multimedia and ExpoGoogle Scholar
  9. 9.
    Debevec P, Taylor C, Malik J (1996) Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In: Proceedings of the ACM SIGGRAPH, pp 11–20Google Scholar
  10. 10.
    de Aguiar E, Stoll C, Theobalt C, Ahmed N, Seidel H, Thrun S (2008) Performance capture from sparse multi-view video. ACM Trans Graph 27(3)Google Scholar
  11. 11.
    Hauswiesner S, Straka M, Reitmayr G (2011) Image-based clothes transfer. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, pp 169–172Google Scholar
  12. 12.
    Henderson S, Feiner S (2011) Augmented reality in the psychomotor phase of a procedural task. In: Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, pp 191–200Google Scholar
  13. 13.
    Hilsmann A, Fechteler P, Eisert P (2013) Pose space image based rendering. In: Proceedings of the Computer Graphics Forum, vol 32, pp 265–274Google Scholar
  14. 14.
    Hondori H, Khademi M, Dodakian L, Cramer S, Lopes CV (2013) A spatial augmented reality rehab system for post-stroke hand rehabilitation. In: Proceedings of the Conference on Medicine Meets Virtual Reality, pp 279–285Google Scholar
  15. 15.
    Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fitzgibbon A (2011) KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp 559–568Google Scholar
  16. 16.
    Klein G, Murray D (2007) Parallel tracking and mapping for small AR workspaces. In: Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented RealityGoogle Scholar
  17. 17.
    Lorensen W, Cline H (1987) Marching cubes: A high resolution 3D surface construction algorithm. In: Proceedings of the ACM SIGGRAPH, vol 21, pp 163–169Google Scholar
  18. 18.
    Malleson C, Klaudiny M, Hilton A, Guillemaut JY (2013) Single-view RGBD-based reconstruction of dynamic human geometry. In: Proceedings of the International Workshop on Dynamic Shape Capture and Analysis, pp 307–314Google Scholar
  19. 19.
    Matusik W, Buehler C, Raskar R, Gortler S, McMillan L (2000) Image-based visual hulls. In: Proceedings of the ACM SIGGRAPH, pp 369–374Google Scholar
  20. 20.
    Pagés R, Berjón D, Morán F (2013) Automatic system for virtual human reconstruction with 3D mesh multi-texturing and facial enhancement. Signal Process Image Commun 28(9):1089–1099CrossRefGoogle Scholar
  21. 21.
    Shotton J, Sharp T, Kipman A, Fitzgibbon A, Finocchio M, Blake A, Cook M, Moore R (2013) Real-time human pose recognition in parts from single depth images. Commun ACM 56(1):116–124CrossRefGoogle Scholar
  22. 22.
    Shum H, Kang SB (2000) Review of image-based rendering techniques. Visual Communications and Image Processing:2–13Google Scholar
  23. 23.
    Velloso E, Bulling A, Gellersen H (2013) MotionMA: Motion modelling and analysis by demonstration. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp 1309–1318Google Scholar
  24. 24.
    Wang Z, Ong S, Nee A (2013) Augmented reality aided interactive manual assembly design. Int J Adv Manuf Technol 69(5-8):1311–1321CrossRefGoogle Scholar
  25. 25.
    Waschbüsch M, Würmlin S, Cotting D, Sadlo F, Gross M (2005) Scalable 3D video of dynamic scenes. Vis Comput 21(8-10):629–638CrossRefGoogle Scholar
  26. 26.
    Würmlin S, Lamboray E, Staadt O, Gross M (2002) 3D video recorder. In: Proceedings of the Pacific Conference on Computer Graphics and Applications, pp 325–334Google Scholar
  27. 27.
    Xu F, Liu Y, Stoll C, Tompkin J, Bharaj G, Dai Q, Seidel HP, Kautz J, Theobalt C (2011) Video-based characters: creating new human performances from a multi-view video database. ACM Trans Graph 30(4)Google Scholar
  28. 28.
    Yamabe T, Nakajima T (2013) Playful training with augmented reality games: case studies towards reality-oriented system design. Multimed Tools Appl 62(1):259–286CrossRefGoogle Scholar
  29. 29.
    Ye G, Liu Y, Deng Y, Hasler N, Ji X, Dai Q, Theobalt C (2013) Free-viewpoint video of human actors using multiple handheld Kinects. IEEE Trans Cybern 43(5):1370–1382CrossRefGoogle Scholar
  30. 30.
    Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334CrossRefGoogle Scholar
  31. 31.
    Zhou Z, Shu B, Zhuo S, Deng X, Tan P, Lin S (2012) Image-based clothes animation for virtual fitting. In: Proceedings of the ACM SIGGRAPH AsiaGoogle Scholar
  32. 32.
    Zitnick C, Kang S, Uyttendaele M, Winder S, Szeliski R (2004) High-quality video view interpolation using a layered representation. ACM Trans Graph 23(3):600–608CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  1. 1.Nara Institute of Science and TechnologyIkomaJapan

Personalised recommendations