Rapid Reconstruction and Simulation of Real Characters in Mixed Reality Environments

Part of the Lecture Notes in Computer Science book series (LNCS, volume 10605)


This paper presents a comparison of latest software and hardware methods for rapid reconstruction of real humans using as an input RGB or RGB-D images, and base on this comparison is introduced the pipeline that produces high realistic reconstructions in a reasonable amount of time, suitable for real-time Virtual Reality (VR), Augmented Reality (AR) as well as Holographic Mixed Reality (HMR). In this work, we also present and compare usage of latest VR and AR Head Mounted displays (HDMs), which are Microsoft Hololens, Oculus Rift and mobile AR. Specifically, we compare the immersion experience, interaction system, field of view and level of presence that each of these technologies provide. We demonstrate our results at Asinou church, a UNESCO Cultural Heritage monument located in Cyprus. Our reconstructed virtual narrator is the real priest of Asinou church which gives a virtual tour in the church. This interactive virtual narrator supports a range of different capabilities like performing gestures, speech and lip synchronization.



The research leading to these results was partially funded by the European Union People Programme (FP7-PEOPLE-2013-ITN) under grant agreement 608013 and was partially funded by the Virtual Multimodal Museum (ViMM), a Coordination and Support Action (CSA), funded under the EU Horizon 2020 programme (CULT-COOP-8-2016).


  1. 1.
    Microsoft Hololens. Accessed 12 May 2017
  2. 2.
    Mixamo. Accessed 12 May 2017
  3. 3.
    Arnold, D., Day, A., Glauert, J., Haegler, S., Jennings, V., Kevelham, B., Laycock, R., Magnenat-Thalmann, N., Maïm, J., Maupu, D., Papagiannakis, G., Thalmann, D., Yersin, B., Rodriguez-Echavarria, K.: Tools for populating cultural heritage environments with interactive virtual humans. In: Open Digital Cultural Heritage Systems, EPOCH Final Event 2008, February 2008Google Scholar
  4. 4.
    Georgopoulos, A.: Data acquisition for the geometric documentation of cultural heritage. In: Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, G. (eds.) Mixed Reality and Gamification for Cultural Heritage, pp. 29–73. Springer, Cham (2017). Scholar
  5. 5.
    Hildenbrand, D.: Foundations of Geometric Algebra Computing, vol. 8. Springer, Heidelberg (2013). Scholar
  6. 6.
    Kateros, S., Georgiou, S., Papaefthymiou, M., Papagiannakis, G., Tsioumas, M.: A comparison of gamified, immersive VR curation methods for enhanced presence and human-computer interaction in digital humanities. Int. J. Heritage Digital Era 4(2), 221–233 (2015). Also presented in The 1st International Workshop on ICT for the Preservation and Transmission of Intangible Cultural Heritage, EUROMED2014CrossRefGoogle Scholar
  7. 7.
    Papaefthymiou, M., Feng, A., Shapiro, A., Papagiannakis, G.: A fast and robust pipeline for populating mobile AR scenes with gamified virtual characters. In: SIGGRAPH Asia 2015 Mobile Graphics and Interactive Applications, SA 2015, pp. 22:1–22:8. ACM, New York (2015)Google Scholar
  8. 8.
    Papaefthymiou, M., Kateros, S., Georgiou, S., Lydatakis, N., Zikas, P., Bachlitzanakis, V., Papagiannakis, G.: Gamified AR/VR character rendering and animation-enabling technologies. In: Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, G. (eds.) Mixed Reality and Gamification for Cultural Heritage. Springer, Cham (2017). Scholar
  9. 9.
    Papagiannakis, G., Schertenleib, S., O’Kennedy, B., Arevalo-Poizat, M., Magnenat-Thalmann, N., Stoddart, A., Thalmann, D.: Mixing virtual and real scenes in the site of ancient pompeii: research articles. Comput. Animat. Virtual Worlds 16(1), 11–24 (2005)CrossRefGoogle Scholar
  10. 10.
    Ramamoorthi, R., Hanrahan, P.: An efficient representation for irradiance environment maps. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001, pp. 497–500. ACM, New York (2001)Google Scholar
  11. 11.
    Shapiro, A., Feng, A., Wang, R., Li, H., Bolas, M., Medioni, G., Suma, E.: Rapid avatar capture and simulation using commodity depth sensors. Comput. Anim. Virtual Worlds 25(3–4), 201–211 (2014)CrossRefGoogle Scholar
  12. 12.
    Vacchetti, L., Lepetit, V., Ponder, M., Papagiannakis, G., Fua, P., Thalmann, D., Thalmann, N.M.: A stable real-time AR framework for training and planning in industrial environments. In: Ong, S.K., Nee, A.Y.C. (eds.) Virtual and Augmented Reality Applications in Manufacturing, pp. 129–145. Springer, London (2004). Scholar
  13. 13.
    Verbiest, F., Proesmans, M., Van Gool, L.: Autonomous mapping of the Priscilla Catacombs. In: Ioannides, M., Magnenat-Thalmann, N., Papagiannakis, G. (eds.) Mixed Reality and Gamification for Cultural Heritage, pp. 75–98. Springer, Cham (2017). Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Institute of Computer ScienceFoundation for Research and Technology HellasHeraklionGreece
  2. 2.Computer Science DepartmentUniversity of CreteHeraklionGreece
  3. 3.ovidVRHeraklionGreece

Personalised recommendations