Advertisement

Multimedia Tools and Applications

, Volume 78, Issue 23, pp 33899–33920 | Cite as

ARStudio

A low-cost virtual studio based on Augmented Reality for video production
  • Ivan A. Aguilar
  • Antonio C. Sementille
  • Silvio R. R. SanchesEmail author
Article
  • 51 Downloads

Abstract

Film, television, and Internet productions have generated content that combines real and virtual elements in the same scene. In a traditional production pipeline, virtual elements are inserted only in the post-production stage, and consequently, these elements are only visualized after every editing process is finalized. This approach can cause problems in situations such as ones where an actor acts alongside virtual characters, due to the lack of precise reference for gaze direction, for example. Such a problem, in the traditional production pipeline, can be identified only in the post-production stage. This can make production more expensive if there is a need to recruit the actors again and reassemble the infrastructure. Costs can also be avoided if part of the production team can follow along with certain retakes or even the entire production stage remotely. In addition, if the generated content can be quickly refined and made available to all the staff still in the production stage, the post-production stage can be considerably simplified. In this paper, we present a low-cost virtual studio called ARStudio that offers a set of functionalities similar to those of professional virtual studios. In addition, our system allows the production team to remotely follow the production stage and allows content refinements to be carried out quickly through tools, usually used in the post-production stage, that are integrated into the system.

Keywords

Virtual studio Augmented Reality Scene composition Production pipeline 

Notes

Supplementary material

11042_2019_8064_MOESM1_ESM.mkv (31 mb)
(MKV 31.0 MB)
11042_2019_8064_MOESM2_ESM.mkv (35.8 mb)
(MKV 35.7 MB)

References

  1. 1.
    Barbosa ECB (2015) Uma metodologia para a geração de conteúdos digitais baseado na utilização de estúdios virtuais com realidade aumentada. Master’s thesis, Universidade Estadual PaulistaGoogle Scholar
  2. 2.
    Bergh VD, Lalioti V (1999) Software chroma keying in an immersive virtual environment. South African Computer Journal 24:155–162Google Scholar
  3. 3.
    Blonde L, Buck M, Galli R, Niem W, Paker Y, Schmidt W, Thomas G (1996) A virtual studio for live broadcasting: the Mona Lisa project. IEEE MultiMedia 3(2):18–29.  https://doi.org/10.1109/93.502291 CrossRefGoogle Scholar
  4. 4.
    Boutaba R, Ren NN, Rasheed Y, Leon-Garcia A (2002) Distributed video production: Tasks, architecture and qos provisioning. Multimed Tools Appl 16 (1):99–136.  https://doi.org/10.1023/A:1013245819429 CrossRefzbMATHGoogle Scholar
  5. 5.
    Cho H, Jung SU, Jee HK (2017) Real-time interactive AR system for broadcasting. In: 2017 IEEE Virtual Reality (VR). IEEE, pp 353–354.  https://doi.org/10.1109/VR.2017.7892322
  6. 6.
    De Gaspari T, Sementille AC, Vielmas DZ, Aguilar IA, Marar JaF (2014) ARSTUDIO: a virtual studio system with augmented reality features. In: Proceedings of the 13th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry. VRCAI ’14. ACM, New York, pp 17–25. http://doi.acm.org/10.1145/2670473.2670491
  7. 7.
    De Goussencourt T, Bertolino P (2015) Using the unity® game engine as a platform for advanced real time cinema image processing. In: 2015 IEEE international conference on image processing (ICIP), pp 4146–4149.  https://doi.org/10.1109/ICIP.2015.7351586
  8. 8.
    Epic Games (2019) Unreal engine. https://www.unrealengine.com/en-US. Accessed 14 May 2019
  9. 9.
    Feng C, Feng T, Ye L (2019) When green screen meets panoramic videos: an interesting video combination framework. In: 2019 IEEE conference on multimedia information processing and retrieval (MIPR). IEEE, pp 480–485.  https://doi.org/10.1109/MIPR.2019.00097
  10. 10.
    Gibbs S, Arapis C, Breiteneder C, Lalioti V, Mostafawy S, Speier J (1998) Virtual studios: an overview. IEEE Multimedia 5(1):18–35.  https://doi.org/10.1109/93.664740 CrossRefGoogle Scholar
  11. 11.
    Hach T, Arias P, Bosch C, Montesa J, Gascó P (2017) Seamless 3D interaction of virtual and real objects in professional virtual studios. SMPTE Motion Imaging Journal 126(1):43–56.  https://doi.org/10.5594/JMI.2016.2632398  https://doi.org/10.5594/JMI.2016.2632398 CrossRefGoogle Scholar
  12. 12.
    Helzle V, Grau O, Knop T (2015) Digital representations of the real world: how to capture, model, and render visual reality. A K Peters/CRC Press, Boca Raton. chap Virtual ProductionGoogle Scholar
  13. 13.
    HIT Lab NZ (2018) ARToolkit. http://www.hitl.washington.edu/artoolkit/. Accessed 10 December 2018
  14. 14.
    Hughes JF, van Dam A, McGuire M, Sklar DF, Foley JD, Feiner SK, Akeley K (2013) Computer graphics: principles and practice, 3rd edn. Addison-Wesley Professional, BostonGoogle Scholar
  15. 15.
    Industrial Light & Magic (2018) OpenEXR. https://www.openexr.com Accessed 3 December 2017
  16. 16.
    Méndez R, Flores J, Castelló E, Viqueira JRR (2018) New distributed virtual tv set architecture for a synergistic operation of sensors and improved interaction between real and virtual worlds. Multimed Tools Appl 77(15):18999–19025.  https://doi.org/10.1007/s11042-017-5353-y CrossRefGoogle Scholar
  17. 17.
    Microsoft (2018) Kinect tools and resources. https://developer.microsoft.com/en-us/windows/kinect/tools. Accessed 11 April 2018
  18. 18.
    Natural Point (2018) OptiTrack. https://optitrack.com. Accessed 10 November 2018
  19. 19.
    Northam L, Istead J, Kaplan CS (2012) A collaborative real time previsualization tool for video games and film. In: ACM SIGGRAPH 2012 posters. SIGGRAPH ’12. ACM, New York, pp 121:1–121:1.  https://doi.org/10.1145/2342896.2343036
  20. 20.
    Owen CB, Zhou J, Tang KH, Xiao F (2006) Handbook of video databases: Design and applications. CRC Press, Boca Raton. chap Augmented imagery for digital video applicationsGoogle Scholar
  21. 21.
    Owens J, Millerson G (2012) Television production. Focal Press, WalthamCrossRefGoogle Scholar
  22. 22.
    Pires K, Simon G (2015) Youtube live and twitch: a tour of user-generated live streaming systems. In: Proceedings of the 6th ACM multimedia systems conference. MMSys ’15. ACM, New York, pp 225–230.  https://doi.org/10.1145/2713168.2713195
  23. 23.
    PTC (2019) Vuforia developer portal. https://developer.vuforia.com/. Accessed 12 March 2019
  24. 24.
    Reinhard E, Ward G, Pattanaik S, Debevec P (2006) The human visual system and {HDR} tone mapping. In: Reinhard E, Ward G, Pattanaik S, Debevec P (eds) High dynamic range imaging, the Morgan Kaufmann series in computer graphics. Morgan Kaufmann, San Francisco, pp 187–221.  https://doi.org/10.1016/B978-012585263-0/50007-5 CrossRefGoogle Scholar
  25. 25.
    Sanches SRR, Tokunaga DM, Silva VF, Sementille AC, Tori R (2012) Mutual occlusion between real and virtual elements in augmented reality based on fiducial markers. In: Proceedings of IEEE workshop on applications of computer vision – WACV 2012. IEEE Computer Society, Washington, pp 49–54.  https://doi.org/10.1109/WACV.2012.6163037
  26. 26.
    Sanches SRR, Silva VF, Nakamura R, Tori R (2013) Objective assessment of video segmentation quality for augmented reality. In: Proceedings of IEEE International Conference on Multimedia and Expo – ICME 2013. IEEE Computer Society, Washington, pp 1–6.  https://doi.org/10.1109/ICME.2013.6607476
  27. 27.
    Saraiji Y (2019) UnityCam. https://github.com/mrayy/UnityCam. Accessed 30 May 2019
  28. 28.
    Shimoda S, Hayashi M, Kanatsugu Y (1989) New chroma-key imagining technique with hi-vision background. IEEE Trans Broadcast 35(4):357–361.  https://doi.org/10.1109/11.40835 CrossRefGoogle Scholar
  29. 29.
    Thomas G (2006) Mixed reality techniques for tv and their application for on-set and pre-visualization in film production. In: International workshop on mixed reality technology for filmmaking, pp 31–36Google Scholar
  30. 30.
    Trottnow J, Götz K, Seibert S, Spielmann S, Helzle V, Einabadi F, Sielaff CKH, Grau O (2015) Intuitive virtual production tools for set and light editing. In: Proceedings of the 12th European conference on visual media production. CVMP ’15. ACM, New York, pp 6:1–6:8.  https://doi.org/10.1145/2824840.2824851
  31. 31.
    Ultrahaptics Ltd (2018) Leap motion. https://www.leapmotion.com. Accessed 5 October 2018
  32. 32.
    Unity Technologies (2019) Unity. https://unity.com. Accessed 14 May 2019
  33. 33.
    Wojdala A (1998) Challenges of virtual set technology. IEEE Multimedia 5(1):50–57.  https://doi.org/10.1109/93.664742 CrossRefGoogle Scholar
  34. 34.
    Wright S (2010) Digital compositing for film and video, 3rd edn. Focal Press, WalthamGoogle Scholar
  35. 35.
    Yang K, Park T (2015) K-motion: visualizing election information for live television broadcasts. Multimed Tools Appl 74(24):11631–11651.  https://doi.org/10.1007/s11042-014-2253-2 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Universidade Estadual Paulista “Júlio de Mesquita Filho”BauruBrazil
  2. 2.Universidade Tecnológica Federal do ParanáCornélio ProcópioBrazil

Personalised recommendations