Skip to main content
SpringerLink
Log in

The Evolution of the Virtual Production Studio as a Game Changer in Filmmaking

  • Chapter
  • First Online:
Creating Digitally

Part of the book series: Intelligent Systems Reference Library ((ISRL,volume 241))

  • 408 Accesses

Abstract

This chapter analyses the key methodological and conceptual shifts in filmmaking brought about by the integrated use of VR technology, game engines, and LED walls in Virtual Production Studios (VPS). VPS combine the real time processing capabilities of game engines with LED walls and camera tracking to produce live in-camera visual effects. In this chapter, we focus not only on the elements that make an integrated VPS system, but also on what makes it qualitatively different from traditional and digital filmmaking. The methodology of the VPS transforms the relationships among display technology (screen), live action (actors), image production (cameras), and postproduction (visual effects or VFX). An understanding of how these technologies work together will form the premise for an analysis of the qualitative and performative shifts that filmmaking is undergoing. There are a number of crucial elements of an integrated VPS technology: a game engine, LED walls, a camera tracking system, a motion tracking system, Simulcam, and a Render. VP has its roots in traditional filmmaking techniques associated with combining different images in one shot, such as back projection and green screen. The back projection technique immerses performers in the environment of the action with an analogue (filmed) or a digital backlot (this technique was used, for example, in the film Oblivion in 2013), but brings the problem of matching the brightness of foreground and background, and, more crucially, it requires the camera on set to be still in order to avoid problems with parallax. The traditional green screen technique allows actors to perform with placeholder props and extras in an empty room lined with an even colour background. Neither of them is ideal. More recently, LED walls have replaced green and blue screen techniques, creating an active stage for actors’ performance in a VPS and a safe passage from digital to virtual backlot for filmmakers. While with a green screen stage and digital backlot the final image is composed on screen, with LED walls and a virtual backlot the final image is displayed on LED walls and updated in real time using a game engine. VFX are created in camera, making the distinction between screens and cameras superfluous, to some extent. VPS systems based on game engines have been tested since 2001, when Peter Jackson used a virtual camera in Lord of the Rings and Steven Spielberg used an iPad attached to a Vive controller to visualise potential shots in a CG environment updated in real time for the film AI. The technological components of a VPS have become progressively more sophisticated, with studios testing and adopting in-house solutions to issues of synchronisation, data management, and lighting. Everything changed when the game company Epic Games partnered with Lux Machina (LED walls), Magnopus (VR/AR), Profile Studios (mocap and VP), Quixel (3D graphics), and ARRI (cameras) to launch an out-of-the-box VPS solution at SIGGRAPH 2019 in Los Angeles with the director of photography Matt Workman. Since these developments are so recent, existing literature generally focuses on technical details and individual components of the VPS system, rather than on its impact on filmmaking. The use of LED walls in a VPS makes a radical methodological and conceptual difference to live performance and to filmmaking itself. The game changer here is the link between on-stage camera and virtual in-game-engine camera that makes changes in real time on the LED wall, as virtual camera movements are updated in sync with the movements of the on-stage camera. The analysis in this paper is supported by a selected review of publications relevant to the subject to address the domains that require further investigation. Publications that offer overviews and integrated analyses are beginning to emerge, particularly as educational institutions start to develop VFX courses that encompass VP, but an in-depth conceptual analysis of the changes in progress to capture this unique historical moment is of critical importance. We present a review of how VP and game engine technologies have evolved and converged, as well as an overview of the impact of VPS on filmmaking.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 59.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Sandin, D.J.: Digital illusion, virtual reality and cinema. In Dodsworth, C. (ed) Digital illusion: entertaining the future with high technology. Addison-Wesley Professional: Boston, Ma. 3–26 (1998)

    Google Scholar 

  2. Martin, K.H.: Atomic fiction answers the call for innovation in welcome to Marwen. VFXV (2019). https://www.vfxvoice.com/atomic-fiction-answers-the-call-for-innovation-in-welcome-to-marwen

  3. Kadner, N.: The Virtual Production Field Guide. Volume 1. Epic Games (2019). https://cdn2.unrealengine.com/Unreal+Engine%2Fvpfieldguide%2FVP-Field-Guide-V1.2.02-5d28ccec9909ff626e42c619bcbe8ed2bf83138d.pdf

  4. Weta Digital: Virtual Production (2021). https://www.wetafx.co.nz/research-and-tech/technology/virtual-production

  5. Lowood, H., Guins, R.: Debugging game history: a critical lexicon. The MIT Press Cambridge, Massachusetts London, England (2016)

    Google Scholar 

  6. Kipnis, L.: Film and changing technologies in the oxford guide to film studies Hill, J. and Church Gibson, P. (eds) Oxford: Oxford University Press, pp. 595–604 (1998)

    Google Scholar 

  7. Musser, C.: Toward a History of Screen Practice. Quart. Rev. Cinema Stud. 9(1)(Winter 1984), 59–69 (1984)

    Google Scholar 

  8. Rizzo, M.: The Art Direction Handbook for Film & Television, 2nd ed. Taylor & Francis Group (2014)

    Google Scholar 

  9. McGowan, F.: The use of rear screen projection in King Kong (2014). https://francismcgowanvfx.wordpress.com/2014/10/05/the-use-of-rear-screen-projection-in-king-kong/

  10. Lightman, H.A.: Filming 2001: a space Odyssey. American Cinematographer (2018). Retrieved on 15 July 2022. https://ascmag.com/articles/filming-2001-a-space-odyssey

  11. Turnock, J.: The Screen on the Set: The Problem of Classical-Studio Rear Projection. Cinema J. 51(2) (Winter 2012), 157–162. University of Texas Press on behalf of the Society for Cinema & Media Studies (2012)

    Google Scholar 

  12. Sherman, T.: “The Premature Birth of Video Art” University (2007). Retrieved from http://videohistoryproject.org/sites/default/files/history/pdf/ShermanThePrematureBirthofVideoArt_2561.pdf

  13. Meigh-Andrews, C.: A History of video art: the development of form and function. Bloomsbury, London (2014)

    Google Scholar 

  14. Daly, K.M.: 2010. How cinema is digital. In Einav, G. (ed.) Transitioned media. a turning point into the digital realm. 135–147 (2010)

    Google Scholar 

  15. Dent, S.: Film's cinema comeback is driven by nostalgia, not logic. Engadget (2016). https://www.engadget.com/2016-02-15-films-cinema-comeback-is-driven-by-nostalgia-not-logic.html

  16. Layzelle, D.: History of the Game Engine: Part 2—The 1990s (2021b). https://ultimategamingparadise.com/features/series/history-of-the-game-engine/history-of-the-game-engine-part-2/

  17. Layzelle, D.: History of the Game Engine: Part 1 (2021a). https://ultimategamingparadise.com/features/series/history-of-the-game-engine/history-of-the-game-engine-part-1

  18. O'Toole-Bateman, C.: The History of the Game Engine: Part 11—Unreal Engine Builds Bridges (2022). https://ultimategamingparadise.com/features/series/history-of-the-game-engine/part-11-unreal-engine/

  19. McWhertor, M.: Former EA CEO John Riccitiello is now head of Unity. Polygon (2014). https://www.polygon.com/2014/10/22/7039683/electronic-arts-john-riccitiello-unity-ceo

  20. Smykill, J.: Apple design award winners announced. ArsTechnica. (2006). https://arstechnica.com/gadgets/2006/08/4937

  21. O'Toole-Bateman, C.: The History of the Game Engine: Part 9—Fight! (2021). https://ultimategamingparadise.com/features/series/history-of-the-game-engine/part-9-fight/

  22. Niklaus, L.: Unity Vs Unreal: Which Fits You Best? Mount CG. https://mountcg.com/unity-vs-unreal/. ND

  23. Ware, C., Osborne, S.: Exploration and virtual camera control in virtual three dimensional environments. ACM SIGGRAPH, 24(2):175–183 (1990)

    Google Scholar 

  24. McKenna, M.: Interactive viewpoint control and three-dimensional operations. Proceedings of the 1992 Symposium on Interactive 3D Graphics (I3D '92). ACM. pp. 53–56. https://doi.org/10.1145/147156.147163

  25. Phillips, C.B., Badler, N.I., Granieri J.: Automatic viewing control for 3D direct manipulation. In ACM SIGGRAPH Symposium on Interactive 3D graphics, pp. 71–74, Cambridge, Massachusetts, USA, 1992. ACM Press

    Google Scholar 

  26. Drucker, S.M., Zeltzer, D.: CamDroid: a system for implementing intelligent camera control. ACM Symposium on Interactive 3D Graphics and Games. (1995)

    Google Scholar 

  27. Brooks, F.P. et. al.: Final technical report: Walkthrough Project, Six generations of building walkthroughs. Tech. Rep. Dept. of Computer Science, Univ of North Carolina, Chapel Hill, N.C. (1992)

    Google Scholar 

  28. Burelli, P.: Virtual cinematography in games: investigating the impact on player experience. International Conference On The Foundations of Digital Games, Chania, Greece, pp. 1–8 (2013)

    Google Scholar 

  29. Christianson D.B., Anderson S.E., He L.-W., Weld D.S., Cohen M.F., Salesin D.H.: Declarative camera control for automatic cinematography. In AAAI (1996), pp. 148–155

    Google Scholar 

  30. Lino, C., Christie, M., Lamarche, F., Schofield, G., Olivier, P.: A real-time cinematography system for interactive 3D environments. In Eurographics ACM SIGGRAPH Symposium on Computer Animation, pp 139–148 (2010)

    Google Scholar 

  31. Lewinski, J.S.: Resident Evil 5 Offers Sneak Peek at Avatar's ‘Virtual Camera'. Wired (2009). https://www.wired.com/2009/02/resident-evil-5-6

  32. Marino, P.: 3D Game-Based filmmaking: the art of machinima. Scottsdale, AZ: Paraglyph Press (2004)

    Google Scholar 

  33. Harwood, T.G. Grussi, B.: Pioneers in machinima: grassroots of virtual production. Wilmington: Vernon Press (2021)

    Google Scholar 

  34. Shonig, J.: The Shape of Motion: Cinema and the Aesthetics of Movement. Oxford University Press (2021)

    Google Scholar 

  35. Carroll, J., Cameron, D.: Machinima: digital performance and emergent authorship. In changing views—worlds in play. Proceedings of DiGRA Conference. (2005) Retrieved 9 November 2021, from http://www.digra.org/dl/db/06276.32151.pdf

  36. Lightman, H.A.: Front projection for 2001: a space Odyssey. In Schwam, S. (ed) The Making of 2001: A Space Odyssey. New York, Modern Library (2000)

    Google Scholar 

  37. Film 4: Tom Cruise on Oblivion. Film4 Interview Special (2013). Retrieved from https://www.youtube.com/watch?v=ielGQM63Z6o&t=252s&ab_channel=Film4

  38. Definition Magazine.: Earth bound: the oblivion movie and how to avoid blue screen (2013). Retrieved on 28 June 2022. https://definitionmagazine.com/features/earth-bound-the-oblivion-movie-and-how-to-avoid-blue-screen

  39. Hive Division: LED Wall Virtual Production - Bye Bye Green Screen (2020). https://www.youtube.com/watch?v=ysIOi_MP_cs&ab_channel=HiveDivision

  40. Mitchell, J.: Frontier Man: Cinematographer Greig Fraser (2022). Retrieved on 15 May https://www.premiumbeat.com/blog/greig-fraser-cinematographer

  41. Atkinson, S.: Interactive ‘making-of’ machines: The performance and materiality of the processes, spaces and labor of VFX production. In Performing Labor in the Media Industries. Fortmueller, K. (ed) Spectator 35 (2) (Fall 2015), 36–46 (2015)

    Google Scholar 

  42. Holben, J.: The Mandalorian: This Is the Way (2020). https://ascmag.com/articles/the-mandalorian

  43. No Film School: How Cutting-Edge ILM Technology Brought ‘The Mandalorian’ to Life Retrieved 31 July 2022 (2020)

    Google Scholar 

  44. Desowitz, B.: ‘Thor: Love and Thunder’ Takes a Hammer to Marvel’s Green Screen Problems (2022). Retrieved on 20 June 2022. https://www.indiewire.com/2022/07/thor-love-and-thunder-stagecraft-ilm-1234738253/

  45. Australian Cinematographers Society: Virtual Production - A Cinematographers’ Conversation (2020). https://www.youtube.com/watch?v=3g_GhufsQ7E&ab_channel=AustralianCinematographersSociety

  46. Roettgers, J.: How video-game engines help create visual effects on movie sets in real time (2019). Retrieved 7 September 2021. https://variety.com/2019/biz/features/video-game-engines-visual-effects-real-time-1203214992

  47. Giardina, C.: NEP Acquires Lux Machina, Halon, Prysm Collective to Launch Virtual Production Business. Behind the Screen (2021). https://www.hollywoodreporter.com/business/digital/nep-acquires-lux-machina-halon-prysm-collective-virtual-production-business-1234991516

  48. Cade, D.L.: Incredible real-time visual effects tech replaces green screens with LED walls. Peta Pixel (2019). https://petapixel.com/2019/08/22/demo-of-incredible-real-time-visual-effects-using-massive-led-walls/

  49. Dalkian, S.: nDisplay Technology Whitepaper. Epic Games (2019). https://cdn2.unrealengine.com/Unreal+Engine%2Fndisplay-whitepaper-final-updates%2FnDisplay_Whitepaper_FINAL-f87f7ae569861e42d965e4bffd1ee412ab49b238.pdf

  50. Baker-Whitelaw, G.: The problem with Disney’s new ‘Star Wars’ VFX technology. ‘The Volume’ (2022). https://www.dailydot.com/unclick/star-wars-disney-the-volume-obi-wan/?fbclid=IwAR1IP3v8zGx8XH_hbpHS3tq3TgmMsm9RZPwKXOqpb4_uKojt2rOl5TaPTwk

  51. Kavakli, M., Cremona, C.: The virtual production studio concept–an emerging game changer in filmmaking. 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (2022), pp. 29–37

    Google Scholar 

  52. Advanced Imaging Society: Virtual Production Workflows & Insights (2020). https://www.youtube.com/watch?v=81f6FPnWMCI&ab_channel=AdvancedImagingSociety

  53. Coldewey, D.: How Oculus squeezed sophisticated tracking into pipsqueak hardware. Techcrunch (2019). https://techcrunch.com/2019/08/22/how-oculus-squeezed-sophisticated-tracking-into-pipsqueak-hardware

  54. Wright, M. Chambers, J. Israel.: Large Scale VFX Pipelines. ACM SIGGRAPH 2017, Talks, Article 59, pp. 1–2 (2021). https://dl.acm.org/doi/https://doi.org/10.1145/3084363.3085021

  55. Kavakli, M.: How to reduce input lag in a virtual production studio. In International Conference on Human–Computer Interaction. Springer International Publishing (2022), pp. 58–65

    Google Scholar 

  56. Brompton: Ecreating Reality (2022). https://www.bromptontech.com/applications/virtual-production/

  57. Oakden, T., Kavakli, M.: Graphics processing for real-time ray tracing using RTX architecture. 14th International Conference on Computer and Automation Engineering (ICCAE 2022) March 25–27, 2022 , pp 1–6 Brisbane, Australia (2022a). http://www.iccae.org/

  58. Oakden, T., Kavakli, M.: Performance analysis of RTX architecture in virtual production and graphics processing, networked entertainment systems (NES) with ICDCS 2022 -Co-Located with the 42nd IEEE Intl. Conf. on Distributed Computing Systems (ICDCS 2022), Technically co-sponsored by the IEEE and the IEEE Computer Society, July 10 - July 13, 2022, Bologna, Italy, pp 1-11 (2022b)

    Google Scholar 

  59. Failes, I.: A virtual production explainer: what it is, and what it could mean for your project. Cartoon Brew (2018). https://www.cartoonbrew.com/vfx/a-virtual-production-explainer-what-it-is-and-what-it-could-mean-for-your-project-166554.html

  60. Technicolor: Technicolor Genesis—Real Time Virtual Production (2019). https://www.youtube.com/watch?v=x5O2E-jvUbE&t=48s&ab_channel=technicolor

  61. Srivastav, S., Singh, R.: Film Viewing Experience: Changes with Technology. Proc. 28th National Conf. Cultural Representation and Power of Media (2017). https://www.academia.edu/35899511/Film_Viewing_Experience_Changes_with_Technology

  62. Latour, B.: Reassembling the social: an introduction to Actor-Network- Theory. Oxford University Press, Oxford (2007)

    Google Scholar 

  63. Weintraub, S.: O'Shea Jackson Jr. on How He Landed His ‘Obi-Wan Kenobi’ Role After Not Getting Lando and Working With The Volume Technology (2021) https://collider.com/obi-wan-kenobi-series-oshea-jackson-jr-comments-lando-the-volume-interview/

  64. Zhang, C.: The why, what, and how of immersive experience. In IEEE Access 8, 90878–90888 (2020)

    Article  Google Scholar 

  65. Ilmaranta, K.: Cinematic Space in Virtual Production (2020). https://www.academia.edu/44002310/Cinematic_Space_in_Virtual_Production

  66. Bennett, J.: Immersive performance environment: a framework for facilitating an actor in Virtual Production. PhD dissertation, Creative Industries Faculty, Queensland Univ. of Technology, Brisbane, Queensland (2020). https://eprints.qut.edu.au/203911/

  67. Seymour, M.: Art of (LED Wall) Virtual Production Sets, Part Two: How you make one. VFX Guide (2020). https://www.fxguide.com/fxfeatured/art-of-led-wall-virtual-production-sets-part-two-how-you-make-one

  68. Debevec, P., Wenger, A., Tchou, C., Gardner, A., Waese, J.: A lighting reproduction approach to live-action compositing. ACM Trans. Graph. (SIGGRAPH 2002) 21(3), 547–556 (2002). https://dl.acm.org/doi/abs/https://doi.org/10.1145/566570.566614

  69. Hedman, P., Srinivasan, P.P., Mildenhall, B., Barron, J.T., Debevec, P.: Baking neural radiance fields for real-time view synthesis (2021). Arxiv, abs/2103.14645, 2021 https://arxiv.org/pdf/2103.14645

  70. Debevec, P., Malik, J.: Recovering high dynamic range radiance maps from photographs. Computer graphics and interactive techniques (SIGGRAPH ‘97). ACM Press/Addison-Wesley Publishing Co., USA, pp. 369–378 (1997) https://www.academia.edu/download/29352455/recovering_high_dynamic_range_radiance_maps_from_photographs.pdf

  71. Hu, K.: The effects of digital video technology on modern film. Master of Science Thesis, Dept. of Television Management, Drexel Univ., UK (2016)

    Google Scholar 

  72. Kohn, E.: ‘Thor: Love and Thunder’ Takes a Hammer to Marvel’s Green Screen Problems (2022). Retrieved 9 July 2022. https://www.indiewire.com/2022/07/thor-love-and-thunder-stagecraft-ilm-1234738253

  73. lasermagnetic: Memories of Working On Where Eagles Dare (2009). Retrieved on 10 July 2022 https://www.filmboards.com/board/p/1747699

  74. Montgomery, J.: Who are you: FXGuide in focus. FXGuide (2002). https://www.fxguide.com/fxfeatured/who_are_you_fxguide_in_focus/?highlight=MIKE%20SEYMOUR

  75. Natale, S.: A short history of superimposition: From spirit photography to early cinema. Early Popular Visual Cult. 10(2), 125–145 (2012)

    Google Scholar 

  76. Salt, B.: Film style and technology in the thirties. Film Quart. Autumn 30(1) (Autumn, 1976), 19–32. Published by: University of California Press (1976)

    Google Scholar 

  77. Shields, M.: How Does Rear Projection Work? (2020) Retrieved 30 May 2022 https://filmschoolrejects.com/rear-projection

  78. Stump, D.: Digital cinematography: fundamentals, tools, techniques, and workflows. Focal Press, Taylor & Francis Group (2014)

    Google Scholar 

  79. Witmer, JD., Fish, A.: ASC members offer their perspectives on what the next century has in store for the art and craft of cinematography. Future of Cinematography: The Next 100 Years (2019). https://ascmag.com/articles/the-next-100-years

Download references

Author information

Authors and Affiliations

  1. University of Sydney and Macquarie University, 1/290 Clovelly Road, Coogee, NSW, 2034, Australia

    Cinzia Cremona

  2. Aston University, Aston St, Birmingham, United Kingdom, B4 7ET, United Kingdom

    Manolya Kavakli

Authors
  1. Cinzia Cremona
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manolya Kavakli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cinzia Cremona .

Editor information

Editors and Affiliations

  1. CREATE—Section for Media Technology, Campus Aalborg (Human Machine Interaction/The Center for Applied Game Research), Department of Architecture, Design and Media Technology; The Technical Faculty of IT and Design, Aalborg University, Aalborg, Denmark

    Anthony L. Brooks

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Cremona, C., Kavakli, M. (2023). The Evolution of the Virtual Production Studio as a Game Changer in Filmmaking. In: Brooks, A.L. (eds) Creating Digitally. Intelligent Systems Reference Library, vol 241. Springer, Cham. https://doi.org/10.1007/978-3-031-31360-8_14

Download citation

Publish with us

Policies and ethics

Search

Navigation