Multimedia Tools and Applications

, Volume 77, Issue 2, pp 2333–2357 | Cite as

Video-based interactive storytelling using real-time video compositing techniques

  • Edirlei Soares de Lima
  • Bruno Feijó
  • Antonio L. Furtado


Interactive storytelling systems usually adopt computer graphics to represent virtual story worlds, which facilitates the dynamic generation of visual content. However, the quality of the images and motion produced by these systems is still inferior compared to the high quality experience found in live-action films. Interactive rates in photorealistic rendering for the film industry will not be possible for decades to come. A promising alternative is the replacement of 3D virtual characters with video sequences with real actors. In this paper, we propose a new method for video-based interactive narratives that uses video compositing algorithms that run at truly interactive frame rates. The proposed method is consistent with plots that are generated by nondeterministic planning algorithms. Moreover, we propose a system of artificial intelligent agents that perform the same roles played by filmmaking professionals. A user evaluation of the proposed method is presented. We believe that future improvements of the techniques proposed in this paper represent an important contribution to the quest for new and more immersive forms of interactive cinema.


Interactive storytelling Video compositing Video-based interactive storytelling Virtual cinematography Interactive cinema 



We would like to thank CNPq (National Council for Scientific and Technological Development), FAPERJ (Carlos Chagas Filho Research Support Foundation of the State of Rio de Janeiro) and FINEP (Brazilian Innovation Agency), which belong to the Ministry of Science, Technology and Innovation, for the financial support. Also we are grateful to the anonymous reviewers for their valuable comments.


  1. 1.
    Ahanger G, Little TDC (1997) A system for customized news delivery from video archives. In: Proceedings of International Conference on Multimedia Computing and Systems, Ottawa, pp 526–533Google Scholar
  2. 2.
    Arijon D (1976) Grammar of the film language. Silman-James Press, Los AngelesGoogle Scholar
  3. 3.
    Barbosa SDJ, Lima ES, Furtado AL, Feijó B (2014) Generation and Dramatization of Detective Stories. SBC Journal on 3D Interactive Systems 5:39–52Google Scholar
  4. 4.
    Barbosa SDJ, Silva FAG, Furtado AL, Casanova MA (2015) Plot generation with character-based decisions. Comput Entertain 12:1–21CrossRefGoogle Scholar
  5. 5.
    Bocconi S (2006) Vox Populi: generating video documentaries from semantically annotated media repositories. PhD Thesis, Technische Universiteit Eindhoven, NetherlandsGoogle Scholar
  6. 6.
    Bredow R, Hastings A, Schaub D, Kramer D, Engle R (2015) From Mocap to Movie: The Polar Express. Course # 28, SIGGRAPH 2015, Los AngelesGoogle Scholar
  7. 7.
    Brown B (2011) Cinematography: theory and practice: image making for cinematographers and directors. Focal Press, WalthamGoogle Scholar
  8. 8.
    Brown M, Lowe D (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–77CrossRefGoogle Scholar
  9. 9.
    Brown SL, Vaughan CC (2009) Play: how it shapes the brain, opens the imagination, and invigorates the soul. Avery Publishing Group, New YorkGoogle Scholar
  10. 10.
    Cavazza M, Charles F, Mead S (2002) Character-based interactive storytelling. IEEE Intell Syst, Special issue AI in Interactive Entertainment 17(4):17–24zbMATHGoogle Scholar
  11. 11.
    Cavazza M, Lugrin J-L, Pizzi D, and Charles F (2007) Madame Bovary on the Holodeck: Immersive Interactive Storytelling. In: Proceedings of the 15th ACM International Conference on Multimedia, Augsburg, pp 651–660Google Scholar
  12. 12.
    Chabert CF, Einarsson P, Jones A, Lamond B, Ma WC, Sylwan S, Hawkins T, Debevec P (2006) Relighting human locomotion with flowed reflectance fields. In: ACM SIGGRAPH 2006 Sketches, New YorkGoogle Scholar
  13. 13.
    Chua TS, Ruan LQ (1995) A video retrieval and sequencing system. ACM Trans Inf Syst 13(4):373–407CrossRefGoogle Scholar
  14. 14.
    Ciarlini AEM, Pozzer CT, Furtado AL, Feijó B (2005) A logic-based tool for interactive generation and dramatization of stories. In: Proceedings of the International Conference on Advances in Computer Entertainment Technology, Valencia, pp 133–140Google Scholar
  15. 15.
    Davenport, G., Murtaugh, M (1995) ConText Towards the Evolving Documentary. In: Proceedings of ACM Multimedia '95, San Francisco, pp 377–389Google Scholar
  16. 16.
    Doria TR, Ciarlini AEM, Andreatta A (2008) A Nondeterministic Model for Controlling the Dramatization of Interactive Stories. In: Proceedings of the ACM Multimedia 2008 - 2nd ACM Workshop on Story Representation, Mechanism and Context - SRMC08, Vancouver, pp 21–26Google Scholar
  17. 17.
    Ekman P, Friesen WV (1971) Constants across cultures in the face and emotion. J Pers Soc Psychol 17:124–129CrossRefGoogle Scholar
  18. 18.
    Foster J (2010) The green screen handbook: real-world production techniques. Sybex, IndianapolisGoogle Scholar
  19. 19.
    Ghallab M, Nau D, Traverso P (2004) Automated planning: theory and practice. Mor-gan Kaufmann Publishers, San FranciscozbMATHGoogle Scholar
  20. 20.
    Haykin SO (2008) Neural networks and learning machines, 3rd edn. Prentice Hall, Upper Saddle RiverGoogle Scholar
  21. 21.
    Joshi N, Matusik W, Avidan S (2006) Natural video matting using camera arrays. ACM Trans Graph 25:779–786CrossRefGoogle Scholar
  22. 22.
    Jung Von Matt/Spree (2010) Last Call, Berlin Accessed 25 Jan 2016
  23. 23.
    Liang C, Xu C, Cheng J, Min W, Lu H (2013) Script-to-movie: a computational framework for story movie composition. IEEE Trans Multimedia 15(2):401–414CrossRefGoogle Scholar
  24. 24.
    Lima ES, Feijó B, Pozzer CT, Ciarlini AEM, Barbosa SDJ, Furtado AL, Silva FGA (2012a) Social Interaction for Interactive Storytelling. In: Proceedings of the 11th International Conference on Entertainment Computing Bremen, Germany, pp 1–15Google Scholar
  25. 25.
    Lima ES, Feijó B, Furtado AL, Pozzer C, Ciarlini A (2012b) Automatic Video Editing For Video-Based Interactive Storytelling. In: Proceedings of the 2012 I.E. International Conference on Multimedia and Expo (ICME), Melbourne, pp 806–811Google Scholar
  26. 26.
    Lima ES, Feijó B, Furtado AL, Pozzer CT, Ciarlini AEM, Silva FG (2012c) A multi-user natural language Interface for interactive storytelling in TV and cinema. In: Proceedings of the XI Brazilian Symposium on Computer Games and Digital Entertainment, Brasília, pp. 154–161Google Scholar
  27. 27.
    Lima ES, Feijó B, Barbosa SDJ, Furtado AL, Ciarlini AEM, Pozzer CT (2014) Draw your own story: paper and pencil interactive storytelling. Entertainment Computing 5(1):33–41CrossRefGoogle Scholar
  28. 28.
    Lima ES, Feijó B, Furtado AL (2015) Storytelling Variants: The Case of Little Red Riding Hood. Proceedings of the 14th International Conference on Entertainment Computing (ICEC 2015), Trondheim, pp 286–300Google Scholar
  29. 29.
    Lima ES, Feijó B, Casanova MA, Furtado AL (2016) Storytelling variants based on semiotic relations. Entertainment Computing 17:31–44CrossRefGoogle Scholar
  30. 30.
    Logtell (2016) Logtell Project Web Site Accessed 25 Jan 2016
  31. 31.
    Mascelli J (1965) The five C's of cinematography: motion picture filming techniques. Silman-James Press, Los AngelesGoogle Scholar
  32. 32.
    Mateas, M (2002) Interactive Drama, art, and artificial intelligence. Doctoral Thesis. School of Computer Science, Carnegie Mellon University, PittsburghGoogle Scholar
  33. 33.
    Mateas M, Vanouse P, Domike S (2000) Generation of ideologically-biased historical documentaries. In: Proceedings of the Seventeenth National Conference on Artificial intelligence and Twelfth Conference on innovative Applications of Artificial intelligence. AAAI Press, pp 236–242Google Scholar
  34. 34.
    Mehlmann G, Endrass B, and André E (2011) Modeling and Interpretation of Multithreaded and Multimodal Dialogue. In: Proceedings of the 13th International Conference on Multimodal Interaction (ICMI 2011), pp 385–392Google Scholar
  35. 35.
    Mehta S, Ramamoorthi R, Meyer M, Hery C (2012) Analytic tangent irradiance environment maps for anisotropic surfaces. Comput Graphics Forum 31(4):1501–1508CrossRefGoogle Scholar
  36. 36.
    Müller W, Spierling U, Stockhausen C (2013) Production and delivery of interactive narratives based on video snippets. In: Proceedings of the 6th international conference on interactive digital storytelling, Istanbul, Turkey, pp 71–82Google Scholar
  37. 37.
    Ng R, Ramamoorthi R, Hanrahan P (2003) All-frequency shadows using non-linear wavelet lighting approximation. ACM Trans Graph 22(3):376–381CrossRefGoogle Scholar
  38. 38.
    Piacenza A, Guerrini F, Adami N, Leonardi R, Porteous J, Teutenberg J, Cavazza M (2011) Generating Story Variants with Constrained Video Recombination. In: Proceedings of the 19th ACM International Conference on Multimedia 2011, Scottsdale, pp 223–232Google Scholar
  39. 39.
    Pizzi D, Cavazza M (2007) Affective storytelling based on Characters' feelings. AAAI Fall Symposium on Intelligent Narrative Technologies, ArlingtonGoogle Scholar
  40. 40.
    Porteous J, Benini S, Canini L, Charles F, Cavazza M, Leonardi R (2010) Interactive storytelling via video content recombination. In: Proceedings of the 18th ACM International Conference on Multimedia 2010, Firenze, pp 1715–1718Google Scholar
  41. 41.
    Porter T, Duff T (1984) Compositing Digital Images. Comput Graph 18(3):253–259CrossRefGoogle Scholar
  42. 42.
    Riedl MO, Young RM (2006) From linear story generation to branching story graphs. IEEE Comput Graph Appl 26(3):23–31CrossRefGoogle Scholar
  43. 43.
    Schoenau-Fog H (2011) Hooked! – Evaluating Engagement as Continuation Desire in Interactive Narratives. In: Fourth International Conference on Interactive Digital Storytelling (ICIDS 2011), Vancouver, pp 219–230Google Scholar
  44. 44.
    Shen YT, Lieberman H, Davenport G (2009) What’s next? Emergent storytelling from video collections. In: Proceeding of International Conference on Human factors in computing systems, ACM Press, pp 809–818Google Scholar
  45. 45.
    Silva FGA, Ciarlini AEM, Siqueira SWM (2010) Nondeterministic planning for generating interactive plots. In: 12th Ibero-American conference on AI, Bahía Blanca. Springer, Argentina, pp 133–143Google Scholar
  46. 46.
    Sun J, Li Y, Kang SB, Shum H-Y (2006) Flash matting. ACM Trans Graph 25(3):772–778CrossRefGoogle Scholar
  47. 47.
    Thompson R, Bowen C (2009) Grammar of the shot. Focal Press, BurlingtonGoogle Scholar
  48. 48.
    Ursu MF, Kegel IC, Williams D, Thomas M, Mayer H, Zsombori V, Uomola ML, Larsson H, Wyver J (2008) ShapeShifting TV: interactive screen media narratives. Multimedia Systems 14(2):115–132CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  • Edirlei Soares de Lima
    • 1
  • Bruno Feijó
    • 2
  • Antonio L. Furtado
    • 2
  1. 1.Department of Computational ModelingRio de Janeiro State University (UERJ)Nova FriburgoBrazil
  2. 2.Department of InformaticsPontifical Catholic University of Rio de JaneiroRio de JaneiroBrazil

Personalised recommendations