SceneMaker: Multimodal Visualisation of Natural Language Film Scripts

  • Eva Hanser
  • Paul Mc Kevitt
  • Tom Lunney
  • Joan Condell
  • Minhua Ma
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6279)


Producing plays, films or animations is a complex and expensive process involving various professionals and media. Our proposed software system, SceneMaker, aims to facilitate this creative process by automatically interpreting natural language film scripts and generating multimodal, animated scenes from them. During the generation of the story content, SceneMaker gives particular attention to emotional aspects and their reflection in fluency and manner of actions, body posture, facial expressions, speech, scene composition, timing, lighting, music and camera work. Related literature and software on Natural Language Processing, in particular textual affect sensing, affective embodied agents, visualisation of 3D scenes and digital cinematography are reviewed. In relation to other work, SceneMaker follows a genre-specific text-to-animation methodology which combines all relevant expressive modalities and is made accessible via web-based and mobile platforms. In conclusion, SceneMaker will enhance the communication of creative ideas providing quick pre-visualisations of scenes.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ma, M.: Automatic Conversion of Natural Language to 3D Animation. PhD Thesis, School of Computing and Intelligent Systems, University of Ulster (2006)Google Scholar
  2. 2.
    Ekman, P., Rosenberg, E.L.: What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system. Oxford University Press, Oxford (1997)Google Scholar
  3. 3.
    Mehrabian, A.: Framework for a Comprehensive Description and Measurement of Emotional States. Genetic, Social, and General Psychology Monographs 121(3), 339–361 (1995)Google Scholar
  4. 4.
    Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988)Google Scholar
  5. 5.
    Francisco, V., Hervás, R., Gervás, P.: Two Different Approaches to Automated Mark Up of Emotions. In: Sattar, A., Kang, B.-h. (eds.) AI 2006. LNCS (LNAI), vol. 4304, pp. 101–114. Springer, Heidelberg (2006)Google Scholar
  6. 6.
    Strapparava, C., Mihalcea, R.: Learning to identify emotions in text. In: Proceedings of the 2008 ACM Symposium on Applied Computing. SAC 2008, pp. 1556–1560. ACM, New York (2008)CrossRefGoogle Scholar
  7. 7.
    Su, W.-P., Pham, B., Wardhani, A.: Personality and Emotion-Based High-Level Control of Affective Story Characters. IEEE Transactions on Visualization and Computer Graphics 13(2), 281–293 (2007)CrossRefGoogle Scholar
  8. 8.
    Liu, H., Lieberman, H., Selker, T.: A model of textual affect sensing using real-world knowledge. In: Proceedings of the 8th International Conference on Intelligent User Interfaces. IUI 2003, pp. 125–132. ACM, New York (2003)CrossRefGoogle Scholar
  9. 9.
    Liu, H., Singh, P.: ConceptNet: A practical commonsense reasoning toolkit. BT Technology Journal 22(4), 211–226 (2004)CrossRefGoogle Scholar
  10. 10.
    Shaikh, M.A.M., Prendinger, H., Ishizuka, M.: A Linguistic Interpretation of the OCC Emotion Model for Affect Sensing from Text. In: Affective Information Processing, pp. 45–73. Springer, London (2009)CrossRefGoogle Scholar
  11. 11.
    Choujaa, D., Dulay, N.: Using screenplays as a source of context data. In: Proceeding of the 2nd ACM International Workshop on Story Representation, Mechanism and Context. SRMC 2008, pp. 13–20. ACM, New York (2008)CrossRefGoogle Scholar
  12. 12.
    Cassell, J., Vilhjálmsson, H.H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: Proceedings of the 28th Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH 2001, pp. 477–486. ACM, New York (2001)CrossRefGoogle Scholar
  13. 13.
    Breitfuss, W., Prendinger, H., Ishizuka, M.: Automated generation of non-verbal behavior for virtual embodied characters. In: Proceedings of the 9th International Conference on Multimodal Interfaces. ICMI 2007, pp. 319–322. ACM, New York (2007)CrossRefGoogle Scholar
  14. 14.
    Prendinger, H., Ishizuka, M.: SCREAM: scripting emotion-based agent minds. In: Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 1. AAMAS 2002, pp. 350–351. ACM, New York (2002)CrossRefGoogle Scholar
  15. 15.
    Gebhard, P.: ALMA - Layered Model of Affect. In: Proceedings of the 4th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2005), Utrecht University, Netherlands, pp. 29–36. ACM, New York (2005)CrossRefGoogle Scholar
  16. 16.
    De Raad, B.: The Big Five Personality Factors. In: The Psycholexical Approach to Personality. Hogrefe & Huber (2000)Google Scholar
  17. 17.
    Kopp, S., Allwood, J., Grammer, K., Ahlsen, E., Stocksmeier, T.: Modeling Embodied Feedback with Virtual Humans. In: Wachsmuth, I., Knoblich, G. (eds.) ZiF Research Group International Workshop. LNCS (LNAI), vol. 4930, pp. 18–37. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  18. 18.
    Pelachaud, C.: Multimodal expressive embodied conversational agents. In: Proceedings of the 13th Annual ACM International Conference on Multimedia. MULTIMEDIA 2005, pp. 683–689. ACM, New York (2005)CrossRefGoogle Scholar
  19. 19.
    Gunes, H., Piccardi, M.: A Bimodal Face and Body Gesture Database for Automatic Analysis of Human Nonverbal Affective Behavior. In: 18th International Conference on Pattern Recognition, ICPR, vol. 1, pp. 1148–1153. IEEE Computer Society, Washington (2006)CrossRefGoogle Scholar
  20. 20.
    Kelleher, J., Doris, T., Hussain, Q., Ó Nualláin, S.: SONAS: Multimodal, Multi-User Interaction with a Modelled Environment. In: Spatial Cognition, pp. 171–184. John Benjamins Publishing Company, Amsterdam (2000)Google Scholar
  21. 21.
    Coyne, B., Sproat, R.: WordsEye: an automatic text-to-scene conversion system. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 487–496. ACM Press, Los Angeles (2001)Google Scholar
  22. 22.
    Kennedy, K., Mercer, R.E.: Planning animation cinematography and shot structure to communicate theme and mood. In: Proceedings of the 2nd International Symposium on Smart Graphics. SMARTGRAPH 2002, vol. 24, pp. 1–8. ACM, New York (2002)CrossRefGoogle Scholar
  23. 23.
    De Melo, C., Paiva, A.: Multimodal Expression in Virtual Humans. Computer Animation and Virtual Worlds 2006 17(3-4), 239–348 (2006)CrossRefGoogle Scholar
  24. 24.
    Liu, Z., Leung, K.: Script visualization (ScriptViz): a smart system that makes writing fun. Soft Computing 10(1), 34–40 (2006)CrossRefGoogle Scholar
  25. 25.
    Rasheed, Z., Sheikh, Y., Shah, M.: On the use of computable features for film classification. IEEE Transactions on Circuits and Systems for Video Technology 15(1), 52–64 (2005)CrossRefGoogle Scholar
  26. 26.
    Shim, H., Kang, B.G.: CAMEO - camera, audio and motion with emotion orchestration for immersive cinematography. In: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology. ACE 2008, vol. 352, pp. 115–118. ACM, New York (2008)CrossRefGoogle Scholar
  27. 27.
    Kuo, F., Chiang, M., Shan, M., Lee, S.: Emotion-based music recommendation by association discovery from film music. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, MULTIMEDIA 2005, pp. 507–510. ACM, New York (2005)CrossRefGoogle Scholar
  28. 28.
    Wahlster, W.: Smartkom: Foundations of Multimodal Dialogue Systems. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  29. 29.
  30. 30.
    Virtual Theatre Interface,
  31. 31.
    Gutiérrez, M., Vexo, F., Thalmann, D.: The Mobile Animator: Interactive Character Animation in Collaborative Virtual Environments. In: Proceedings of the IEEE Virtual Reality 2004, VR 2004, pp. 125–132. IEEE Computer Society, Washington (2004)CrossRefGoogle Scholar
  32. 32.
  33. 33.
    Tesniere, L.: Elements de syntaxe structurale. Klincksieck, Paris (1959)Google Scholar
  34. 34.
    Fellbaum, C.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)MATHGoogle Scholar
  35. 35.
  36. 36.
    Strapparava, C., Valitutti, A.: WordNet-Affect: an affective extension of WordNet. In: Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004, pp. 1083–1086 (April 2004)Google Scholar
  37. 37.
    Humanoid Animation Working Group,
  38. 38.
    FreeTTS 1.2 - A speech synthesizer written entirely in the JavaTM programming language,

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Eva Hanser
    • 1
  • Paul Mc Kevitt
    • 1
  • Tom Lunney
    • 1
  • Joan Condell
    • 1
  • Minhua Ma
    • 2
  1. 1.School of Computing & Intelligent Systems, Faculty of Computing & EngineeringUniversity of Ulster, MageeDerryNorthern Ireland
  2. 2.School of Computing, Faculty of Business, Computing and LawUniversity of DerbyDerbyEngland

Personalised recommendations