Skip to main content
Book cover

International Conference on Knowledge-Based and Intelligent Information and Engineering Systems

KES 2010: Knowledge-Based and Intelligent Information and Engineering Systems pp 430–439Cite as

SceneMaker: Multimodal Visualisation of Natural Language Film Scripts

  • Conference paper

Part of the Lecture Notes in Computer Science book series (LNAI,volume 6279)

Abstract

Producing plays, films or animations is a complex and expensive process involving various professionals and media. Our proposed software system, SceneMaker, aims to facilitate this creative process by automatically interpreting natural language film scripts and generating multimodal, animated scenes from them. During the generation of the story content, SceneMaker gives particular attention to emotional aspects and their reflection in fluency and manner of actions, body posture, facial expressions, speech, scene composition, timing, lighting, music and camera work. Related literature and software on Natural Language Processing, in particular textual affect sensing, affective embodied agents, visualisation of 3D scenes and digital cinematography are reviewed. In relation to other work, SceneMaker follows a genre-specific text-to-animation methodology which combines all relevant expressive modalities and is made accessible via web-based and mobile platforms. In conclusion, SceneMaker will enhance the communication of creative ideas providing quick pre-visualisations of scenes.

Keywords

  • Multiagent System
  • Speech Synthesis
  • Multimodal Interface
  • Virtual Human
  • Story Character

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-642-15384-6_46
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   109.00
Price excludes VAT (USA)
  • ISBN: 978-3-642-15384-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   149.00
Price excludes VAT (USA)
Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ma, M.: Automatic Conversion of Natural Language to 3D Animation. PhD Thesis, School of Computing and Intelligent Systems, University of Ulster (2006)

    Google Scholar 

  2. Ekman, P., Rosenberg, E.L.: What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system. Oxford University Press, Oxford (1997)

    Google Scholar 

  3. Mehrabian, A.: Framework for a Comprehensive Description and Measurement of Emotional States. Genetic, Social, and General Psychology Monographs 121(3), 339–361 (1995)

    Google Scholar 

  4. Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988)

    Google Scholar 

  5. Francisco, V., Hervás, R., Gervás, P.: Two Different Approaches to Automated Mark Up of Emotions. In: Sattar, A., Kang, B.-h. (eds.) AI 2006. LNCS (LNAI), vol. 4304, pp. 101–114. Springer, Heidelberg (2006)

    Google Scholar 

  6. Strapparava, C., Mihalcea, R.: Learning to identify emotions in text. In: Proceedings of the 2008 ACM Symposium on Applied Computing. SAC 2008, pp. 1556–1560. ACM, New York (2008)

    CrossRef  Google Scholar 

  7. Su, W.-P., Pham, B., Wardhani, A.: Personality and Emotion-Based High-Level Control of Affective Story Characters. IEEE Transactions on Visualization and Computer Graphics 13(2), 281–293 (2007)

    CrossRef  Google Scholar 

  8. Liu, H., Lieberman, H., Selker, T.: A model of textual affect sensing using real-world knowledge. In: Proceedings of the 8th International Conference on Intelligent User Interfaces. IUI 2003, pp. 125–132. ACM, New York (2003)

    CrossRef  Google Scholar 

  9. Liu, H., Singh, P.: ConceptNet: A practical commonsense reasoning toolkit. BT Technology Journal 22(4), 211–226 (2004)

    CrossRef  Google Scholar 

  10. Shaikh, M.A.M., Prendinger, H., Ishizuka, M.: A Linguistic Interpretation of the OCC Emotion Model for Affect Sensing from Text. In: Affective Information Processing, pp. 45–73. Springer, London (2009)

    CrossRef  Google Scholar 

  11. Choujaa, D., Dulay, N.: Using screenplays as a source of context data. In: Proceeding of the 2nd ACM International Workshop on Story Representation, Mechanism and Context. SRMC 2008, pp. 13–20. ACM, New York (2008)

    CrossRef  Google Scholar 

  12. Cassell, J., Vilhjálmsson, H.H., Bickmore, T.: BEAT: the Behavior Expression Animation Toolkit. In: Proceedings of the 28th Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH 2001, pp. 477–486. ACM, New York (2001)

    CrossRef  Google Scholar 

  13. Breitfuss, W., Prendinger, H., Ishizuka, M.: Automated generation of non-verbal behavior for virtual embodied characters. In: Proceedings of the 9th International Conference on Multimodal Interfaces. ICMI 2007, pp. 319–322. ACM, New York (2007)

    CrossRef  Google Scholar 

  14. Prendinger, H., Ishizuka, M.: SCREAM: scripting emotion-based agent minds. In: Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 1. AAMAS 2002, pp. 350–351. ACM, New York (2002)

    CrossRef  Google Scholar 

  15. Gebhard, P.: ALMA - Layered Model of Affect. In: Proceedings of the 4th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2005), Utrecht University, Netherlands, pp. 29–36. ACM, New York (2005)

    CrossRef  Google Scholar 

  16. De Raad, B.: The Big Five Personality Factors. In: The Psycholexical Approach to Personality. Hogrefe & Huber (2000)

    Google Scholar 

  17. Kopp, S., Allwood, J., Grammer, K., Ahlsen, E., Stocksmeier, T.: Modeling Embodied Feedback with Virtual Humans. In: Wachsmuth, I., Knoblich, G. (eds.) ZiF Research Group International Workshop. LNCS (LNAI), vol. 4930, pp. 18–37. Springer, Heidelberg (2008)

    CrossRef  Google Scholar 

  18. Pelachaud, C.: Multimodal expressive embodied conversational agents. In: Proceedings of the 13th Annual ACM International Conference on Multimedia. MULTIMEDIA 2005, pp. 683–689. ACM, New York (2005)

    CrossRef  Google Scholar 

  19. Gunes, H., Piccardi, M.: A Bimodal Face and Body Gesture Database for Automatic Analysis of Human Nonverbal Affective Behavior. In: 18th International Conference on Pattern Recognition, ICPR, vol. 1, pp. 1148–1153. IEEE Computer Society, Washington (2006)

    CrossRef  Google Scholar 

  20. Kelleher, J., Doris, T., Hussain, Q., Ó Nualláin, S.: SONAS: Multimodal, Multi-User Interaction with a Modelled Environment. In: Spatial Cognition, pp. 171–184. John Benjamins Publishing Company, Amsterdam (2000)

    Google Scholar 

  21. Coyne, B., Sproat, R.: WordsEye: an automatic text-to-scene conversion system. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, pp. 487–496. ACM Press, Los Angeles (2001)

    Google Scholar 

  22. Kennedy, K., Mercer, R.E.: Planning animation cinematography and shot structure to communicate theme and mood. In: Proceedings of the 2nd International Symposium on Smart Graphics. SMARTGRAPH 2002, vol. 24, pp. 1–8. ACM, New York (2002)

    CrossRef  Google Scholar 

  23. De Melo, C., Paiva, A.: Multimodal Expression in Virtual Humans. Computer Animation and Virtual Worlds 2006 17(3-4), 239–348 (2006)

    CrossRef  Google Scholar 

  24. Liu, Z., Leung, K.: Script visualization (ScriptViz): a smart system that makes writing fun. Soft Computing 10(1), 34–40 (2006)

    CrossRef  Google Scholar 

  25. Rasheed, Z., Sheikh, Y., Shah, M.: On the use of computable features for film classification. IEEE Transactions on Circuits and Systems for Video Technology 15(1), 52–64 (2005)

    CrossRef  Google Scholar 

  26. Shim, H., Kang, B.G.: CAMEO - camera, audio and motion with emotion orchestration for immersive cinematography. In: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology. ACE 2008, vol. 352, pp. 115–118. ACM, New York (2008)

    CrossRef  Google Scholar 

  27. Kuo, F., Chiang, M., Shan, M., Lee, S.: Emotion-based music recommendation by association discovery from film music. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, MULTIMEDIA 2005, pp. 507–510. ACM, New York (2005)

    CrossRef  Google Scholar 

  28. Wahlster, W.: Smartkom: Foundations of Multimodal Dialogue Systems. Springer, Heidelberg (2006)

    CrossRef  Google Scholar 

  29. ScriptRight, http://www.scriptright.com

  30. Virtual Theatre Interface, http://www.accad.ohio-state.edu/VT

  31. Gutiérrez, M., Vexo, F., Thalmann, D.: The Mobile Animator: Interactive Character Animation in Collaborative Virtual Environments. In: Proceedings of the IEEE Virtual Reality 2004, VR 2004, pp. 125–132. IEEE Computer Society, Washington (2004)

    CrossRef  Google Scholar 

  32. Connexor, http://www.connexor.eu/technology/machinese

  33. Tesniere, L.: Elements de syntaxe structurale. Klincksieck, Paris (1959)

    Google Scholar 

  34. Fellbaum, C.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998)

    MATH  Google Scholar 

  35. Lexical Conceptual Structure Database, http://www.umiacs.umd.edu/~bonnie/LCS_Database_Documentation.html

  36. Strapparava, C., Valitutti, A.: WordNet-Affect: an affective extension of WordNet. In: Proceedings of the 4th International Conference on Language Resources and Evaluation, LREC 2004, pp. 1083–1086 (April 2004)

    Google Scholar 

  37. Humanoid Animation Working Group, http://www.h-anim.org

  38. FreeTTS 1.2 - A speech synthesizer written entirely in the JavaTM programming language, http://freetts.sourceforge.net/docs/index.php

Download references

Author information

Authors and Affiliations

  1. School of Computing & Intelligent Systems, Faculty of Computing & Engineering, University of Ulster, Magee, Derry, Londonderry, BT48 7JL, Northern Ireland

    Eva Hanser, Paul Mc Kevitt, Tom Lunney & Joan Condell

  2. School of Computing, Faculty of Business, Computing and Law, University of Derby, Derby, DE22 1GB, England

    Minhua Ma

Authors
  1. Eva Hanser
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Mc Kevitt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tom Lunney
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joan Condell
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Minhua Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Engineering, Cardiff University, The Parade, CF24 3AA, Cardiff, UK

    Rossitza Setchi

  2. Dept. of Computer Science and Software Engineering, University of Portsmouth, BUckingham Building, Lion Terrace, PO1 3HE, Portsmouth, UK

    Ivan Jordanov

  3. KES International, 145-157 St. John Street, EC1V 4PY, London, UK

    Robert J. Howlett

  4. School of Electrical and Information Engineering, University of South Australia, Adelaide, Mawson Lakes Campus, 5095, SA, Australia

    Lakhmi C. Jain

Rights and permissions

Reprints and Permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Hanser, E., Kevitt, P.M., Lunney, T., Condell, J., Ma, M. (2010). SceneMaker: Multimodal Visualisation of Natural Language Film Scripts. In: Setchi, R., Jordanov, I., Howlett, R.J., Jain, L.C. (eds) Knowledge-Based and Intelligent Information and Engineering Systems. KES 2010. Lecture Notes in Computer Science(), vol 6279. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-15384-6_46

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-15384-6_46

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-15383-9

  • Online ISBN: 978-3-642-15384-6

  • eBook Packages: Computer ScienceComputer Science (R0)