Multimedia Tools and Applications

, Volume 59, Issue 2, pp 407–439 | Cite as

Semantic annotation of narrative media objects

  • Vincenzo Lombardo
  • Rossana Damiano


This paper addresses the annotation of the narrative features of media objects. Based on a relevant narratological and computational background, we introduce an ontology–based model called Drammar, an annotation schema for the narrative features of media objects based on Drammar and a software tool, Cinematic, for annotating these objects and validating the annotation. Annotated media objects can also be automatically edited into sequences, with the twofold goal of testing the validity of the annotation—through the reconstruction of the baseline sequence—and exploring the possibility of alternative sequences. The software tool encodes both the narrative model and the annotation itself in ontological format, and relies on external ontologies for representing world knowledge and limit the arbitrariness of the annotation. The paper opens the way to the design of a general annotation schema for narrative multimedia with the long–term goal of building large corpora of annotated video material and of bridging the gap between the low–level signal analysis and the high–level semantic representation of the narrative content of the media objects. Finally, the paper illustrates a few projects elaborated with the Drammar annotation and the Cinematic tool, with purposes of artistic research and cross–media analysis, that provide an empirical validation of the annotation process.


Narrative media object Multimedia corpus annotation Automatic video editing 



This work is partially supported by the CADMOS Project (Character–centered Annotation of Dramatic Media ObjectS), funded by Regione Piemonte, Pole of Innovation for the Digital Creativity and Multimedia, 2010–2012.

We thank the anonymous reviewers who put a lot of time and effort in helping us to improve the paper’s quality and clarity.

We thank Giulio Valente and Stefano Vajna de Pava for their contribution in the implementation of Cinematic and Fabrizio Nunnari for software engineering; Antonio Pizzo for his contribution to the theoretical aspects of this work; the artist Pino Cappellano and the graduate student Irene Fornerone for their projects in Cinematic.


  1. 1.
    Arndt R, Troncy R, Staab S, Hardman L, Vacura M (2007) COMM: designing a well-founded multimedia ontology for the web. The semantic web, pp 30–43Google Scholar
  2. 2.
    Bates J, Loyall A, Reilly W (1994) An architecture for action, emotion, and social behaviour. Artificial Social Systems. LNAI, vol 830Google Scholar
  3. 3.
    Bocconi S, Nack F, Hardman L (2008) Automatic generation of matter-of-opinion video documentaries. Web Semantics 6(2):139–150. Semantic MultimediaCrossRefGoogle Scholar
  4. 4.
    Bratman ME, Israel DJ, Pollack ME (1988) Plans and resource-bounded practical reasoning. Comput Intell 4:349–355CrossRefGoogle Scholar
  5. 5.
    Butler S, Parkes A (1997) Film sequence generation strategies for automatic intelligent video editing. Appl Artif Intell 11:367–388CrossRefGoogle Scholar
  6. 6.
    Campbell J (1949) The hero with a thousand faces. Princeton University Press, PrincetonGoogle Scholar
  7. 7.
    Carroll N (2001) Beyond esthetics: philosophical essays. Cambridge University Press, New YorkCrossRefGoogle Scholar
  8. 8.
    Cavazza M, Charles F, Mead S (2002) Interacting with virtual characters in interactive storytelling. In: Proc. of the first int. joint conf. on autonomous agents and multiagent systemsGoogle Scholar
  9. 9.
    Coleridge ST (1817, 1985) Biographia literaria. Princeton University PressGoogle Scholar
  10. 10.
    Cua J, Ong E, Manurung R, Pease A (2010) Representing story plans in SUMO. In: Proceedings of the NAACL HLT 2010 second workshop on computational approaches to linguistic creativity. Association for Computational Linguistics, pp 40–48Google Scholar
  11. 11.
    Damiano R, Gena C, Lombardo V, Nunnari F, Pizzo A (2008) A stroll with carletto: adaptation in drama-based tours with virtual characters. User Model User-Adapt Interact 18(5):417–453CrossRefGoogle Scholar
  12. 12.
    Damiano R, Lombardo V (2009) Value-driven characters for storytelling and drama. In: Proc. of AI*IA10, LNCS 5883-0441Google Scholar
  13. 13.
    Damiano R, Lombardo V (2011) An architecture for directing value-driven artificial characters. Agents for Games and Simulations II: Trends in Techniques, Concepts and Design, pp 76–90Google Scholar
  14. 14.
    Damiano R, Pizzo A (2008) Emotions in drama characters and virtual agents. In: AAAI spring symposium on emotion, personality, and social behaviorGoogle Scholar
  15. 15.
    Dasiopoulou S, Tzouvaras V, Kompatsiaris I, Strintzis M (2010) Enquiring MPEG-7 based multimedia ontologies. Multimed Tools Appl 46(2):331–370CrossRefGoogle Scholar
  16. 16.
    Davenport G, Morgenroth L (1994) Video database design: convivial storytelling tools. Technical report, MIT ICGoogle Scholar
  17. 17.
    Davis M (2003) Editing out video editing. IEEE Multimed 10(12):2–12Google Scholar
  18. 18.
    Dennett DC (1987) The intentional stance. The MIT Press, CambridgeGoogle Scholar
  19. 19.
    Egri L (1946) The art of dramatic writing. Simon and Schuster, New YorkGoogle Scholar
  20. 20.
    Feagin SL (2007) On Noel Carrol on narrative closure. Philos Stud (135):17–25Google Scholar
  21. 21.
    Freytag G (1863) Die Technik des dramas. Hirzel, LeipzigGoogle Scholar
  22. 22.
    Genette G (1983) Narrative discourse: an essay in method. Cornell University PressGoogle Scholar
  23. 23.
    Greimas A (1977) Elements of a narrative grammar. Diacritics 7(1):23–40CrossRefGoogle Scholar
  24. 24.
    Hardman L, Obrenović Ž, Nack F, Kerhervé B, Piersol K (2008) Canonical processes of semantically annotated media production. Multimedia Syst 14(6):327–340CrossRefGoogle Scholar
  25. 25.
    Hartmann K, Hartmann S, Feustel M (2005) Motif definition and classification to structure non-linear plots and to control the narrative flow in interactive dramas. In: Third International Conference, Virtual Storytelling, ICVS, using virtual reality technologies for storytelling, pp 158–167Google Scholar
  26. 26.
    Jewell M, Lawrence K, Tuffield M, Prugel-Bennett A, Millard D, Nixon M, Shadbolt N (2005) OntoMedia: an ontology for the representation of heterogeneous media. In: Proceeding of SIGIR workshop on mutlimedia information retrieval. ACM SIGIR.Google Scholar
  27. 27.
    Jung B, Song J, Lee Y (2007) A narrative-based abstraction framework for story-oriented video. ACM Transactions on Multimedia 3(2):1–28Google Scholar
  28. 28.
    Lavandier Y (1994) La dramaturgie. Le clown et l’enfant, CergyGoogle Scholar
  29. 29.
    Levesque H, Pirri F, Reiter R (1998) Foundations for the situation calculus. Electron Trans Artif Intell 2:159–178MathSciNetGoogle Scholar
  30. 30.
    Lohse L, Thomas M (2005) Storytelling for the small screen: authoring and producing reconfigurable cinematic narrative for sit-back enjoyment. In: Proceedings of int. conf. on virtual storytelling 2005, LNCS 3805/2005.Google Scholar
  31. 31.
    Lombardo V, Damiano R (2010) Narrative annotation and editing of video. In: ICIDS’10, proceedings of the third joint conference on interactive digital storytelling. Springer-Verlag, pp 62–73Google Scholar
  32. 32.
    Lombardo V, Nunnari F, Damiano R, Pizzo A, Gena C (2008) The canonical processes of a dramatized approach to information presentation. Multimedia Systems 14(6):385–393CrossRefGoogle Scholar
  33. 33.
    Madhwacharyula CL, Davis M, Mulhem P, Kankanhalli MS (2006) Metadata handling: a video perspective. ACM Transactions on Multimedia 2(4):358–388CrossRefGoogle Scholar
  34. 34.
    Manovich L (2001) The language of new media. The MIT PressGoogle Scholar
  35. 35.
    Mateas M, Stern A (2003) Integrating plot, character and natural language processing in the interactive drama Fa¸ade. In: TIDSE 03Google Scholar
  36. 36.
    Mateas M, Stern A (2005) Structuring content in the fa¸ade interactive drama architecture. In: Proceedings of artificial intelligence and interactive digital entertainment.Google Scholar
  37. 37.
    Mateas M, Vanouse P, Domike S (2000) Generation of ideologically-biased historical documentaries. In: Seventeenth national conference on artificial intelligence (AAAI), pp 36–42Google Scholar
  38. 38.
    McKee R (1997) Story. Harper Collins, New YorkGoogle Scholar
  39. 39.
    Nack F (1996) AUTEUR: the application of video semantics and theme representation in automated video editing. PhD thesis, Lancaster UniversityGoogle Scholar
  40. 40.
    Nack F, Lindsay AT (1999) Everything you wanted to know about mpeg-7. IEEE Multimedia 6(3):65–77CrossRefGoogle Scholar
  41. 41.
    Nack F, Putz W (2001) Designing annotation before it’s needed. In: Proceedings of the 9th ACM international conference on multimedia, pp 251–260Google Scholar
  42. 42.
    Norling E, Sonenberg L (2004) Creating interactive characters with BDI agents. In: Proceedings of the Australian workshop on interactive entertainment IE2004.Google Scholar
  43. 43.
    Ortony A, Clore G, Collins A (1988) The cognitive structure of emotions. Cambrigde University Press.Google Scholar
  44. 44.
    Peinado F, Cavazza M, Pizzi D (2008) Revisiting character-based affective storytelling under a narrative BDI framework. In: Proc. of ICIDIS08. Erfurt, GermanyGoogle Scholar
  45. 45.
    Peinado F, Gervás P (2004) Transferring game mastering laws to interactive digital storytelling. In: Proceedings of the 2nd international conference on technologies for interactive digital storytelling and entertainmentGoogle Scholar
  46. 46.
    Pizzi D, Charles F, Lugrin J, Cavazza M (2007) Interactive storytelling with literary feelings. In: The second international conference on affective Computing and Intelligent Interaction (ACII2007). Springer, Lisbon, PortugalGoogle Scholar
  47. 47.
    Polti G (1895). Les trente-six situations dramatiques. Mercure de France, ParisGoogle Scholar
  48. 48.
    Prince G (2003) A dictionary of narratology. University of Nebraska PressGoogle Scholar
  49. 49.
    Propp V (1968) Morphology of the folktale. University of Texas PressGoogle Scholar
  50. 50.
    Ryan ML (2004) Narrative across media: the language of storytelling. University of Nebraska PressGoogle Scholar
  51. 51.
    Riedl M, Young M (2006) From linear story generation to branching story graphs. IEEE Journal of Computer Graphics and Applications, pp 23–31Google Scholar
  52. 52.
    Rimmon-Kenan S (1983) Narrative fiction: contemporary poetics. RoutledgeGoogle Scholar
  53. 53.
    Schank RC, Abelson RP (1977) Scripts, plans goals and understanding. Lawrence Erlbaum, HillsdalezbMATHGoogle Scholar
  54. 54.
    Seger L (1990) Creating unforgettable characters. Henry Holt and Company, New YorkGoogle Scholar
  55. 55.
    Sgouros N (1999) Dynamic generation, management and resolution of interactive plots. Artif Intell 107(1):29–62zbMATHCrossRefGoogle Scholar
  56. 56.
    Smith TGA, Davenport G (1992) The stratification system. a design environment for random access video. In: ACM workshop on networking and operating system support for digital audio and videoGoogle Scholar
  57. 57.
    Spierling U (2007) Adding aspects of “implicit creation” to the authoring process in interactive storytelling. In: Proceedings of virtual storytelling 2007. LNCS, 4871:13Google Scholar
  58. 58.
    Swartjes I, Theune M (2006) A fabula model for emergent narrative. Technologies for Interactive Digital Storytelling and Entertainment (TIDSE). LNCS, vol 4326, pp 95–100Google Scholar
  59. 59.
    Szilas N (2003) Idtension: a narrative engine for interactive drama. In: Proc. 1st international conference on Technologies for Interactive Digital Storytelling and Entertainment (TIDSE 2003). Darmstadt, GermanyGoogle Scholar
  60. 60.
    Theune M, Faas S, Heylen D, Nijholt A (2003) The virtual storyteller: story creation by intelligent agents. In: Proceedings TIDSE 03. Fraunhofer IRB Verlag, pp 204–215Google Scholar
  61. 61.
    Ursu MF, Kegel I, Williams D, Thomas M, Mayer H, Zsombori V, Tuomola ML, Larsson H, Wyver J (2008) Shapeshifting tv: interactive screen media narratives. Multimedia Syst 14(2):115–132CrossRefGoogle Scholar
  62. 62.
    van Fraassen B (1973) Values and the heart’s command. J Philos 70(1):5–19CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.CIRMA and Dipartimento di InformaticaUniversità di TorinoTorinoItaly

Personalised recommendations