International Conference on Human Interface and the Management of Information

HIMI 2015: Human Interface and the Management of Information. Information and Knowledge Design pp 261-272 | Cite as

Fusing Text and Image Data with the Help of the OWLnotator

  • Giuseppe AbramiEmail author
  • Alexander  Mehler
  • Dietmar Pravida
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9172)


A central challenge for any approach to mining multimedia data concerns the availability of a unified semantics that allows for the fusion of multicodal information objects. To meet this challenge, a format is needed that enables the representation of multimedia data even across the border of different (e.g. iconic and symbolic) codes using the same ontology. In this paper, we introduce the OWLnotator as a first step to meeting this dual challenge by example of text-image relations. The OWLnotator is presented as part of the eHumanities Desktop, a browser-based, platform-independent environment for the support of collaborative research in the digital humanities. It focuses on modeling and analyzing multicodal, multimedia information objects as studied in the humanities. The eHumanities Desktop contains a wide range of tools for managing, analyzing and sharing resources based on a scalable concept of access permissions. Within this framework, we introduce the OWLnotator as a tool for annotating intra- and intermedia relations of artworks. The OWLnotator allows for modeling relations of symbolic and iconic signs of various levels of resolution: ranging from the level of elementary constituents to the one of complete texts and images. To this end, the OWLnotator integrates TEILex (a system for interrelating corpus and lexicon data as part of the eHumanities Desktop) with the expressiveness of OWL-based ontologies in order to meet the first part of our twofold challenge. As an evaluation, we illustrate the OWLnotator by means of “Illustrations of Goethes Faust”.


Multimedia Data Access Permission Digital Humanity Sign Aggregate Descriptive Metadata 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Abrami, G., Freiberg, M., Warner, P.: Managing and annotating historical multimodal corpora with the eHumanities desktop - an outline of the current state of the LOEWE project illustrations of goethe’s faust. In: Proceedings of the Historical Corpora Conference, 6–9 December 2012, Frankfurt (2012)Google Scholar
  2. 2.
    Bateman, J.A.: Text and Image: A Critical Introduction to the Visual/Verbal Divide. Taylor & Francis, NewYork (2014)Google Scholar
  3. 3.
    Bateman, J.A., Kamps, T., Kleinz, J., Reichenberger, K.: Towards constructive text, diagram, and layout generation for information presentation. Comput. Linguist. 27(3), 409–449 (2001)CrossRefGoogle Scholar
  4. 4.
    von Boehn, M.: Faust und die Kunst. In: Goethe, Johann Wolfgang: Faust. Hundertjahrs-Ausgabe, pp. 3–221. Askanischer Verlag, Berlin (1924)Google Scholar
  5. 5.
    CIDOC CRM Special Interest Group (SIG): Definition of the CIDOC Conceptual Reference Model, 5.0.4 edn. (2011).
  6. 6.
    Dieckmann, L.: Prometheus: the distributed digital image archive for research and education. In: L’Art et la Mesure: Histoire de l’art et méthodes quantitatives, sous la direction de Béatrice Joyeux-Prunel, avec la collaboration de Luc Sigalo Santos, pp. 141–151. Paris (2010)Google Scholar
  7. 7.
    Dieckmann, L., Kliemann, A., Warnke, M.: Meta-Image: forschungsumgebung für den bilddiskurs in der kunstgeschichte. cms-j. Comput.- und Medienservice 35, 11–17 (2012)Google Scholar
  8. 8.
    Giesen, S.: Den Faust, dächt’ ich, gäben wir ohne Holzschnitte und Bildwerke: Goethes “Faust” in der europäischen Kunst des 19. Jahrhunderts. Ph.D. thesis, Technische Hochschule Aachen, Aachen (1998)Google Scholar
  9. 9.
    Gleim, R., Mehler, A., Ernst, A.: SOA implementation of the eHumanities Desktop. In: Proceedings of the Workshop on Service-oriented Architectures (SOAs) for the Humanities: Solutions and Impacts, Digital Humanities 2012, Hamburg, Germany (2012)Google Scholar
  10. 10.
    Gleim, R., Warner, P., Mehler, A.: eHumanities Desktop - an architecture for flexible annotation in iconographic research. In: Proceedings of the 6th International Conference on Web Information Systems and Technologies (WEBIST 2010), Valencia, 7–10 April 2010Google Scholar
  11. 11.
    Hollink, L., Schreiber, A.T., Wielinga, B.J., Worring, M.: Classification of user image descriptions. Int. J. Hum.-Comput. Stud. 61(5), 601–626 (2004)CrossRefGoogle Scholar
  12. 12.
    ICOM-CIDOC Working Group Data Harvesting and Interchange: LIDO - Lightweight Information Describing Objects, 1.0 edn. (2010).
  13. 13.
    Jussen, B. (ed.): Atlas des Historischen Bildwissens 1: Liebig. Digitale Bibliothek, Berlin (2009)Google Scholar
  14. 14.
    Jussen, B. (ed.): Atlas des Historischen Bildwissens 2: Reklamesammelbilder. Digitale Bibliothek, Berlin (2009)Google Scholar
  15. 15.
    Kress, G., Leeuwen, T.: Multimodal Discourse. Arnold, London (2001)Google Scholar
  16. 16.
    Kuper, H.G., Loebel, J.M.: Hyperimage: of layers, labels and links. In: Proceedings of RENEW the 5th edition of the International Conference on the Histories of Media Art, Science and Technology. Riga (2014)Google Scholar
  17. 17.
    Kuper, H.G., Loebel, J.M.: Yenda - Picture Knowledge, Open-Source semantische virtuelle Forschungsumgebung (2015).
  18. 18.
    Loebel, J.M., Kuper, H.G., Arnold, M., Decker, E.: Hachiman digital handscrolls semantische annreicherung mit hyperimage und yenda. In: Bienert, A., Hemsley, J., Santos, P. (eds.) Elektronische Medien & Kunst, Kultur und Historie, pp. 262–267. Konferenzband, Berlin (2014)Google Scholar
  19. 19.
    Lücking, A., Pfeiffer, T.: Framing multimodal technical communication. with focal points in speech-gesture-integration and gaze recognition. In: Mehler, A., Romary, L. (eds.) Handbook of Technical Communication, Handbooks of Applied Linguistics, vol. 8, chap. 18, pp. 591–644. De Gruyter Mouton, Berlin and Boston (2012)Google Scholar
  20. 20.
    Poser, H.: Wissenschaftstheorie. Eine philosophosche Einführung, 2nd edn. Reclam, Stuttgart (2012)Google Scholar
  21. 21.
    Taboada, M., Habel, C.: Rhetorical relations in multimodal documents. Discourse Stud. 15(1), 59–85 (2013)CrossRefGoogle Scholar
  22. 22.
    Wegner, W.: Die Faustdarstellung vom 16. Jahrhundert bis zur Gegenwart. Erasmus Buchhandlung, Amsterdam (1962)Google Scholar
  23. 23.
    Weidenmann, B.: Multicodierung und Multimodalität im Lernprozess. In: Issing, L.J., Klimsa, P. (eds.) Information und Lernen mit Multimedia, pp. 45–62. Beltz, Weinheim (1997)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Giuseppe Abrami
    • 1
    Email author
  • Alexander  Mehler
    • 1
  • Dietmar Pravida
    • 2
  1. 1.Johann Wolfgang Goethe-Universität Frankfurt am MainFrankfurtGermany
  2. 2.Frankfurter Goethe-Museum/Freies Deutsches HochstiftFrankfurtGermany

Personalised recommendations