Multimedia Tools and Applications

, Volume 61, Issue 1, pp 105–129 | Cite as

Enabling context-aware multimedia annotation by a novel generic semantic problem-solving platform

  • Ruben Verborgh
  • Davy Van Deursen
  • Erik Mannens
  • Chris Poppe
  • Rik Van de Walle


Automatic generation of metadata, facilitating the retrieval of multimedia items, potentially saves large amounts of manual work. However, the high specialization degree of feature extraction algorithms makes them unaware of the context they operate in, which contains valuable and often necessary information. In this paper, we show how Semantic Web technologies can provide a context that algorithms can interact with. We propose a generic problem-solving platform that uses Web services and various knowledge sources to find solutions to complex requests. The platform employs a reasoner-based composition algorithm, generating an execution plan that combines several algorithms as services. It then supervises the execution of this plan, intervening in case of errors or unexpected behavior. We illustrate our approach by a use case in which we annotate the names of people depicted in a photograph.


Annotation Metadata generation Semantic Web Service composition Web services 



The research activities as described in this paper were funded by Ghent University, the Interdisciplinary Institute for Broadband Technology (IBBT), the Institute for the Promotion of Innovation by Science and Technology in Flanders (IWT), the Fund for Scientific Research Flanders (FWO Flanders), and the European Union.


  1. 1.
    Atrey PK, Hossain MA, El Saddik A, Kankanhalli MS (2010) Multimodal fusion for multimedia analysis: a survey. Multimedia Syst 16(6):345–379. doi: 10.1007/s00530-010-0182-0 CrossRefGoogle Scholar
  2. 2.
    Berners-Lee T, Connolly D (2009) Notation3 (N3): a readable RDF syntax. W3C recommendation. Accessed 12 December 2010
  3. 3.
    Berners-Lee T, Connolly D, Kagal L, Scharf Y, Hendler J (2008) N3Logic: a logical framework for the World Wide Web. Theory and Practice of Logic Programming 8(3):249–269. doi: 10.1017/S1471068407003213 MathSciNetzbMATHCrossRefGoogle Scholar
  4. 4.
    Berners-Lee T, Hendler J, Lassila O (2001) The Semantic Web. Sci Am 284(5):34CrossRefGoogle Scholar
  5. 5.
    Bizer C, Heath T, Berners-Lee T (2009) Linked Data - the story so far. Int J Semant Web Inf 5(3):1–22. doi 10.4018/jswis.2009070101 CrossRefGoogle Scholar
  6. 6.
    Bloehdorn S, Petridis K, Saathoff C, Simou N, Tzouvaras V, Avrithis Y, Handschuh S, Kompatsiaris Y, Staab S, Strintzis M (2005) Semantic annotation of images and videos for multimedia analysis. In: The Semantic Web: research and applications, pp 592–607Google Scholar
  7. 7.
    Corkill DD (1991) Blackboard systems. AI Expert 6(9):40–47Google Scholar
  8. 8.
    DBpedia (2010) Accessed 12 December 2010
  9. 9.
    De Roo J (2010) Euler proof mechanism. Accessed 12 December 2010
  10. 10.
    Fensel D, Bussler C (2002) Semantic Web enabled Web services. In: 2nd annual diffuse conferenceGoogle Scholar
  11. 11.
    Finzi A, Pirri F, Reiter R (2000) Open world planning in the situation calculus. In: Proceedings of the national conference on artificial intelligence, pp 754–760Google Scholar
  12. 12.
    Freebase (2010) Accessed 12 December 2010
  13. 13.
    Hanjalic A, Lienhart R, Ma WY, Smith RJ (2008) The holy grail of multimedia information retrieval: so close or yet so far away? In: Proceedings of the IEEE Multimedia, vol 96(4), pp 541–547.
  14. 14.
    Horrocks I, Patel-Schneider PF, Boley H, Tabet S (2004) SWRL: a Semantic Web rule language combining OWL and RuleML. W3C member submission. Accessed 12 December 2010
  15. 15.
    Hristoskova A, Volckaert B, Turck FD (2009) Dynamic composition of semantically annotated web services through QoS-aware HTN planning algorithms. In: ICIW ’09: Proceedings of the 2009 fourth international conference on Internet and Web applications and services. IEEE Computer Society, Washington, pp 377–382Google Scholar
  16. 16.
    Junghans M, Agarwal S, Studer R (2010) Towards practical Semantic Web service discovery. In: The Semantic Web: research and applications, 7th extended Semantic Web conference, ESWC 2010, Heraklion, Crete, Greece, May 30–June 3, 2010, Proceedings. Springer, New YorkGoogle Scholar
  17. 17.
    Klusch M, Fries B, Khalid M (2005) OWLS-MX: Hybrid OWL-S service matchmaking. In: Proceedings of 1st intl. AAAI fall symposium on agents and the Semantic WebGoogle Scholar
  18. 18.
    Klyne G, Carrol JJ (2004) Resource description framework (RDF): concepts and abstract syntax. W3C recommendation. Accessed 12 December 2010
  19. 19.
    Li L, Horrocks I (2003) A software framework for matchmaking based on Semantic Web technology. In: WWW ’03: Proceedings of the 12th international conference on World Wide Web. ACM Press, New York, pp 331–339. doi: 10.1145/775152.775199 Google Scholar
  20. 20.
    Martin D, Burstein M, Hobbs J, Lassila O (2004) OWL-S: Semantic markup for web services. W3C member submission. Accessed 12 December 2010
  21. 21.
    Medjahed B, Bouguettaya A, Elmagarmid AK (2003) Composing web services on the Semantic Web. VLDB J 12(4):333–351. doi: 10.1007/s00778-003-0101-5 CrossRefGoogle Scholar
  22. 22.
    Osmun T (2010) Euler Eye installation, demo, and deep taxonomy benchmark. Accessed 12 December 2010
  23. 23.
    Osmun T (2010) Eye deep taxonomy benchmark results. Accessed 12 December 2010
  24. 24.
    Prud’hommeaux E, Seaborne A (2008) SPARQL query language for RDF. W3C recommendation. Accessed 12 December 2010
  25. 25.
    Rahurkar M, Tsai S, Dagli C, Huang T (2010) Image interpretation using large corpus: wikipedia. Proc IEEE 98(8):1509–1525. doi: 10.1109/JPROC.2010.2050410. CrossRefGoogle Scholar
  26. 26.
    Redavid D, Iannone L, Payne T (2007) OWL-S atomic services composition with SWRL rules. In: Proceedings of the 4th Italian Semantic Web workshop.
  27. 27.
    Roy PV, Haridi S (2004) Concepts, techniques, and models of computer programming. MIT Press, CambridgeGoogle Scholar
  28. 28.
    Shin D, Lee K, Suda T (2009) Automated generation of composite Web services based on functional semantics. Web Semantics: ScienceGoogle Scholar
  29. 29.
    Smith JR, Schirling P (2006) Metadata standards roundup. IEEE MultiMedia 13:84–88. doi: 10.1109/MMUL.2006.34 CrossRefGoogle Scholar
  30. 30.
    Sycara K, Paolucci M, Ankolekar A, Srinivasan N (2003) Automated discovery, interaction and composition of Semantic Web services. Journal of Web Semantics 1(1):27–46CrossRefGoogle Scholar
  31. 31.
    Troncy R, Mannens E, Pfeiffer S, Van Deursen D (2010) Media fragments URI 1.0. W3C working draft. Accessed 12 December 2010
  32. 32.
    Verborgh R (2010) Generic semantic problem-solving platform demonstration: road trip. Accessed 12 December 2010
  33. 33.
    Verborgh R, Van Deursen D, De Roo J, Mannens E, Van de Walle R (2010) SPARQL endpoints as front-end for multimedia processing algorithms. In: Proceedings of the fourth international workshop on service matchmaking and resource retrieval in the Semantic Web at the 9th international Semantic Web conference (ISWC 2010)Google Scholar
  34. 34.
    Verstockt S, Van Leuven S, Van de Walle R, Dermaut E, Torelle S, Gevaert W (2009) Actor recognition for interactive querying and automatic annotation in digital video. In: 13th, Proceedings of the IASTED international conference on Internet and multimedia systems and applications. ACTA Press, Honolulu, pp 149–155Google Scholar
  35. 35.
    Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154. doi: 10.1023/B:VISI.0000013087.49260.fb CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Ruben Verborgh
    • 1
  • Davy Van Deursen
    • 1
  • Erik Mannens
    • 1
  • Chris Poppe
    • 1
  • Rik Van de Walle
    • 1
  1. 1.ELIS—Multimedia LabGhent University—IBBTLedeberg-GhentBelgium

Personalised recommendations