Multimedia Tools and Applications

, Volume 75, Issue 22, pp 14019–14038 | Cite as

Towards robust and reliable multimedia analysis through semantic integration of services

  • Ben De Meester
  • Ruben Verborgh
  • Pieter Pauwels
  • Wesley De Neve
  • Erik Mannens
  • Rik Van de Walle


Thanks to ubiquitous Web connectivity and portable multimedia devices, it has never been so easy to produce and distribute new multimedia resources such as videos, photos, and audio. This ever-increasing production leads to an information overload for consumers, which calls for efficient multimedia retrieval techniques. Multimedia resources can be efficiently retrieved using their metadata, but the multimedia analysis methods that can automatically generate this metadata are currently not reliable enough for highly diverse multimedia content. A reliable and automatic method for analyzing general multimedia content is needed. We introduce a domain-agnostic framework that annotates multimedia resources using currently available multimedia analysis methods. By using a three-step reasoning cycle, this framework can assess and improve the quality of multimedia analysis results, by consecutively (1) combining analysis results effectively, (2) predicting which results might need improvement, and (3) invoking compatible analysis methods to retrieve new results. By using semantic descriptions for the Web services that wrap the multimedia analysis methods, compatible services can be automatically selected. By using additional semantic reasoning on these semantic descriptions, the different services can be repurposed across different use cases. We evaluated this problem-agnostic framework in the context of video face detection, and showed that it is capable of providing the best analysis results regardless of the input video. The proposed methodology can serve as a basis to build a generic multimedia annotation platform, which returns reliable results for diverse multimedia analysis problems. This allows for better metadata generation, and improves the efficient retrieval of multimedia resources.


Multimedia analysis Reasoning cycle Semantic reasoning Web services 


  1. 1.
    (2014). World Telecommunication/ICT Indicators database 2014, 18 edn. International Telecommunication Union (ITU)Google Scholar
  2. 2.
    Atrey PK, Hossain MA, El Saddik A, Kankanhalli MS (2010) Multimodal fusion for multimedia analysis: a survey. Multimedia systems 16(6):345–379CrossRefGoogle Scholar
  3. 3.
    Beckett D (2004) RDF/XML syntax specification (revised). Accessed on April 15th, 2013
  4. 4.
    Berners-Lee T, Hendler J, Lassila O, et al. (2001) The semantic web. Sci. Am. 284(5):28–37CrossRefGoogle Scholar
  5. 5.
    Chamin Morikawa CM, Kiyoharu Aizawa KA (2012) Iconic visual queries for face image retrieval. Journal of Convergence 3(3):39–46Google Scholar
  6. 6.
    De Roo J (2013) Euler Yet another proof Engine. Accessed on April 7th
  7. 7.
    Drummond N, Shearer R (2006) The open world assumption. Presentation at eSI Workshop: The Closed World of Databases meets the Open World of the Semantic WebGoogle Scholar
  8. 8.
    Erdmann M, Maedche A, Schnurr HP, Staab S (2000) From manual to semi-automatic semantic annotation: About ontology-based text annotation tools. In: Proceedings of the COLING-2000 Workshop on Semantic Annotation and Intelligent Content, pp. 79–85. Association for Computational LinguisticsGoogle Scholar
  9. 9.
    Fensel D, Bussler C (2002) Semantic web enabled web services. In: Jarke M, Lakemeyer G, Koehler J (eds) Proceedings of the 25th Annual German Conference on AI: KI 2002: Advances in Artificial Intelligence, vol 25, pp 319–319. Springer, AachenGoogle Scholar
  10. 10.
    Hanani U, Shapira B, Shoval P (2001) Information filtering: Overview of issues, research and systems. User Model User-Adap Inter 11(3):203–259. doi:10.1023/A:1011196000674 CrossRefMATHGoogle Scholar
  11. 11.
    Hauptmann AG (2005) Lessons for the future from a decade of informedia video analysis research. In: Leow WK, Lew M, Chua TS, Ma WY, Chaisorn L, Bakker E (eds) Image and Video Retrieval, Lecture Notes in Computer Science, vol 3568, pp 1–10. Springer, BerlinGoogle Scholar
  12. 12.
    Hjelmås E, Low BK (2001) Face detection: A survey. Comp Vision Image Underst 83(3):236–274. doi:10.1006/cviu.2001.0921 CrossRefMATHGoogle Scholar
  13. 13.
    Huang YP, Lai SL (2012) Novel query-by-humming/singing method with fuzzy inference system. Journal of Convergence 3(4):1–8Google Scholar
  14. 14.
    Jaeger MC, Rojec-Goldmann G, Muhl G (2004) QoS aggregation for web service composition using workflow patterns. In: Proceedings of the Eighth IEEE International Conference on Enterprise Distributed Object Computing (EDOC), vol 8, pp 149–159. IEEE, MontereyCrossRefGoogle Scholar
  15. 15.
    Lanthaler M, Gütl C (2013) Hydra: A Vocabulary for Hypermedia-Driven Web APIs. In: Bizer C, Heath T, Berners-Lee T, Hausenblas M, Auer S (eds) Proceedings of the WWW2013 Workshop on Linked Data on the Web (LDOW), vol 6. Rio de Janeiro, BrazilGoogle Scholar
  16. 16.
    Ma M, Park DW, Kim SK, An S (2012) Online recognition of handwritten korean and english characters. Journal of Information Processing Systems 8(4):653–669. doi:10.3745/JIPS.2012.8.4.653 CrossRefGoogle Scholar
  17. 17.
    Menasce DA (2004) Composing web wervices: A QoS view. IEEE Internet Computing 8(6):88–90. doi:10.1109/MIC.2004.57 CrossRefGoogle Scholar
  18. 18.
    Ohkawara T, Aikebaier A, Enokido T, Takizawa M (2012) Quorums-based replication of multimedia objects in distributed systems. Human-centric Computing and Information Sciences 2(1):11. doi:10.1186/2192-1962-2-11 CrossRefGoogle Scholar
  19. 19.
    Pauwels P, Bod R (2013) Including the power of interpretation through a simulation of Peirce’s process of inquiry. Literary and Linguistic Computing (LLC) 28(3):452–460CrossRefGoogle Scholar
  20. 20.
    Sarkar K, Nasipuri M, Ghose S (2012) Machine learning based keyphrase extraction: Comparing decision trees, naïve bayes, and artificial neural networks. Journal of Information Processing Systems 8(4):693–712. doi:10.3745/JIPS.2012.8.4.693 CrossRefGoogle Scholar
  21. 21.
    Satone M, Kharate GK (2012) Face recognition based on pca on wavelet subband of average-half-face. Journal of Information Processing Systems 8(3):483–494. doi:10.3745/JIPS.2012.8.3.483 CrossRefGoogle Scholar
  22. 22.
    Schapire RE (2003) The boosting approach to machine learning: An overview. Nonlinear Estimation and Classification. Lecture Notes in Statistics 171(7):149–172MathSciNetCrossRefMATHGoogle Scholar
  23. 23.
    Silas S, Ezra K, Blessing Rajsingh E (2012) A novel fault tolerant service selection framework for pervasive computing. Human-centric Computing and Information Sciences 2(1):5. doi:10.1186/2192-1962-2-5 CrossRefGoogle Scholar
  24. 24.
    Smith A (2013) Smartphone ownership – 2013 update. Pew Research Center: Washington DC:12Google Scholar
  25. 25.
    Smith DR (1985) The design of divide and conquer algorithms. Sci Comput Program 5(0):37–58. doi:10.1016/0167-6423(85)90003-6 MathSciNetCrossRefMATHGoogle Scholar
  26. 26.
    Smith JR, Schirling P (2006) Metadata standards roundup. MultiMedia IEEE 13(2):84–88CrossRefGoogle Scholar
  27. 27.
    Verborgh R, Steiner T, Van Deursen D, De Roo J, Van de Walle R, Gabarró Vallés J (2013) Capturing the functionality of Web services with functional descriptions. Multimedia Tools and Applications 64(2):365–387CrossRefGoogle Scholar
  28. 28.
    Verborgh R, Van Deursen D, Mannens E, Poppe C, Van de Walle R (2012) Enabling context-aware multimedia annotation by a novel generic semantic problem-solving platform. Multimedia Tools and Applications 61(1):105–129. doi:10.1007/s11042-010-0709-6 CrossRefGoogle Scholar
  29. 29.
    Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 1, pp. 511–518. Kauai, HI, USAGoogle Scholar
  30. 30.
    Wolfgang P (1994) Design patterns for object-oriented software development, 1 edn. Addison-Wesley (C)Google Scholar
  31. 31.
    Yen NY, Kuo SYF (2012) An intergrated approach for internet resources mining and searching. Journal of Convergence 3(2):37–44Google Scholar
  32. 32.
    Zadeh LA (1988) Fuzzy logic. Computer 21(4):83–93. doi:10.1109/2.53 CrossRefGoogle Scholar
  33. 33.
    Zeng L, Benatallah B, Ngu AH, Dumas M, Kalagnanam J, Chang H (2004) QoS-aware middleware for web services composition. IEEE Trans Softw Eng 30(5):311–327CrossRefGoogle Scholar
  34. 34.
    Zhu Y, Jin Q (2012) An adaptively emerging mechanism for context-aware service selections regulated by feedback distributions. Human-centric Computing and Information Sciences 2(1):15. doi:10.1186/2192-1962-2-15 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Ben De Meester
    • 1
  • Ruben Verborgh
    • 1
  • Pieter Pauwels
    • 2
  • Wesley De Neve
    • 3
    • 4
  • Erik Mannens
    • 1
  • Rik Van de Walle
    • 1
  1. 1.Ghent University - iMinds - Multimedia LabLedeberg-GhentBelgium
  2. 2.Ghent University - Department of Architecture and Urban Planning GhentBelgium
  3. 3.Multimedia Lab, Ghent University – iMindsLedeberg-GhentBelgium
  4. 4.Image and Video Systems Lab, KAISTYuseong-guRepublic of Korea

Personalised recommendations