Multimedia Systems

, Volume 17, Issue 2, pp 149–164 | Cite as

A middleware to enhance current multimedia retrieval systems with content-based functionalities

  • Gorka Marcos
  • Arantza Illarramendi
  • Igor G. Olaizola
  • Julián Flórez
Regular Paper


Nowadays the retrieval of multimedia assets is mainly performed by text-based retrieval systems with powerful and stable indexing mechanisms. Migration from those systems to content-aware multimedia retrieval systems is a common aim for companies from very diverse sectors. In this paper we present a semantic middleware designed to achieve a seamless integration with existing systems. This middleware outsources the semantic functionalities (e.g. knowledge extraction, semantic query expansion,…) that are not covered by traditional systems, thereby allowing the use of complementary content-based techniques. We include a list of key criteria to successfully deploy this middleware, which provides semantic support to many different steps of the retrieval process. Both the middleware and the design criteria are validated by two real complementary deployments in two very different industrial domains.


Multimedia information retrieval Content-based Semantic middleware 


  1. 1.
    Arens, Y., Knoblock, C.A., Shen, W.M.: Query reformulation for dynamic information integration. J. Intell. Inf. Syst. 6(2–3), 99–130 (1996)CrossRefGoogle Scholar
  2. 2.
    Baer, D., Groenewoud, P., Kapetanios, E., Keuser, S.: A semantics based interactive query formulation technique. In: Proceedings of the Second International Workshop on User Interfaces to Data Intensive Systems (UIDIS’01), p. 43. IEEE Computer Society, Washington, DC, USA (2001)Google Scholar
  3. 3.
    Baeza-Yates, R.A., Ribeiro-Neto, B.A.: Modern Information Retrieval. ACM Press/Addison-Wesley, New York (1999)Google Scholar
  4. 4.
    Bailer, W., Schallauer, P.: The detailed audiovisual profile: Enabling interoperability between mpeg-7 based systems. In: Proceedings of 12th Multimedia Modellinig Conference, pp. 217–224. Beijing (2006)Google Scholar
  5. 5.
    Bailer, W., Schallauer, P., Neuschmied, H.: Mpeg-7 detailed audiovisual profile. Tech. rep., Joanneum Research (2007)Google Scholar
  6. 6.
    Bechhofer, S., Volz, R., Lord, P.W.: Cooking the semantic web with the owl api. In: International Semantic Web Conference, pp. 659–675 (2003)Google Scholar
  7. 7.
    Bennett, M.: 2008 enterprise search vendors: the new fab 4... and 1/2. Tech. rep., New Idea Engineering, Inc., vol. 5, No. 1 (2008)Google Scholar
  8. 8.
    Bhogal, J., Macfarlane, A., Smith, P.: A review of ontology based query expansion. Int. J. Inform. Process. Manage. 43(4), 866–886 (2007)CrossRefGoogle Scholar
  9. 9.
    Bürger, T.: An intelligent media framework for multimedia content. In: Proceedings of International Workshop on Semantic Web Annotations for Multimedia (SWAMM) (2006)Google Scholar
  10. 10.
    Candela, L., Castelli, D., Pagano, P., Simi, M.: The digital library information mediator layer. In: Agosti, M., Thanos, C. (Eds) IRCDL, pp. 29–36. DELOS: a Network of Excellence on Digital Libraries (2006)Google Scholar
  11. 11.
    Castells, P., Fernández, M., Vallet, D.: An adaptation of the vector-space model for ontology-based information retrieval. IEEE Trans. Knowl. Data Eng. 19, 261–272 (2007)CrossRefGoogle Scholar
  12. 12.
    Chatterjee, K., Chen, S.C.: A novel indexing and access mechanism using affinity hybrid tree for content-based image retrieval in multimedia databases. Int. J. Semantic Comput. 1(2) ,147–170 (2007)CrossRefMathSciNetGoogle Scholar
  13. 13.
    Faaborg, A.J.: A goal-oriented user interface for personalized semantic search. Ph.D. thesis, B.A. Information Science Cornell University (2003)Google Scholar
  14. 14.
    Google-Labs: Google sets: Automatically create sets of items from a few examples.Google Scholar
  15. 15.
    Grau, B.C., Motik, B.: Owl 1.1 web ontology language: Model-theoretic semantics. Tech. rep., Oxford University, Oxford (2008)Google Scholar
  16. 16.
    Haarslev, V., Möller, R.: Racer system description. In: Goré, R., Leitsch, A., Nipkow T (eds.) International Joint Conference on Automated Reasoning, IJCAR’2001, June 18–23, Siena, Italy, pp. 701–705. Springer-Verlag, New York (2001)Google Scholar
  17. 17.
    Karvounarakis, G., Alexaki, S., Christophides, V., Plexousakis, D., Scholl, M.: Rql: a declarative query language for rdf. In: Proceedings of the 11th international conference on World Wide Web, pp. 592–603. ACM, New York (2002)Google Scholar
  18. 18.
    Kerschberg, L., Weishar, D.: Conceptual models and architectures for advanced information systems. Appl. Intell. 13(2) ,149–164 (2000)CrossRefGoogle Scholar
  19. 19.
    Kim, W., Seo, J.: Classifying schematic and data heterogeneity in multidatabase systems. Computer 24(12), 12–18 (1991)CrossRefGoogle Scholar
  20. 20.
    Knuth, D.E.: backus normal form vs. backus naur form. Commun. ACM 7(12), 735–736 (1964)CrossRefGoogle Scholar
  21. 21.
    Köhler, J., Philippi, S., Specht, M., Rüegg, A.: Ontology based text indexing and querying for the semantic web. Know.-Based Syst. 19(8), 744–754 (2006)CrossRefGoogle Scholar
  22. 22.
    Larson, R.R.: Information retrieval systems, 3rd edn. Encyclopedia of Library and Information Sciences (2010)Google Scholar
  23. 23.
    Lew, M.S., Sebe, N., Djeraba, C., Jain, R.: Content-based multimedia information retrieval: State of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl. 2(1), 1–19 (2006)CrossRefGoogle Scholar
  24. 24.
    Luo, H.: Concept-based large-scale video database browsing and retrieval via visualization. Ph.D. thesis, The University of North Carolina at Charlotte, Charlotte, NC, USA (Adviser-Fan, Jianping) (2007)Google Scholar
  25. 25.
    Marcos, G., Alonso, K., Illarramendi, A., Olaizola, I.G., Flórez, J.: Dms-1 driven data model to enable a semantic middleware for multimedia information retrieval in a broadcaster. In: Proceedings of the 4th International Workshop on Semantic Media Adaptation and Personalization (2009)Google Scholar
  26. 26.
    Marcos, G., Krämer, P., Illarramendi, A., Olaizola, I.G., Flórez, J.: Semantic middleware to enhance multimedia retrieval in a broadcaster. In: Luke, D., Hardman, L., Hauptmann, A., Paulus, D., Staab, S. (eds.) Semantic Multimedia, Third International Conference on Semantic and Digital Media Technologies, SAMT 2008 Koblenz, Germany, December 2008, Proceedings, pp. 74–88. Springer-Verlag, Berlin (2008)Google Scholar
  27. 27.
    Marcos, G., Smithers, T., Jimánez, I., Toro, C.: Meta level: Enabler for semantic steered multimedia retrieval in an industrial design domain. Syst. Sci. 2, 15–22 (2007)Google Scholar
  28. 28.
    Meghini, C., Sebastiani, F., Straccia, U.: A model of multimedia information retrieval. J. ACM 48(5), 909–970 (2001)CrossRefMathSciNetGoogle Scholar
  29. 29.
    Naphade, M.R., Huang, T.S.: Extracting semantics from audiovisual content: The final frontier in multimedia retrieval. IEEE Trans. Neural Networks 13(4), 793–810 (2002)CrossRefGoogle Scholar
  30. 30.
    Neo, S.Y., Zhao, J., Kan, M.Y., Chua, T.S.: Video retrieval using high level features: exploiting query matching and confidence-based weighting. In: Image and Video Retrieval, pp. 143–152. Springer, Berlin (2006)Google Scholar
  31. 31.
    Nitto, E.D., Pianciamore, M., Selvini, P.: The role of agents in knowledge management. In: WOA, pp. 29–34 (2002)Google Scholar
  32. 32.
    Olaizola, I.G., Marcos, G., Krämer, P., Flórez, J., Sierra, B.: Architecture for semi-automatic multimedia analysis by hypothesis reinforcement. In: IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (2009)Google Scholar
  33. 33.
    Pan, Z.: Benchmarking dl reasoners using realistic ontologies. In: OWLED (2005)Google Scholar
  34. 34.
    Piasecki, M., Beran, B.: A semantic annotation tool for hydrologic sciences. Earth Sci. Inform. 2(3), 157–168 (2009)CrossRefGoogle Scholar
  35. 35.
    Ruthven, I., Lalmas, M.: A survey on the use of relevance feedback for information access systems. Knowl. Eng. Rev. 18(2), 95–145 (2003)CrossRefGoogle Scholar
  36. 36.
    Sarris, N.: Rushes-project. D5.5 Report on validation and performance evaluation. Tech. rep., Athens Technology Centre (2009)Google Scholar
  37. 37.
    Schach, S.R.: Object-Oriented and Classical Software Engineering. McGraw-Hill, New York (2007)Google Scholar
  38. 38.
    Schallauer, P., Bailer, W., Thallinger, G.: A description infrastructure for audiovisual media processing systems based on mpeg-7. J. Univers. Knowl. Manage. 1(1), 26–35 (2006)Google Scholar
  39. 39.
    Schreer, O., Ardeo, L.F., Sotiriou, D., Sadka, A., Izquierdo, E.: User requirements for multimedia indexing and retrieval of unedited audio-visual footage—rushes. In: Proceedings of 9th Int. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS) (2008)Google Scholar
  40. 40.
    Sevilmis, N., Stork, A., Smithers, T., Posada, J., Pianciamore, M., Castro, R., Jimenez, I., Marcos, G., Mauri, M., Selvini, P., Thelen, B., Zecchino, V.: Knowledge sharing by information retrieval in the semantic web. In: ESWC, pp. 471–485 (2005)Google Scholar
  41. 41.
    Shamsfard, M., Nematzadeh, A., Motiee, S.: Orank: An ontology based system for ranking documents. Int. J. Comput. Sci. 1, 225–231 (2006)Google Scholar
  42. 42.
    Simou, N., Kollias, S.: Fire : A fuzzy reasoning engine for impecise knowledge. In: K-Space PhD Workshop. K-Space PhD Students Workshop, Berlin, Germany (2007)Google Scholar
  43. 43.
    Simou, N., Saathoff, C., Dasiopoulou, S., Spyrou, E., Voisine, N., Tzouvaras, V., Kompatsiaris, I., Avrithis, Y.S., Staab, S.: An ontology infrastructure for multimedia reasoning. In: Atzori, L., Giusto, D.D., Leonardi, R., Pereira, F. (eds.) VLBV, Lecture Notes in Computer Science, vol. 3893, pp. 51–60. Springer, Berlin (2005)Google Scholar
  44. 44.
    Snoek, C.G.M., Huurnink, B., Hollink, L., de Rijke, M., Schreiber, G., Worring, M.: Adding semantics to detectors for video retrieval. IEEE Trans. Multimedia 9(5), 975–986 (2007)CrossRefGoogle Scholar
  45. 45.
    Soergel, D.: Indexing Languages and Thesauri: Construction and Maintenance. Wiley, New York (1974)Google Scholar
  46. 46.
    Srinivasan, U., Pfeiffer, S., Nepal, S., Lee, M., Gu, L., Barrass, S.: A survey of mpeg-1 audio, video and semantic analysis techniques. Multimedia Tools Appl. 27(1), 105–141 (2005)CrossRefGoogle Scholar
  47. 47.
    Stamou, G., Kollias, S.: Multimedia Content and the Semantic Web: Standards, Methods and Tools. Wiley, New York (2005)CrossRefGoogle Scholar
  48. 48.
    Thelen, B., Sevilmis, N., Stork, A., Castro, R., Jimánez, I., Marcos, G., Posada, J., Smithers, T., Mauri, M., Pianciamore, M., Selvini, P., Zecchino, V.: Information management on the basis of semantic-web techniques, or a google for developers. In: VDI-Verlag - Gesellschaft Fahrzeug- und Verkehrstechnik: Erprobung und Simulation in der Fahrzeugentwicklung: Mess- und Versuchstechnik, pp. 167–180. VDI-Verlag, Würzburg (2005)Google Scholar
  49. 49.
    Troncy, R., Bailer, W., Hausenblas, M., Hofmair, P., Schlatte, R.: Enabling multimedia metadata interoperability by defining formal semantics of mpeg-7 profiles. In: SAMT, pp. 41–55 (2006)Google Scholar
  50. 50.
    Tzitzikas, Y., Spyratos, N., Constantopoulos, P.: Query translation for mediators over ontology-based information sources. In: Proceedings of the Second Hellenic Conference on AI, pp. 423–436. Springer, London (2002)Google Scholar
  51. 51.
    Wang, H., Liu, S., Chia, L.T.: Does ontology help in image retrieval?: a comparison between keyword, text ontology and multi-modality ontology approaches. In: ACM Multimedia, pp. 109–112 (2006)Google Scholar
  52. 52.
    Wilkinson, J.: Material exchange format (mxf) descriptive metadata scheme—1 (standard, dynamic). Tech. rep., Pro-MPEG (2003)Google Scholar
  53. 53.
    Yu, Y., Kim, J., Shin, K., Jo, G.S.: Recommendation system using location-based ontology on wireless internet: an example of collective intelligence by using ’mashup’ applications. Expert Syst. Appl. 36(9), 11675–11681 (2009)CrossRefGoogle Scholar
  54. 54.
    Zhang, Y.j.: Semantic-Based Visual Information Retrieval. IRM Press, Italy (2006)Google Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  • Gorka Marcos
    • 1
  • Arantza Illarramendi
    • 2
  • Igor G. Olaizola
    • 1
  • Julián Flórez
    • 1
  1. 1.VicomtechSan SebastiánSpain
  2. 2.University of the Basque CountrySan SebastiánSpain

Personalised recommendations