Skip to main content
SpringerLink
Log in

A middleware to enhance current multimedia retrieval systems with content-based functionalities

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Nowadays the retrieval of multimedia assets is mainly performed by text-based retrieval systems with powerful and stable indexing mechanisms. Migration from those systems to content-aware multimedia retrieval systems is a common aim for companies from very diverse sectors. In this paper we present a semantic middleware designed to achieve a seamless integration with existing systems. This middleware outsources the semantic functionalities (e.g. knowledge extraction, semantic query expansion,…) that are not covered by traditional systems, thereby allowing the use of complementary content-based techniques. We include a list of key criteria to successfully deploy this middleware, which provides semantic support to many different steps of the retrieval process. Both the middleware and the design criteria are validated by two real complementary deployments in two very different industrial domains.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Fast ESP Search Engine web site http://www.microsoft.com/enterprisesearch/en/us/Fast.aspx.

  2. WIDE project(IST-2001-34417) http://www.ist-wide.info.

  3. RUSHES project(FP6-045189) http://www.rushes-project.eu.

References

  1. Arens, Y., Knoblock, C.A., Shen, W.M.: Query reformulation for dynamic information integration. J. Intell. Inf. Syst. 6(2–3), 99–130 (1996)

    Article  Google Scholar 

  2. Baer, D., Groenewoud, P., Kapetanios, E., Keuser, S.: A semantics based interactive query formulation technique. In: Proceedings of the Second International Workshop on User Interfaces to Data Intensive Systems (UIDIS’01), p. 43. IEEE Computer Society, Washington, DC, USA (2001)

  3. Baeza-Yates, R.A., Ribeiro-Neto, B.A.: Modern Information Retrieval. ACM Press/Addison-Wesley, New York (1999)

    Google Scholar 

  4. Bailer, W., Schallauer, P.: The detailed audiovisual profile: Enabling interoperability between mpeg-7 based systems. In: Proceedings of 12th Multimedia Modellinig Conference, pp. 217–224. Beijing (2006)

  5. Bailer, W., Schallauer, P., Neuschmied, H.: Mpeg-7 detailed audiovisual profile. Tech. rep., Joanneum Research (2007)

  6. Bechhofer, S., Volz, R., Lord, P.W.: Cooking the semantic web with the owl api. In: International Semantic Web Conference, pp. 659–675 (2003)

  7. Bennett, M.: 2008 enterprise search vendors: the new fab 4... and 1/2. Tech. rep., New Idea Engineering, Inc., vol. 5, No. 1 (2008)

  8. Bhogal, J., Macfarlane, A., Smith, P.: A review of ontology based query expansion. Int. J. Inform. Process. Manage. 43(4), 866–886 (2007)

    Article  Google Scholar 

  9. Bürger, T.: An intelligent media framework for multimedia content. In: Proceedings of International Workshop on Semantic Web Annotations for Multimedia (SWAMM) (2006)

  10. Candela, L., Castelli, D., Pagano, P., Simi, M.: The digital library information mediator layer. In: Agosti, M., Thanos, C. (Eds) IRCDL, pp. 29–36. DELOS: a Network of Excellence on Digital Libraries (2006)

  11. Castells, P., Fernández, M., Vallet, D.: An adaptation of the vector-space model for ontology-based information retrieval. IEEE Trans. Knowl. Data Eng. 19, 261–272 (2007)

    Article  Google Scholar 

  12. Chatterjee, K., Chen, S.C.: A novel indexing and access mechanism using affinity hybrid tree for content-based image retrieval in multimedia databases. Int. J. Semantic Comput. 1(2) ,147–170 (2007)

    Article  MathSciNet  Google Scholar 

  13. Faaborg, A.J.: A goal-oriented user interface for personalized semantic search. Ph.D. thesis, B.A. Information Science Cornell University (2003)

  14. Google-Labs: Google sets: Automatically create sets of items from a few examples.

  15. Grau, B.C., Motik, B.: Owl 1.1 web ontology language: Model-theoretic semantics. Tech. rep., Oxford University, Oxford (2008)

  16. Haarslev, V., Möller, R.: Racer system description. In: Goré, R., Leitsch, A., Nipkow T (eds.) International Joint Conference on Automated Reasoning, IJCAR’2001, June 18–23, Siena, Italy, pp. 701–705. Springer-Verlag, New York (2001)

  17. Karvounarakis, G., Alexaki, S., Christophides, V., Plexousakis, D., Scholl, M.: Rql: a declarative query language for rdf. In: Proceedings of the 11th international conference on World Wide Web, pp. 592–603. ACM, New York (2002)

  18. Kerschberg, L., Weishar, D.: Conceptual models and architectures for advanced information systems. Appl. Intell. 13(2) ,149–164 (2000)

    Article  Google Scholar 

  19. Kim, W., Seo, J.: Classifying schematic and data heterogeneity in multidatabase systems. Computer 24(12), 12–18 (1991)

    Article  Google Scholar 

  20. Knuth, D.E.: backus normal form vs. backus naur form. Commun. ACM 7(12), 735–736 (1964)

    Article  Google Scholar 

  21. Köhler, J., Philippi, S., Specht, M., Rüegg, A.: Ontology based text indexing and querying for the semantic web. Know.-Based Syst. 19(8), 744–754 (2006)

    Article  Google Scholar 

  22. Larson, R.R.: Information retrieval systems, 3rd edn. Encyclopedia of Library and Information Sciences (2010)

  23. Lew, M.S., Sebe, N., Djeraba, C., Jain, R.: Content-based multimedia information retrieval: State of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl. 2(1), 1–19 (2006)

    Article  Google Scholar 

  24. Luo, H.: Concept-based large-scale video database browsing and retrieval via visualization. Ph.D. thesis, The University of North Carolina at Charlotte, Charlotte, NC, USA (Adviser-Fan, Jianping) (2007)

  25. Marcos, G., Alonso, K., Illarramendi, A., Olaizola, I.G., Flórez, J.: Dms-1 driven data model to enable a semantic middleware for multimedia information retrieval in a broadcaster. In: Proceedings of the 4th International Workshop on Semantic Media Adaptation and Personalization (2009)

  26. Marcos, G., Krämer, P., Illarramendi, A., Olaizola, I.G., Flórez, J.: Semantic middleware to enhance multimedia retrieval in a broadcaster. In: Luke, D., Hardman, L., Hauptmann, A., Paulus, D., Staab, S. (eds.) Semantic Multimedia, Third International Conference on Semantic and Digital Media Technologies, SAMT 2008 Koblenz, Germany, December 2008, Proceedings, pp. 74–88. Springer-Verlag, Berlin (2008)

  27. Marcos, G., Smithers, T., Jimánez, I., Toro, C.: Meta level: Enabler for semantic steered multimedia retrieval in an industrial design domain. Syst. Sci. 2, 15–22 (2007)

    Google Scholar 

  28. Meghini, C., Sebastiani, F., Straccia, U.: A model of multimedia information retrieval. J. ACM 48(5), 909–970 (2001)

    Article  MathSciNet  Google Scholar 

  29. Naphade, M.R., Huang, T.S.: Extracting semantics from audiovisual content: The final frontier in multimedia retrieval. IEEE Trans. Neural Networks 13(4), 793–810 (2002)

    Article  Google Scholar 

  30. Neo, S.Y., Zhao, J., Kan, M.Y., Chua, T.S.: Video retrieval using high level features: exploiting query matching and confidence-based weighting. In: Image and Video Retrieval, pp. 143–152. Springer, Berlin (2006)

  31. Nitto, E.D., Pianciamore, M., Selvini, P.: The role of agents in knowledge management. In: WOA, pp. 29–34 (2002)

  32. Olaizola, I.G., Marcos, G., Krämer, P., Flórez, J., Sierra, B.: Architecture for semi-automatic multimedia analysis by hypothesis reinforcement. In: IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (2009)

  33. Pan, Z.: Benchmarking dl reasoners using realistic ontologies. In: OWLED (2005)

  34. Piasecki, M., Beran, B.: A semantic annotation tool for hydrologic sciences. Earth Sci. Inform. 2(3), 157–168 (2009)

    Article  Google Scholar 

  35. Ruthven, I., Lalmas, M.: A survey on the use of relevance feedback for information access systems. Knowl. Eng. Rev. 18(2), 95–145 (2003)

    Article  Google Scholar 

  36. Sarris, N.: Rushes-project. D5.5 Report on validation and performance evaluation. Tech. rep., Athens Technology Centre (2009)

  37. Schach, S.R.: Object-Oriented and Classical Software Engineering. McGraw-Hill, New York (2007)

    Google Scholar 

  38. Schallauer, P., Bailer, W., Thallinger, G.: A description infrastructure for audiovisual media processing systems based on mpeg-7. J. Univers. Knowl. Manage. 1(1), 26–35 (2006)

    Google Scholar 

  39. Schreer, O., Ardeo, L.F., Sotiriou, D., Sadka, A., Izquierdo, E.: User requirements for multimedia indexing and retrieval of unedited audio-visual footage—rushes. In: Proceedings of 9th Int. Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS) (2008)

  40. Sevilmis, N., Stork, A., Smithers, T., Posada, J., Pianciamore, M., Castro, R., Jimenez, I., Marcos, G., Mauri, M., Selvini, P., Thelen, B., Zecchino, V.: Knowledge sharing by information retrieval in the semantic web. In: ESWC, pp. 471–485 (2005)

  41. Shamsfard, M., Nematzadeh, A., Motiee, S.: Orank: An ontology based system for ranking documents. Int. J. Comput. Sci. 1, 225–231 (2006)

    Google Scholar 

  42. Simou, N., Kollias, S.: Fire : A fuzzy reasoning engine for impecise knowledge. In: K-Space PhD Workshop. K-Space PhD Students Workshop, Berlin, Germany (2007)

  43. Simou, N., Saathoff, C., Dasiopoulou, S., Spyrou, E., Voisine, N., Tzouvaras, V., Kompatsiaris, I., Avrithis, Y.S., Staab, S.: An ontology infrastructure for multimedia reasoning. In: Atzori, L., Giusto, D.D., Leonardi, R., Pereira, F. (eds.) VLBV, Lecture Notes in Computer Science, vol. 3893, pp. 51–60. Springer, Berlin (2005)

  44. Snoek, C.G.M., Huurnink, B., Hollink, L., de Rijke, M., Schreiber, G., Worring, M.: Adding semantics to detectors for video retrieval. IEEE Trans. Multimedia 9(5), 975–986 (2007)

    Article  Google Scholar 

  45. Soergel, D.: Indexing Languages and Thesauri: Construction and Maintenance. Wiley, New York (1974)

    Google Scholar 

  46. Srinivasan, U., Pfeiffer, S., Nepal, S., Lee, M., Gu, L., Barrass, S.: A survey of mpeg-1 audio, video and semantic analysis techniques. Multimedia Tools Appl. 27(1), 105–141 (2005)

    Article  Google Scholar 

  47. Stamou, G., Kollias, S.: Multimedia Content and the Semantic Web: Standards, Methods and Tools. Wiley, New York (2005)

    Book  Google Scholar 

  48. Thelen, B., Sevilmis, N., Stork, A., Castro, R., Jimánez, I., Marcos, G., Posada, J., Smithers, T., Mauri, M., Pianciamore, M., Selvini, P., Zecchino, V.: Information management on the basis of semantic-web techniques, or a google for developers. In: VDI-Verlag - Gesellschaft Fahrzeug- und Verkehrstechnik: Erprobung und Simulation in der Fahrzeugentwicklung: Mess- und Versuchstechnik, pp. 167–180. VDI-Verlag, Würzburg (2005)

  49. Troncy, R., Bailer, W., Hausenblas, M., Hofmair, P., Schlatte, R.: Enabling multimedia metadata interoperability by defining formal semantics of mpeg-7 profiles. In: SAMT, pp. 41–55 (2006)

  50. Tzitzikas, Y., Spyratos, N., Constantopoulos, P.: Query translation for mediators over ontology-based information sources. In: Proceedings of the Second Hellenic Conference on AI, pp. 423–436. Springer, London (2002)

  51. Wang, H., Liu, S., Chia, L.T.: Does ontology help in image retrieval?: a comparison between keyword, text ontology and multi-modality ontology approaches. In: ACM Multimedia, pp. 109–112 (2006)

  52. Wilkinson, J.: Material exchange format (mxf) descriptive metadata scheme—1 (standard, dynamic). Tech. rep., Pro-MPEG (2003)

  53. Yu, Y., Kim, J., Shin, K., Jo, G.S.: Recommendation system using location-based ontology on wireless internet: an example of collective intelligence by using ’mashup’ applications. Expert Syst. Appl. 36(9), 11675–11681 (2009)

    Article  Google Scholar 

  54. Zhang, Y.j.: Semantic-Based Visual Information Retrieval. IRM Press, Italy (2006)

    Google Scholar 

Download references

Acknowledgments

We would like to thank Professor Ray Larson of the University of California and the Image group of the National University of Athens for so generously sharing their work with us. We also would like to thank the European Commission for co-funding this work and to the engineers of the Basque Television (ETB), Italdesign, FAST and Schenck for their kind support.

Author information

Authors and Affiliations

  1. Vicomtech, San Sebastian Technological Park, San Sebastián, Spain

    Gorka Marcos, Igor G. Olaizola & Julián Flórez

  2. University of the Basque Country, San Sebastián, Spain

    Arantza Illarramendi

Authors
  1. Gorka Marcos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arantza Illarramendi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Igor G. Olaizola
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Julián Flórez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gorka Marcos.

Additional information

Communicated by B. Prabhakaran.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Marcos, G., Illarramendi, A., Olaizola, I.G. et al. A middleware to enhance current multimedia retrieval systems with content-based functionalities. Multimedia Systems 17, 149–164 (2011). https://doi.org/10.1007/s00530-010-0217-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-010-0217-6

keywords

Search

Navigation