A Hybrid Approach for Multi-faceted IR in Multimodal Domain

  • Serwah Sabetghadam
  • Ralf Bierig
  • Andreas Rauber
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8685)


We present a model for multimodal information retrieval, leveraging different information sources to improve the effectiveness of a retrieval system. This method takes into account multifaceted IR in addition to the semantic relations present in data objects, which can be used to answer complex queries, combining similarity and semantic search. By providing a graph data structure and utilizing hybrid search in addition to structured search techniques, we take advantage of relations in data to improve retrieval. We tested the model with ImageCLEF 2011 Wikipedia collection, as a multimodal benchmark data collection, for an image retrieval task.


Multimodal Information Retrieval Graph Hybrid Search Facet Spreading Activation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berber, T., Vahid, A.H., Ozturkmenoglu, O., Hamed, R.G., Alpkocak, A.: Demir at imageclefwiki 2011: Evaluating different weighting schemes in information retrieval. In: CLEF (2011)Google Scholar
  2. 2.
    Berthold, M.R., Brandes, U., Kotter, T., Mader, M., Nagel, U., Thiel, K.: Pure spreading activation is pointless. In: CIKM (2009)Google Scholar
  3. 3.
    Crestani, F.: Application of spreading activation techniques in information retrieval. Artificial Intelligence Review, 11 (1997)Google Scholar
  4. 4.
    Delbru, R., Toupikov, N., Catasta, M., Tummarello, G.: A node indexing scheme for web entity retrieval. In: Aroyo, L., Antoniou, G., Hyvönen, E., ten Teije, A., Stuckenschmidt, H., Cabral, L., Tudorache, T. (eds.) ESWC 2010, Part II. LNCS, vol. 6089, pp. 240–256. Springer, Heidelberg (2010)Google Scholar
  5. 5.
    Duan, L., Li, W., Tsang, I.W.-H., Xu, D.: Improving web image search by bag-based reranking. IEEE Transactions on Image Processing 20(11) (2011)Google Scholar
  6. 6.
    Elbassuoni, S., Blanco, R.: Keyword search over RDF graphs. In: CIKM (2011)Google Scholar
  7. 7.
    Fergus, R., Fei-Fei, L., Perona, P., Zisserman, A.: Learning object categories from google’s image search. In: Proc. of Intl. Conf. on Computer Vision (2005)Google Scholar
  8. 8.
    Kasneci, G., Suchanek, F., Ifrim, G., Ramanath, M., Weikum, G.: Naga: Searching and ranking knowledge. In: ICDE (2008)Google Scholar
  9. 9.
    Lazaridis, M., Axenopoulos, A., Rafailidis, D., Daras, P.: Multimedia search and retrieval using multimodal annotation propagation and indexing techniques. Signal Processing: Image Comm. (2012)Google Scholar
  10. 10.
    Magatti, D., Steinke, F., Bundschus, M., Tresp, V.: Combined Structured and Keyword-Based Search in Textually Enriched Entity-Relationship Graphs. In: Proceedings of the Workshop on Automated Knowledge Base Construction (2011)Google Scholar
  11. 11.
    Martinet, J., Satoh, S.: An information theoretic approach for automatic document annotation from intermodal analysis. In: Workshop on Multimodal Information Retrieval (2007)Google Scholar
  12. 12.
    Rocha, C., Schwabe, D., Aragao, M.P.: A hybrid approach for searching in the semantic web. In: WWW (2004)Google Scholar
  13. 13.
    Sabetghadam, S., Lupu, M., Rauber, A.: Astera - a generic model for multimodal information retrieval. In: Proc. of Integrating IR Technologies for Professional Search Workshop (2013)Google Scholar
  14. 14.
    Sabetghadam, S., Lupu, M., Rauber, A.: A combined approach of structured and non-structured IR in multimodal domain. In: ICMR (2014)Google Scholar
  15. 15.
    Shekarpour, S., Auer, S., Ngomo, A., Gerber, D., Hellmann, S., Stadler, C.: Keyword-driven sparql query generation leveraging background knowledge. In: Web Intelligence and Intelligent Agent Technology (WI-IAT), IEEE/WIC/ACM International Conference on Web Intelligence, vol. 1. IEEE (2011)Google Scholar
  16. 16.
    Srinivasan, S., Slaney, M.: A bipartite graph model for associating images and text. In: Workshop on Multimodal Information Retrieval (2007)Google Scholar
  17. 17.
    Tsikrika, T., Popescu, A., Kludas, J.: Overview of the wikipedia image retrieval task at imageclef 2011. In: CLEF (2011)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Serwah Sabetghadam
    • 1
  • Ralf Bierig
    • 1
  • Andreas Rauber
    • 1
  1. 1.Institute of Software Technology and Interactive SystemsVienna University of TechnologyAustria

Personalised recommendations