Using Evidences Based on Natural Language to Drive the Process of Fusing Multimodal Sources

  • Sergio Navarro
  • Fernando Llopis
  • Rafael Muñoz
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5723)


This paper focuses on the proposal and evaluation of two multimodal fusion techniques in the field of Visual Information Retrieval (VIR). These proposals are based on two widely used fusion strategies in the VIR area, the multimodal blind relevance feedback and the multimodal re-ranking strategy. Unlike the existent techniques, our alternative proposals are guided by the evidence found in the natural language annotations related to the images. The results achieved by our runs in two different ImageCLEF tasks, 3rd place in the Wikipedia task [1] and 4th place within all the automatic runs in the photo task [2], jointly with the results obtained in later experiments presented in this paper show us that the use of conceptual information associated with an image can improve significantly the performance of the original multimodal fusion techniques used.


Query Term Query Expansion Image Annotation CBIR System Photo Task 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Navarro, S., Muñoz, R., Llopis, F.: A Textual Approach based on Passages Using IR-n in WikipediaMM Task 2008. In: Online Working Notes, CLEF 2008 (2008)Google Scholar
  2. 2.
    Navarro, S., Llopis, F., Muñoz, R.: Different Multimodal Approaches using IR-n in ImageCLEFphoto 2008. In: Online Working Notes, CLEF 2008 (2008)Google Scholar
  3. 3.
    Grubinger, M.: Analysis and Evaluation of Visual Information Systems Performance. PhD thesis, Engineering and Science Victoria University (2007)Google Scholar
  4. 4.
    Díaz-Galiano, M., García-Cumbreras, M., Martín-Valdivia, M., Montejo-Raez, A., Urea-López, L.: Sinai at imageclef 2007. In: Working Notes of the 2007 CLEF Workshop, Budapest, Hungary (September 2007)Google Scholar
  5. 5.
    Gao, S., Chevallet, J.P., Le, T.H.D., Pham, T.T., Lim, J.H.: Ipal at imageclef 2007 mixing features, models and knowledge. In: Working Notes of the 2007 CLEF Workshop, Budapest, Hungary (September 2007)Google Scholar
  6. 6.
    Robertson, S.E., Sparck Jones, K.: Relevance weighting of search terms. Journal of the American Society for Information Science 27(3), 129–146 (1977)CrossRefGoogle Scholar
  7. 7.
    Xu, J., Croft, W.B.: Improving the effectiveness of information retrieval with local context analysis. ACM Trans. Inf. Syst. 18(1), 79–112 (2000)CrossRefGoogle Scholar
  8. 8.
    Llopis, F., Vicedo, J.L., Ferrández, A.: IR-n System at CLEF-2002. In: Peters, C., Braschler, M., Gonzalo, J. (eds.) CLEF 2002. LNCS, vol. 2785, pp. 291–300. Springer, Heidelberg (2003)Google Scholar
  9. 9.
    Navarro, S., Muñoz, R., Llopis, F.: A Multimodal Approach to the Medical Retrieval Task using IR-n. In: Online Working Notes, CLEF 2008 (2008)Google Scholar
  10. 10.
    Liu, H., Junzhong Gu, Z.L.: Improving the Effectiveness of Local Context Analysis Based on Semantic Similarity. In: 2007 International Conference on Convergence Information Technology, ICCIT 2007 (2007)Google Scholar
  11. 11.
    Reid, N.: The photographic collections in st andrews university library. Scottish Archives 5, 83–90 (1999)Google Scholar
  12. 12.
    Clough, P., Grubinger, M., Deselaers, T., Hanbury, A., Müller, H.: Overview of the imageclef 2006 photographic retrieval and object annotation tasks. In: Peters, C., Clough, P., Gey, F.C., Karlgren, J., Magnini, B., Oard, D.W., de Rijke, M., Stempfhuber, M. (eds.) CLEF 2006. LNCS, vol. 4730, pp. 579–594. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  13. 13.
    Grubinger, M., Clough, P., Hanbury, A., Müller, H.: Overview of the ImageCLEFphoto 2007 photographic retrieval task. In: Peters, C., Jijkoun, V., Mandl, T., Müller, H., Oard, D.W., Peñas, A., Petras, V., Santos, D. (eds.) CLEF 2007. LNCS, vol. 5152, pp. 433–444. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  14. 14.
    Tsikrika, T., Kludas, J.: Overview of the wikipediaMM task at ImageCLEF 2008. In: Peters, C., Giampiccol, D., Ferro, N., Petras, V., Gonzalo, J., Peñas, A., Deselaers, T., Mandl, T., Jones, G., Kurimo, M. (eds.) Evaluating Systems for Multilingual and Multimodal Information Access. LNCS, vol. 5706, pp. 539–550. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  15. 15.
    Amati, G., Van Rijsbergen, C.J.: Probabilistic Models of information retrieval based on measuring the divergence from randomness. ACM TOIS 20(4), 357–389 (2002)CrossRefGoogle Scholar
  16. 16.
    Deselaers, T., Keysers, D., Ney, H.: Features for image retrieval: An experimental comparison. Information Retrieval 11(2), 77–107 (2008)CrossRefGoogle Scholar
  17. 17.
    Demerdash, O.E., Kosseim, L., Bergler, S.: CLaC at ImageCLEFPhoto 2008. In: Online Working Notes, CLEF 2008 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Sergio Navarro
    • 1
  • Fernando Llopis
    • 1
  • Rafael Muñoz
    • 1
  1. 1.Natural Language Processing and Information Systems GroupUniversity of AlicanteSpain

Personalised recommendations