Information Fusion for Combining Visual and Textual Image Retrieval in ImageCLEF@ICPR

  • Xin Zhou
  • Adrien Depeursinge
  • Henning Müller
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6388)


In the ImageCLEF image retrieval competition multimodal image retrieval has been evaluated over the past seven years. For ICPR 2010 a contest was organized for the fusion of visual and textual retrieval as this was one task where most participants had problems. In this paper, classical approaches such as the maximum combinations (combMAX), the sum combinations (combSUM) and the multiplication of the sum and the number of non–zero scores (combMNZ) were employed and the trade–off between two fusion effects (chorus and dark horse effects) was studied based on the sum of n maxima. Various normalization strategies were tried out. The fusion algorithms are evaluated using the best four visual and textual runs of the ImageCLEF medical image retrieval task 2008 and 2009. The results show that fused runs outperform the best original runs and multi–modality fusion statistically outperforms single modality fusion. The logarithmic rank penalization shows to be the most stable normalization. The dark horse effect is in competition with the chorus effect and each of them can produce best fusion performance depending on the nature of the input data.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Dietterich, T.G.: Ensemble methods in machine learning. In: Kittler, J., Roli, F. (eds.) MCS 2000. LNCS, vol. 1857, pp. 1–15. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  2. 2.
    Müller, H., Michoux, N., Bandon, D., Geissbuhler, A.: A review of content-based image retrieval systems in medicine–clinical benefits and future directions. International Journal of Medical Informatics 73(1), 1–23 (2004)CrossRefGoogle Scholar
  3. 3.
    Hersh, W., Müller, H., Kalpathy-Cramer, J., Kim, E., Zhou, X.: The consolidated ImageCLEFmed medical image retrieval task test collection. Journal of Digital Imaging 22(6), 648–655 (2009)CrossRefGoogle Scholar
  4. 4.
    Müller, H., Kalpathy–Cramer, J., Eggel, I., Bedrick, S., Radhouani, S., Bakke, B., Kahn Jr., C.E., Hersh, W.: Overview of the CLEF 2009 medical image retrieval track. In: Peters, C., Caputo, B., Gonzalo, J., Jones, G.J.F., Kalpathy-Cramer, J., Müller, H., Tsikrika, T. (eds.) CLEF 2009. LNCS, vol. 6242, pp. 72–84. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Müller, H., Kalpathy-Cramer, J.: The ImageCLEF medical retrieval task at icpr 2010 — information fusion to combine viusal and textual information. In: Ünay, D., Çataltepe, Z., Aksoy, S. (eds.) ICPR 2010. LNCS, vol. 6388, pp. 101–110. Springer, Heidelberg (2010)Google Scholar
  6. 6.
    Wu, S., Mcclean, S.: Performance prediction of data fusion for information retrieval. Information Processing & Management 42(4), 899–915 (2006)CrossRefGoogle Scholar
  7. 7.
    Valet, L., Mauris, G., Bolon, P.: A statistical overview of recent literature in information fusion. IEEE Aerospace and Electronic Systems Magazine 16(3), 7–14 (2001)CrossRefGoogle Scholar
  8. 8.
    Croft, W.B.: Combining approaches to information retrieval. In: Advances in Information Retrieval, pp. 1–36. Springer US, Heidelberg (2000)Google Scholar
  9. 9.
    Kludas, J., Bruno, E., Marchand-Maillet, S.: Information fusion in multimedia information retrieval. In: Proceedings of 5th International Workshop on Adaptive Multimedia Retrieval (AMR), vol. 4918, pp. 147–159. ACM, New York (June 2008)Google Scholar
  10. 10.
    Dwork, C., Kumar, R., Naor, M., Sivakumar, D.: Rank aggregation methods for the web. In: WWW 2001: Proceedings of the 10th International Conference on World Wide Web, New York, NY, USA, pp. 613–622 (2001)Google Scholar
  11. 11.
    Renda, E.M., Straccia, U.: Web metasearch: rank vs. score based rank aggregation methods. In: SAC 2003: Proceedings of the 2003 ACM Symposium on Applied Computing, pp. 841–846. ACM Press, New York (2003)Google Scholar
  12. 12.
    Fox, E.A., Shaw, J.A.: Combination of multiple searches. In: Text REtrieval Conference, pp. 243–252 (1993)Google Scholar
  13. 13.
    Aslam, J.A., Montague, M.: Models for metasearch. In: SIGIR 2001: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 276–284. ACM, New York (2001)Google Scholar
  14. 14.
    Montague, M., Aslam, J.A.: Condorcet fusion for improved retrieval. In: CIKM 2002: Proceedings of the Eleventh International Conference on Information and Knowledge Management, pp. 538–548. ACM, New York (2002)Google Scholar
  15. 15.
    Lillis, D., Toolan, F., Collier, R., Dunnion, J.: Probfuse: a probabilistic approach to data fusion. In: SIGIR 2006: Proceedings of the 29th ACM SIGIR Conference on Research and Development in Information Retrieval, New York, USA, pp. 139–146 (2006)Google Scholar
  16. 16.
    Vogt, C.C., Cottrell, G.W.: Fusion via a linear combination of scores. Information Retrieval 1(3), 151–173 (1999)CrossRefGoogle Scholar
  17. 17.
    Lee, J.H.: Analyses of multiple evidence combination. In: SIGIR 1997: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 267–276. ACM, New York (1997)Google Scholar
  18. 18.
    Wu, S., Crestani, F., Bi, Y.: Evaluating score normalization methods in data fusion. In: Information Retrieval Technology, AIRS 2006, pp. 642–648 (2006)Google Scholar
  19. 19.
    Wu, S., Bi, Y., Zeng, X., Han, L.: Assigning appropriate weights for the linear combination data fusion method in information retrieval. Information Processing & Management 45(4), 413–426 (2009)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Xin Zhou
    • 1
  • Adrien Depeursinge
    • 1
    • 2
  • Henning Müller
    • 1
    • 2
  1. 1.Geneva University Hospitals and University of GenevaSwitzerland
  2. 2.University of Applied Sciences Western SwitzerlandSierreSwitzerland

Personalised recommendations