Information Fusion in Multimedia Information Retrieval

  • Jana Kludas
  • Eric Bruno
  • Stéphane Marchand-Maillet
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4918)


In retrieval, indexing and classification of multimedia data an efficient information fusion of the different modalities is essential for the system’s overall performance. Since information fusion, its influence factors and performance improvement boundaries have been lively discussed in the last years in different research communities, we will review their latest findings. They most importantly point out that exploiting the feature’s and modality’s dependencies will yield to maximal performance. In data analysis and fusion tests with annotated image collections this is undermined.


Support Vector Machine Information Fusion Fusion Result Decision Fusion Rank Aggregation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Benitez, A.B., Chang, S.F.: Multimedia knowledge integration, summarization and evaluation. In: Workshop on Multimedia Data Mining, pp. 23–26 (2002)Google Scholar
  2. 2.
    Ross, A., Jain, A.K.: Multimodal biometrics: An overview. In: EUSIPCO Proc. of 12th European Signal Processing Conference (EUSIPCO), pp. 1221–1224 (2004)Google Scholar
  3. 3.
    Fassinut-Mombot, B., Choquel, J.B.: A new probabilistic and entropy fusion approach for management of information sources. Information Fusion 5, 35–47 (2004)CrossRefGoogle Scholar
  4. 4.
    Vogt, C.C., Cottrell, G.W.: Fusion via a linear combination of scores. Information Retrieval 1(3), 151–173 (1999)CrossRefGoogle Scholar
  5. 5.
    Bruno, E., Moenne-Loccoz, N., Marchand-Maillet, S.: Design of multimodal dissimilarity spaces for retrieval of multimedia documents. IEEE Transaction on Pattern Analysis and Machine Intelligence (to appear, 2008)Google Scholar
  6. 6.
    Brown, G., Yao, X.: On the effectiveness of negative correlation learning. In: First UK Workshop on Computational Intelligence (UKCI 2001) (2001)Google Scholar
  7. 7.
    Chechik, G., Tishby, N.: Extracting relevant structures with side information. In: Advances in Neural Information Processing Systems, vol. 15 (2003)Google Scholar
  8. 8.
    Taylor, G., Kleeman, L.: Fusion of multimodal visual cues for model-based object tracking. In: Australasian Conference on Robotics and Automation (2003)Google Scholar
  9. 9.
    Barnard, K., Johnson, M.: Word sense disambiguation with pictures. Artificial Intelligence 167, 13–30 (2005)CrossRefGoogle Scholar
  10. 10.
    Tumer, K., Gosh, J.: Linear order statistics combiners for pattern classification. Combining Artificial Neural Networks, 127–162 (1999)Google Scholar
  11. 11.
    Valet, L., Bolon, P., Mauris, G.: A statistical overview of recent literature in information fusion. In: Proceedings of the Third International Conference on Information Fusion, vol. 1, pp. MOC3/22 – MOC3/29 (2000)Google Scholar
  12. 12.
    Wu, L., Cohen, P.R., Oviatt, S.L.: From members to team to committee - a robust approach to gestural and multimodal recognition. Transactions on Neural Networks 13 (2002)Google Scholar
  13. 13.
    Llinas, J., Bowman, C., Rogova, G., Steinberg, A., Waltz, E., White, F.: Revisiting the jdl data fusion model II. Information Fusion, 1218–1230 (2004)Google Scholar
  14. 14.
    Kokar, M.M., Weyman, J., Tomasik, J.A.: Formalizing classes of information fusion systems. Information Fusion 5, 189–202 (2004)CrossRefGoogle Scholar
  15. 15.
    Poh, N., Bengio, S.: How do correlation and variance of base-experts affect fusion in biometric authentication tasks? IEEE Transactions on Acoustics, Speech, and Signal Processing 53, 4384–4396 (2005)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Ueda, N., Nakano, R.: Generalization error of ensemble estimators. IEEE International Conference on Neural Networks 1, 90–95 (1996)Google Scholar
  17. 17.
    Koval, O., Pun, T., Voloshynovskiy, S.: Error exponent analysis of person identification based on fusion of dependent/independent modalities. In: Proceedings of SPIE-IS&T Electronic Imaging 2007, Security, Steganography, and Watermarking of Multimedia Contents IX (2007)Google Scholar
  18. 18.
    Aarabi, P., Dasarathy, B.V.: Robust speech processing using multi-sensor, multi-source information fusion - an overview of the state of the art. Information Fusion 5, 77–80 (2004)CrossRefGoogle Scholar
  19. 19.
    Zhao, R., Grosky, W.I.: Narrowing the semantic gap - improved text-based web document retrieval using visual features. IEEE Transactions on Multimedia 4(2), 189–200 (2002)CrossRefGoogle Scholar
  20. 20.
    Rosen, B.E.: Ensemble learning using decorrelated neural networks. Connections Science 8, 373–384 (1996)CrossRefGoogle Scholar
  21. 21.
    Dass, S.C., Jain, A.K., Nandakumar, K.: A principled approach to score level fusion in multimodal biometric systems. In: Proceedings of Audio- and Video-based Biometric Person Authentication (AVBPA), pp. 1049–1058 (2005)Google Scholar
  22. 22.
    Beitzel, S.M., Chowdury, A., Jensen, E.C.: Disproving the fusion hypothesis: An analysis of data fusion via effective information retrieval strategies. In: ACM symposium on Applied computing, pp. 823–827 (2003)Google Scholar
  23. 23.
    Wu, S., McClean, S.: Performance prediction of data fusion for information retrieval. Information Processing and Management 42, 899–915 (2006)CrossRefGoogle Scholar
  24. 24.
    Squire, D.M., Müller, W., Müller, H., Raki, J.: Content-based query of image databases, inspirations from text retrieval: inverted files, frequency-based weights and relevance feedback. Pattern Recognition Letters (Selected Papers from The 11th Scandinavian Conference on Image Analysis SCIA 1999) 21(13-14), 1193–1198 (2000)zbMATHGoogle Scholar
  25. 25.
    Joachims, T., Shawe-Taylor, J., Cristianini, N.: Composite kernels for hypertext categorization, pp. 250–257. Morgan Kaufmann, San Francisco (2001)Google Scholar
  26. 26.
    Kolenda, T., Winther, O., Hansen, L.K., Larsen, J.: Independent component analysis for understanding multimedia content. Neural Networks for Signal Processing, 757–766 (2002)Google Scholar
  27. 27.
    Westerveld, T., de Vries, A.P.: Multimedia retrieval using multiple examples. In: Enser, P.G.B., Kompatsiaris, Y., O’Connor, N.E., Smeaton, A.F., Smeulders, A.W.M. (eds.) CIVR 2004. LNCS, vol. 3115, Springer, Heidelberg (2004)CrossRefGoogle Scholar
  28. 28.
    Wu, Y., Chen-Chuan Chang, K., Chang, E.Y., Smith, J.R.: Optimal multimodal fusion for multimedia data analysis. In: MULTIMEDIA 2004: Proceedings of the 12th annual ACM international conference on Multimedia, pp. 572–579. ACM Press, New York (2004)CrossRefGoogle Scholar
  29. 29.
    Yan, R., Hauptmann, A.G.: The combination limit in multimedia retrieval. In: MULTIMEDIA 2003: Proceedings of the eleventh ACM international conference on Multimedia, pp. 339–342. ACM Press, New York (2003)CrossRefGoogle Scholar
  30. 30.
    Li, C., Biswas, G.: Unsupervised clustering with mixed numeric and nominal data - a new similarity based agglomerative system. In: International Workshop on AI and Statistics, pp. 327–346 (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Jana Kludas
    • 1
  • Eric Bruno
    • 1
  • Stéphane Marchand-Maillet
    • 1
  1. 1.University of GenevaSwitzerland

Personalised recommendations