Optimizing Multimedia Retrieval Using Multimodal Fusion and Relevance Feedback Techniques

  • Apostolos Axenopoulos
  • Stavroula Manolopoulou
  • Petros Daras
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7131)


This paper introduces a novel approach for search and retrieval of multimedia content. The proposed framework retrieves multiple media types simultaneously, namely 3D objects, 2D images and audio files, by utilizing an appropriately modified manifold learning algorithm. The latter, which is based on Laplacian Eigenmaps, is able to map the mono-modal low-level descriptors of the different modalities into a new low-dimensional multimodal feature space. In order to accelerate search and retrieval and make the framework suitable even for large-scale applications, a new multimedia indexing scheme is adopted. The retrieval accuracy of the proposed method is further improved through relevance feedback, which enables users to refine their queries by marking the retrieved results as relevant or non-relevant. Experiments performed on a multimodal dataset demonstrate the effectiveness and efficiency of our approach. Finally, the proposed framework can be easily extended to involve as many heterogeneous modalities as possible.


Multimodal Search Multimedia Indexing Relevance Feedback 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Lai, P.L., Fyfe, C.: Canonical correlation analysis using artificial neural networks. In: Proc. European Symposium on Artificial Neural Networks, ESANN (1998)Google Scholar
  2. 2.
    Li, D., Dimitrova, N., Li, M., Sethi, I.K.: Multimedia Content Processing through Cross-Modal Association. In: Proceedings of the Eleventh ACM International Conference on Multimedia (MM 2003), USA (2003)Google Scholar
  3. 3.
    Zhang, H., Weng, J.: Measuring Multi-Modality Similarities Via Subspace Learning for Cross-Media Retrieval. In: Zhuang, Y.-T., Yang, S.-Q., Rui, Y., He, Q. (eds.) PCM 2006. LNCS, vol. 4261, pp. 979–988. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    Yang, Y., Xu, D., Nie, F., Luo, J., Zhuang, Y.: Ranking with Local Regression and Global Alignment for Cross Media Retrieval. ACM MM, Beijing, China (2009)Google Scholar
  5. 5.
    Salton, G., Buckley, C.: Improving retrieval performance by relevance feedback. Journal of the American Society for Information Science 41, 288–297 (1990)CrossRefGoogle Scholar
  6. 6.
    Schettini, R., Ciocca, G., Gagliardi, I.: Content-based color image retrieval with relevance feedback. In: International Conf. on Image Processing, Kobe, Japan (1999)Google Scholar
  7. 7.
    Zhang, H., Meng, F.: Multi-modal Correlation Modeling and Ranking for Retrieval. In: Muneesawang, P., Wu, F., Kumazawa, I., Roeksabutr, A., Liao, M., Tang, X. (eds.) PCM 2009. LNCS, vol. 5879, pp. 637–646. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  8. 8.
    He, J., Li, M., Zhang, H.J., Tong, H., Zhang, C.: Manifold-Ranking Based Image Retrieval. ACM MM, New York USA (2004)Google Scholar
  9. 9.
    Rui, Y., Huang, T.S.: Optimizing learning in image retrieval. In: IEEE Conf. Computer Vision and Pattern Recognition, South Carolina (2000)Google Scholar
  10. 10.
    Belkin, M., Niyogi, P.: Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15(6), 1373–1396 (2003)CrossRefzbMATHGoogle Scholar
  11. 11.
    Amato, G., Savino, P.: Approximate similarity search in metric spaces using inverted files. In: Proceedings of the 3rd International Conference on Scalable Information Systems (InfoScale 2008), pp. 1–10. ICST (2008)Google Scholar
  12. 12.
    Vanamali, T.P., Godil, A., Dutagaci, H., Furuya, T., Lian, Z., Ohbuchi, R.: SHREC 2010 Track: Generic 3D Warehouse. In: Proceedings of the Eurographics/ACM SIGGRAPH Symposium on 3D Object Retrieval (2010)Google Scholar
  13. 13.
    Saul, L.K., Roweis, S.T.: Think Globally, Fit Locally: Unsupervised Learning of Low Dimensional Manifolds. Journal of Machine Learning Research (2003)Google Scholar
  14. 14.
    Vranic, D.: 3d model retrieval. Ph.D. Dissertation, University of Leipzig (2004)Google Scholar
  15. 15.
    Daras, P., Axenopoulos, A.: A 3D Shape Retrieval Framework Supporting Multimodal Queries. International Journal of Computer Vision (July 2009), doi:10.1007/s11263-009-0277-2Google Scholar
  16. 16.
    Wichern, Xue, Thornburg, Mechteley, Spanias: Segmentation, Indexing, and Retrieval for Environmental and Natural Sounds. IEEE Transactions on Audio, Speech and Language Processing (March 2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Apostolos Axenopoulos
    • 1
  • Stavroula Manolopoulou
    • 1
  • Petros Daras
    • 1
  1. 1.Centre for Research and Technology HellasInformatics and Telematics InstituteThessalonikiGreece

Personalised recommendations