Understanding Multimedia Document Semantics for Cross-Media Retrieval

  • Fei Wu
  • Yi Yang
  • Yueting Zhuang
  • Yunhe Pan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3767)


Multimedia Document (MMD) such as Web Page and Multimedia cyclopedias is composed of media objects of different modalities, and its integrated semantics is always expressed by the combination of all media objects in it. Since the contents in MMDs are enormous and the amount of them is increasing rapidly, effective management of MMDs is in great demand. Meanwhile, it is meaningful to provide users cross-media retrieval facilities so that users can query media objects by examples of different modalities, e.g. users may query an MMD (or an image) by submitting a audio clip and vice versa. However, there exist two challenges to achieve the above goals. First, how can we represent an MMD and fuse media objects together to achieve Cross-index and facilitate Cross-media retrieval? Second, how can we understand MMD semantics? Taking into account of the two problems, we give the definition of MMD and propose a manifold learning method to discover MMD semantics in this paper. We first construct an MMD semi-semantic graph (SSG) and then adopt Multidimensional scaling to create an MMD semantic space (MMDSS). We also propose two periods’ feedbacks. The first one is used to refine SSG and the second one is adopted to introduce new MMD that is not in the MMDSS into MMDSS. Since all of the MMDs and their component media objects of different modalities lie in MMDSS, cross-media retrieval can be easily performed. Experiment results are encouraging and indicate that the performance of the proposed approach is effective.


Cross-media Retrieval Multimedia Document Manifold 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Zhang, H.J., Zhong, D.: Schema for visual feature based image retrieval [A]. In: Proceedings of Storage and Retrieval for Image and Video Database, USA, pp. 36–46 (1995)Google Scholar
  2. 2.
    Wang, J.Z., Wiederhold, G., Firschein, O., Wei, S.X.: Content-based image indexing and searching using Daubechies’ wavelets. International Journal on Digital Libaries 1, 311–328 (1997)CrossRefGoogle Scholar
  3. 3.
    Chang, E., Goh, K., Sychay, G., Wu, G.: CBSA: Content-Based Soft Annotation for Multimodal Image Retrieval Using Bayes Point Machine. IEEE Trans on Circuits and Systems for Video Technology 13(1) (January 2003)Google Scholar
  4. 4.
    He, X., Ma, W.Y., Zhang, H.J.: Learning an Image Manifold for Retrieval. In: ACM Multimedia Conference, New York (2004)Google Scholar
  5. 5.
    Maddage, N.C., Xu., C., Kankanhalli, M.S., Shao, X.: Content-based Music Structure Analysis with Applications to Music Semantics Understanding. In: ACM Multimedia Conference, New York (2004)Google Scholar
  6. 6.
    Guo, G., Li, S.Z.: Content-based audio classification and retrieval by support vector machines. IEEE Transactions on Neural Networks 14(1), 209–215 (2003)CrossRefGoogle Scholar
  7. 7.
    Wold, E., Blum, T., Keislar, D., Wheaton, J.: Content-based classification,search and retrieval of audio. IEEE Multimedia Mag. 3, 27–36 (1996)CrossRefGoogle Scholar
  8. 8.
    Smoliar, S.W., Zhang, H.: Content based video indexing and retrieval. Multimedia, IEEE 1(2), 62–72 (Summer 1994)Google Scholar
  9. 9.
    Fan, J., Elmagarmid, A.K., Zhu, X., Aref, W.G., Wu, L.: ClassView: hierarchical video shot classification, indexing, and accessing. IEEE Transactions on Multimedia 6(1), 70–86 (2004)CrossRefGoogle Scholar
  10. 10.
    Wu, M.Y., Chiu, C.Y., Chao, S., Yang, S., Lin, H.C.: Content-Based Retrieval for Human Motion Data. In: 16th IPPR Conference on Computer Vision, Graphics and Image Processing CVGIP 2003 (2003)Google Scholar
  11. 11.
    Müller, M., Röder, T., Clausen, M.: Efficient Content-Based Retrieval of Motion Capture Data. Proceedings of ACM SIGGRAPH (2005)Google Scholar
  12. 12.
    Wang, Z., Liu, J.: Multimedia content analysis using audio and visual information [J]. IEEE Signal Processing Magazine 17(6), 12–36 (2000)CrossRefGoogle Scholar
  13. 13.
    Beyer, K., Goldstein, J., Ramakrishnan, R., Shaft, U.: When is nearest neighbor meaningful? In: International Conference on Database Theory, pp. 217–235 (1999)Google Scholar
  14. 14.
    Yang, J., Zhuang, Y.T., Li, Q.: Search for multi-modality data in digital libraries. In: Proceedings of 2nd IEEE Pacific-rim Conference on Multimedia, Beijing, China, pp. 482–489 (2001)Google Scholar
  15. 15.
    Seung, H.S., Lee, D.: The manifold ways of perception. Science 290 (December 22, 2000)Google Scholar
  16. 16.
    Tenenbaum, J.B., Silva, V.D., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290 (December 22, 2000)Google Scholar
  17. 17.
    Kruskal, J.B., Wish, M.: Multidimensional Scaling. Sage Publications, Beverly Hills (1977)Google Scholar
  18. 18.
    Zhuang, Y., Wu, C., Wu, F., Liu, X.: Improving Web-based Learning: Automatic Annotation of Multimedia Semantics and Cross-Media Indexing. In: Liu, W., Shi, Y., Li, Q. (eds.) ICWL 2004. LNCS, vol. 3143, pp. 255–262. Springer, Heidelberg (2004)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2005

Authors and Affiliations

  • Fei Wu
    • 1
  • Yi Yang
    • 1
  • Yueting Zhuang
    • 1
  • Yunhe Pan
    • 1
  1. 1.College of Computer Science and EngineeringZhejiang UniversityHangzhouP.R. China

Personalised recommendations