Understanding Multimedia Document Semantics for Cross-Media Retrieval
Multimedia Document (MMD) such as Web Page and Multimedia cyclopedias is composed of media objects of different modalities, and its integrated semantics is always expressed by the combination of all media objects in it. Since the contents in MMDs are enormous and the amount of them is increasing rapidly, effective management of MMDs is in great demand. Meanwhile, it is meaningful to provide users cross-media retrieval facilities so that users can query media objects by examples of different modalities, e.g. users may query an MMD (or an image) by submitting a audio clip and vice versa. However, there exist two challenges to achieve the above goals. First, how can we represent an MMD and fuse media objects together to achieve Cross-index and facilitate Cross-media retrieval? Second, how can we understand MMD semantics? Taking into account of the two problems, we give the definition of MMD and propose a manifold learning method to discover MMD semantics in this paper. We first construct an MMD semi-semantic graph (SSG) and then adopt Multidimensional scaling to create an MMD semantic space (MMDSS). We also propose two periods’ feedbacks. The first one is used to refine SSG and the second one is adopted to introduce new MMD that is not in the MMDSS into MMDSS. Since all of the MMDs and their component media objects of different modalities lie in MMDSS, cross-media retrieval can be easily performed. Experiment results are encouraging and indicate that the performance of the proposed approach is effective.
KeywordsCross-media Retrieval Multimedia Document Manifold
Unable to display preview. Download preview PDF.
- 1.Zhang, H.J., Zhong, D.: Schema for visual feature based image retrieval [A]. In: Proceedings of Storage and Retrieval for Image and Video Database, USA, pp. 36–46 (1995)Google Scholar
- 3.Chang, E., Goh, K., Sychay, G., Wu, G.: CBSA: Content-Based Soft Annotation for Multimodal Image Retrieval Using Bayes Point Machine. IEEE Trans on Circuits and Systems for Video Technology 13(1) (January 2003)Google Scholar
- 4.He, X., Ma, W.Y., Zhang, H.J.: Learning an Image Manifold for Retrieval. In: ACM Multimedia Conference, New York (2004)Google Scholar
- 5.Maddage, N.C., Xu., C., Kankanhalli, M.S., Shao, X.: Content-based Music Structure Analysis with Applications to Music Semantics Understanding. In: ACM Multimedia Conference, New York (2004)Google Scholar
- 8.Smoliar, S.W., Zhang, H.: Content based video indexing and retrieval. Multimedia, IEEE 1(2), 62–72 (Summer 1994)Google Scholar
- 10.Wu, M.Y., Chiu, C.Y., Chao, S., Yang, S., Lin, H.C.: Content-Based Retrieval for Human Motion Data. In: 16th IPPR Conference on Computer Vision, Graphics and Image Processing CVGIP 2003 (2003)Google Scholar
- 11.Müller, M., Röder, T., Clausen, M.: Efficient Content-Based Retrieval of Motion Capture Data. Proceedings of ACM SIGGRAPH (2005)Google Scholar
- 13.Beyer, K., Goldstein, J., Ramakrishnan, R., Shaft, U.: When is nearest neighbor meaningful? In: International Conference on Database Theory, pp. 217–235 (1999)Google Scholar
- 14.Yang, J., Zhuang, Y.T., Li, Q.: Search for multi-modality data in digital libraries. In: Proceedings of 2nd IEEE Pacific-rim Conference on Multimedia, Beijing, China, pp. 482–489 (2001)Google Scholar
- 15.Seung, H.S., Lee, D.: The manifold ways of perception. Science 290 (December 22, 2000)Google Scholar
- 16.Tenenbaum, J.B., Silva, V.D., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290 (December 22, 2000)Google Scholar
- 17.Kruskal, J.B., Wish, M.: Multidimensional Scaling. Sage Publications, Beverly Hills (1977)Google Scholar