The Visual Computer

, Volume 27, Issue 1, pp 21–34 | Cite as

Graph-based multi-space semantic correlation propagation for video retrieval

  • Bailan Feng
  • Juan Cao
  • Xiuguo Bao
  • Lei Bao
  • Yongdong Zhang
  • Shouxun Lin
  • Xiaochun Yun
Original Article

Abstract

By introducing the concept detection results to the retrieval process, concept-based video retrieval (CBVR) has been successfully used for semantic content-based video retrieval application. However, how to select and fuse the appropriate concepts for a specific query is still an important but difficult issue. In this paper, we propose a novel and effective concept selection method, named graph-based multi-space semantic correlation propagation (GMSSCP), to explore the relationship between the user query and concepts for video retrieval application. Compared with traditional methods, GMSSCP makes use of a manifold-ranking algorithm to collectively explore the multi-layered relationships between the query and concepts, and the expansion result is more robust to noises. Parallel to this, GMSSCP has a query-adapting property, which can enhance the process of concept correlation propagation and selection with strong pertinence of query cues. Furthermore, it can dynamically update the unified propagation graph by flexibly introducing the multi-modal query cues as additional nodes, and is not only effective for automatic retrieval but also appropriate for the interactive case. Encouraging experimental results on TRECVID datasets demonstrate the effectiveness of GMSSCP over the state-of-the-art concept selection methods. Moreover, we also apply it to the interactive retrieval system—VideoMap and gain an excellent performance and user experience.

Keywords

Concept-based video retrieval Concept selection and fusion Multi-space integration and propagation Manifold ranking 

References

  1. 1.
    Astam, J.A., Yilmaz, E.: Inferring document relevance via average precision. In: Proc. ACM Int. Conf. on Research and Development in Information Retrieval, pp. 601–602, New York, USA (2006) Google Scholar
  2. 2.
    Cao, J., Zhang, Y.D., Feng, B.L., Hua, X.F., Bao, L., Zhang, X.: MCG-ICT-CAS TRECVID2008 search task report. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008) Google Scholar
  3. 3.
    Cao, J., Jing, H.F., Ngo, C.W., Zhang, Y.D.: Distribution-based concept selection for concept-based video retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 19–24, Beijing, China (2009) Google Scholar
  4. 4.
    Cao, J., Zhang, Y.D., Guo, J.B., Bao, L., Li, J.T.: VideoMap: An interactive video retrieval system of MCG-ICT-CAS. In: Proc. ACM Int. Conf. on Image and Video Retrieval, Santorini, Greece (2009) Google Scholar
  5. 5.
    Fellbaum, C., et al.: WordNet: An Electronic Lexical Database. MIT Press, Cambridge (1998) MATHGoogle Scholar
  6. 6.
    Haubold, A., Natsev, A., Naphade, M.: Semantic multimedia retrieval using lexical query expansion and model-based reranking. In: Proc. IEEE Int. Conf. on Multimedia and Expo, pp. 1761–1764, Toronto, Canada (2006) Google Scholar
  7. 7.
    Hauptmann, A., Rong, Y., Lin, W.H., Christel, M., Wactlar, H.: Can high-level concepts fill the semantic gap in video retrieval? A case study with broadcast news. IEEE Trans. Multimedia 9(5), 958–966 (2007) CrossRefGoogle Scholar
  8. 8.
    He, J.R., Li, M.J., Zhang, H.J., Tong, H.H., Zhang, C.S.: Manifold-ranking based image retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 9–16, New York, USA (2004) Google Scholar
  9. 9.
    Jiang, Y.G., Ngo, C.W., Yang, J.: Towards optimal bag-of-features for object categorization and semantic video retrieval. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 494–501, Amsterdam, the Netherlands (2007) Google Scholar
  10. 10.
    Li, X., Wang, D., Li, J., Zhang, B.: Video search in concept subspace: A text-like paradigm. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 603–610, Amsterdam, the Netherlands (2007) Google Scholar
  11. 11.
    Naphade, M.R., Kennedy, L., Kender, J.R., Chang, S.F., Smith, J.R., Over, P., Hauptmann, A.: A light scale concept ontology for multimedia understanding for TRECVID2005. IBM Research Technical Report (2005) Google Scholar
  12. 12.
    Naphade, M.R., Smith, J.R., Tesic, J., Chang, S.F., Hsu, W., Kennedy, L., Hauptmann, A., Curtis, J.: Large-scale concept ontology for multimedia. In: Proc. ACM Int. Conf. on Multimedia, pp. 86–91, Santa Barbara, CA (2006) Google Scholar
  13. 13.
    Natsev, A.P., Haubold, A., Tesic, J., Xie, L., Yan, R.: Semantic concept based query expansion and re-ranking for multimedia retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 991–1000, Augsburg, Germany (2007) Google Scholar
  14. 14.
    Neo, S.Y., Zhao, J., Kan, M.Y., Chua, T.S.: Video retrieval using high level features: Exploiting query matching and confidence-based weighting. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 143–152, Heidelberg, Germany (2006) Google Scholar
  15. 15.
    Ngo, C.W., Zhang, H.J., Pone, T.C.: Recent advances in content based video analysis. J. Image Graph. 1(3), 445–468 (2001) CrossRefGoogle Scholar
  16. 16.
    Ngo, C.W., Jiang, Y.G., Wei, X.Y., Zhao, W.L., Wang, F., Wu, X., Tan, H.K.: Beyond semantic search: What you observe may not be what you think. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008) Google Scholar
  17. 17.
    Over, P., Awad, G., Rose, T., Fiscus, J., Kraaij, W., Semeaton, A.F.: TRECVID 2008-goals, tasks, data, evaluation mechanisms and metrics. In: Proc. TRECVID Workshop, Gaithersburg, MD (2008) Google Scholar
  18. 18.
    Qi, G.J., Hua, X.S., Rui, Y., Tang, J.H., Mei, T., Zhang, H.J.: Correlative multi-label video annotation. In: Proc. ACM Int. Conf. on Multimedia, pp. 17–26, Augsburg, Germany (2007) Google Scholar
  19. 19.
    Rasiwasia, N., Moreno, P.J., Vasconcelos, N.: Bridging the gap: Query by semantic example. IEEE Trans. Multimedia 9(5), 923–938 (2007) CrossRefGoogle Scholar
  20. 20.
    Smeaton, A.F., Over, P., Kraaij, W.: Evaluation campaigns and trecvid. In: Proc. ACM Int. Workshop on Multimedia Information Retrieval, pp. 321–330, New York, USA (2006) Google Scholar
  21. 21.
    Snoek, C.G.M., Worring, M.: Concept-based video retrieval. Found. Trends Inf. Retr. 4(2), 215–322 (2009) Google Scholar
  22. 22.
    Snoek, C.G.M., Worring, M., Gemert, J.C.V., Geusebroek, J.M., Smeulders, A.W.M.: The challenge problem for automated detection of 101 semantic concepts in multimedia. In: Proc. ACM Int. Conf. on Multimedia, pp. 421–430, Santa Barbara, CA (2006) Google Scholar
  23. 23.
    Snoek, C.G.M., Huurnink, B., Hollink, L., Rijke, M.D., Schreiber, G., Worring, M.: Adding semantics to detectors for video retrieval. IEEE Trans. Multimedia 9(5), 975–986 (2007) CrossRefGoogle Scholar
  24. 24.
    Snoek, C.G.M., Worring, M., Rooij, O.D., Sande, K.E.A.V., Yan, R., Hauptmann, A.: VideOlympics: Real-time evaluation of multimedia retrieval systems. IEEE Multimedia 15(1), 86–91 (2008) CrossRefGoogle Scholar
  25. 25.
    Tang, J.H., Hua, X.S., Qi, G.J., Wang, M., Mei, T., Wu, X.Q.: Structure-sensitive manifold ranking for video concept detection. In: Proc. ACM Int. Conf. on Multimedia, pp. 852–861, Augsburg, Germany (2007) Google Scholar
  26. 26.
    Tang, J., Hua, X.S., Wang, M., Gu, Z., Qi, G.J., Wu, X.: Correlative linear neighborhood propagation for video annotation. IEEE Trans. Syst. Man Cybern., B Cybern. 39(2), 409–416 (2009) CrossRefGoogle Scholar
  27. 27.
    Tang, J.H., Yan, S.C., Hong, R.C., Qi, G.J., Chua, T.S.: Inferring semantic concepts from community-contributed images and noisy tags. In: Proc. ACM Int. Conf. on Multimedia, pp. 223–232, Beijing, China (2009) Google Scholar
  28. 28.
    Tesic, J., Natsev, A., Smith, J.R.: Cluster-based data modeling for semantic video search. In: Proc. ACM Int. Conf. on Image and Video Retrieval, pp. 595–602, Amsterdam, the Netherlands (2007) Google Scholar
  29. 29.
    Wan, H.L., Morshed, U., Hu, H., et al.: Texture feature and its application in CBIR. J. Comput.-Aided Des. Comput. Graph. 15(2), 195–199 (2003) Google Scholar
  30. 30.
    Wang, D., Li, X., Li, J., Zhang, B.: The importance of query-concept-mapping for automatic video retrieval. In: Proc. ACM Int. Conf. on Multimedia, pp. 285–288, Augsburg, Germany (2007) Google Scholar
  31. 31.
    Wei, X.Y., Ngo, C.W.: Ontology-enriched semantic space for video search. In: Proc. ACM Int. Conf. on Multimedia, pp. 981–990, Augsburg, Germany (2007) Google Scholar
  32. 32.
    Wei, X.Y., Ngo, C.W.: Fusing semantics, observability, reliability and diversity of concept detectors for video search. In: Proc. ACM Int. Conf. on Multimedia, pp. 26–31, Vancouver, Canada (2008) Google Scholar
  33. 33.
    Wei, X.Y., Ngo, C.W., Jiang, Y.G.: Selection of concept detectors for video search by ontology-enriched semantic spaces. IEEE Trans. Multimedia 10(6), 1085–1096 (2008) CrossRefGoogle Scholar
  34. 34.
    Yuan, X., Hua, X.S., Wang, M., Wu, X.: Manifold-ranking based video concept detection on large database and feature pool. In: Proc. ACM Int. Conf. on Multimedia, pp. 623–626, Santa Barbara, CA (2006) Google Scholar
  35. 35.
    Yanagawa, A., et al.: Columbia University’s baseline detectors for 374 LSCOM semantic visual concepts. Technical report, Columbia University (2007) Google Scholar
  36. 36.
    Zhou, D., Weston, J., Gretton, A., Bousquet, O., Scho, B.: Ranking on data manifolds. In: Proc. Annual Conf. on Neural Information Processing Systems, pp. 169–176, Cambridge, MA (2003) Google Scholar
  37. 37.
    Zhou, D., Bousquet, O., Lal, T.N., Weston, J., Scholkopf, B.: Learning with local and global consistency. In: Proc. Annual Conf. on Neural Information Processing Systems, pp. 321–328, Cambridge, MA (2003) Google Scholar
  38. 38.
    Zhou, X.S., Rui, Y., Huang, T.: Water-Filling: A novel way for image structural feature extraction. In: Proc. IEEE Int. Conf. on Image Processing, pp. 570–574, Kobe, Japan (1999) Google Scholar

Copyright information

© Springer-Verlag 2010

Authors and Affiliations

  • Bailan Feng
    • 1
    • 2
  • Juan Cao
    • 1
  • Xiuguo Bao
    • 3
  • Lei Bao
    • 1
    • 2
  • Yongdong Zhang
    • 1
  • Shouxun Lin
    • 1
  • Xiaochun Yun
    • 3
  1. 1.Laboratory of Advanced Computing Research, Institute of Computing TechnologyChinese Academy of SciencesBeijingChina
  2. 2.Graduate University of the Chinese Academy of SciencesBeijingChina
  3. 3.National Computer Network Emergency Response Technical/Coordination Center of ChinaBeijingChina

Personalised recommendations