Journal of Zhejiang University SCIENCE C

, Volume 11, Issue 11, pp 850–859 | Cite as

Salient object extraction for user-targeted video content association

  • Jia Li
  • Han-nan Yu
  • Yong-hong TianEmail author
  • Tie-jun Huang
  • Wen Gao


The increasing amount of videos on the Internet and digital libraries highlights the necessity and importance of interactive video services such as automatically associating additional materials (e.g., advertising logos and relevant selling information) with the video content so as to enrich the viewing experience. Toward this end, this paper presents a novel approach for user-targeted video content association (VCA). In this approach, the salient objects are extracted automatically from the video stream using complementary saliency maps. According to these salient objects, the VCA system can push the related logo images to the users. Since the salient objects often correspond to important video content, the associated images can be considered as content-related. Our VCA system also allows users to associate images to the preferred video content through simple interactions by the mouse and an infrared pen. Moreover, by learning the preference of each user through collecting feedbacks on the pulled or pushed images, the VCA system can provide user-targeted services. Experimental results show that our approach can effectively and efficiently extract the salient objects. Moreover, subjective evaluations show that our system can provide content-related and user-targeted VCA services in a less intrusive way.

Key words

Salient object extraction User-targeted video content association Complementary saliency maps 

CLC number



Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Achanta, R., Hemami, S., Estrada, F., Susstrunk, S., 2009. Frequency-Tuned Salient Region Detection. IEEE Conf. on Computer Vision and Pattern Recognition, p.1597–1604. [doi:10.1109/CVPR.2009.5206596]Google Scholar
  2. Allili, M.S., Ziou, D., 2007. Object of Interest Segmentation and Tracking by Using Feature Selection and Active Contours. IEEE Conf. on Computer Vision and Pattern Recognition, p.1–8.Google Scholar
  3. Brasnett, P., Bober, M., 2007. Proposed Improvements to Image Signature XM 31.0. MPEG Doc No. M14983.Google Scholar
  4. Chang, C.H., Hsieh, K.Y., Chung, M.C., Wu, J.L., 2008. Visa: Virtual Spotlighted Advertising. Proc. ACM Int. Conf. on Multimedia, p.837–840.Google Scholar
  5. Elazary, L., Itti, L., 2008. Interesting objects are visually salient. J. Vis., 8(3), Article No. 3. [doi:10.1167/8.3.3]Google Scholar
  6. Friedland, G., Jantz, K., Rojas, R., 2005. Siox: Simple Interactive Object Extraction in Still Images. IEEE Int. Symp. on Multimedia, p.7–14.Google Scholar
  7. Gao, W., Tian, Y.H., Huang, T.J., Yang, Q., 2010. Vlogging: a survey of video blogging technology on the web. ACM Comput. Surv., 42(4), Article No. 15. [doi:10.1145/1749 603.1749606]Google Scholar
  8. Guo, J.L., Mei, T., Liu, F.L., Hua, X.S., 2009. Adon: an Intelligent Overlay Video Advertising System. SIGIR, p.628–629.Google Scholar
  9. Hou, X.D., Zhang, L.Q., 2007. Saliency Detection: a Spectral Residual Approach. IEEE Conf. on Computer Vision and Pattern Recognition, p.1–8. [doi:10.1109/CVPR.2007.383 267]Google Scholar
  10. Hua, G., Liu, Z.C., Zhang, Z.Y., Wu, Y., 2006. Iterative localglobal energy minimization for automatic extraction of objects of interest. IEEE Trans. Pattern Anal. Mach. Intell., 28(10):1701–1706. [doi:10.1109/TPAMI.2006.209]CrossRefGoogle Scholar
  11. Itti, L., Koch, C., Niebur, E., 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell., 20(11):1254–1259.CrossRefGoogle Scholar
  12. Ko, B.C., Nam, J.Y., 2006. Automatic Object-of-Interest Segmentation from Natural Images. IEEE Int. Conf. on Pattern Recognition, p.45–48.Google Scholar
  13. Kwak, S.Y., Ko, B.C., Byun, H., 2005. Automatic salient-object extraction using the contrast map and salient points. LNCS, 3332:138–145.Google Scholar
  14. Lee, J.C., 2008. Hacking the nintendo Wii remote. IEEE Perv. Comput., 7(3):39–45. [doi:10.1109/MPRV.2008.53]CrossRefGoogle Scholar
  15. Lee, J.T., Lee, H.D., Park, H.S., Song, Y.I., Rim, H.C., 2009. Finding Advertising Keywords on Video Scripts. SIGIR, p.686–687.Google Scholar
  16. Lekakos, G., Papakiriakopoulos, D., Chorianopoulos, K., 2001. An Integrated Approach to Interactive and Personalized TV Advertising. Workshop on Personalization in Future TV.Google Scholar
  17. Li, Y., Wan, K.W., Yan, X., Xu, C.S., 2005. Real Time Advertisement Insertion in Baseball Video Based on Advertisement Effect. Proc. ACM Int. Conf. on Multimedia, p.343–346.Google Scholar
  18. Liao, W.S., Chen, K.T., Hsu, W.H., 2008. Adimage: Video Advertising by Image Matching and Ad Scheduling Optimization. SIGIR, p.767–768.Google Scholar
  19. Liu, H.Y., Jiang, S.Q., Huang, Q.M., Xu, C.S., 2008. A Generic Virtual Content Insertion System Based on Visual Attention Analysis. Proc. ACM Int. Conf. on Multimedia, p.379–388.Google Scholar
  20. Liu, T., Sun, J., Zheng, N.N., Tang, X.O., Shum, H.Y., 2007. Learning to Detect a Salient Object. IEEE Conf. on Computer Vision and Pattern Recognition, p.1–8.Google Scholar
  21. Martin, D., Fowlkes, C., Tai, D., Malik, J., 2001. A Database of Human Segmented Natural Images and Its Application to Evaluating Segmentation Algorithms and Measuring Ecological Statistics. IEEE ICCV, p.416–423.Google Scholar
  22. Mei, T., Hua, X.S., Yang, L.J., Li, S.P., 2007. Videosense—Towards Effective Online Video Advertising. Proc. ACM Int. Conf. on Multimedia, p.1075–1084.Google Scholar
  23. Movahedi, V., Elder, J.H., 2010. Design and Perceptual Validation of Performance Measures for Salient Object Segmentation. IEEE Computer Society Workshop on Perceptual Organization in Computer Vision, p.49–56.Google Scholar
  24. Park, K.T., Moon, Y.S., 2007. Automatic Extraction of Salient Objects Using Feature Maps. Int. Conf. on Acoustics, Speech, and Signal Processing, p.617–620.Google Scholar
  25. Pinneli, S., Chandler, D.M., 2008. A Bayesian Approach to Predicting the Perceived Interest of Objects. 15th IEEE Int. Conf. on Image Processing, p.2584–2587. [doi:10.1109/ICIP.2008.4712322]Google Scholar
  26. Srinivasan, S.H., Sawant, N., Wadhwa, S., 2007. Vadeo-Video Advertising System. Proc. ACM Int. Conf. on Multimedia, p.455–456.Google Scholar
  27. Thawani, A., Gopalan, S., Sridhar, V., 2004. Context Aware Personalized Ad Insertion in an Interactive TV Environment. Workshop on Personalization in Future TV.Google Scholar
  28. Walther, D., Koch, C., 2006. Modeling attention to salient proto-objects. Neur. Networks, 19(9):1395–1407. [doi:10.1016/j.neunet.2006.10.001]zbMATHCrossRefGoogle Scholar
  29. Wang, J.Q., Fang, Y.K., Lu, H.Q., 2008. Online Video Advertising Based on User’s Attention Relevancy Computing. IEEE Int. Conf. on Multimedia and Expo, p.1161–1164. [doi:10.1109/ICME.2008.4607646]Google Scholar

Copyright information

© ?Journal of Zhejiang University Science? Editorial Office and Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Jia Li
    • 1
    • 2
  • Han-nan Yu
    • 3
  • Yong-hong Tian
    • 3
    Email author
  • Tie-jun Huang
    • 3
  • Wen Gao
    • 1
    • 3
  1. 1.Key Lab of Intelligent Information Processing, Institute of Computing TechnologyChinese Academy of SciencesBeijingChina
  2. 2.Graduate University of Chinese Academy of SciencesBeijingChina
  3. 3.National Engineering Lab for Video Technology (NELVT), School of EE & CSPeking UniversityBeijingChina

Personalised recommendations