Multimedia Systems

, Volume 18, Issue 3, pp 201–214 | Cite as

@ICT: attention-based virtual content insertion

  • Huiying LiuEmail author
  • Qingming Huang
  • Changsheng Xu
  • Shuqiang Jiang
Regular Paper


In this paper, we propose an attention-based virtual content insertion solution, called @ICT. Virtual content insertion (VCI) is an emerging application of video analysis and has been used in video augmentation and advertisement insertion. An ideal VCI solution should make the inserted virtual content being noticed by audiences and at the same time should not interfere with audiences’ viewing experience on the original content. To balance these two conflicting issues, meaning high attention and low intrusiveness, we choose higher attentive shots as insertion time while determine insertion place and content interdependently by considering lower attention together with visual consistency. We also propose a measurement of intrusiveness from the viewpoint of visual attention. Furthermore, @ICT includes an in-scene insertion module, which embeds the virtual content into the videos with higher vividness and lower intrusiveness. @ICT is able to obtain an optimal balance between the noticing of the virtual content by audiences and disruption of viewing experience to the original content. It needs little prior knowledge and is applied to general videos. Extensive quantitative and qualitative evaluations on the VCI result have verified the effectiveness of the solution.


Video content analysis Virtual content insertion Visual attention 


  1. 1.
    Chang, C.-H., Hsieh, K.-Y., Chung, M.-C., Wu, J.-L.: ViSA: virtual spotlighted advertising. In: Proceedings of the 16th ACM International Conference on Multimedia, pp. 837–840 (2008)Google Scholar
  2. 2.
    Chen, X., Yang, J., Zhang, J. Waibel, A. Automatic detection and recognition of signs from natural scenes. IEEE Trans. Image Process. 13(1), 87–99 (2004)Google Scholar
  3. 3.
    Deshpande, S., Naphade, P., Rao, C.V.K., Bhadada, K., Rangan, P.V.: Method and apparatus for including virtual ads in video presentations. US Patent, 7158666 (2007)Google Scholar
  4. 4.
    Duan, L.-Y., Xu, M., Tian, Q., Xu, C.-S., Jin, J.S.: A Unified framework for semantic shot classification in sports video. IEEE Trans. Multimedia 7(6), 1066–1083 (2005)CrossRefGoogle Scholar
  5. 5.
    Dufaux, F., Konrad, J.: Efficient, robust, and fast global motion estimation for video coding. IEEE Trans. Image Process. 9(3), 497–501 (2000)CrossRefGoogle Scholar
  6. 6.
    Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)CrossRefGoogle Scholar
  7. 7.
    Hu, Y., Rajan, D., Chia, L.-T.: Robust subspace analysis for detecting visual attention regions in images. In: Proceedings of the 13th ACM International Conference on Multimedia, pp. 716–724 (2005)Google Scholar
  8. 8.
    Itti, L., Baldi, P.: A principled approach to detecting surprising events in video. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 631–637 (2005)Google Scholar
  9. 9.
    Itti, L., Koch, C., Niebur, E.: “A model of saliency-based visual attention for rapid scene analysis”. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)CrossRefGoogle Scholar
  10. 10.
    Kreitman, H., Bar-El, D., Amir, Y., Tirosh, E.: Method and system for perspectively distorting an image and implanting same into a video stream. US Patent 5,731,846 (1998)Google Scholar
  11. 11.
    Li, Y., Wah Wan, K., Yan, X., Xu, C.: Real time advertisement insertion in baseball video based on advertisement effect. In: Proceedings of the 13th ACM International Conference on Multimedia, pp. 343–346 (2005)Google Scholar
  12. 12.
    Li, L., Mei, T., Hua, X.-S.: GameSense: game-like in-image advertising. Multimedia Tools Appl. 49(1), 145–166 (2010)CrossRefGoogle Scholar
  13. 13.
    Liu, C., Liu, H., Jiang, S., Huang, Q., Zheng Y., Zhang W.: JDL at Trecvid 2006 shot boundary detection. In: TRECVID 2006 WorkshopGoogle Scholar
  14. 14.
    Liu, H., Qiu, X., Huang, Q., Jiang, S., Xu, C.: Advertise gently—in-image advertising with low intrusiveness. In: Proceedings of the IEEE International Conference on Image Processing (2009)Google Scholar
  15. 15.
    Liu, H., Jiang, S., Huang Q., Xu, C.: A generic virtual content insertion system based on visual attention analysis. In: Proceedings of the 16th ACM International Conference on Multimedia, pp. 379–388 (2008)Google Scholar
  16. 16.
    Liu, H., Jiang, S., Huang Q., Xu, C.: Lower attentive region detection for virtual content insertion. In: Proceedings of the IEEE International Conference on Multimedia & Expo (2008)Google Scholar
  17. 17.
    Li, H., Edwards, S.M., Lee, J.-H.: Measuring the intrusiveness of advertisements: scale development and validation. J. Advert. 31(2), 37–47 (2002)Google Scholar
  18. 18.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  19. 19.
    Ma, Y.-F., Hua, X.-S., Lu, L., Zhang, H.-J.: A generic framework of user attention model and its application in video summarization. IEEE Trans. Multimedia 7(5), 907–919 (2005)CrossRefGoogle Scholar
  20. 20.
    Mei, T., Hua, X.-S., Li, S.: Contextual in-image advertising. In: Proceedings of the 16th ACM International Conference on Multimedia, pp. 439–448 (2008)Google Scholar
  21. 21.
    Mei, T., Hua, X.-S., Yang, L., Li, S.: VideoSense-towards effective online video advertising. In: Proceedings of the 15th ACM International Conference on Multimedia, pp. 1075–1084 (2007)Google Scholar
  22. 22.
    Mei, T., Guo, J., Hua, X.-S., Liu, F.: “AdOn: toward contextual overlay in-video advertising”. Multimedia Syst 16, 335–344 (2010)CrossRefGoogle Scholar
  23. 23.
    Meur, O.L., Callet, P.L., Barba, D., Thoreau, D.: A coherent computational approach to model bottom-up visual attention. IEEE Trans. Pattern Anal. Mach. Intell. 28(5), 802–816 (2006)CrossRefGoogle Scholar
  24. 24.
    Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)MathSciNetCrossRefGoogle Scholar
  25. 25.
    Rolls, E.T., Aggelopoulos, N.C., Zheng, F.: Effective size of receptive fields of inferior temporal visual cortex neurons in natural scenes. J. Neurosci. 23(1), 339–348 (2003)Google Scholar
  26. 26.
    Rosser, R.J., Das, S., Tan, Y.: Method of tracking scene motion for live video insertion systems. US Patent 5,808,695 (1998)Google Scholar
  27. 27.
    Sharir, A., Tamir, M., Wilf, I.: Method and apparatus for automatic electronic replacement of billboards in a video image. US Patent 6,384,871 (2002)Google Scholar
  28. 28.
    Soodak, R.E.: “Two-dimensional modeling of visual receptive fields using Gaussian subunits”. Proc. Natl. Acad. Sci. USA 83, 9259–9263 (1986)CrossRefGoogle Scholar
  29. 29.
    Sun, Y., Fisher, R.B., Wang, F., Gomes, H.M.: A computer vision model for visual-object-based attention and eye movements. Comput. Vis. Image Underst. 112(2), 126–142 (2008)CrossRefGoogle Scholar
  30. 30.
    Wan, K., Xu, C.: Automatic content placement in sports highlights. In: Proceedings of the IEEE International Conference on Multimedia & Expo, pp. 1893–1896 (2006)Google Scholar
  31. 31.
    Whitley, D.: A genetic algorithm tutorial. Stat. Comput. 4, 65–85 (1994)CrossRefGoogle Scholar
  32. 32.
    Xu, C., Wan, K.W., Bui, S.H., Tian, Q.: Implanting virtual advertisement into broadcast soccer video. In: Pacific-Rim Conference on Multimedia, pp. 264–271 (2004)Google Scholar
  33. 33.
    Yu, X., Yan, X., Chi, T.T.P., Cheong, L.F.: Inserting 3D projected virtual content into broadcast tennis video. In: Proceedings of the 14th ACM International Conference on Multimedia, pp. 619–622 (2006)Google Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Huiying Liu
    • 1
    Email author
  • Qingming Huang
    • 2
  • Changsheng Xu
    • 3
  • Shuqiang Jiang
    • 1
  1. 1.Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of SciencesBeijingChina
  2. 2.Graduate University of Chinese Academy of SciencesBeijingChina
  3. 3.National Lab of Pattern Recognition, Institute of Automation, Chinese Academy of SciencesBeijingChina

Personalised recommendations