Advertisement

Automatic salient object sequence rebuilding for video segment analysis

  • Tie Liu
  • Haibin Duan
  • Yuanyuan Shang
  • Zejian Yuan
  • Nanning Zheng
Research Paper

Abstract

Detection of salient object sequences from video data is challenging when the salient object changes between consecutive frames. In this study, we addressed the salient object sequence rebuilding problem with video segment analysis. We reformulated the problem as a binary labeling problem, analyzed the potential salient object sequences in the video using a clustering method, and separated the salient object sequence from the background by applying an energy optimization method. Our proposed approach determines whether temporal consecutive pixels belong to the same salient object sequence. The conditional random field is then learned to effectively integrate the salient features and the sequence consecutive constraints. A dynamic programming algorithm was developed to resolve the energy minimization problem efficiently. Experimental results confirmed the ability of our approach to address the salient object rebuilding problem in automatic visual attention applications and video content analysis.

Keywords

salient object video attention sequence segment analysis conditional random model 

Notes

Acknowledgements

This work was supported by National Key RD Program of China (Grant No. 2016YFB100 1001), National Natural Science Foundation of China (Grant No. 61603022), and China Postdoctoral Science Foundation and Aeronautical Science Foundation of China (Grant No. 20135851042).

References

  1. 1.
    Borji A, Itti L. State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intel, 2011, 35: 185–207CrossRefGoogle Scholar
  2. 2.
    Ma L, Chen L, Zhang X J, et al. A waterborne salient ship detection method on SAR imagery. Sci China Inf Sci, 2015, 58: 089301Google Scholar
  3. 3.
    Liu T, Yuan Z J, Sun J, et al. Learning to detect a salient object. IEEE Trans Pattern Anal Mach Intel, 2011, 33: 353–367CrossRefGoogle Scholar
  4. 4.
    Liu T, Zheng N N, Yuan Z J, et al. Video attention: learning to detect a salient object sequence. In: Proceedings of the International Conference on Pattern Recognition, Tampa, 2008. 1–4Google Scholar
  5. 5.
    Feng J, Wei Y, Tao L, et al. Salient object detection by composition. In: Proceedings of the IEEE International Conference on Computer Vision, Barcelona, 2011. 1028–1035Google Scholar
  6. 6.
    Yang C, Zhang L, Lu H, et al. Saliency detection via graph-based manifold ranking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, 2013. 3166–3173Google Scholar
  7. 7.
    Santella A, Agrawala M, Decarlo D, et al. Gaze-based interaction for semi-automatic photo cropping. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Montr´eal, 2006. 771–780CrossRefGoogle Scholar
  8. 8.
    Chen L Q, Xie X, Fan X, et al. A visual attention mode for adapting images on small displays. Multim Syst, 2003, 9: 353–364CrossRefGoogle Scholar
  9. 9.
    Rother C, Bordeaux L, Hamadi Y, et al. Autocollage. In: Proceedings of the International Conference and Exhibition on Computer Graphics and Interactive Techniques (SIGGRAPH), Boston, 2006. 847–852Google Scholar
  10. 10.
    Jiang H, Wang J, Yuan Z, et al. Automatic salient object segmentation based on context and shape prior. In: Proceedings of the British Machine Vision Conference. Durham: BMVA Press, 2011Google Scholar
  11. 11.
    Jiang H, Wang J, Yuan Z, et al. Salient object detection: a discriminative regional feature integration approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Portland, 2013. 2083–2090Google Scholar
  12. 12.
    Zhao Y, Lu S J, Qian H L, et al. Robust mesh deformation with salient features preservation. Sci China Inf Sci, 2016, 59: 052106CrossRefGoogle Scholar
  13. 13.
    Wu X M, Du M N, Chen W H, et al. Salient object detection via region contrast and graph regularization. Sci China Inf Sci, 2016, 59: 032104CrossRefGoogle Scholar
  14. 14.
    Comaniciu D, Ramesh V, Meer P. Real-time tracking of nonrigid objects using mean shift. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hilton Head Island, 2000. 142–149Google Scholar
  15. 15.
    Wei Y, Sun J, Tang X, et al. Interactive offline tracking for color objects. In: Proceedings of the IEEE International Conference on Computer Vision, Rio de Janeiro, 2007Google Scholar
  16. 16.
    Zhang G, Yuan Z, Zheng N N, et al. Visual saliency based object tracking. 1111In: Proceedings of the Asian Conference on Computer Vision. Berlin: Springer, 2009. 193–203Google Scholar
  17. 17.
    Liu D, Chen T. A topic-motion model for unsupervised video object discovery. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, 2007Google Scholar
  18. 18.
    Yun X, Jing Z L, Xiao G, et al. A compressive tracking based on time-space Kalman fusion model. Sci China Inf Sci, 2016, 59: 012106CrossRefGoogle Scholar
  19. 19.
    Yang Y X, Yang J, Zhang Z X, et al. High-speed visual target tracking with mixed rotation invariant description and skipping searching. Sci China Inf Sci, 2017, 60: 062401CrossRefGoogle Scholar
  20. 20.
    Doucet A, Freitas N, Gordon N. Sequential Monte Carlo Methods in Practice. Berlin: Springer, 2001CrossRefMATHGoogle Scholar
  21. 21.
    Lafferty J, McCallum A, Pereira F. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the International Conference on Machine Learning. San Francisco: Morgan Kaufmann Publishers, 2001. 282–289Google Scholar
  22. 22.
    Shi J, Malik J. Normalized cuts and image segmentation. IEEE Trans Pattern Anal Mach Intel, 2000, 22: 888–905CrossRefGoogle Scholar
  23. 23.
    Blake A, Rother C, Brown M, et al. Interactive image segmentation using an adaptive GMMRF model. 1111In: Proceedings of the European Conference on Computer Vision. Berlin: Springer, 2004. 428–441Google Scholar
  24. 24.
    Sun J, Zhang W, Tang X, et al. Bi-directional tracking using trajectory segment analysis. In: Proceedings of the IEEE International Conference on Computer Vision, Beijing, 2005Google Scholar
  25. 25.
    Zhang Y, Su A, Zhu X, et al. Salient object detection approach in UAV video. In: Proceedings of the 8th International Symposium on Multispectral Image Processing and Pattern Recognition. Bellingham: SPIE Proceedings, 2013. 8224Google Scholar

Copyright information

© Science China Press and Springer-Verlag GmbH Germany 2017

Authors and Affiliations

  • Tie Liu
    • 1
  • Haibin Duan
    • 1
  • Yuanyuan Shang
    • 2
  • Zejian Yuan
    • 3
  • Nanning Zheng
    • 3
  1. 1.School of Automation Science and Electrical EngineeringBeihang UniversityBeijingChina
  2. 2.Information Engineering CollegeCapital Normal UniversityBeijingChina
  3. 3.The School of Electronic and Information EngineeringXi’an Jiaotong UniversityXi’anChina

Personalised recommendations