Advertisement

Multimedia Tools and Applications

, Volume 76, Issue 8, pp 11081–11095 | Cite as

Simultaneously retargeting and super-resolution for stereoscopic video

Article
  • 132 Downloads

Abstract

This paper presents a novel approach that is able to resize stereoscopic video to fit various display environments with different aspect-ratios, while preserving the prominent content, keeping temporally consistent, adapting depth, as well as increasing the resolution. Our proposed approach can deal with retargeting and super-resolution problems simultaneously via replacing the down-sampling matrix appearing in super-resolution algorithm with a novel one, named as content-aware-sampling matrix, derived from retargeting method. The new matrix can sample the image into any resolution while preserving its important information as much as possible. Our approach can be roughly subdivided into three steps. In the first step, we calculate the overall saliency map for a shot, while considering the conspicuous information such as motion, depth, and structures. In the second step, given a certain resolution, we compute the retargeting parameters by a global optimization and formulate them into a matrix. Finally, we substitute the matrix into the objective function of super-resolution to achieve high visual quality images with expected resolution. In addition, we propose a novel single image super-resolution method inspired by a blind image deblurring method. The experimental results based on user studies verify the effectiveness of our approach. And the comparisons with the-state-of-the-art single image super-resolution methods validate the potential of our super-resolution method.

Keywords

Stereoscopic video Retargeting Super-resolution 

References

  1. 1.
    Almeida MSC, Almeida LB (2010) Blind and semi-blind deblurring of natural images. IEEE Trans Image Process 19:36–52MathSciNetCrossRefGoogle Scholar
  2. 2.
    Almeida MSC, Figueiredo MAT (2011) New stopping criteria for iterative blind image deblurring based on residual whiteness measures. In: IEEE statistical signal processing workshop (SSP)Google Scholar
  3. 3.
    Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. ACM Trans Graph (TOG) - Proc ACM SIGGRAPH 2007 26Google Scholar
  4. 4.
    Bevilacqua M, Roumy A, More CGMLA (2012) Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 23rd British machine vision conference (BMVC)Google Scholar
  5. 5.
    Brust H, Tech G, Muller K (2009) Report on generation of mixed spatial resolution stereo data base. Tech. rep. MOBILE3DTV projectGoogle Scholar
  6. 6.
    Chang CH, Liang CK, Chuang YY (2011) Content-aware display adaptation and interactive editing for stereoscopic images. IEEE Trans Multimed 13:589–601CrossRefGoogle Scholar
  7. 7.
    Dollar P, Zitnick CL (2013) Structured forests for fast edge detection. In: IEEE international conference on computer visionGoogle Scholar
  8. 8.
    Farsiu S, Robinson MD, Elad M., Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13:1327–1344CrossRefGoogle Scholar
  9. 9.
    Garcia DC, Dorea C, de Queiroz RL (2012) Super resolution for multiview images using depth information. IEEE Trans Circ Syst Video Technol 22:1249–1256CrossRefGoogle Scholar
  10. 10.
    Guthier B, Kiess J, Kopf S, Effelsberg W (2013) Seam carving for stereoscopic video. In: IEEE IVMSP workshopGoogle Scholar
  11. 11.
    He L, Qi H, Zaretzki R (2013) Beta process joint dictionary learning for coupled feature spaces with application to single image super-resolution. In: IEEE conference on computer vision and pattern recognition (CVPR)Google Scholar
  12. 12.
    Kopf S, Guthier B, Hipp C, Kiess J, Effelsberg W (2014) Warping-based video retargeting for stereoscopic video. In: IEEE international conference on image processing (ICIP)Google Scholar
  13. 13.
    Li X, Orchard MT (2001) New edge-directed interpolation. IEEE Trans Image Process 10:1521– 1527CrossRefGoogle Scholar
  14. 14.
    Liu C. (2009) Beyond pixels: exploring new representations and applications for motion analysis. Ph.D. thesis. Massachusetts Institute of TechnologyGoogle Scholar
  15. 15.
    Liu C, Sun D (2011) A bayesian approach to adaptive video super resolution. In: IEEE Conference on computer vision and pattern recognition (CVPR)Google Scholar
  16. 16.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60:91–110CrossRefGoogle Scholar
  17. 17.
    Niu Y, Liu F, Feng WC, Jin H (2012) Aesthetics-based stereoscopic photo cropping for heterogeneous displays. IEEE Trans Multimed 14:783–796CrossRefGoogle Scholar
  18. 18.
    Rubinstein M, Shamir A, Avidan S (2008) Improved seam carving for video retargeting. ACM Trans Graph (TOG) - Proc ACM SIGGRAPH 2008 27Google Scholar
  19. 19.
    Timofte R, Smet VD, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International conference on computer vision (ICCV)Google Scholar
  20. 20.
    Timofte R, Smet VD, Gool LV (2013) Anchored neighborhood regression for fast example-based super-resolution. In: IEEE International conference on computer vision (ICCV)Google Scholar
  21. 21.
    Timofte R, Smet VD, Gool LV (2014) A+: adjusted anchored neighborhood regression for fast super-resolution. In: 12th Asian conference on computer visionGoogle Scholar
  22. 22.
    Utsugi K, Shibahara T, Koike T, Takahashi K, Naemura T (2010) Seam carving for stereo images. In: 3DTV-conference: the true vision - capture, transmission and display of 3D video (3DTV-CON)Google Scholar
  23. 23.
    Villena S, Vega M, Molina R, Katsaggelos AK (2009) Bayesian super-resolution image reconstruction using an l1 prior. In: Proceedings of 6th international symposium on image and signal processing and analysisGoogle Scholar
  24. 24.
    Wang YS, Tai CL, Sorkine O, Lee TY (2008) Optimized scale-and-stretch for image resizing. ACM Trans Graph (TOG) - Proc ACM SIGGRAPH Asia 2008 27Google Scholar
  25. 25.
    Yang CY, Yang MH (2013) Fast direct super-resolution by simple functions. In: IEEE International conference oncomputer vision (ICCV)Google Scholar
  26. 26.
    Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19:2861–2873MathSciNetCrossRefGoogle Scholar
  27. 27.
    Yu J, Wang Z (2014) 3d facial motion tracking by combining online appearance model and cylinder head model in particle filtering. Sci Chin Inf Sci 57:274–280Google Scholar
  28. 28.
    Yu J, Wang Z (2015) A video, text and speech driven realistic 3d virtual head for human-machine interface. IEEE Trans Cybern 45:977–988Google Scholar
  29. 29.
    Zeyde R, Elad M, Protter M (2012) On single image scale-up using sparse representations. Curves Surf 2011 6920:711–730MathSciNetMATHGoogle Scholar
  30. 30.
    Zhang J, Cao Y, Wang Z (2013) A simultaneous method for 3d video super-resolution and high-quality depth estimation. In: IEEE International conference on image processing (ICIP)Google Scholar
  31. 31.
    Zhang J, Cao Y, Zha ZJ, Zheng Z, Chen CW, Wang Z (2014) A unified scheme for super-resolution and depth estimation from asymmetric stereoscopic video. IEEE Trans Circ Syst Video Technol. doi: 10.1109/TCSVT.2014.2367356
  32. 32.
    Zhang J, Cao Y, Zheng Z, Chen C, Wang Z (2014) A new closed loop method of super-resolution for multi-view images. Mach Vis Appl 25:1685–1695CrossRefGoogle Scholar
  33. 33.
    Zhu Y, Zhang Y, Yuille AL (2014) Single image super-resolution using deformable patches. In: IEEE Conference on computer vision and pattern recognition (CVPR)Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of AutomationUniversity of Science and Technology of ChinaHefeiPeople’s Republic of China
  2. 2.Institute of Intelligent MachinesChinese Academy of SciencesHeiFeiPeople’s Republic of China

Personalised recommendations