Skip to main content
SpringerLink
Log in

Image retargeting using nonparametric semantic segmentation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

In this paper, a new full-automatic approach to content aware image retargeting is proposed. Most image retargeting approaches does not incorporate content information and only use local appearance information. However, there are some approaches which use high level information such as saliency regions, objects mask and depth information. Such methods do not use semantic labelling for each object. In this paper, object masks as well as their semantic class labels are used to propose a new approach to image retargeting. To do so, semantic segmentation of image is provided. Hence, a nonparametric approach to semantic segmentation is employed which is fast with no need to any learning model. This makes it simple and applicable to any dataset. To evaluate the proposed approach, besides presenting visual examples, we performed a set of subjective evaluations too. The obtained results show that our method outperforms other retargeting approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Achanta R, Susstrun S (2009)“Saliency Detection for Content-aware Image Resizing,” in International Conference on Image Processing (ICIP), pp. 1005–1008

  2. Arbelaez P, Maire M, Fowlkes C, Malik J (2011)“Contour detection and hierarchical image segmentation,” IEEE Trans. on PAMI

  3. Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. ACM Trans Graph 26:10

    Article  Google Scholar 

  4. Gärtner B, Matousek J (2012) Approximation Algorithms and Semidefinite Programming. Springer

  5. Gonfaus JM et al (2010) “Harmony Potentials for Joint Classification and Segmentation,” in Computer Vision and Pattern Recognition, pp. 3280–3287

  6. Gould S, Zhang Y (2012) “PatchMatchGraph: Building a Graph of Dense Patch Correspondences for Label Transfer,” in European Conference on Computer Vision, pp. 439–452

  7. Gould S, Zhang Y (2012) “PatchMatchGraph: Building a Graph of Dense Patch Correspondences for Label Transfer,” in European Conference on Computer Vision (ECCV), pp. 439–452

  8. Grant M, Boyd S (2012) “cvx Users’ Guide,”

  9. Jain A, Gupta A, Davis LS (2010) “Learning What and How of Contextual Models for Scene Labeling,” in European Conference on Computer Vision, pp. 199–212

  10. Karni Z, Freedman D, Gotsman C (2009) Energy-based image deformation. Comput Graph Forum 28:1257–1268

    Article  Google Scholar 

  11. Lazebnik S, Schmid C, Ponce J (2006) “Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories,” in Computer Vision and Pattern Analysis, pp. 2169–2178

  12. Liu C, Yuen J, Torralba A (2011) Nonparametric scene parsing via label transfer. Pattern Anal Mach Intell 33:2368–2382

    Article  Google Scholar 

  13. Lowe D (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vision 60:91–110

    Article  Google Scholar 

  14. Manseld A, Gehler P, Gool LV, Rother C (2012) “Scene Carving: Scene Consistent Image Retargeting,” in European Conference on Computer Vision, pp. 143–156

  15. Myeong H, Lee KM (2013) “Tensor-based High-order Semantic Relation Transfer for Semantic Scene Segmentation,” in Computer Vision and Pattern Recognition, pp. 3073–3080

  16. Pritch Y, Kav-Venaki E, and Peleg S (2009)“Shift-map image editing,” in International Conference on Computer Vision, pp. 151–158

  17. Razzaghi P, Samavi S (2014) A new fast approach to nonparametric scene parsing. Pattern Recogn Lett 42:56–64

    Article  Google Scholar 

  18. Rubinstein M, Shamir A, Avidan S (2009) “Multi-operator Media Retargeting,” ACM Transactions on Graphics (TOG), vol. 28, p. 23

  19. Setlur V et al (2005) “Automatic image retargeting,” in international conference on Mobile and ubiquitous multimedia pp. 59–68

  20. Shamir A, Sorkine O (2009)“Visual media retargeting,” in SIGGRAPH ASIA Courses, p. 11

  21. Shotton J, Johnson M, Cipolla R (2008) “Semantic texton forests for image categorization and segmentation,” in Computer Vision and Pattern Recognition, pp. 1–8

  22. Shotton J, Winn J, Rother C, Criminisi A (2006) “TextonBoost: Joint Appearance, Shape and Context Modeling for Multi-Class Object Recognition and Segmentation,” in European Conference on Computer Vision, pp. 1–15

  23. Vaquero D et al (2010)“A survey of image retargeting techniques,” SPIE Optical Engineering+Applications, pp. 779814–779814

  24. Wang YS, Tai CL, Sorkine O, Lee TY (2008) Optimized scale-and-stretch for image resizing. ACM Trans Graph 27:118

    Google Scholar 

  25. Weijer JVD, Schmid CC (2006) “Coloring Local Feature Extraction,” in European Conference on Computer Vision, pp. 334–348

  26. Wu L et al (2012) Semantic aware sport image resizing jointly using seam carving and warping. Multimed Tools Appl 70:721–739

    Article  Google Scholar 

  27. Yang C et al (2013) “Saliency Detection via Graph-Based Manifold Ranking,” in Computer Vision and Pattern Recognition, pp. 3166–3173

  28. Yao J, Fidler S, Urtasun R (2012) “Describing the Scene as a Whole: Joint Object Detection, Scene Classification and Semantic Segmentation,” in Computer Vision and Pattern Recognition, pp. 702–709

Download references

Author information

Authors and Affiliations

  1. Electrical and Computer Engineering Department, Isfahan University of Technology, Isfahan, Iran

    Parvin Razzaghi & Shadrokh Samavi

  2. Electrical and Computer Engineering Department, McMaster University, Hamilton, Canada

    Shadrokh Samavi

Authors
  1. Parvin Razzaghi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shadrokh Samavi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Parvin Razzaghi.

Appendix

Appendix

The modified matrices of the Eq. 2 are defined as follows:

$$ \tilde{P}=\Big[{\begin{array}{cccc}\hfill {p}_{11}\hfill & \hfill {p}_{21}\hfill & \hfill \dots \hfill & \hfill \begin{array}{cccc}\hfill {p}_{N1}\hfill & \hfill {p}_{12}\hfill & \hfill \dots \hfill & \hfill \begin{array}{cc}\hfill {p}_{N2}\hfill & \hfill \begin{array}{cccc}\hfill \dots \hfill & \hfill {p}_{1M}\hfill & \hfill \dots \hfill & \hfill {p}_{NM}\Big]\hfill \end{array}\hfill \end{array}\hfill \end{array}\hfill \end{array}}^T $$
$$ \tilde{W}={\left[\begin{array}{cccc}\hfill {W}_{N\times N}\hfill & \hfill {0}_{N\times N}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times N}\hfill \\ {}\hfill {0}_{N\times N}\hfill & \hfill {W}_{N\times N}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times N}\hfill \\ {}\hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill \\ {}\hfill {0}_{N\times N}\hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill {W}_{N\times N}\hfill \end{array}\right]}_{\left(N\times M\right)\times \left(N\times M\right)} $$
$$ {W}_t^h={\left[\begin{array}{cccc}\hfill {W}_{N\times F}^h\hfill & \hfill {0}_{N\times F}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times F}\hfill \\ {}\hfill {0}_{N\times F}\hfill & \hfill {W}_{N\times F}^h\hfill & \hfill \dots \hfill & \hfill {0}_{N\times F}\hfill \\ {}\hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill \\ {}\hfill {0}_{N\times F}\hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill {W}_{N\times F}^h\hfill \end{array}\right]}_{\left(N\times M\right)\times \left(F\times M\right)} $$
$$ {\tilde{W}}^{*}=\Big[\begin{array}{cc}\hfill {w}_{11}^{*}\hfill & \hfill \begin{array}{cccc}\hfill \dots \hfill & \hfill {w}_{1F}^{*}\hfill & \hfill \dots \hfill & \hfill \begin{array}{ccc}\hfill {w}_{M1}^{*}\hfill & \hfill \dots \hfill & \hfill {w}_{MF}^{*}\Big]{}_{1\times \left(F\times M\right)}\hfill \end{array}\hfill \end{array}\hfill \end{array} $$
$$ {\tilde{C}}_1={\left[\begin{array}{cccc}\hfill {C}_{N\times M}\hfill & \hfill {0}_{N\times M}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times M}\hfill \\ {}\hfill {0}_{N\times M}\hfill & \hfill {C}_{N\times M}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times M}\hfill \\ {}\hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill \\ {}\hfill {0}_{N\times M}\hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill {C}_{N\times M}\hfill \end{array}\right]}_{\left(N\times M\right)\times \left(M\times M\right)} $$
$$ {\tilde{C}}_2={\left[\overset{M}{\overbrace{\begin{array}{cccc}\hfill {c}_{N\times 1}\hfill & \hfill {0}_{N\times 1}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times 1}\hfill \\ {}\hfill {0}_{N\times 1}\hfill & \hfill {c}_{N\times 1}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times 1}\hfill \\ {}\hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill \\ {}\hfill {0}_{N\times 1}\hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill {c}_{N\times 1}\hfill \end{array}}}\begin{array}{c}\hfill \kern0.24em \dots \kern0.6em \hfill \\ {}\hfill \dots \kern0.24em \hfill \\ {}\hfill \dots \kern0.24em \hfill \\ {}\hfill \dots \hfill \end{array}\begin{array}{cccc}\hfill {c}_{N\times 1}\hfill & \hfill {0}_{N\times 1}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times 1}\hfill \\ {}\hfill {0}_{N\times 1}\hfill & \hfill {c}_{N\times 1}\hfill & \hfill \dots \hfill & \hfill {0}_{N\times 1}\hfill \\ {}\hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill \dots \hfill \\ {}\hfill {0}_{N\times 1}\hfill & \hfill \dots \hfill & \hfill \dots \hfill & \hfill {c}_{N\times 1}\hfill \end{array}\right]}_{\left(N\times M\right)\times \left(M\times M\right)} $$
$$ {\tilde{C}}^{*}=\Big[\begin{array}{cc}\hfill {c}_{11}^{*}\hfill & \hfill \begin{array}{cccc}\hfill \dots \hfill & \hfill {c}_{1M}^{*}\hfill & \hfill \dots \hfill & \hfill \begin{array}{ccc}\hfill {c}_{M1}^{*}\hfill & \hfill \dots \hfill & \hfill {c}_{MM}^{*}\Big]{}_{1\times \left(M\times M\right)}\hfill \end{array}\hfill \end{array}\hfill \end{array} $$

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Razzaghi, P., Samavi, S. Image retargeting using nonparametric semantic segmentation. Multimed Tools Appl 74, 11517–11536 (2015). https://doi.org/10.1007/s11042-014-2249-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-2249-y

Keywords

Search

Navigation