Skip to main content
SpringerLink
Log in

Context-aware saliency detection for image retargeting using convolutional neural networks

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Image retargeting is the task of making images capable of being displayed on screens with different sizes. This work should be done so that high-level visual information and low-level features such as texture remain as intact as possible to the human visual system. At the same time, the output image may have different dimensions. Thus, simple methods such as scaling and cropping are not adequate for this purpose. In recent years, researchers have tried to improve the existing retargeting methods, and they have introduced new ones. However, a specific method cannot be utilized to retarget all types of images. In other words, different images require different retargeting methods. Image retargeting has a close relationship to image saliency detection, which is a relatively new image processing task. Earlier saliency detection methods were based on local and global but low-level image information. These methods are called bottom-up processes. On the other hand, newer approaches are top-down and mixed methods that consider the high level and semantic knowledge of the image too. In this paper, we introduce the proposed methods in both saliency detection and retargeting. For the saliency detection, the use of image context and semantic segmentation are examined, and a novel mixed bottom-up and top-down saliency detection method is introduced. After saliency detection, a modified version of an existing retargeting technique is utilized for retargeting the images. The results suggest that the proposed image retargeting pipeline has excellent performance compared to other tested methods. Also, the subjective evaluations on the Pascal dataset can be used as a retargeting quality assessment dataset for further research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  1. Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. In: ACM Transactions on Graphics (TOG), vol. 26, p. 10. ACM

  2. Borji A, Itti L (2015) Cat2000: A large scale fixation dataset for boosting saliency research. arXiv preprint arXiv:1505.03581

  3. Cho D, Park J, Oh TH, Tai YW, So Kweon I (2017) Weakly-and self-supervised learning for content-aware deep image retargeting. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4558–4567

  4. Cornia M, Baraldi L, Serra G, Cucchiara R (2018) Predicting human eye fixations via an lstm-based saliency attentive model. IEEE Trans Image Process 27(10):5142–5154

    Article  MathSciNet  Google Scholar 

  5. Esmaeili SA, Singh B, Davis LS (2017) Fast-at: Fast automatic thumbnail generation using deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4622–4630

  6. Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2012) The PASCAL Visual Object Classes Challenge (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html

  7. Fang Y, Fang Z, Yuan F, Yang Y, Yang S, Xiong NN (2017) Optimized multioperator image retargeting based on perceptual similarity measure. IEEE Transactions on Systems, Man, and Cybernetics: Systems 47(11):2956–2966

    Article  Google Scholar 

  8. Fujimura M, Imamura K, Kuroda H (2017) Application of saliency map to restraint scheme of attack to digital watermark using seam carving. In: 2017 IEEE International Conference on Consumer Electronics-Taiwan (ICCE-TW), pp. 347–348. IEEE

  9. Goferman S, Zelnik-Manor L, Tal A (2011) Context-aware saliency detection. IEEE Trans Pattern Anal Mach Intell 34(10):1915–1926

    Article  Google Scholar 

  10. Guo C, Zhang L (2010) A novel multiresolution spatiotemporal saliency detection model and its applications in image and video compression. IEEE Trans Image Process 19(1):185–198

    Article  MathSciNet  Google Scholar 

  11. Hooshmand M, Soroushmehr SMR, Khadivi P, Samavi S, Shirani S (2013) Visual sensor network lifetime maximization by prioritized scheduling of nodes. J Netw Comput Appl 36(1):409–419

    Article  Google Scholar 

  12. Hu W, Hu R, Xie N, Ling H, Maybank S (2014) Image classification using multiscale information fusion based on saliency driven nonlinear diffusion filtering. IEEE Trans Image Process 23(4):1513–1526

    Article  MathSciNet  Google Scholar 

  13. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis & Machine Intelligence 11:1254–1259

    Article  Google Scholar 

  14. Kim J, Han D, Tai YW, Kim J (2016) Salient region detection via high-dimensional color transform and local spatial support. IEEE Trans Image Process 25(1):9–23

    Article  MathSciNet  Google Scholar 

  15. Kruthiventi SS, Ayush K, Babu RV (2017) Deepfix: a fully convolutional neural network for predicting human eye fixations. IEEE Trans Image Process 26(9):4446–4456

    Article  MathSciNet  Google Scholar 

  16. Li C, Wand M (2015) Approximate translational building blocks for image decomposition and synthesis. ACM Transactions on Graphics (TOG) 34(5), 158

  17. Li H, Chen J, Lu H, Chi Z (2017) Cnn for saliency detection with low-level feature integration. Neurocomputing 226:212–220

    Article  Google Scholar 

  18. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL (2014) Microsoft coco: Common objects in context. In: European conference on computer vision, pp. 740–755. Springer

  19. Ma L, Lin W, Deng C, Ngan KN (2012) Image retargeting quality assessment: a study of subjective scores and objective metrics. IEEE Journal of Selected Topics in Signal Processing 6(6):626–639

    Article  Google Scholar 

  20. Panozzo D,Weber O, Sorkine O (2012) Robust image retargeting via axis-aligned deformation. In: Computer Graphics Forum, vol. 31, pp. 229–236. Wiley Online Library

  21. Pritch Y, Kav-Venaki E, Peleg S (2009) Shift-map image editing. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 151–158. IEEE

  22. Rabbani N, Nazari B, Sadri S, Rikhtehgaran R (2017) Efficient Bayesian approach to saliency detection based on dirichlet process mixture. IET Image Process 11(11):1103–1113

    Article  Google Scholar 

  23. Razzaghi P, Samavi S (2015) Image retargeting using nonparametric semantic segmentation. Multimed Tools Appl 74(24):11517–11536

    Article  Google Scholar 

  24. Rubinstein M, Shamir A, Avidan S (2008) Improved seam carving for video retargeting. In: ACM transactions on graphics (TOG), vol. 27, p. 16. ACM

  25. Rubinstein M, Shamir A, Avidan S (2009) Multi-operator media retargeting. ACM Transactions on graphics (TOG) 28(3), 23

  26. Rubinstein M, Gutierrez D, Sorkine O, Shamir A (2010) A comparative study of image retargeting. In: ACM transactions on graphics (TOG), vol. 29, p. 160. ACM

  27. Shafieyan F, Karimi N, Mirmahboub B, Samavi S, Shirani S (2017) Image retargeting using depth assisted saliency map. Signal Process Image Commun 50:34–43

    Article  Google Scholar 

  28. Shao F, Lin W, Lin W, Jiang Q, Jiang G (2017) QoE-guided warping for stereoscopic image retargeting. IEEE Trans Image Process 26(10):4790–4805

    Article  MathSciNet  Google Scholar 

  29. Simakov D, Caspi Y, Shechtman E, Irani M (2008) Summarizing visual data using bidirectional similarity. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE

  30. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  31. Tang F, Dong W, Meng Y, Ma C, Wu F, Li X, Lee TY (2018) Image retargetability. arXiv preprint arXiv:1802.04392

  32. Tian H, Fang Y, Zhao Y, Lin W, Ni R, Zhu Z (2014) Salient region detection by fusing bottom-up and top-down features extracted from a single image. IEEE Trans Image Process 23(10):4389–4398

    Article  MathSciNet  Google Scholar 

  33. Wang L, Chen C (2016) Study on the image retargeting by using semantic concepts. International Journal of Signal Processing, Image Processing and Pattern Recognition 9(3):281–290

    Article  Google Scholar 

  34. Wang Q, Yuan Y (2014) High quality image resizing. Neurocomputing 131:348–356

    Article  Google Scholar 

  35. Wang YS, Tai CL, Sorkine O, Lee TY (2008) Optimized scale-and-stretch for image resizing. In: ACM Transactions on Graphics (TOG), vol. 27, p. 118. ACM

  36. Wang Q, Yuan Y, Yan P, Li X (2013) Saliency detection by multiple-instance learning. IEEE transactions on cybernetics 43(2):660–672

    Article  Google Scholar 

  37. Wang K, Lin L, Lu J, Li C, Shi K (2015) Pisa: Pixelwise image saliency by aggregating complementary appearance contrast measures with edge-preserving coherence. IEEE Trans Image Process 24(10):3019–3033

    Article  MathSciNet  Google Scholar 

  38. Wang W, Shen J, Yang R, Porikli F (2018) Saliency-aware video object segmentation. IEEE Trans Pattern Anal Mach Intell 40(1):20–33

    Article  Google Scholar 

  39. Wolf L, Guttmann M, Cohen-Or D (2007) Non-homogeneous content-driven video-retargeting. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–6. IEEE

  40. Yan Q, Xu L, Shi J, Jia J (2013) Hierarchical saliency detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1155–1162

  41. Yan B, Li K, Yang X, Hu T (2015) Seam searching-based pixel fusion for image retargeting. IEEE Transactions on Circuits and Systems for Video Technology 25(1):15–23

    Article  Google Scholar 

  42. Zhao H. Pyramid scene parsing network. https://github.com/hszhao/PSPNet/

  43. Zhao R, Ouyang W, Li H, Wang X (2015) Saliency detection by multi-context deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1265–1274

  44. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881–2890

Download references

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Isfahan University of Technology, Isfahan, 84156-83111, Iran

    Mahdi Ahmadi, Nader Karimi & Shadrokh Samavi

  2. Department of Electrical and Computer Engineering, McMaster University, Hamilton, L8S 4L8, Canada

    Shadrokh Samavi

Authors
  1. Mahdi Ahmadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nader Karimi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shadrokh Samavi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shadrokh Samavi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmadi, M., Karimi, N. & Samavi, S. Context-aware saliency detection for image retargeting using convolutional neural networks. Multimed Tools Appl 80, 11917–11941 (2021). https://doi.org/10.1007/s11042-020-10185-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10185-0

Keywords

Search

Navigation