CMAIR: content and mask-aware image retargeting

  • Hon-Hang ChangEmail author
  • Timothy K. Shih
  • Carl K. Chang
  • Wallapak Tavanapong


In recent years, more and more image retargeting techniques have been proposed to facilitate our daily life, in particular those based on the use of seam carving, warping or the combination of them. However, these techniques can only retarget the source picture into the same shape of a square, and these approaches cannot be reshape into a circular, a polygon or other shapes. This paper focuses on creating a graphics editing system, named CMAIR (Content and Mask-Aware Image Retargeting), for image retargeting, which can retarget the source images into different shapes of image to highlight the salient objects of primary region of interest. CMAIR effectively supports removal of unimportant pixels, and frames as many surrounding objects inside the provided mask as possible. Also, we propose a unique irregular interpolation method to produce four possible target images, and an evaluation mechanism to decide the best candidate image as the final output with the consideration of image saliency. The results show that not only the source image can be placed into different targeted shapes of mask, but also the salient objects are retained and highlighted as much as possible.


Image retargeting Seam carving Irregular interpolation Image saliency Region of interest 



The authors would like to express thanks to National Central University, Taiwan and the Iowa State University, America for providing a comfortable experimental environment and adequate research facilities.


  1. 1.
    Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. In: ACM Transactions on Graphics (TOG), vol 26. ACM, pp 10Google Scholar
  2. 2.
    Barnes C, Shechtman E, Finkelstein A, Goldman DB (2009) Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graph 28(3):24–1CrossRefGoogle Scholar
  3. 3.
    Bo Y, Sun K, Liu L (2013) Matching-area-based seam carving for video retargeting. IEEE Trans Circuits Syst Video Technol 23(2):302–310CrossRefGoogle Scholar
  4. 4.
    Bo Y, Yuan B, Bo Y (2014) Effective video retargeting with jittery assessment. IEEE Trans Multimedia 16(1):272–277CrossRefGoogle Scholar
  5. 5.
    Castillo S, Judd T, Gutierrez D (2011) Using eye-tracking to assess different image retargeting methods. In: Proceedings of the ACM SIGGRAPH Symposium on Applied Perception in Graphics and Visualization, ACM, pp 7–14Google Scholar
  6. 6.
    Cho S, Choi H, Matsushita Y, Lee S (2009) Image retargeting using importance diffusion. In: 2009 16th IEEE International Conference on Image processing (ICIP), IEEE, pp 977–980Google Scholar
  7. 7.
    Deng C, Lin W, Cai J (2012) Content-based image compression for arbitrary-resolution display devices. IEEE Trans Multimedia 14(4):1127–1139CrossRefGoogle Scholar
  8. 8.
    Dong W, Zhou Ning, Paul J-C, Zhang X (2009) Optimized image resizing using seam carving and scaling. In: ACM Transactions on Graphics (TOG), vol 28, ACM, pp 125Google Scholar
  9. 9.
    Fan X, Xie X, Zhou H-Q, Ma W-Y (2003) Looking into video frames on small displays. In: Proceedings of the Eleventh ACM International Conference on Multimedia, ACM, pp 247–250Google Scholar
  10. 10.
    Fang Y, Chen Z, Lin W, Lin C-W (2012) Saliency detection in the compressed domain for adaptive image retargeting. IEEE Trans Image Process 21(9):3888–3901MathSciNetCrossRefGoogle Scholar
  11. 11.
    Fang Y, Lin W, Lee B-S, Lau C-T, Chen Z, Lin C-W (2012) Bottom-up saliency detection model based on human visual sensitivity and amplitude spectrum. IEEE Trans Multimedia 14(1):187–198CrossRefGoogle Scholar
  12. 12.
    Fang Y, Zeng K, Wang Z, Lin W, Fang Z, Lin C-W (2014) Objective quality assessment for image retargeting based on structural similarity. cIEEE J Emerging Sel Top Circuits Syst 4(1):95–105CrossRefGoogle Scholar
  13. 13.
    Grundmann Matthias, Kwatra Vivek, Han Mei, Essa Irfan (2010) Discontinuous seam-carving for video retargeting. In: 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp 569–576Google Scholar
  14. 14.
    Guo Y, Liu F, Shi J, Zhou Z-H, Gleicher M (2009) Image retargeting using mesh parametrization. IEEE Trans Multimedia 11(5):856–867CrossRefGoogle Scholar
  15. 15.
    Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259CrossRefGoogle Scholar
  16. 16.
    Kiess J, Kopf S, Guthier B, Effelsberg W (2010) Seam carving with improved edge preservation. In: ISAndamp;t/SPIE electronic imaging, International Society for Optics and Photonics, pp 75420g–75420gGoogle Scholar
  17. 17.
    Kopf S, Guthier B, Lemelson H, Effelsberg W (2009) Adaptation of web pages and images for mobile applications. In: ISAndamp;t/SPIE Electronic Imaging, International Society for Optics and Photonics, pp 72560c–72560cGoogle Scholar
  18. 18.
    Lee J-H, Kim K, Park J, Park JY, Jung S-W (2016) Image recomposition using seam carving and insertion considering the rule of thirds. IEIE Transaction on Smart Processing and Computing 5(1):1–4CrossRefGoogle Scholar
  19. 19.
    Li B, Duan L-Y, Wang J, Ji R, Lin C-W, Gao W (2014) Spatiotemporal grid flow for video retargeting. IEEE Trans Image Process 23(4):1615–1628MathSciNetCrossRefGoogle Scholar
  20. 20.
    Liang Y, Liu Y-J, Gutierrez D (2017) Objective quality prediction of image retargeting algorithms. IEEE Trans Vis Comput Graph 23(2):1099–1110CrossRefGoogle Scholar
  21. 21.
    Lin S-S, Yeh I-C, Lin C-H, Lee T-Y (2013) Patch-based image warping for content-aware retargeting. IEEE Trans Multimedia 15(2):359–368CrossRefGoogle Scholar
  22. 22.
    Liu F, Gleicher M (2005) Automatic image retargeting with fisheye-view warping. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, ACM, pp 153– 162Google Scholar
  23. 23.
    Liu F, Gleicher M (2006) Video retargeting: automating pan and scan. In: Proceedings of the 14th ACM International Conference on Multimedia, ACM, pp 241–250Google Scholar
  24. 24.
    Liu Hao, Xie Xing, Ma Wei-Ying, Zhang Hong-Jiang (2003) Automatic browsing of large pictures on mobile devices. In: Proceedings of the Eleventh ACM International Conference on Multimedia, ACM, pp 148–155Google Scholar
  25. 25.
    Liu Y-J, Xi L, Xuan Y-M, Chen W-F, Fu X-L (2011) Image retargeting quality assessment. In: Computer Graphics Forum, vol 30. Wiley online library, pp 583–592Google Scholar
  26. 26.
    Ma L, Lin W, Deng C, Ngan KN (2012) Image retargeting quality assessment: a study of subjective scores and objective metrics. IEEE J Sel Top Sign Proces 6(6):626–639CrossRefGoogle Scholar
  27. 27.
    Ma L, Lin W, Deng C, Ngan KN (2012) Study of subjective and objective quality assessment of retargeted images. In: 2012 IEEE International Symposium on Circuits and Systems (ISCAS), IEEE, pp 2677–2680Google Scholar
  28. 28.
    Ma L, Xu L, Zhang Y, Yan Y, Ngan KN (2016) No-reference retargeted image quality assessment based on pairwise rank learning. IEEE Trans Multimedia 18 (11):2228–2237CrossRefGoogle Scholar
  29. 29.
    Ni B, Xu M, Cheng B, Wang M, Yan S, Qi T. (2013) Learning to photograph a compositional perspective. IEEE Trans Multimedia 15(5):1138–1151CrossRefGoogle Scholar
  30. 30.
    Pritch Y, Kav-Venaki E, Peleg S (2009) Shift-map image editing. In: 2009 IEEE 12th International Conference on Computer Vision, IEEE, pp 151–158Google Scholar
  31. 31.
    Qi S, Chi Y-TJ, Peter AM, Ho J (2016) Casair: Content and shape-aware image retargeting and its applications. IEEE Trans Image Process 25(5):2222–2232MathSciNetCrossRefGoogle Scholar
  32. 32.
    Qu Z, Wang J, Xu M, Lu H (2013) Context-aware video retargeting via graph model. IEEE Trans Multimedia 15(7):1677–1687CrossRefGoogle Scholar
  33. 33.
    Rubinstein M, Gutierrez D, Sorkine O, Shamir A (2010) A comparative study of image retargeting. In: ACM Transactions on Graphics (TOG), vol 29. ACM, pp 160Google Scholar
  34. 34.
    Rubinstein M, Shamir A, Avidan S (2008) Improved seam carving for video retargeting. In: ACM Transactions on Graphics (TOG), vol 27. ACM, pp 16Google Scholar
  35. 35.
    Rubinstein M, Shamir A, Avidan S (2009) Multi-operator media retargeting. In: ACM Transactions on Graphics (TOG), vol 28, ACM, pp 23Google Scholar
  36. 36.
    Santella A, Agrawala M, DeCarlo D, Salesin D, Cohen M (2006) Gaze-based interaction for semi-automatic photo cropping. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, pp 771–780Google Scholar
  37. 37.
    Shamir A, Sorkine O (2009) Visual media retargeting. In: ACM SIGGRAPH ASIA 2009 Courses, ACM, pp 11Google Scholar
  38. 38.
    Simakov Denis, Caspi Yaron, Shechtman Eli, Irani Michal (2008) Summarizing visual data using bidirectional similarity. In: 2008. CVPR 2008. IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1–8Google Scholar
  39. 39.
    Suh B, Ling H, Bederson BB, Jacobs DW (2003) Automatic thumbnail cropping and its effectiveness. In: Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, ACM, pp 95–104Google Scholar
  40. 40.
    Tao C, Jia J, Sun H (2007) Active window oriented dynamic video retargeting. In: Proceedings of the Workshop on Dynamical Vision, ICCVGoogle Scholar
  41. 41.
    Wang Y-S, Fu H, Sorkine O, Lee T-Y, Seidel H-P (2009) Motion-aware temporal coherence for video resizing. ACM Trans Graph (TOG) 28(5):127Google Scholar
  42. 42.
    Wang Y-S, Tai C-L, Sorkine O, Lee T-Y (2008) Optimized scale-and-stretch for image resizing. In: ACM Transactions on Graphics (TOG), vol 27. ACM, pp 118Google Scholar
  43. 43.
    Wolf L, Guttmann M, Cohen-Or D (2007) Non-homogeneous content-driven video-retargeting. In: 2007. ICCV 2007. IEEE 11th International Conference on Computer Vision, IEEE, pp 1–6Google Scholar
  44. 44.
    Wu L, Cao L, Min X, Wang J (2014) A hybrid image retargeting approach via combining seam carving and grid warping. J Multimed 9(4):483–492Google Scholar
  45. 45.
    Wu L, Gong Y, Yuan X, Zhang X, Cao L (2014) Semantic aware sport image resizing jointly using seam carving and warping. Multimed Tools Appl 70 (2):721–739CrossRefGoogle Scholar
  46. 46.
    Yen T-C, Tsai C-M, Lin C-W (2011) Maintaining temporal coherence in video retargeting using mosaic-guided scaling. IEEE Trans Image Process 20(8):2339–2351MathSciNetCrossRefGoogle Scholar
  47. 47.
    Zhang L, Wang M, Nie L, Hong L, Rui Y, Qi T (2015) Retargeting semantically-rich photos. IEEE Trans Multimedia 17(9):1538–1549CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Taoyuan CityTaiwan (R.O.C.)
  2. 2.Iowa State UniversityAmesUSA
  3. 3.Iowa State UniversityAmesUSA

Personalised recommendations