Skip to main content

Advertisement

SpringerLink
Log in

Scale and Object Aware Image Thumbnailing

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In this paper we study effective approaches to create thumbnails from input images. Since a thumbnail will eventually be presented to and perceived by a human visual system, a thumbnailing algorithm should consider several important issues in the process including thumbnail scale, object completeness and local structure smoothness. To address these issues, we propose a new thumbnailing framework named scale and object aware thumbnailing (SOAT), which contains two components focusing respectively on saliency measure and thumbnail warping/cropping. The first component, named scale and object aware saliency (SOAS), models the human perception of thumbnails using visual acuity theory, which takes thumbnail scale into consideration. In addition, the “objectness” measurement (Alexe et al. 2012) is integrated in SOAS, as to preserve object completeness. The second component uses SOAS to guide the thumbnailing based on either retargeting or cropping. The retargeting version uses the thin-plate-spline (TPS) warping for preserving structure smoothness. An extended seam carving algorithm is developed to sample control points used for TPS model estimation. The cropping version searches a cropping window that balances the spatial efficiency and SOAS-based content preservation. The proposed algorithms were evaluated in three experiments: a quantitative user study to evaluate thumbnail browsing efficiency, a quantitative user study for subject preference, and a qualitative study on the RetargetMe dataset. In all studies, SOAT demonstrated promising performances in comparison with state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Notes

  1. We use saliency to indicate the importance measurement used in general, which is not limited to the visual attention-based saliency.

  2. Rigorously speaking, we should use the resulting image \(J\) instead of \(I\) to shrink into the thumbnail size. This however requests the size of \(J\) to be known beforehand which may not be true for some thumbnailing algorithms. In addition, though smaller than \(I\), \(J\) is still much larger than the final thumbnail. Therefore the approximation using \(I\) does not bring significant difference in practice.

  3. http://www.vision.ee.ethz.ch/~calvin/objectness/

  4. There is no overlap between these 30 subjects and the 20 subjects in the experiment of our previous study (Sun and Ling 2011).

  5. http://people.csail.mit.edu/mrub/index.html#code_seamcarving

  6. http://www.dabi.temple.edu/~hbling/code/auto_thumb.zip

References

  • Alexe, B., Deselaers, T., & Ferrari, V. (2012). Measuring the objectness of image windows. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(11), 2189–2202.

    Article  Google Scholar 

  • Avidan, S., & Shamir, A. (2007). Seam carving for content-aware image resizing. ACM Transactions on Graphics, 26(3), 10.

    Article  Google Scholar 

  • Bookstein, F. (1989). Principal warps: thin-plate splines and the decomposition of deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11(6), 567–585.

    Article  MATH  Google Scholar 

  • Chen, L. Q., Xie, X., Fan, X., Ma, W. Y., Zhang, H. J., & Zhou, H. Q. (2003). A visual attention model for adapting images on small displays. Multimedia Systems, 9, 353–364.

    Google Scholar 

  • Chen, R., Freedman, D., Karni, Z., Gotsman, C., & Liu, L. (2010). Content-aware image resizing by quadratic programming. IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), (pp. 1–8).

  • Ding, Y., Xiao, J., Yu, J. (2011). Importance filtering for image retargeting. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 89–96).

  • El-Alfy, H., Jacobs, D., Davis, L. (2007). Multi-scale video cropping. Proceedings of the 15th International Conference on Multimedia, ACM, New York, MULTIMEDIA ’07, (pp. 97–106).

  • Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A. (2008). The PASCAL visual object classes challenge 2008 (VOC2008) results. http://www.pascal-network.org/challenges/VOC/voc2008/workshop/index.html. Accessed 18 Feb 2013.

  • Grundmann, M., Kwatra, V., Han, M., & Essa, I. (2010). Discontinuous seam-carving for video retargeting. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 569–576).

  • Guo, Y., Liu, F., Shi, J., Zhou, Z. H., & Gleicher, M. (2009). Image retargeting using mesh parametrization. IEEE Transactions on Multimedia, 11(5), 856–867.

    Article  Google Scholar 

  • Itti, L., Koch, C., & Niebur, E. (1998). A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(11), 1254–1259.

    Article  Google Scholar 

  • Judd, T., Durand, F., & Torralba, A. (2011). Fixations on low-resolution images. Journal of Vision, 11(4), 1–20.

    Article  Google Scholar 

  • Karni, Z., Freedman, D., Gotsman, C. (2009). Energy-based image deformation. Proceedings of the Symposium on Geometry Processing, Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, SGP ’09, (pp. 1257–1268).

  • Kennedy, L., van Zwol, R., Torzec, N., Tseng, B. (2011) Learning crop regions for content-aware generation of thumbnail images. Proceedings of the 1st ACM International Conference on Multimedia Retrieval, ACM, New York, ICMR ’11, (pp. 30:1–30:8).

  • Kim, J. S., Kim, J. H., & Kim, C. S. (2009). Adaptive image and video retargeting technique based on fourier analysis. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 1730–1737).

  • Krähenbühl, P., Lang, M., Hornung, A., & Gross, M. (2009). A system for retargeting of streaming video. ACM Transactions on Graphics, 28(5), 126:1–126:10.

    Article  Google Scholar 

  • Lam, H., Baudisch, P. (2005). Summary thumbnails: Readable overviews for small screen web browsers. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, New York, CHI ’05, (pp. 681–690).

  • Li, X., Ling, H. (2009). Learning based thumbnail cropping. IEEE International Conference on Multimedia and Expo, (ICME 2009), (pp. 558–561).

  • Liu, F., Gleicher, M. (2005). Automatic image retargeting with fisheye-view warping. Proceedings of the 18th annual ACM symposium on User interface software and technology, ACM, New York, UIST ’05, (pp. 153–162).

  • Liu, F., Gleicher, M. (2006). Region enhanced scale-invariant saliency detection. IEEE International Conference on Multimedia and Expo, (pp. 1477–1480).

  • Luo, Y., Yuan, J., Xue, P., Tian, Q. (2010). Saliency density maximization for object detection and localization. Proceedings of the 10th Asian Conference on Computer vision: Volume Part III, ACCV’10, (pp. 396–408).

  • Mannos, J., & Sakrison, D. (1974). The effects of a visual fidelity criterion of the encoding of images. IEEE Transactions on Information Theory, 20(4), 525–536.

    Article  MATH  Google Scholar 

  • Mansfield, A., Gehler, P., Van Gool, L., Rother, C. (2010). Scene carving: scene consistent image retargeting. Proceedings of the 11th European Conference on Computer Vision: Part I, Springer-Verlag, Berlin, Heidelberg, ECCV’10, (pp. 143–156).

  • Marchesotti, L., Cifarelli, C., Csurka, G. (2009). A framework for visual saliency detection with applications to image thumbnailing. IEEE 12th International Conference on Computer Vision, (pp. 2232–2239).

  • Niu, Y., Liu, F., Li, X., & Gleicher, M. (2012). Image resizing via non-homogeneous warping. Multimedia Tools Application, 56(3), 485–508.

    Article  Google Scholar 

  • Peli, E. (2001). Contrast sensitivity function and image discrimination. Journal of the Optical Society of America A, 18(2), 283–293.

    Article  Google Scholar 

  • Pritch, Y., Kav-Venaki, E., Peleg S. (2009). Shift-map image editing. IEEE 12th International Conference on Computer Vision, (pp. 151–158).

  • Rubinstein, M., Shamir, A., & Avidan, S. (2008). Improved seam carving for video retargeting. ACM Transactions on Graphics, 27(3), 16:1–16:9.

    Article  Google Scholar 

  • Rubinstein, M., Shamir, A., & Avidan, S. (2009). Multi-operator media retargeting. ACM Transactions on Graphics, 28(3), 23:1–23:11.

    Article  Google Scholar 

  • Rubinstein, M., Gutierrez, D., Sorkine, O., Shamir, A. (2010a). A benchmark for image retargeting. http://people.csail.mit.edu/mrub/retargetme/.

  • Rubinstein, M., Gutierrez, D., Sorkine, O., & Shamir, A. (2010b). A comparative study of image retargeting. ACM Transactions on Graphics, 29(6), 160:1–160:10.

    Article  Google Scholar 

  • Simakov, D., Caspi, Y., Shechtman, E., & Irani, M. (2008). Summarizing visual data using bidirectional similarity. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (pp. 1–8).

  • Suh, B., Ling, H., Bederson, B.B., Jacobs, D.W. (2003). Automatic thumbnail cropping and its effectiveness. Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology, ACM, New York, UIST ’03, (pp. 95–104).

  • Sun, J., Ling, H. (2011). Scale and object aware image retargeting for thumbnail browsing. IEEE International Conference on Computer Vision (ICCV), (pp. 1511–1518).

  • Van Nes, F. L., & Bouman, M. A. (1967). Spatial modulation transfer in the human eye. Journal of the Optical Society of America, 57(3), 401–406.

    Article  Google Scholar 

  • Wang, Y., & Zhu, S. C. (2008). Perceptual scale-space and its applications. International Journal of Computer Vision, 80(1), 143–165.

    Article  Google Scholar 

  • Wang, Y. S., Tai, C. L., Sorkine, O., & Lee, T. Y. (2008). Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics, 27(5), 118:1–118:8.

    Google Scholar 

  • Wang, Y. S., Lin, H. C., Sorkine, O., & Lee, T. Y. (2010). Motion-based video retargeting with optimized crop-and-warp. ACM Transactions on Graphics, 29, 90:1–90:9.

    Google Scholar 

  • Wolf, L., Guttmann, M., Cohen-Or, D. (2007). Non-homogeneous content-driven video-retargeting. IEEE 11th International Conference on Computer Vision (ICCV), (pp. 1–6).

  • Wu, H., Wang, Y. S., Feng, K. C., Wong, T. T., Lee, T. Y., & Heng, P. A. (2010). Resizing by symmetry-summarization. ACM Transactions on Graphics, 29(6), 159:1–159:10.

    Article  Google Scholar 

Download references

Acknowledgments

We thank B. Alexe for making the objectness code available, M. Rubinstein for seam carving code, G. Teodoro for proofreading the manuscript, and all anonymous reviewers for valuable suggestions which help significantly improve the manuscript. This work was supported in part by NSF Grant IIS-1218156.

Author information

Authors and Affiliations

  1. Department of Computer and Information Sciences, Temple University, 1805 N Broad St., Philadelphia, PA, USA

    Jin Sun & Haibin Ling

Authors
  1. Jin Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haibin Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haibin Ling.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Sun, J., Ling, H. Scale and Object Aware Image Thumbnailing. Int J Comput Vis 104, 135–153 (2013). https://doi.org/10.1007/s11263-013-0618-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-013-0618-z

Keywords

Search

Navigation