Skip to main content
SpringerLink
Log in

Introducing Image Saliency Information into Content Based Indexing and Emotional Impact Analysis

  • Chapter
  • First Online:
Visual Content Indexing and Retrieval with Psycho-Visual Models

Part of the book series: Multimedia Systems and Applications ((MMSA))

  • 447 Accesses

Abstract

We propose in this chapter to highlight the impact of visual saliency information in Content Based Image Retrieval (CBIR) systems. We firstly present results of subjective evaluations for emotion analysis with and without use of saliency to reduce the image size and conclude that image reduction to more salient regions implies a better evaluation of emotional impact. We also test eye-tracking methods to validate our results and conclusions. Those experiments lead us to study saliency to improve the image description for indexing purpose. We first show the influence of selecting salient features for relevant image indexing and retrieval. Then, we propose a novel approach that makes use of saliency in an information gain criterion to improve the selection of a visual dictionary in the well-known Bags of Visual Words approach. Our experiments will underline the effectiveness of the proposal. Finally, we present some results on emotional impact recognition using CBIR descriptors and Bags of Visual Words approach with image saliency information.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Our saliency values are computed using the Graph-Based Visual Saliency (GBVS) software http://www.klab.caltech.edu/~harel/share/gbvs.php which implements also Itti et al.’s algorithm.

  2. 2.

    Those from the chosen detectors.

  3. 3.

    We used the descriptors provided by Jegou et al. available at http://lear.inrialpes.fr/people/jegou/data.php.

References

  1. Abdel-Hakim, A.E., Farag, A.A.: CSIFT: A SIFT Descriptor with color invariant characteristics. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2006)

    Google Scholar 

  2. Bay, H., Tuytelaars, T., Van Gool, L.: Surf: speeded up robust features. Lecture Notes in Computer Science, vol. 3951, pp. 404–417. Springer, Berlin (2006)

    Google Scholar 

  3. Beke, L., Kutas, G., Kwak, Y., Sung, G.Y., Park, D., Bodrogi, P.: Color preference of aged observers compared to young observers. Color. Res. Appl. 33(5), 381–394 (2008)

    Article  Google Scholar 

  4. Borji, A., Sihite, D., Itti, L.: Quantitative analysis of human-model agreement in visual saliency modeling: A comparative study. IEEE Trans. Image Process. 22(1), 55–69 (2013)

    Article  MathSciNet  Google Scholar 

  5. Boyatziz, C., Varghese, R.: Children’s emotional associations with colors. J. Gen. Psychol. 155, 77–85 (1993)

    Article  Google Scholar 

  6. Bradley, M.M., Codispoti, M., Sabatinelli, D., Lang, P.J.: Emotion and motivation ii: sex differences in picture processing. Emotion 1(3), 300–319 (2001)

    Article  Google Scholar 

  7. Csurka, G., Bray, C., Dance, C., Fan, L.: Visual categorization with bags of keypoints. In: Workshop on Statistical Learning in Computer Vision, ECCV, pp. 1–22 (2004)

    Google Scholar 

  8. Denis, P., Courboulay, V., Revel, A., Gbehounou, S., Lecellier, F., Fernandez-Maloigne, C.: Improvement of natural image search engines results by emotional filtering. EAI Endorsed Trans. Creative Technologies 3(6), e4 (2016). https://hal.archives-ouvertes.fr/hal-01261237

    Article  Google Scholar 

  9. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)

    Article  Google Scholar 

  10. Gao, K., Lin, S., Zhang, Y., Tang, S., Ren, H.: Attention model based sift keypoints filtration for image retrieval. In: Proceedings of IEEE International Conference on Computer and Information Science, pp. 191–196 (2008)

    Google Scholar 

  11. Gbèhounou, S., Lecellier, F., Fernandez-Maloigne, C., Courboulay, V.: Can Salient Interest Regions Resume Emotional Impact of an Image?, pp. 515–522 Springer, Berlin (2013). doi:10.1007/978-3-642-40261-6_62. http://dx.doi.org/10.1007/978-3-642-40261-6_62

  12. Gbehounou, S., Lecellier, F., Fernandez-Maloigne, C.: Evaluation of local and global descriptors for emotional impact recognition. J. Vis. Commun. Image Represent. 38, 276–283 (2016)

    Article  Google Scholar 

  13. Gbèhounou, S., Lecellier, F., Fernandez-Maloigne, C.: Evaluation of local and global descriptors for emotional impact recognition. J. Vis. Commun. Image Represent. 38(C), 276–283 (2016). doi:10.1016/j.jvcir.2016.03.009. http://dx.doi.org/10.1016/j.jvcir.2016.03.009

  14. González-Díaz, I., Buso, V., Benois-Pineau, J.: Perceptual modeling in the problem of active object recognition in visual scenes. Pattern Recognition 56, 129–141 (2016). doi:10.1016/j.patcog.2016.03.007. http://dx.doi.org/10.1016/j.patcog.2016.03.007

  15. Gordoa, A., Rodriguez-Serrano, J.A., Perronnin, F., Valveny, E.: Leveraging category-level labels for instance-level image retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3045–3052 (2012)

    Google Scholar 

  16. Harel, J., Koch, C., Perona, P.: Graph-based visual saliency. In: Advances in Neural Information Processing Systems, pp. 545–552. MIT Press, Cambridge (2007)

    Google Scholar 

  17. Harris, C., Stephens, M.: A combined corner and edge detector. In: Proceedings of the 4th Alvey Vision Conference, pp. 147–151 (1988)

    Google Scholar 

  18. Itti, L., Koch, C., Niebur, E.: A model of saliency-based visual attention for rapid scene analysis. IEEE Trans. Pattern Anal. Mach. Intell. 20(11), 1254–1259 (1998)

    Article  Google Scholar 

  19. Jegou, H., Douze, M., Schmid, C.: Hamming embedding and weak geometric consistency for large scale image search. In: Proceedings of the 10th European Conference on Computer Vision: Part I, ECCV’08, pp. 304–317. Springer, Berlin (2008)

    Google Scholar 

  20. Jégou, H., Douze, M., Schmid, C., Pérez, P.: Aggregating local descriptors into a compact image representation. In: Proceedings of the 23rd IEEE Conference on Computer Vision & Pattern Recognition, pp. 3304–3311. IEEE Computer Society, New York (2010)

    Google Scholar 

  21. Kaya, N., Epps, H.H.: Color-emotion associations: Past experience and personal preference. In: AIC Colors and Paints, Interim Meeting of the International Color Association (2004)

    Google Scholar 

  22. Ke, Y., Sukthankar, R.: PCA-SIFT: a more distinctive representation for local image descriptors. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 506–513 (2004)

    Google Scholar 

  23. Kootstra, G., de Boer, B., Schomaker, L.: Predicting eye fixations on complex visual stimuli using local symmetry. Cogn. Comput. 3(1), 223–240 (2011)

    Article  Google Scholar 

  24. Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International affective picture system (IAPS): affective ratings of pictures and instruction manual. technical report A-8. Technical Report, University of Florida (2008)

    Google Scholar 

  25. Le Meur, O., Le Callet, P., Barba, D., Thoreau, D.: A coherent computational approach to model bottom-up visual attention. IEEE Trans. Pattern Anal. Mach. Intell. 28(5), 802–817 (2006)

    Article  Google Scholar 

  26. Li, Y., Zhou, Y., Yan, J., Niu, Z., Yang, J.: Visual saliency based on conditional entropy. Lecture Notes in Computer Science, vol. 5994, pp. 246–257. Springer, Berlin (2010)

    Google Scholar 

  27. Liu, W., Xu, W., Li, L.: A tentative study of visual attention-based salient features for image retrieval. In: Proceedings of the 7th World Congress on Intelligent Control and Automation, pp. 7635–7639 (2008)

    Google Scholar 

  28. Liu, N., Dellandréa, E., Chen, L.: Evaluation of features and combination approaches for the classification of emotional semantics in images. In: International Conference on Computer Vision Theory and Applications (2011)

    Google Scholar 

  29. Lowe, D.G.: Object recognition from local scale-invariant features. In: International Conference on Computer Vision, vol. 2, pp. 1150–1157 (1999)

    Google Scholar 

  30. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2004)

    Article  Google Scholar 

  31. Lucassen, M.P., Gevers, T., Gijsenij, A.: Adding texture to color: quantitative analysis of color emotions. In: Proceedings of CGIV (2010)

    Google Scholar 

  32. Machajdik, J., Hanbury, A.: Affective image classification using features inspired by psychology and art theory. In: Proceedings of the international conference on Multimedia, pp. 83–92 (2010)

    Google Scholar 

  33. Mikolajczyk, K., Schmid, C.: Indexing based on scale invariant interest points. In: Proceedings of the 8th IEEE International Conference on Computer Vision, vol. 1, pp. 525–531 (2001)

    Google Scholar 

  34. Mikolajczyk, K., Schmid, C.: An affine invariant interest point detector. In: Computer Vision-ECCV. Lecture Notes in Computer Science, vol. 2350, pp. 128–142. Springer, Berlin (2002)

    Google Scholar 

  35. Mikolajczyk, K., Tuytelaars, T., Schmid, C., Zisserman, A., Matas, J., Schaffalitzky, F., Kadir, T., Van Gool, L.: A comparison of affine region detectors. Int. J. Comput. Vision 65(1-2), 43–72 (2005)

    Article  Google Scholar 

  36. Mindru, F., Tuytelaars, T., Van Gool, L., Moons, T.: Moment invariants for recognition under changing viewpoint and illumination. Comput. Vis. Image Underst. 94(1–3), 3–27 (2004)

    Article  Google Scholar 

  37. Moravec, H.P.: Towards automatic visual obstacle avoidance. In: Proceedings of the 5th International Joint Conference on Artificial Intelligence, vol. 2, pp. 584–584. Morgan Kaufmann, San Francisco (1977)

    Google Scholar 

  38. Nauge, M., Larabi, M.C., Fernandez-Maloigne, C.: A statistical study of the correlation between interest points and gaze points. In: Human Vision and Electronic Imaging, p. 12. Burlingame (2012)

    Google Scholar 

  39. Nistér, D., Stewénius, H.: Scalable recognition with a vocabulary tree. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 2161–2168 (2006)

    Google Scholar 

  40. Ou, L.C., Luo, M.R., Woodcock, A., Wright, A.: A study of colour emotion and colour preference. part i: Colour emotions for single colours. Color. Res. Appl. 29(3), 232–240 (2004)

    Google Scholar 

  41. Ou, L.C., Luo, M.R., Woodcock, A., Wright, A.: A study of colour emotion and colour preference. part ii: Colour emotions for two-colour combinations. Color. Res. Appl. 29(4), 292–298 (2004)

    Google Scholar 

  42. Ou, L.C., Luo, M.R., Woodcock, A., Wright, A.: A study of colour emotion and colour preference. Part iii: colour preference modeling. Color. Res. Appl. 29(5), 381–389 (2004)

    Google Scholar 

  43. Paleari, M., Huet, B.: Toward emotion indexing of multimedia excerpts. In: Proceedings on Content-Based Multimedia Indexing, International Workshop, pp. 425–432 (2008)

    Google Scholar 

  44. Perreira Da Silva, M., Courboulay, V., Prigent, A., Estraillier, P.: Evaluation of preys/predators systems for visual attention simulation. In: Proceedings of the International Conference on Computer Vision Theory and Applications, pp. 275–282, INSTICC (2010)

    Google Scholar 

  45. Perronnin, F.: Universal and adapted vocabularies for generic visual categorization. IEEE Trans. Pattern Anal. Mach. Intell. 30(7), 1243–1256 (2008)

    Article  Google Scholar 

  46. Philbin, J., Chum, O., Isard, M., Sivic, J., Zisserman, A.: Object retrieval with large vocabularies and fast spatial matching. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, MI (2007)

    Google Scholar 

  47. Rosten, E., Drummond, T.: Fusing points and lines for high performance tracking. In: Proceedings of the IEEE International Conference on Computer Vision, vol. 2, pp. 1508–1511 (2005)

    Google Scholar 

  48. Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Proceedings of the European Conference on Computer Vision, vol. 1, pp. 430–443 (2006)

    Google Scholar 

  49. Schmid, C., Mohr, R., Bauckhage, C.: Evaluation of interest point detectors. Int. J. Comput. Vision 37(2), 151–172 (2000)

    Article  MATH  Google Scholar 

  50. Sivic, J., Zisserman, A.: Video Google: A text retrieval approach to object matching in videos. In: Proceedings of the International Conference on Computer Vision, pp. 1470–1477 (2003)

    Google Scholar 

  51. Smith, S.M., Brady, J.M.: Susan—a new approach to low level image processing. Int. J. Comput. Vision 23(1), 45–78 (1997)

    Article  Google Scholar 

  52. Solli, M., Lenz, R.: Color harmony for image indexing. In: Proceedings of the 12th International Conference on Computer Vision Workshops, pp. 1885–1892 (2009)

    Google Scholar 

  53. Solli, M., Lenz, R.: Emotion related structures in large image databases. In: Proceedings of the ACM International Conference on Image and Video Retrieval, pp. 398–405. ACM, New York (2010)

    Google Scholar 

  54. Tuytelaars, T., Mikolajczyk, K.: Local invariant feature detectors: a survey. Found. Trends Comput. Graph. Vis. 3(3), 177–280 (2008)

    Article  Google Scholar 

  55. Urruty, T., Gbèhounou, S., Le, T.L., Martinet, J., Fernandez-Maloigne, C.: Iterative random visual words selection. In: Proceedings of International Conference on Multimedia Retrieval, ICMR’14, pp. 249–256. ACM, New York (2014)

    Google Scholar 

  56. van de Sande, K.E.A., Gevers, T., Snoek, C.G.M.: Evaluating color descriptors for object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1582–1596 (2010)

    Article  Google Scholar 

  57. Wang, W., Yu, Y.: Image emotional semantic query based on color semantic description. In: Proceedings of the The 4th International Conference on Machine Leraning and Cybernectics, vol. 7, pp. 4571–4576 (2005)

    Google Scholar 

  58. Wei, K., He, B., Zhang, T., He, W.: Image Emotional classification based on color semantic description. Lecture Notes in Computer Science, vol. 5139, pp. 485–491. Springer, Berlin (2008)

    Google Scholar 

  59. Yanulevskaya, V., Van Gemert, J.C., Roth, K., Herbold, A.K., Sebe, N., Geusebroek, J.M.: Emotional valence categorization using holistic image features. In: Proceedings of the 15th IEEE International Conference on Image Processing, pp. 101–104 (2008)

    Google Scholar 

  60. Zdziarski, Z., Dahyot, R.: Feature selection using visual saliency for content-based image retrieval. In: Proceedings of the IET Irish Signals and Systems Conference, pp. 1–6 (2012)

    Google Scholar 

  61. Zhang, J., Marszalek, M., Lazebnik, S., Schmid, C.: Local features and kernels for classification of texture and object categories: a comprehensive study. Int. J. Comput. Vis. 73(2), 213–238 (2007)

    Article  Google Scholar 

  62. Zhang, L., Tong, M.H., Marks, T.K., Shan, H., Cottrell, G.W.: Sun: A Bayesian framework for saliency using natural statistics. J. Vis. 8(7), 1–20 (2008)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Jules SAS, Blagnac, France

    Syntyche Gbehounou

  2. Xlim, University of Poitiers, CNRS, Poitiers, France

    Thierry Urruty, François Lecellier & Christine Fernandez-Maloigne

Authors
  1. Syntyche Gbehounou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thierry Urruty
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. François Lecellier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christine Fernandez-Maloigne
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to François Lecellier .

Editor information

Editors and Affiliations

  1. LaBRI UMR 5800, Univ. Bordeaux, CNRS, Bordeaux INP, Univ. Bordeaux, Talence, France

    Jenny Benois-Pineau

  2. LS2N, UMR CNRS 6004, Université de Nantes, Nantes Cedex 3, France

    Patrick Le Callet

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Gbehounou, S., Urruty, T., Lecellier, F., Fernandez-Maloigne, C. (2017). Introducing Image Saliency Information into Content Based Indexing and Emotional Impact Analysis. In: Benois-Pineau, J., Le Callet, P. (eds) Visual Content Indexing and Retrieval with Psycho-Visual Models. Multimedia Systems and Applications. Springer, Cham. https://doi.org/10.1007/978-3-319-57687-9_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-57687-9_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-57686-2

  • Online ISBN: 978-3-319-57687-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation