Saliency-based selection of visual content for deep convolutional neural networks

Application to architectural style classification
  • A. Montoya ObesoEmail author
  • J. Benois-Pineau
  • M. S. García Vázquez
  • A. A. Ramírez Acosta


The automatic description of digital multimedia content was mainly developed for classification tasks, retrieval systems and massive ordering of data. Preservation of cultural heritage is a field of high importance of application of these methods. We address classification problem in cultural heritage such as classification of architectural styles in digital photographs of Mexican cultural heritage. In general, the selection of relevant content in the scene for training classification models makes the models more efficient in terms of accuracy and training time. Here we use a saliency-driven approach to predict visual attention in images and use it to train a Deep Convolutional Neural Network. Also, we present an analysis of the behavior of the models trained under the state-of-the-art image cropping and the saliency maps. To train invariant models to rotations, data augmentation of training set is required, which posses problems of filling normalization of crops, we study were different padding techniques and we find an optimal solution. The results are compared with the state-of-the-art in terms of accuracy and training time. Furthermore, we are studying saliency cropping in training and generalization for another classical task such as weak labeling of massive collections of images containing objects of interest. Here the experiments are conducted on a large subset of ImageNet database. This work is an extension of preliminary research in terms of image padding methods and generalization on large scale generic database.


Data selection Visual attention prediction Cultural heritage Deep learning 



This work was sponsored by CONACYT and SIP2017.


  1. 1.
    Alexe B, Deselaers T, Ferrari V (2012) Measuring the objectness of image windows. IEEE Trans Pattern Anal Mach Intell 34(11):2189–2202CrossRefGoogle Scholar
  2. 2.
    Ali H, Seifert C, Jindal N, Paletta L, Paar G (2007) Window detection in facades. In: 2007 14th international conference on image analysis and processing, ICIAP 2007. IEEE, pp 837–842Google Scholar
  3. 3.
    Benois-Pineau J, Callet PL (eds) (2017) Visual content indexing and retrieval with psychovisual models. Springer, HeidelbergGoogle Scholar
  4. 4.
    Benois-Pineau J, Mitrea M (2017) Extraction of saliency in images and video: Problems, methods and applications. A survey. In: 2017 Seventh international conference on image processing theory, tools and applications (IPTA). IEEE, Montreal, Canada.
  5. 5.
    Berg AC, Grabler F, Malik J (2007) Parsing images of architectural scenes. In: IEEE 11th international conference on 2007 computer vision, ICCV 2007. IEEE, pp 1–8Google Scholar
  6. 6.
    Bhowmik N, Gouet-Brunet V, Bloch G, Besson S (2017) Combination of image descriptors for the exploration of cultural photographic collections. J Electron Imag 26(1):011,019–011,019CrossRefGoogle Scholar
  7. 7.
    Buso V, González-díaz I, Benois-Pineau J (2015) Goal-oriented top-down probabilistic visual attention model for recognition of manipulated objects in egocentric videos. Sig Proc Image Commun 39:418–431. CrossRefGoogle Scholar
  8. 8.
    Buswell GT (1935) How people look at pictures: a study of the psychology and perception in artGoogle Scholar
  9. 9.
    Bylinskii Z, Recasens A, Borji A, Oliva A, Torralba A, Durand F (2016) Where should saliency models look next?. In: European conference on computer vision. Springer, pp 809–824Google Scholar
  10. 10.
    de Carvalho Soares R, da Silva IR, Guliato D (2012) Spatial locality weighting of features using saliency map with a bag-of-visual-words approach. In: 2012 IEEE 24th international conference on tools with artificial intelligence (ICTAI), vol 1. IEEE, pp 1070–1075Google Scholar
  11. 11.
    De San Roman PP, Benois-Pineau J, Domenger JP, De Rugy A, Paclet F, Cataert D (2017) Saliency driven object recognition in egocentric videos with deep cnn: toward application in assistance to neuroprostheses Computer Vision and Image UnderstandingGoogle Scholar
  12. 12.
    Ghodrati A, Diba A, Pedersoli M, Tuytelaars T, Van Gool L (2017) Deepproposals: hunting objects and actions by cascading deep convolutional layers. Int J Comput Vis 124(2):115–131. CrossRefGoogle Scholar
  13. 13.
    Girshick R (2015) Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision, pp 1440–1448Google Scholar
  14. 14.
    Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 580–587Google Scholar
  15. 15.
    González-Díaz I, Buso V, Benois-Pineau J (2016) Perceptual modeling in the problem of active object recognition in visual scenes. Pattern Recogn 56:129–141CrossRefGoogle Scholar
  16. 16.
    GPU NDIDL (2015) Training systemGoogle Scholar
  17. 17.
    Harel J, Koch C, Perona P (2007) Graph-based visual saliency. In: Advances in neural information processing systems, pp 545–552Google Scholar
  18. 18.
    He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778Google Scholar
  19. 19.
    Howard AG (2013) Some improvements on deep convolutional neural network based image classification. arXiv:1312.5402
  20. 20.
    Itti L, Koch C (2001) Computational modelling of visual attention. Nature Rev Neuroscience 2(3):194CrossRefGoogle Scholar
  21. 21.
    Jia Y, Shelhamer E, Donahue J, Karayev S, Long J, Girshick R, Guadarrama S, Darrell T (2014) Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM international conference on multimedia. ACM, pp 675–678Google Scholar
  22. 22.
    Krizhevsky A, Sutskever I, Hinton G (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105Google Scholar
  23. 23.
    Liu Z, Wang J, Liu W (2005) Building extraction from high resolution imagery based on multi-scale object oriented classification and probabilistic hough transform. In: 2005 Proceedings of the IEEE international geoscience and remote sensing symposium, 2005. IGARSS’05, vol 4. IEEE, pp 2250–2253Google Scholar
  24. 24.
    Llamas J, Lerones PM, Zalama E, Gómez-garcía-bermejo J (2016) Applying deep learning techniques to cultural heritage images within the inception project. In: Euro-mediterranean conference. Springer, pp 25–32Google Scholar
  25. 25.
    Mahadevan V, Vasconcelos N (2013) Biologically inspired object tracking using center-surround saliency mechanisms. IEEE Trans Pattern Anal Mach Intell 35 (3):541–554CrossRefGoogle Scholar
  26. 26.
    Mathe S, Sminchisescu C (2012) Dynamic eye movement datasets and learnt saliency models for visual action recognition. Computer Vision–ECCV 2012:842–856Google Scholar
  27. 27.
    Mathias M, Martinovic A, Weissenberg J, Haegler S, Van Gool L (2011) Automatic architectural style recognition. ISPRS-international archives of the photogrammetry. Remote Sens Spatial Inform Sci 3816:171–176Google Scholar
  28. 28.
    Nesterov Y (1983) A method of solving a convex programming problem with convergence rate o (1/k2). In: Soviet mathematics Doklady, vol 27, pp 372–376Google Scholar
  29. 29.
    Obeso AM, Benois-Pineau J, Acosta AAR, Vázquez MSG (2016) Architectural style classification of mexican historical buildings using deep convolutional neural networks and sparse features. J Electron Imag 26(1):011,016. CrossRefGoogle Scholar
  30. 30.
    Obeso AM, Reyes LMA, Rodriguez ML, Cruz MHM, Vázquez MSG, Benois-Pineau J, Fuentes LMZ, Martinez EC, Secundino JAF, Martinez JLR et al (2016) Image annotation for mexican buildings database. In: International society for optics and photonics of the SPIE optical engineering+ applications, pp 99,700y–99,700yGoogle Scholar
  31. 31.
    Obeso AM, Vázquez MSG, Acosta AAR, Benois-Pineau J (2017) Connoisseur: classification of styles of mexican architectural heritage with deep learning and visual attention prediction. In: Proceedings of the 15th international workshop on content-based multimedia indexing, vol 16. ACMGoogle Scholar
  32. 32.
    Papushoy A, Bors AG (2015) Image retrieval based on query by saliency content. Digital Signal Process 36:156–173MathSciNetCrossRefGoogle Scholar
  33. 33.
    Pont-Tuset J, Arbeláez P, Barron JT, Marques F, Malik J (2017) Multiscale combinatorial grouping for image segmentation and object proposal generation. IEEE Trans Pattern Anal Mach Intell 39(1):128–140. CrossRefGoogle Scholar
  34. 34.
    Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: towards real-time object detection with region proposal networks. In: Advances in neural information processing systems, pp 91–99Google Scholar
  35. 35.
    Ren X, Gu C (2010) Figure-ground segmentation improves handled object recognition in egocentric video. In: 2010 IEEE conference on Computer vision and pattern recognition (CVPR). IEEE, pp 3137–3144Google Scholar
  36. 36.
    Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, Huang Z, Karpathy A, Khosla A, Bernstein M, Berg AC, Fei-Fei L (2015) Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV) 115(3):211–252. MathSciNetCrossRefGoogle Scholar
  37. 37.
    San Biagio M, Bazzani L, Cristani M, Murino V (2014) Weighted bag of visual words for object recognition. In: 2014 IEEE international conference on image processing (ICIP). IEEE, pp 2734–2738Google Scholar
  38. 38.
    Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, LeCun Y (2013) Overfeat: integrated recognition, localization and detection using convolutional networks. CoRR arXiv:1312.6229
  39. 39.
    Shalunts G (2015) Architectural style classification of building facade towers. In: International symposium on visual computing. Springer, pp 285–294Google Scholar
  40. 40.
    Shalunts G, Haxhimusa Y, Sablatnig R (2011) Architectural style classification of building facade windows. In: International symposium on visual computing. Springer, pp 280–289Google Scholar
  41. 41.
    Shalunts G, Haxhimusa Y, Sablatnig R (2012) Classification of gothic and baroque architectural elements. In: 2012 19th international conference on systems, signals and image processing (IWSSIP). IEEE, pp 316–319Google Scholar
  42. 42.
    Sharma G, Jurie F, Schmid C (2012) Discriminative spatial saliency for image classification. In: 2012 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp 3506–3513Google Scholar
  43. 43.
    Sikora T, Makai B (1995) Shape-adaptive dct for generic coding of video. IEEE Trans Circuit Syst Video Technol 5(1):59–62CrossRefGoogle Scholar
  44. 44.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556
  45. 45.
    Su Y, Zhao Q, Zhao L, Gu D (2014) Abrupt motion tracking using a visual saliency embedded particle filter. Pattern Recogn 47(5):1826–1834CrossRefGoogle Scholar
  46. 46.
    Sutskever I, Martens J, Dahl G, Hinton G (2013) On the importance of initialization and momentum in deep learning. In: International conference on machine learning, pp 1139–1147Google Scholar
  47. 47.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9Google Scholar
  48. 48.
    Uijlings JRR, Van De Sande KEA, Gevers T, Smeulders AWM (2013) Selective search for object recognition. Int J Comput Vis 104(2):154–171. CrossRefGoogle Scholar
  49. 49.
    Viola PA, Jones MJ (2001) Rapid object detection using a boosted cascade of simple features. In: 2001 IEEE computer society conference on computer vision and pattern recognition (CVPR 2001), with CD-ROM, 8-14 December 2001, Kauai, HI, USA, pp 511–518.
  50. 50.
    Wang Q, Yuan Y, Yan P (2013) Visual saliency by selective contrast. IEEE Trans Circuit Syst Video Technol 23(7):1150–1155CrossRefGoogle Scholar
  51. 51.
    Wang Q, Yuan Y, Yan P, Li X (2013) Saliency detection by multiple-instance learning. IEEE Trans Cybern 43(2):660–672CrossRefGoogle Scholar
  52. 52.
    Xu Z, Tao D, Zhang Y, Wu J, Tsoi AC (2014) Architectural style classification using multinomial latent logistic regression. In: European conference on computer vision. Springer, pp 600–615Google Scholar
  53. 53.
    Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer, pp 818–833Google Scholar
  54. 54.
    Zhang B, Song Y, Guan SU, Zhang Y (2010) Historic chinese architectures image retrieval by svm and pyramid histogram of oriented gradients features. Int J Soft Comput 5(2):19–28CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Instituto Politécnico NacionalCiudad de MéxicoMéxico
  2. 2.Université de BordeauxBordeauxFrance
  3. 3.MIRAL R&D&ISan DiegoUSA

Personalised recommendations