Multimedia Tools and Applications

, Volume 77, Issue 20, pp 26173–26189 | Cite as

A robust CBIR framework in between bags of visual words and phrases models for specific image datasets

  • Achref Ouni
  • Thierry UrrutyEmail author
  • Muriel Visani


One objective of the Content Based Image Retrieval research field is to propose new methodologies and tools to manage the increasing number of images available. Linked to a specific context of small expert datasets without prior knowledge, our research work focuses on improving the discriminative power of the image representation while keeping the same efficiency for retrieval. Based on the well-known bag of visual words model, we propose three different methodologies inspired by the visual phrase model effectiveness and a compression technique which ensures the same effectiveness for retrieval than the BoVW model. Our experimental results study the performance of our proposals on different well known benchmark datasets and show its good performance compared to other recent approaches.


Image representation CBIR Bag of visual words Visual phrases Expert dataset 



This research is supported by the Poitou-Charentes Regional Founds for Research activities and the European Regional Development Founds (ERDF) inside the e-Patrimoine project from the ax 1 of the NUMERIC Program.


  1. 1.
    Aggarwal CC (2015) Data mining—the textbook. Springer, BerlinzbMATHGoogle Scholar
  2. 2.
    Alqasrawi Y, Neagu D, Cowling PI (2013) Fusing integrated visual vocabularies-based bag of visual words and weighted colour moments on spatial pyramid layout for natural scene image classification. Signal Image Video Process 7(4):759–775CrossRefGoogle Scholar
  3. 3.
    Babenko A, Lempitsky VS (2015) Aggregating local deep features for image retrieval. In: 2015 IEEE international conference on computer vision, ICCV 2015, Santiago, Chile, December 7–13, 2015. IEEE Computer Society, pp 1269–1277Google Scholar
  4. 4.
    Babenko A, Slesarev A, Chigorin A, Lempitsky VS (2014) Neural codes for image retrieval. In: Fleet D J, Pajdla T, Schiele B, Tuytelaars T (eds) Computer vision—ECCV 2014—13th European conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I, volume 8689 of Lecture Notes in Computer Science. Springer, Berlin, pp 584–599Google Scholar
  5. 5.
    Bay H, Tuytelaars T, Gool L (2006) Surf: speeded up robust features. In: Leonardis A, Bischof H, Pinz A (eds) Computer vision—ECCV 2006, volume 3951 of Lecture Notes in Computer Science. Springer, Berlin, pp 404–417CrossRefGoogle Scholar
  6. 6.
    Carpineto C, Romano G (2012) A survey of automatic query expansion in information retrieval. ACM Comput Surv 44(1):1:1–1:50CrossRefGoogle Scholar
  7. 7.
    Chatoux H, Lecellier F, Fernandez-Maloigne C (2016) Comparative study of descriptors with dense key points. In: 23rd international conference on pattern recognition, ICPR 2016, Cancún, Mexico, December 4–8, 2016, pp 1988–1993Google Scholar
  8. 8.
    Csurka G, Bray C, Dance C, Fan L (2004) Visual categorization with bags of keypoints. In: Workshop on statistical learning in computer vision, ECCV, pp 1–22Google Scholar
  9. 9.
    Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2012) The PASCAL visual object classes challenge 2012 (VOC2012) Results.
  10. 10.
    Girshick RB, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE conference on computer vision and pattern recognition, CVPR 2014, Columbus, OH, USA, June 23–28, 2014. IEEE Computer Society, pp 580–587Google Scholar
  11. 11.
    Harris Z (1954) Distributional structure. Word 10(23):146–162CrossRefGoogle Scholar
  12. 12.
    He K, Zhang X, Ren S, Sun J (2015) Deep residual learning for image recognition. CoRR arXiv:1512.03385
  13. 13.
    He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. CoRR arXiv:1502.01852
  14. 14.
    He K, Zhang X, Ren S, Sun J (2016) Identity mappings in deep residual networks. CoRR arXiv:1603.05027
  15. 15.
    Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. CoRR arXiv:1502.03167
  16. 16.
    Jégou H, Douze M, Schmid C (2008) Hamming embedding and weak geometric consistency for large scale image search. In: Forsyth AZD, Torr P (eds) European conference on computer vision, volume I of LNCS. Springer, pp 304–317Google Scholar
  17. 17.
    Jégou H, Douze M, Schmid C, Pérez P (2010) Aggregating local descriptors into a compact image representation. In: 23rd IEEE conference on computer vision & pattern recognition (CVPR ’10). IEEE Computer Society, San Francisco, pp 3304–3311Google Scholar
  18. 18.
    Jiang W, Zhao Z, Su F (2016) Bayes pooling of visual phrases for object retrieval. Multimedia Tools Appl 75(15):9095–9119CrossRefGoogle Scholar
  19. 19.
    Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25. Curran Associates Inc., pp 1097–1105Google Scholar
  20. 20.
    Lazebnik S, Schmid C, Ponce J (2006) Beyond bags of features: spatial pyramid matching for recognizing natural scene categories. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR 2006), 17–22 June 2006, New York, pp 2169–2178Google Scholar
  21. 21.
    Li F, Fergus R, Perona P (2007) Learning generative visual models from few training examples: an incremental bayesian approach tested on 101 object categories. Comput Vis Image Underst 106(1):59–70CrossRefGoogle Scholar
  22. 22.
    Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model. In: Schuurmans D, Wellman MP (eds) Proceedings of the thirtieth AAAI conference on artificial intelligence, February 12–17, 2016. AAAI Press, Phoenix, pp 1266–1272Google Scholar
  23. 23.
    Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2015) Action2activity: recognizing complex activities from sensor data. In: Proceedings of the 24th international conference on artificial intelligence, IJCAI’15. AAAI Press, pp 1617–1623Google Scholar
  24. 24.
    Lowe DG (1999) Object recognition from local scale-invariant features. In: International conference on computer vision, vol 2, pp 1150–1157Google Scholar
  25. 25.
    Nistér D, Stewénius H (2006) Scalable recognition with a vocabulary tree. In: IEEE conference on computer vision and pattern recognition (CVPR), vol 2, pp 2161–2168Google Scholar
  26. 26.
    Ouni A, Urruty T, Visani M (2017) Improving the discriminative power of bag of visual words model. In: MultiMedia modeling—23rd international conference, MMM 2017, Reykjavik, Iceland, January 4–6, 2017, Proceedings, Part II, pp 245–256Google Scholar
  27. 27.
    Pedrosa G, Traina A (2013) From bag-of-visual-words to bag-of-visual-phrases using n-grams. In: 2013 26th SIBGRAPI—conference on graphics, patterns and images (SIBGRAPI), pp 304–311Google Scholar
  28. 28.
    Perronnin F, Dance CR (2007) Fisher kernels on visual vocabularies for image categorization. In: 2007 IEEE computer society conference on computer vision and pattern recognition (CVPR 2007), 18–23 June 2007. IEEE Computer Society, MinneapolisGoogle Scholar
  29. 29.
    Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of the IEEE conference on computer vision and pattern recognitionGoogle Scholar
  30. 30.
    Ren Y, Bugeau A, Benois-Pineau J (2014) Bag-of-bags of words irregular graph pyramids vs spatial pyramid matching for image retrieval. In: 2014 4th international conference on image processing theory, tools and applications (IPTA), pp 1–6Google Scholar
  31. 31.
    Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. CoRR arXiv:1409.1556
  32. 32.
    Sivic J, Zisserman A (2003) Video Google: a text retrieval approach to object matching in videos. In: Proceedings of the international conference on computer vision, pp 1470–1477Google Scholar
  33. 33.
    Szegedy C, Ioffe S, Vanhoucke V (2016) Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR arXiv:1602.07261
  34. 34.
    Szegedy C, Liu W, Jia Y, Sermanet P, Reed SE, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going deeper with convolutions. CoRR arXiv:1409.4842
  35. 35.
    van de Sande KEA, Gevers T, Snoek CGM (2010) Evaluating color descriptors for object and scene recognition. IEEE Trans Pattern Anal Mach Intell 32(9):1582–1596CrossRefGoogle Scholar
  36. 36.
    Wang JZ, Li J, Wiederhold G (2001) Simplicity: semantics-sensitive integrated matching for picture libraries. IEEE Trans Pattern Anal Mach Intell 23(9):947–963CrossRefGoogle Scholar
  37. 37.
    Wang K, Yin Q, Wang W, Wu S, Wang L (2016) A comprehensive survey on cross-modal retrieval. CoRR arXiv:1607.06215
  38. 38.
    Yang Y, Newsam SD (2011) Spatial pyramid co-occurrence for image classification. In: Metaxas DN, Quan L, Sanfeliu A, Gool LJV (eds) IEEE international conference on computer vision, ICCV 2011, Barcelona, Spain, November 6–13, 2011. IEEE Computer Society, pp 1465–1472Google Scholar
  39. 39.
    Yeganli F, Nazzal M, Özkaramanli H (2015) Image super-resolution via sparse representation over multiple learned dictionaries based on edge sharpness and gradient phase angle. Signal Image Video Process 9:285–293CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.XLIM, UMR CNRS 7252University of PoitiersPoitiersFrance
  2. 2.Laboratory L3iUniversity of La RochelleLa RochelleFrance
  3. 3.Vietnam-France ICT LaboratoryUniversity of Science and Technology of HanoïHanoiVietnam
  4. 4.Laboratory LaBRIUniversity of BordeauxBordeauxFrance

Personalised recommendations