Multimedia Tools and Applications

, Volume 77, Issue 5, pp 5475–5501 | Cite as

Vector space model adaptation and pseudo relevance feedback for content-based image retrieval

  • H. Karamti
  • M. Tmar
  • M. Visani
  • T. Urruty
  • F. Gargouri


Image retrieval is an important problem for researchers in computer vision and content-based image retrieval (CBIR) fields. Over the last decades, many image retrieval systems were based on image representation as a set of extracted low-level features such as color, texture and shape. Then, systems calculate similarity metrics between features in order to find similar images to a query image. The disadvantage of this approach is that images visually and semantically different may be similar in the low level feature space. So, it is necessary to develop tools to optimize retrieval of information. Integration of vector space models is one solution to improve the performance of image retrieval. In this paper, we present an efficient and effective retrieval framework which includes a vectorization technique combined with a pseudo relevance model. The idea is to transform any similarity matching model (between images) to a vector space model providing a score. A study on several methodologies to obtain the vectorization is presented. Some experiments have been undertaken on Wang, Oxford5k and Inria Holidays datasets to show the performance of our proposed framework.


Vectorization CBIR Neural network Late fusion Vector space model Pseudo-relevance feedback 


  1. 1.
    Aidos H, Duarte JMM, Fred ALN (2014) Identifying regions of interest for discriminating Alzheimer’s disease from mild cognitive impairment. IEEE International Conference on Image Processing (ICIP), pp 21–25Google Scholar
  2. 2.
    Angelova A, Zhu S (2013) Efficient object detection and segmentation for fine-grained recognition. IEEE Conference on Computer Vision and Pattern Recognition, pp 811–818Google Scholar
  3. 3.
    Anh ND, Bao PT, Nam BN, Hoang NH (2010) A New CBIR System Using SIFT Combined with Neural Network and Graph-Based Segmentation. Intelligent Information and Database Systems, Second International Conference, ACIIDS, pp 294–301Google Scholar
  4. 4.
    Alpkocak A, Kilinc D, Expansion TB (2010) Re-ranking approaches for multimodal image retrieval using text-based methods ImageCLEF, Experimental Evaluation in Visual Information Retrieval, pp 261–275Google Scholar
  5. 5.
    Arandjelovic R, Gronat P, Torii A, Pajdla T, Sivic J (2015) NetVLAD: CNN architecture for weakly supervised place recognitionGoogle Scholar
  6. 6.
    Arandjelovic R, Zisserman A (2012) Three things everyone should know to improve object retrieval. IEEE Conference on Computer Vision and Pattern Recognition, pp 2911–2918Google Scholar
  7. 7.
    Atreya A, Elkan C (2010) Latent semantic indexing (LSI) fails for TREC collections. SIGKDD Explorations, pp 5–10Google Scholar
  8. 8.
    Babenko A, Lempitsky VS (2015) Aggregating Deep Convolutional Features for Image RetrievalGoogle Scholar
  9. 9.
    Babenko A, Slesarev A, Chigorin A, Lempitsky VS (2014) Neural codes for image retrieval. Computer Vision - ECCV, pp 584–599Google Scholar
  10. 10.
    Becker CJ, Rigamonti R, Lepetit V, Fua P (2013) Supervised feature learning for curvilinear structure segmentation. Medical Image Computing and Computer-Assisted Intervention - MICCAI, pp 526– 533Google Scholar
  11. 11.
    Canny J (1986) A computational approach to edge detection. IEEE Transaction on Pattern Analysis Machine Intelligence. pp 679–698Google Scholar
  12. 12.
    Chatzichristofis SA, Yiannis SB (2008) CEDD color and edge directivity descriptor: A compact descriptor for image indexing and retrieval. Computer Vision Systems, 6th International Conference, pp 312–322Google Scholar
  13. 13.
    Claveau V, Tavenard R, d’appariement LA (2010) Vectorisation des processus document-requte. Conference en Recherche d’Infomations et Applications. Conference en Recherche d’Infomations et Applications (CORIA), pp 313–324Google Scholar
  14. 14.
    Delaitre V, Laptev I, Sivic J (2010) Recognizing human actions in still images: a study of bag-of-features and part-based representations. British Machine Vision Conference - BMVC, pp 1–11Google Scholar
  15. 15.
    Deserno TM, Guld MO, Plodowski B, Spitzer K, Wein BB, Schubert H, Ney H, Seidl T (2008) Extended query refinement for medical image retrieval. Journal Digital Imaging pp 280–289Google Scholar
  16. 16.
    Donahue J, Jia Y, Vinyals O, Hoffman J, Zhang N, Tzeng E, Trevor D (2014) DeCAF A deep convolutional activation feature for generic visual recognition. Proceedings of the 31th International Conference on Machine Learning - ICML, pp 647–655Google Scholar
  17. 17.
    Douze M, Jegou H, Sandhawalia H, Amsaleg L, Schmid C (2009) Evaluation of GIST descriptors for web-scale image search. Proceedings of the 8th ACM International Conference on Image and Video Retrieval CIVRGoogle Scholar
  18. 18.
    Flickner M, Sawhney HS, Ashley J, Huang Q, Dom B, Gorkani M, Hafner J, Lee D, Petkovic D, Steele D, Yanker P (1995) Query by image and video content: The QBIC System. IEEE Computer, pp 23–32Google Scholar
  19. 19.
    Gong Y, Lazebnik S (2011) Iterative quantization: A procrustean approach to learning binary codes. The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 817–824Google Scholar
  20. 20.
    Gong Y, Wang L, Guo R, Lazebnik S (2014) Multi-scale orderless pooling of deep convolutional activation features. Computer Vision - ECCV, pp 392–407Google Scholar
  21. 21.
    Gony J, Cord M, Philipp-Foliguet S, Gosselin Philippe H, Precioso F, Jordan M (2007) RETIN: a smart interactive digital media retrieval system. Proceedings of the 6th ACM International Conference on Image and Video Retrieval, pp 93–96Google Scholar
  22. 22.
    Hsu W, Rodney Long L, Antani SK (2007) SPIRS: A framework for content-based image retrieval from large biomedical databases. (MEDINFO)- Proceedings of the 12th World Congress on Health (Medical) Informatics - Building Sustainable Health Systems, pp 188–192Google Scholar
  23. 23.
    Jain AK, Nandakumar K, Ross A (2005) Score norMalization in multimodal biometric systems. Pattern Recognition, pp 2270–2285Google Scholar
  24. 24.
    Jegou H, Chum O (2012) Negative Evidences and Co-occurences in Image Retrieval: The Benefit of PCA and Whitening. Computer Vision. ECCV 12th European Conference on Computer Vision, pp 774–787Google Scholar
  25. 25.
    Jegou H, Zisserman A (2014) Triangulation embedding and democratic aggregation for image search. IEEE Conference on Computer Vision and Pattern Recognition - CVPR, pp 3310–3317Google Scholar
  26. 26.
    Jordan C, Watters CR (2004) Extending the Rocchio relevance feedback algorithm to provide contextual retrieval dvances in Web intelligence. Second International Atlantic Web Intelligence Conference (AWIC), pp 135–144Google Scholar
  27. 27.
    Kalantidis Y, Mellina C, Osindero S (2016) Cross-dimensional weighting for aggregated deep convolutional features. Computer Vision - ECCV, pp 685–701Google Scholar
  28. 28.
    Karamti H (2013) Vectorisation du modle d’appariement pour la recherche d’images par le contenu. Conference en Recherche d’Infomations et Applications (CORIA), pp 335–340Google Scholar
  29. 29.
    Karamti H, Tmar M, Benammar A (2012) A new relevance feedback approach for multimedia retrieval. IKE, pp 129–135Google Scholar
  30. 30.
    Karamti H, Tmar M, Gargouri F (2014) Content-based image retrieval system using neural network. 11th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA), pp 723–728Google Scholar
  31. 31.
    Karamti H, Tmar M, Gargouri F (2015) Content-based image retrieval system with relevance feedback. WEBIST 2015 - Proceedings of the 11th International Conference on Web Information Systems and Technologies, pp 287–292Google Scholar
  32. 32.
    Kasutani E, Yamada A (2001) The MPEG-7 color layout descriptor: a compact image feature description for high-speed image/video segment retrieval. ICIP (1), pp 674–677Google Scholar
  33. 33.
    Kulkarni S, Srinivasan B, Ramakrishna MV (1999) Vector-space image model (VSIM) for content-based retrieval. 10th International Workshop on Database and Expert Systems Applications, pp 899–903Google Scholar
  34. 34.
    Lee H, Grosse RB, Ranganath R, Ng AY (2011) Unsupervised learning of hierarchical representations with convolutional deep belief networks Commun. ACM, pp 95–103Google Scholar
  35. 35.
    Lindeberg T (1996) Edge Detection and Ridge Detection with Automatic Scale Selection. Conference on Computer Vision and Pattern Recognition CVPR, pp 465–470Google Scholar
  36. 36.
    Liu W, Wang J, Ji R, Jiang Y-G, Chang S-F (2012) Supervised hashing with kernels. Conference on Computer Vision and Pattern Recognition, Providence, RI, pp 2074–2081Google Scholar
  37. 37.
    Lowe DG (1999) Object recognition from local scale-invariant features. ICCV, pp 1150–1157Google Scholar
  38. 38.
    Lowe DG (2004) Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, pp 91–110Google Scholar
  39. 39.
    Mandal S, Sudarshan VP, Nagaraj Y, Dean-Ben XL, Razansky D (2015) Multiscale edge detection and parametric shape modeling for boundary delineation in optoacoustic images. 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp 707–710Google Scholar
  40. 40.
    Manning CD, Raghavan P, Schutze H (2008) Introduction to information retrieval. Cambridge University PressGoogle Scholar
  41. 41.
    Miao J, Huang JX, Ye Z (2012) Proximity-based rocchio’s model for pseudo relevance. The 35th International ACM-SIGIR conference on research and development in Information Retrieval, pp 535–544Google Scholar
  42. 42.
    Ming A, Ma H (2007) A blob detector in color images. Proceedings of the 6th ACM International Conference on Image and Video Retrieval, pp 364–370Google Scholar
  43. 43.
    Ng JY-H, Yang F, Davis LS (2015) Exploiting local features from deep networks for image retrieval. IEEE Conference on Computer Vision and Pattern Recognition Workshops - CVPR, pp 53–61Google Scholar
  44. 44.
    Ortiz-Jaramillo B, Benitez-Restrepo H, Garcia-Alvarez JC, Castellanos-Dominguez CG (2010) Region of Interest Extraction based on Multiresolution Analysis for Infrared Nondestructive Testing. 10th Quantitative Infrared Thermography Conference QIRTGoogle Scholar
  45. 45.
    Park DK, Jeon YS, Won CS (2000) Efficient use of local edge histogram descriptor. Proceedings of the ACM Multimedia 2000 Workshops, pp 51–54Google Scholar
  46. 46.
    Paulin M, Douze M, Harchaoui Z, Mairal J, Perronnin F, Schmid C (2015) Local convolutional features with unsupervised training for image retrieval. IEEE International Conference on Computer Vision - ICCV, pp 91–99Google Scholar
  47. 47.
    Philbin J, Chum O, Isard M, Sivic J, Zisserman A (2007) Object retrieval with large vocabularies and fast spatial matching. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)Google Scholar
  48. 48.
    Philbin J, Isard M, Sivic J, Zisserman Andrew (2010) Descriptor learning for efficient retrieval. Computer Vision - (ECCV) 2010, 11th European Conference on Computer Vision, pp 677–691Google Scholar
  49. 49.
    Ramanathan V, Mishra S, Mitra P (2011) Quadtree decomposition based extended vector space model for image retrieval. IEEE Workshop on Applications of Computer Vision WACV, pp 139– 144Google Scholar
  50. 50.
    Ramos H, Ribeiro AL, Giro PM (1994) A Two-dimensional Vector Model of Ferromagnetic Hysteresis. Journal of Magnetism and Magnetic Materials, pp 574–577Google Scholar
  51. 51.
    Rosten E, Drummond T (2006) Machine learning for high-speed corner detection. Computer Vision - ECCV, 9th European Conference on Computer Vision, ppp 430–443Google Scholar
  52. 52.
    Ruthven I, Lalmas M (2003) A survey on the use of relevance feedback for information access systems Knowledge Engineering Review, pp 95–145Google Scholar
  53. 53.
    Salembier P (2002) Overview of the MPEG-7 Standard and of Future Challenges for Visual Information Analysis. EURASIP Journal of Advanced Signal Proceedings, pp 343–353Google Scholar
  54. 54.
    Salembier P, Sikora T (2002) Introduction to MPEG-7: Multimedia content description interface. Wiley,Google Scholar
  55. 55.
    Salton G, Wong A, Yang CS (1975) A Vector Space Model for Automatic Indexing Commun. ACM, pp 613–620Google Scholar
  56. 56.
    Schettini R, Ciocca G, Gagliardi I (2009) Feature extraction for content-based image retrieval. Encyclopedia of Database Systems, pp 1115–1119Google Scholar
  57. 57.
    Sheikholeslami G, Chang W, SemQuery AZ (2002) Semantic clustering and querying on heterogeneous features for visual data. IEEE Transactions on Knowledge and Data Engineering, pp 988– 1002Google Scholar
  58. 58.
    Smeulders AWM, Worring M, Santini S, Gupta A, Jain RC (2000) Content-based image retrieval at the end of the early years. IEEE Transactions Pattern Analysis Machine Intelligence, pp 1349– 1380Google Scholar
  59. 59.
    Stathopoulos V, Jose JM (2011) Bayesian probabilistic models for image retrieval. Proceedings of the Second Workshop on Applications of Pattern Analysis (WAPA), pp 41–47Google Scholar
  60. 60.
    Teran L, Mordohai P (2014) 3D interest point detection via discriminative learning. Computer Vision - ECCV - 13th European Conference, pp 159–173Google Scholar
  61. 61.
    Tolias G (2015) Ronan Sicre and Herve Jegou Particular object retrieval with integral max-pooling of CNN activationsGoogle Scholar
  62. 62.
    Tsai C-F, Hu Y-H, Chen Z-Y (2015) Factors affecting rocchio-based pseudorelevance feedback in image retrieval. JASIST, pp 40–57Google Scholar
  63. 63.
    Wang D, Tan X (2014) C-SVDDNet: An Effective Single-Layer Network for Unsupervised Feature LearningGoogle Scholar
  64. 64.
    Wang Y, Gong M, Wang T, Cohen-Or D, Zhang H, Chen B (2013) Projective analysis for 3D shape segmentation. ACM Transactions, pp 192–192Google Scholar
  65. 65.
    Wengert C, Douze M, Jegou H (2011) Bag-of-colors for improved image search. Proceedings of the 19th International Conference on Multimedia, pp 1437–1440Google Scholar
  66. 66.
    Westerveld T, De Vries AP, Van Ballegooij A, De Jong F, Hiemstra D (2003) A Probabilistic Multimedia Retrieval Model and Its Evaluation. EURASIP Journal of Advanced Signal Proceedings, pp 186– 198Google Scholar
  67. 67.
    Winder SAJ, Hua G, Brown MA (2009) Picking the best DAISY. IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR, pp 178–185Google Scholar
  68. 68.
    Yan R, Hauptmann AG, Jin R (2003) Negative pseudo-relevance feedback in content-based video retrieval. Proceedings of the Eleventh ACM International Conference on Multimedia, pp 343– 346Google Scholar
  69. 69.
    Ye Z, Huang JX (2014) A simple term frequency transformation model for effective pseudo relevance feedback. The 37th International ACM-SIGIR Conference on Research and Development in Information Retrieval, pp 323–332Google Scholar
  70. 70.
    Yue J, Li Z, Lu L, Fu Z (2011) Content-based image retrieval using color and texture fused features. Mathematical and Computer Modelling, pp 1121–1127Google Scholar
  71. 71.
    Zhang D, Lu G (2001) Content-Based Shape Retrieval Using Different Shape Descriptors: A Comparative Study. Proceedings of the IEEE International Conference on Multimedia and ExpoGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.National School of Engineers of SfaxUniversity of SfaxSfaxTunisia
  2. 2.Institut Supérieur d’Informatique et de Multimédia de SfaxSfaxTunisia
  3. 3.Computer Science Lab (L3i)University of La RochelleLa RochelleFrance
  4. 4.Vietnam-France, ICT Lab, USTHHanoiFrance
  5. 5.University of PoitiersPoitiersFrance

Personalised recommendations