Multimedia Tools and Applications

, Volume 75, Issue 20, pp 12669–12692 | Cite as

Multi-scale local structure patterns histogram for describing visual contents in social image retrieval systems

  • Jamil Ahmad
  • Muhammad Sajjad
  • Seungmin Rho
  • Sung Wook Baik


Content based image retrieval systems rely heavily on the set of features extracted from images. Effective image representation emerges as a crucial step in such systems. A key challenge in visual content representation is to reduce the so called ‘semantic gap’. It is the inability of existing methods to describe contents in a human-oriented way. Content representation methods inspired by the human vision system have shown promising results in image retrieval. Considerable work has been carried out during the past two decades for developing methods to extract descriptors inspired by the human vision system and attempt to retrieve visual contents efficiently according to the user needs, thereby reducing the semantic gap. Despite the extensive research being conducted in this area, limitations in current image retrieval systems still exist. This paper presents a descriptor for personalized social image collections which utilizes the local structure patterns in salient edge maps of images at multiple scales. The human visual system at the basic level is sensitive to edges, corners, intersections, and other such intensity variations in images generating local structure patterns. Analyzing these patterns at multiple scales allow the most salient fine-grained and coarse-grained features to be captured. The features are accumulated in a local structure patterns histogram to index images allowing flexible querying of visual contents. The retrieval results show that the proposed descriptor ranks well among similar state-of-the-art methods for large social image collections.


Content based retrieval Social images Local structure patterns Multi-scale Feature histogram 



This research is supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2013R1A1A2012904).


  1. 1.
    Ahmad J, Jan Z, Khan SM (2014) A Fusion of Labeled-Grid Shape Descriptors with Weighted Ranking Algorithm for Shapes Recognition. World Applied Sciences Journal 31(6):1207–1213Google Scholar
  2. 2.
    Ahmad J, Sajjad M, Mehmood I, Baik SW (2015) SSH : Salient structures histogram for content based image retrieval. In: 18th IEEE International Conference on Network-Based Information Systems (NBiS), Taipei, Taiwan, 2015Google Scholar
  3. 3.
    Ahonen T, Hadid A, Pietikainen M (2006) Face description with local binary patterns: Application to face recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(12):2037–2041CrossRefMATHGoogle Scholar
  4. 4.
    Canny J (1986) A computational approach to edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 6:679–698CrossRefGoogle Scholar
  5. 5.
    Corel Dataset. Accessed 10 July 2015
  6. 6.
    Emerson CW, Siu-Ngan Lam N, Ouattrochi D (1999) Multi-scale fractal analysis of image texture and patterns. Photogrammetric Engineering and Remote Sensing 65:51–62Google Scholar
  7. 7.
    Facebook Has a Quarter of a Trillion User Photos (2015)
  8. 8.
  9. 9.
    Felzenszwalb P, Oberlin JG (2014) Multiscale fields of patterns. In: Advances in neural information processing systems, pp 82–90Google Scholar
  10. 10.
    Han J, Ma K-K (2002) Fuzzy color histogram and its use in color image retrieval. IEEE Transactions on Image Processing 11(8):944–952CrossRefGoogle Scholar
  11. 11.
    Hansen T (2003) A neural model of early vision: contrast, contours, corners and surfaces. University of Ulm Faculty of Computer Science: 35–42Google Scholar
  12. 12.
    Hansen T, Neumann H (2004) Neural mechanisms for the robust representation of junctions. Neural Computation 16(5):1013–1037CrossRefMATHGoogle Scholar
  13. 13.
    Heikkilä M, Pietikäinen M, Schmid C (2009) Description of interest regions with local binary patterns. Pattern recognition 42(3):425–436CrossRefMATHGoogle Scholar
  14. 14.
    Huang J, Kumar SR, Mitra M, Zhu W-J, Zabih R (1997) Image indexing using color correlograms. In: Proceedings of the IEEE Computer Society Conference on Computer vision and pattern recognition, 1997, pp 762–768Google Scholar
  15. 15.
    Huang S, Wang W, Zhang H (2014) Retrieving images using saliency detection and graph matching. In: IEEE International Conference on Image Processing (ICIP) 2014, pp 3087–3091Google Scholar
  16. 16.
    INRIA Holidays dataset (2015) Accessed 10 July 2015
  17. 17.
    Jégou H, Douze M, Schmid C, Pérez P (2010) Aggregating local descriptors into a compact image representation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp 3304–3311Google Scholar
  18. 18.
    Jégou H, Douze M, Schmid C (2008) Hamming embedding and weak geometry consistency for large scale image search-extended versionGoogle Scholar
  19. 19.
    Kasutani E, Yamada A (2001) The MPEG-7 color layout descriptor: a compact image feature description for high-speed image/video segment retrieval. In: Proceedings of IEEE International Conference on Image Processing, 2001. pp 674–677Google Scholar
  20. 20.
    Li J, Wang JZ (2008) Real-time computerized annotation of pictures. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(6):985–1002CrossRefGoogle Scholar
  21. 21.
    Lin R-J, Lin W-S (2014) A computational visual saliency model based on statistics and machine learning. Journal of vision 14(9):1CrossRefGoogle Scholar
  22. 22.
    Liu G-H, Li Z-Y, Zhang L, Xu Y (2011) Image retrieval based on micro-structure descriptor. Pattern Recognition 44(9):2123–2133CrossRefGoogle Scholar
  23. 23.
    Liu G-H, Yang J-Y (2013) Content-based image retrieval using color difference histogram. Pattern Recognition 46(1):188–198CrossRefGoogle Scholar
  24. 24.
    Liu G-H, Zhang L, Hou Y-K, Li Z-Y, Yang J-Y (2010) Image retrieval based on multi-texton histogram. Pattern Recognition 43(7):2380–2389CrossRefMATHGoogle Scholar
  25. 25.
    Lu G, Teng S (1999) A novel image retrieval technique based on vector quantization. In: Proceedings of International Conference on Computational Intelligence for Modeling, Control and Automation, 1999. pp 36–41Google Scholar
  26. 26.
    Manjunath BS, Ma W-Y (1996) Texture features for browsing and retrieval of image data. IEEE Transactions on Pattern Analysis and Machine Intelligence 18(8):837–842CrossRefGoogle Scholar
  27. 27.
    Manjunath BS, Ohm J-R, Vasudevan VV, Yamada A (2001) Color and texture descriptors. IEEE Transactions on Circuits and Systems for Video Technology, 11(6):703–715CrossRefGoogle Scholar
  28. 28.
    Messing DS, Van Beek P, Errico JH (2001) The mpeg-7 colour structure descriptor: image description using colour and local spatial information. In: Proceedings of the International Conference on Image Processing, 2001, pp 670–673Google Scholar
  29. 29.
    Mu Y, Yan S, Liu Y, Huang T, Zhou B (2008) Discriminative local binary patterns for human detection in personal album. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2008, pp 1–8Google Scholar
  30. 30.
    Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence 24(7):971–987CrossRefMATHGoogle Scholar
  31. 31.
    Olshausen BA, Field DJ (2004) Sparse coding of sensory inputs. Current opinion in neurobiology 14(4):481–487CrossRefGoogle Scholar
  32. 32.
    Ortega M, Rui Y, Chakrabarti K, Porkaew K, Mehrotra S, Huang TS (1998) Supporting ranked boolean similarity queries in MARS. IEEE Transactions on Knowledge and Data Engineering 10(6):905–925CrossRefGoogle Scholar
  33. 33.
    Pass G, Zabih R, Miller J (1997) Comparing images using color coherence vectors. In: Proceedings of the fourth ACM international conference on Multimedia, 1997. ACM, pp 65–73Google Scholar
  34. 34.
    Portilla J, Simoncelli EP (2000) A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40(1):49–70CrossRefMATHGoogle Scholar
  35. 35.
    Rahimi M, Moghaddam ME (2013) A content-based image retrieval system based on Color Ton Distribution descriptors. SIViP 1–14. doi: 10.1007/s11760-013-0506-6
  36. 36.
    Smeulders AW, Worring M, Santini S, Gupta A, Jain R (2000) Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1349–1380CrossRefGoogle Scholar
  37. 37.
    Subrahmanyam M, Maheshwari R, Balasubramanian R (2012) Expert system design using wavelet and color vocabulary trees for image retrieval. Expert Systems with Applications 39(5):5104–5114CrossRefGoogle Scholar
  38. 38.
    Subrahmanyam M, Wu QJ, Maheshwari R, Balasubramanian R (2013) Modified color motif co-occurrence matrix for image indexing and retrieval. Computers & Electrical Engineering 39(3):762–774CrossRefGoogle Scholar
  39. 39.
    Takala V, Ahonen T, Pietikäinen M (2005) Block-based methods for image retrieval using local binary patterns. In: Image analysis. Springer, pp 882–891Google Scholar
  40. 40.
    Terzić K, Rodrigues JM, du Buf JH (2015) BIMP: A real-time biological model of multi-scale keypoint detection in V1. Neurocomputing 150:227–237CrossRefGoogle Scholar
  41. 41.
    van Ginneken B, ter Haar Romeny BM (2003) Multi-scale texture classification from generalized locally orderless images. Pattern Recognition 36(4):899–911CrossRefGoogle Scholar
  42. 42.
    Vinje WE, Gallant JL (2000) Sparse coding and decorrelation in primary visual cortex during natural vision. Science 287(5456):1273–1276CrossRefGoogle Scholar
  43. 43.
    Vipparthi SK, Murala S, Nagar SK (2015) Dual directional multi-motif XOR patterns: A new feature descriptor for image indexing and retrieval. Optik-International Journal for Light and Electron Optics 126(15):1467–1473CrossRefGoogle Scholar
  44. 44.
    Vipparthi SK, Nagar S (2014) Expert image retrieval system using directional local motif XoR patterns. Expert Systems with Applications 41(17):8016–8026CrossRefGoogle Scholar
  45. 45.
    Wang X, Wang Z (2013) A novel method for image retrieval based on structure elements’ descriptor. Journal of Visual Communication and Image Representation 24(1):63–74CrossRefGoogle Scholar
  46. 46.
    Yang M, Kpalma K, Ronsin J (2008) A survey of shape feature extraction techniques. Pattern recognition:43–90Google Scholar
  47. 47.
    Yao C-H, Chen S-Y (2003) Retrieval of translated, rotated and scaled color textures. Pattern Recognition 36(4):913–929MathSciNetCrossRefGoogle Scholar
  48. 48.
    Zhang D, Lu G (2004) Review of shape representation and description techniques. Pattern recognition 37(1):1–19CrossRefGoogle Scholar
  49. 49.
    Zhang J, Tan T (2002) Brief review of invariant texture analysis methods. Pattern recognition 35(3):735–747CrossRefMATHGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.College of Electronics and Information EngineeringSejong UniversitySeoulRepublic of Korea
  2. 2.Department of Computer ScienceIslamia CollegePeshawarPakistan
  3. 3.Department of MultimediaSungkyul UniversityAnyangRepublic of Korea

Personalised recommendations