Multi-level diversification approach of semantic-based image retrieval results

  • Mariam BouchakwaEmail author
  • Yassine Ayadi
  • Ikram Amous
Regular Paper


With the increasing popularity of social photograph-sharing Web sites, a huge mass of digital images, associated with a set of tags voluntarily introduced by amateur photographers, is daily hosted and consequently, the Tag-based social Image Retrieval technique has been widely adopted. However, tag-based queries are often too ambiguous and abstract to be considered as an efficient solution for the retrieval of the most relevant images that meet the users’ needs. As an alternative, the Semantic-based social Image Retrieval technique has emerged for the purpose of retrieving the relevant images covering as much possible the topics that a given ambiguous query (q) may have. Actually, the diversification strategies are a great challenge for researchers. In this context, we jointly investigate two processes at the ambiguous query preprocessing and postprocessing levels. On the one hand, we propose a Tag-based Query Semantic Reformulation process, which aims at reformulating the tag-based users’ queries, according to multiple semantic facets of the different images’ views, by using a set of predefined ontological semantic rules. On the other hand, we propose a Multi-level Image Diversification process that can first perform a two-level-based image clustering offline, and second, filter and re-rank the image cluster retrieval results according to their pertinence versus the reformulated query online. The experimental results and statistical analysis performed on a collection of 25.000 socio-tagged images shared on Flickr demonstrate the effectiveness of the proposed technique, which is compared with the research technique based on one-level-based image clustering, tag-based image research technique and recent CBIR techniques.


Socio-tagged images Semantic annotation Semantic retrieval Semantic rules k-means clustering Search result diversification 



  1. 1.
    Chathurani, N.W.U.D., Geva, S., Chandran, V., Cynthujah, V.: An effective content based image retrieval system based on global representation and multi-level searching. In: Proceedings of the 10th International Conference on Industrial and Information Systems (ICIIS), pp. 158–163 (2015)Google Scholar
  2. 2.
    Wang, X.Y., Zhang, B.B., Yang, H.Y.: Content-based image retrieval by integrating color and texture features. Multimed. Tools Appl. 68(3), 545–569 (2014)CrossRefGoogle Scholar
  3. 3.
    Atlam, H.F., Attiya, G., El-Fishawy, N.: Integration of color and texture features in CBIR system. Int. J. Comput. Appl. 164(3), 23–29 (2017)Google Scholar
  4. 4.
    Liu, G.H., Yang, J.Y.: Content-based image retrieval using color difference histogram. Pattern Recogn. 46(1), 188–198 (2013)CrossRefGoogle Scholar
  5. 5.
    Chang, B.M., Tsai, H.H., Chou, W.L.: Using visual features to design a content-based image retrieval method optimized by particle swarm optimization algorithm. Eng. Appl. Artif. Intell. 26(10), 2372–2382 (2013)CrossRefGoogle Scholar
  6. 6.
    Liu, G.H., Li, Z.Y., Zhang, L., Xu, Y.: Image retrieval based on micro-structure descriptor. Pattern Recogn. 44(9), 2123–2133 (2011)CrossRefGoogle Scholar
  7. 7.
    ElAlami, M.E.: A new matching strategy for content based image retrieval system. Appl. Soft Comput. 14, 407–418 (2014)CrossRefGoogle Scholar
  8. 8.
    Jiji, G.W., DuraiRaj, P.J.: Content-based image retrieval techniques for the analysis of dermatological lesions using particle swarm optimization technique. Appl. Soft Comput. 30, 650–662 (2015)CrossRefGoogle Scholar
  9. 9.
    Wang, X.Y., Yu, Y.J., Yang, H.Y.: An effective image retrieval scheme using color, texture and shape features. Comput. Stand. Interfaces 33(1), 59–68 (2011)CrossRefGoogle Scholar
  10. 10.
    Pass, G., Zabih, R.: Comparing images using joint histograms. Multimed. Syst. 7(3), 234–240 (1999)CrossRefGoogle Scholar
  11. 11.
    Wang, X.Y., Li, Y.W., Yang, H.Y., Chen, J.W.: An image retrieval scheme with relevance feedback using feature reconstruction and SVM reclassification. Neurocomputing 127, 214–230 (2014)CrossRefGoogle Scholar
  12. 12.
    Yousuf, M., Mehmood, Z., Habib, H.A., Mahmood, T., Saba, T., Rehman, A., Rashid, M.: A novel technique based on visual words fusion analysis of sparse features for effective content-based image retrieval. Math. Probl. Eng. 2018, 1–13 (2018)CrossRefGoogle Scholar
  13. 13.
    Mehmood, Z., Abbas, F., Mahmood, T., Javid, M.A., Rehman, A., Nawaz, T.: Content-based image retrieval based on visual words fusion versus features fusion of local and global features. Arab. J. Sci. Eng. 43(12), 7265–7284 (2018)CrossRefGoogle Scholar
  14. 14.
    Jabeen, S., Mehmood, Z., Mahmood, T., Saba, T., Rehman, A., Mahmood, M.T.: An effective content-based image retrieval technique for image visuals representation based on the bag-of-visual-words model. PLoS ONE 13(4), e0194526 (2018)CrossRefGoogle Scholar
  15. 15.
    Sharif, U., Mehmood, Z., Mahmood, T., Javid, M.A., Rehman, A., Saba, T.: Scene analysis and search using local features and support vector machine for effective content-based image retrieval. Artif. Intell. Rev. 2018, 1–25 (2018)Google Scholar
  16. 16.
    Sarwar, A., Mehmood, Z., Saba, T., Qazi, K.A., Adnan, A., Jamal, H.: A novel method for content-based image retrieval to improve the effectiveness of the bag-of-words model using a support vector machine. J. Inf. Sci. 45(1), 117–135 (2019)CrossRefGoogle Scholar
  17. 17.
    Mehmood, Z., Gul, N., Altaf, M., Mahmood, T., Saba, T., Rehman, A., Mahmood, M.T.: Scene search based on the adapted triangular regions and soft clustering to improve the effectiveness of the visual-bag-of-words model. EURASIP J. Image Video Process. 2018(1), 48 (2018)CrossRefGoogle Scholar
  18. 18.
    Mehmood, Z., Anwar, S.M., Altaf, M.: A novel image retrieval based on rectangular spatial histograms of visual words. Kuwait J. Sci. 45(1), 54–69 (2018)MathSciNetGoogle Scholar
  19. 19.
    Smeulders, A.W., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)CrossRefGoogle Scholar
  20. 20.
    Adebayo, S., McLeod, K., Tudose, I., Osumi-Sutherland, D., Burdett, T., Baldock, R., Parkinson, H.: PhenoImageShare: an image annotation and query infrastructure. J. Biomed. Semant. 7(1), 35–44 (2016)CrossRefGoogle Scholar
  21. 21.
    Kurtz, C., Rubin, D.L.: Utilisation de relations ontologiques pour la comparaison d’images décrites par des annotations sémantiques. In: Proceedings of the (EGC), pp. 609–614 (2014)Google Scholar
  22. 22.
    Lingutla, N.T., Preece, J., Todorovic, S., Cooper, L., Moore, L., Jaiswal, P.: AISO: annotation of image segments with ontologies. J. Biomed. Semant. 5(1), 50–54 (2014)CrossRefGoogle Scholar
  23. 23.
    Hollink, L., Nguyen, G., Schreiber, G., Wielemaker, J., Wielinga, B., Worring, M.: Adding spatial semantics to image annotations. In: Proceedings of the 4th International Workshop on Knowledge Markup and Semantic Annotation (ISWC), pp. 31–40 (2004)Google Scholar
  24. 24.
    Zhang, C., Chai, J., Jin, R.: User term feedback in interactive text-based image retrieval. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 51–58 (2005)Google Scholar
  25. 25.
    Weinberger, K.Q., Slaney, M., Van Zwol, R.: Resolving tag ambiguity. In: Proceedings of the 16th ACM International Conference on Multimedia (MM), pp. 111–120 (2008)Google Scholar
  26. 26.
    Ksibi, A., Ammar, A.B., Amar, C.B.: Adaptive diversification for tag-based social image retrieval. Int. J. Multimed. Inf. Retr. 3(1), 29–39 (2014)CrossRefGoogle Scholar
  27. 27.
    Van Leuken, R.H., Garcia, L., Olivares, X., Van Zwol, R.: Visual diversification of image search results. In: Proceedings of the 9th International Conference on World Wide Web (WWW), pp. 341–350 (2009)Google Scholar
  28. 28.
    Radlinski, F., Bennett, P.N., Carterette, B., Joachims, T.: Redundancy, diversity and interdependent document relevance. SIGIR Forum. 43(2), 46–52 (2009)CrossRefGoogle Scholar
  29. 29.
    Hoque, E., Hoeber, O., Gong, M.: Evaluating the tradeoffs between diversity and precision for web image search using concept-based query expansion. In: Proceedings of the 11th International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), vol. 3, pp. 130–133 (2011)Google Scholar
  30. 30.
    Carpineto, C., Romano, G.: A survey of automatic query expansion in information retrieval. ACM Comput. Surv. 44(1), 1–50 (2012)zbMATHCrossRefGoogle Scholar
  31. 31.
    Lioma, C., Moens, M.F., Azzopardi, L.: Collaborative annotation for pseudo relevance feedback. In: Proceedings of Exploiting Semantic Annotation in Information Retrieval (ESAIR), pp. 25–35 (2008)Google Scholar
  32. 32.
    Naouar, F., Hlaoua, L., Omri, M.N.: Relevance feedback in collaborative information retrieval based on validated annotation. In: Proceedings of the 1st International Conference on Reasoning and Optimization in Information Systems, vol. 13 (2013)Google Scholar
  33. 33.
    Guo, Q., Liu, W., Lin, Y., Lin, H.: Query expansion based on user quality in folksonomy. In: Proceedings of Asia Information Retrieval Symposium (AIRS), pp. 396–405 (2012)Google Scholar
  34. 34.
    Kitanovski, I., Strezoski, G., Dimitrovski, I., Madjarov, G., Loskovska, S.: Multimodal medical image retrieval system. Multimed. Tools Appl. 76(2), 2955–2978 (2017)CrossRefGoogle Scholar
  35. 35.
    Garg, N., Gatica-Perez, D.: Tagging and retrieving images with co-occurrence models: from corel to flickr. In: Proceedings of the 1st ACM Workshop on Large-Scale Multimedia Retrieval and Mining, pp. 105–112 (2009)Google Scholar
  36. 36.
    Jin, S., Lin, H., Su, S.: Query expansion based on folksonomy tag co-occurrence analysis. In: Proceedings of the International Conference on Granular Computing, pp. 300–305 (2009)Google Scholar
  37. 37.
    Broder, A., Ciccolo, P., Gabrilovich, E., Pang, B.: Domain-specific query augmentation using folksonomy tags: the case of contextual advertising. In: Proceedings of the 1st Workshop on Information Retrieval for Advertising (2008)Google Scholar
  38. 38.
    Ksibi, A., Ammar, A.B., Amar, C.B.: Enhanced context-based query-to-concept mapping in social image retrieval. In: Proceedings of the 11th International Workshop on Content-Based Multimedia Indexing (CBMI), pp. 85–89 (2013)Google Scholar
  39. 39.
    Wang, J., Davison, B.D.: Explorations in tag suggestion and query expansion. In: Proceeding of the 2008 ACM Workshop on Search in Social Media (SSM), pp. 43–50 (2008)Google Scholar
  40. 40.
    Biancalana, C., Micarelli, A.: Social tagging in query expansion: a new way for personalized web search. In: Proceedings of the 12th IEEE International Conference on Computational Science and Engineering (CSE), pp. 1060–1065 (2009)Google Scholar
  41. 41.
    Abbasi, R.: Query expansion in folksonomies. In: International Conference on Semantic and Digital Media Technologies (SAMT), pp. 1–16 (2010)Google Scholar
  42. 42.
    He, Y., Li, Y., Lei, J., Leung, C.H.: A framework of query expansion for image retrieval based on knowledge base and concept similarity. Neurocomputing 204, 26–32 (2016)CrossRefGoogle Scholar
  43. 43.
    Otegi, A., Arregi, X., Ansa, O., Agirre, E.: Using knowledge-based relatedness for information retrieval. Knowl. Inf. Syst. 44(3), 689–718 (2015)CrossRefGoogle Scholar
  44. 44.
    Hoque, E., Hoeber, O., Gong, M.: Balancing the trade-offs between diversity and precision for web image search using concept-based query expansion. J. Emerg. Technol. Web Intell. 4(1), 26–34 (2012)Google Scholar
  45. 45.
    Leong, C.W., Hassan, S., Ruiz, M.E., Mihalcea, R.: Improving query expansion for image retrieval via saliency and picturability. In: International Conference of the Cross-Language Evaluation Forum for European Languages (CLEF), pp. 137–142 (2011)Google Scholar
  46. 46.
    Yuvarani, M., Iyengar, N.C.S., Kannan, A.: Improved concept-based query expansion using Wikipedia. Int. J. Commun. Netw. Distrib. Syst. 11(1), 26–41 (2013)CrossRefGoogle Scholar
  47. 47.
    Ayadi, Y., Amous, I., Gargouri, F.: Toward an automatic annotation approach based on ontological enrichment for advanced research. Int. J. Eng. Technol. 13(2), 80–89 (2013)Google Scholar
  48. 48.
    Alromima, W., Moawad, I.F., Elgohary, R., Aref, M.: Ontology-based query expansion for Arabic text retrieval. Int. J. Adv. Comput. Sci. Appl. 7(8), 223–230 (2016)Google Scholar
  49. 49.
    Devi, M.U., Gandhi, G.M.: Wordnet and ontology based query expansion for semantic information retrieval in sports domain. J. Comput. Sci. 11(2), 361–371 (2014)CrossRefGoogle Scholar
  50. 50.
    Charhad, M., Zrigui, M., Quénot, G.: Une approche conceptuelle pour la modélisation et la structuration sémantique des documents vidéos. In: Proceedings of the 3rd International Conference: Sciences of Electronic, Technologies of Information and Telecommunications (SETIT), pp. 1–7 (2005)Google Scholar
  51. 51.
    Bouchakwa, M., Ayadi, Y., Amous, I.: Modeling the semantic content of the socio-tagged images based on the extended conceptual graphs formalism. In: Proceedings of the 14th International Conference on Advances in Mobile Computing and Multi Media (MOMM), pp. 35–39 (2016)Google Scholar
  52. 52.
    De Andrade, D.O.S., Maia, L.F., De Figueirêdo, H.F., Viana, W., Trinta, F., De Souza Baptista, C.: Photo annotation: a survey. Multimed. Tools Appl. 77(1), 423–457 (2018)CrossRefGoogle Scholar
  53. 53.
    Van Leuken, R.H., Garcia, L., Olivares, X., Van Zwol, R.: Visual diversification of image search results. In: Proceedings of the 18th International Conference on World Wide Web (WWW), pp. 341–350 (2009)Google Scholar
  54. 54.
    Wang, M., Yang, K., Hua, X.S., Zhang, H.J.: Towards a relevant and diverse search of social images. IEEE Trans. Multimed. 12(8), 829–842 (2010)CrossRefGoogle Scholar
  55. 55.
    Sun, F., Wang, M., Wang, D., Wang, X.: Optimizing social image search with multiple criteria: relevance, diversity, and typicality. Neurocomputing 95, 40–47 (2012)CrossRefGoogle Scholar
  56. 56.
    Carbonell, J., Goldstein, J.: The use of MMR, diversity-based re-ranking for reordering documents and producing summaries. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 335–336 (1998)Google Scholar
  57. 57.
    Agrawal, R., Gollapudi, S., Halverson, A., Ieong, S.: Diversifying search results. In: Proceedings of the 2nd ACM International Conference on Web Search and Data Mining (WSDM), pp. 5–14 (2009)Google Scholar
  58. 58.
    Santos, R.L., Macdonald, C., Ounis, I.: Exploiting query reformulations for web search result diversification. In: Proceedings of the 19th International Conference on World Wide Web (WWW), pp. 881–890 (2010)Google Scholar
  59. 59.
    Santos, R.L., Macdonald, C., Ounis, I.: Selectively diversifying web search results. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM), pp. 1179–1188 (2010)Google Scholar
  60. 60.
    Song, K., Tian, Y., Gao, W., Huang, T.: Diversifying the image retrieval results. In: Proceedings of the 14th ACM International Conference on Multimedia (ICM), pp. 707–710 (2006)Google Scholar
  61. 61.
    Qian, X., Lu, D., Wang, Y., Zhu, L., Tang, Y.Y., Wang, M.: Image re-ranking based on topic diversity. IEEE Trans. Image Process. 26(8), 3734–3747 (2017)MathSciNetzbMATHCrossRefGoogle Scholar
  62. 62.
    Yan, Y., Liu, G., Wang, S., Zhang, J., Zheng, K.: Graph-based clustering and ranking for diversified image search. Multimed. Syst. 23(1), 41–52 (2017)CrossRefGoogle Scholar
  63. 63.
    Lashari, S.A., Ibrahim, R.: A framework for medical images classification using soft set. Procedia Technol. 11(2013), 548–556 (2013)CrossRefGoogle Scholar
  64. 64.
    Wen, W., Hao, Z., Cai, R., Shao, Z.: Gaussian process learning for image classification based on low-level features. In: Proceedings of the 8th International Conference on Natural Computation (CNC), pp. 237–241 (2012)Google Scholar
  65. 65.
    Cai, D., He, X., Li, Z., Ma, W.Y., Wen, J.R.: Hierarchical clustering of WWW image search results using visual, textual and link information. In: Proceedings of the 12th Annual ACM International Conference on Multimedia (MULTIMEDIA), pp. 952–959 (2004)Google Scholar
  66. 66.
    Sabarad, A.K., Kankudti, M.H., Meena, S.M., Husain, M.: Color and texture feature extraction using Apache Hadoop framework. In: Proceedings of the 2015 International Conference on Computing Communication Control and Automation (ICCUBEA), pp. 585–588 (2015)Google Scholar
  67. 67.
    Zhu, H., Shen, Z., Shang, L., Zhang, X.: Parallel image texture feature extraction under Hadoop cloud platform. In: Proceedings of the International Conference on Intelligent Computing (ICIC), pp. 459–465 (2014)Google Scholar
  68. 68.
    Qayyum, A., Anwar, S.M., Awais, M., Majid, M.: Medical image retrieval using deep convolutional neural network. Neurocomputing 266, 8–20 (2017)CrossRefGoogle Scholar
  69. 69.
    Li, Y.D., Hao, Z.B., Lei, H.: Survey of convolutional neural network. Int. J. Comput. Appl. 36(9), 2508–2515 (2016)Google Scholar
  70. 70.
    Lyndon, D., Kumar, A., Kim, J., Leong, P.H. W., Feng, D.: Convolutional neural networks for medical clustering. In: CLEF (Working Notes) (2015)Google Scholar
  71. 71.
    Mehmood, Z., Rashid, M., Rehman, A., Saba, T., Dawood, H., Dawood, H.: Effect of complementary visual words versus complementary features on clustering for effective content-based image search. J. Intell. Fuzzy Syst. 2019, 1–14 (2019)Google Scholar
  72. 72.
    Cha, G.H.: Kernel principal component analysis for content based image retrieval. In: Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), pp. 844–849 (2005)Google Scholar
  73. 73.
    Wang, S., Jing, F., He, J., Du, Q., Zhang, L.: Igroup: presenting web image search results in semantic clusters. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 587–596 (2007)Google Scholar
  74. 74.
    Ramage, D., Heymann, P., Manning, C.D., Garcia-Molina, H.: Clustering the tagged web. In: Proceedings of the 2nd ACM International Conference on Web Search and Data Mining (WSDM), pp. 54–63 (2009)Google Scholar
  75. 75.
    Yin, Z., Li, R., Mei, Q., Han, J.: Exploring social tagging graph for web object classification. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), pp. 957–966 (2009)Google Scholar
  76. 76.
    Wang, X.J., Zhang, L., Jing, F., Ma, W.-Y.: Annosearch: image auto-annotation by search. Proc. Comput. Vis. Pattern Recogn. 2, 1483–1490 (2006)Google Scholar
  77. 77.
    Bouchakwa, M., Ayadi, Y., Amous, I.: Semantic pattern-based automatic annotation process of images shared on social networks. In: Proceedings of the 30th IBIMA Conference (IBIMA), pp. 19–38 (2017)Google Scholar
  78. 78.
    Sowa, J.F.: Conceptual structures: information processing in mind and machine. Addison-Wesley publishing Company, Reading (1984)zbMATHGoogle Scholar
  79. 79.
    Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1), 37–46 (1960)CrossRefGoogle Scholar
  80. 80.
    MacQueen, J.: Some methods for classification and analysis of multivariate observations. In: Proceedings of the 5th Berkeley Symposium On Mathematical Statistics and Probabilities, vol. 1, pp. 281–296 (1967)Google Scholar
  81. 81.
    Thorndike, R.L.: Who belongs in the family? Psychometrika 18(4), 267–276 (1953)CrossRefGoogle Scholar
  82. 82.
    Rousseeuw, P.J., Kaufman, L.: Finding groups in data. Ser. Probab. Math. Stat. 34(1), 111–112 (1990)zbMATHGoogle Scholar
  83. 83.
    Tibshirani, R., Walther, G., Hastie, T.: Estimating the number of clusters in a data set via the gap statistic. J R. Stat. Soc. 63(2), 411–423 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  84. 84.
    Bhusare, B.B., Bansode, S.M.: Centroids initialization for K-means clustering using improved pillar algorithm. Int. J. Adv. Res. Comput. Eng. Technol. 3(4), 1317–1322 (2014)Google Scholar
  85. 85.
    Chris, B., Ellen, M.: Evaluating evaluation measure stability. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 33–40 (2000)Google Scholar
  86. 86.
    Ellen M.V.: The philosophy of information retrieval evaluation. In: CLEFEL, pp. 355–370 (2001)Google Scholar
  87. 87.
    Im, D.H., Park, G.D.: Linked tag: image annotation using semantic relationships between image tags. Int. J. Multimed. Tools Appl. 74(7), 2273–2287 (2015)CrossRefGoogle Scholar
  88. 88.
    Ricardo, B., Berthier, R.: Modern Information Retrieval. ACM Press, NewYork (1999)Google Scholar
  89. 89.
    Mehmood, Z., Anwar, S.M., Ali, N., Habib, H.A., Rashid, M.: A novel image retrieval based on a combination of local and global histograms of visual words. Math. Probl. Eng. 2016, 12 (2016)CrossRefGoogle Scholar
  90. 90.
    Tian, X., Jiao, L., Liu, X., Zhang, X.: Feature integration of EODH and Color-SIFT: application to image retrieval based on codebook. Signal Process. Image Commun. 29(4), 530–545 (2014)CrossRefGoogle Scholar
  91. 91.
    Ali, N., Bajwa, K.B., Sablatnig, R., Mehmood, Z.: Image retrieval by addition of spatial information based on histograms of triangular regions. Comput. Electr. Eng. 54, 539–550 (2016)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.MIRACL Laboratory, Technopole of SfaxUniversity of SfaxSfaxTunisia

Personalised recommendations