Multimedia Systems

, Volume 22, Issue 4, pp 509–523 | Cite as

The effects of multiple query evidences on social image retrieval

  • Zhiyong Cheng
  • Jialie ShenEmail author
  • Haiyan Miao
Special Issue Paper


System performance assessment and comparison are fundamental for large-scale image search engine development. This article documents a set of comprehensive empirical studies to explore the effects of multiple query evidences on large-scale social image search. The search performance based on the social tags, different kinds of visual features and their combinations are systematically studied and analyzed. To quantify the visual query complexity, a novel quantitative metric is proposed and applied to assess the influences of different visual queries based on their complexity levels. Besides, we also study the effects of automatic text query expansion with social tags using a pseudo relevance feedback method on the retrieval performance. Our analysis of experimental results shows a few key research findings: (1) social tag-based retrieval methods can achieve much better results than content-based retrieval methods; (2) a combination of textual and visual features can significantly and consistently improve the search performance; (3) the complexity of image queries has a strong correlation with retrieval results’ quality—more complex queries lead to poorer search effectiveness; and (4) query expansion based on social tags frequently causes search topic drift and consequently leads to performance degradation.


Query evidence Social image retrieval Performance  Evaluation Experimentation 



Jialie Shen and Zhiyong Cheng are partially supported by Mobile plus Cloud Computing Theme Research Program Award—Microsoft Research. Portions of this work appeared in [7, 52].


  1. 1.
    Jeffries, A.: The man behind flickr on making the service ‘awesome again’. The Verge. 2013–03–20. Retrieved 2013–08–29Google Scholar
  2. 2.
    Benavent, J., Benavent, X., Ves, E., Granados, R., Serrano, A.G.: Experiences at ImageCLEF 2010 using CBIR and TBIR mixing information approaches. In: Cross-Language Evaluation Forum CLEF 2010 (2010)Google Scholar
  3. 3.
    Carmel, D., Yom-Tov, E., Darlow, A., Pelleg, D.: What makes a query difficult? In: Proceedings of ACM SIGIR (2006)Google Scholar
  4. 4.
    Carpineto, C., Romano, G.: A survey of automatic query expansion in information retrieval. ACM Comput. Surv. 44(1), 1–50 (2012)CrossRefzbMATHGoogle Scholar
  5. 5.
    Chen, S., Goodman, J.: An empirical study of smoothing techniques for language modeling. Technical report (1998)Google Scholar
  6. 6.
    Cheng, P., Yeh, J., Ke, H., Chien, B., Yang, W.: NCTU-ISU’s evaluation for the user-centered search task at ImageCLEF 2004. In: Cross-Language Evaluation Forum CLEF 2004 (2004)Google Scholar
  7. 7.
    Cheng, Z., Ren, J., Shen, J., Miao, H.: The effects of heterogeneous information combination on large scale social image search. In: Proceedings of ACM ICIMCS (2011)Google Scholar
  8. 8.
    Cheng, Z., Ren, J., Shen, J., Miao, H.: Building a large scale test collection for effective benchmarking of mobile landmark search. In: Proceedings of MMM (2013)Google Scholar
  9. 9.
    Chua, T., Tang, J., Hong, R., Li, H., Luo, Z., Zheng, Y.: NUS-WIDE: a real-world web image database from National University of Singapore. In: Proceedings of ACM CIVR (2009)Google Scholar
  10. 10.
    Cronen-Townsend, S., Zhou, Y., Croft, W.B.: Predicting query performance. In: Proceedings of ACM SIGIR (2002)Google Scholar
  11. 11.
    Datta, R., Joshi, D., Li, J., Wang, J.: Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40(2), Article 5 (2008)Google Scholar
  12. 12.
    Deselaers, T., Keysers, D., Ney, H.: Features for image retrieval: an experimental comparison. Inf. Retr. 11(2), 77–107 (2008)CrossRefGoogle Scholar
  13. 13.
    Fan, J., Shen, Y., Zhou, N., Gao, Y.: Harvesting large-scale weakly-tagged image databases from the web. In: Proceedings of IEEE CVPR (2010)Google Scholar
  14. 14.
    Gao, Y., Wang, F., Luan, H., Chua, T.: Brand data gathering from live social media streams. In: Proceedings of ACM ICMR (2014)Google Scholar
  15. 15.
    Gao, Y., Wang, M., Tao, D., Ji, R., Dai, Q.: 3-d object retrieval and recognition with hypergraph analysis. IEEE Trans. Multimed. 21(9), 4290–4303 (2012)MathSciNetGoogle Scholar
  16. 16.
    Gao, Y., Wang, M., Zha, Z., Shen, J., Li, X., Wu, X.: Visual-textual joint relevance learning for tag-based social image search. IEEE Trans. Image Process 22(1), 363–376 (2013)MathSciNetCrossRefGoogle Scholar
  17. 17.
    Gao, Y., Wang, M., Zha, Z., Tian, Q., Dai, Q., Zhang, N.: Less is more: efficient 3-d object retrieval with query view selection. IEEE Trans. Image Process 13(5), 1007–1018 (2011)Google Scholar
  18. 18.
    Geng, B., Yang, L., Xu, C., Hua, X., Li, S.: The role of attractiveness in web image search. In: Proceedings of ACM MM (2011)Google Scholar
  19. 19.
    Grauman, K., Fergus, R.: Learning binary hash codes for large-scale image search. In: Machine learning for computer vision, pp. 49–87. Springer (2013)Google Scholar
  20. 20.
    Guillaumin, M., Mensink, T., Verbeek, J., Schmid, C.: TagProp: Discriminative metric learning in nearest neighbor models for image auto-annotation. In: Proceedings of IEEE ICCV (2009)Google Scholar
  21. 21.
    He, B., Ounis, I.: Inferring query performance using pre-retrieval predictors. In: String processing and information retrieval (2004)Google Scholar
  22. 22.
    He, B., Ounis, I.: Query performance prediction. Inf. Syst. 31(7), 585–594 (2006)CrossRefGoogle Scholar
  23. 23.
    Huang, J., Kumar, S., Mitra, M., Zhu, W.J., Zabih, R.: Image indexing using color correlogram. In: Proceedings of IEEE CVPR (1997)Google Scholar
  24. 24.
    Huiskes, M., Lew, M.: The MIR flickr retrieval evaluation. In: Proceedings of ACM MIR (2008)Google Scholar
  25. 25.
    Jeon, J., Lavrenko, V., Manmatha, R.: Automatic image annotation and retrieval using crossmedia relevance models. In: Proceedings of ACM SIGIR (2003)Google Scholar
  26. 26.
    Jones, K., Walker, S., Robertson, S.: A probabilistic model of information retrieval: development and comparative experiments—part 2. Information Processing and Management 36(6), 809–840 (2000)CrossRefGoogle Scholar
  27. 27.
    Kennedy, L., Naaman, M.: Generating diverse and representative image search results for landmarks. In: Proceedings of WWW (2008)Google Scholar
  28. 28.
    Kurashima, T., Iwata, T., Irie, G., Fujimura, K.: Travel route recommendation using geotags in photo sharing sites. In: Proceedings of CIKM (2010)Google Scholar
  29. 29.
    Li, X., Snoek, C., Worring, M.: Learning social tag relevance by neighbor voting. IEEE Trans. Multimed. 11(7), 1310–1320 (2009)CrossRefGoogle Scholar
  30. 30.
    Li, X., Snoek, C., Worring, M.: Unsupervised multi-feature tag relevance learning for social image retrieval. In: Proceedings of ACM CIVR (2010)Google Scholar
  31. 31.
    Li, Y., Geng, B., Yang, L., Xu, C., Bian, W.: Query difficulty estimation for image retrieval. Neurocomputing 95, 48–53 (2012)CrossRefGoogle Scholar
  32. 32.
    Li, Y., Geng, B., Zha, Z., Tao, D., Yang, L., Xu, C.: Difficulty guided image retrieval using linear multiview embedding. In: Proceedings of ACM MM (2011)Google Scholar
  33. 33.
    Liu, D., Hua, X., Yang, L., Wang, M., Zhang, H.: Tag ranking. In: Proceedings of WWW (2009)Google Scholar
  34. 34.
    Liu, D., Hua, X., Zhang, H.: Content-based tag processing for internet social images. Multimed. Tools Appl. 51(2), 723–728 (2011)CrossRefGoogle Scholar
  35. 35.
    Liu, D., Wang, M., Yang, Y., Hua, X., Zhang, H.: Tag quality improvement for social images. In: Proceedings of IEEE ICME (2009)Google Scholar
  36. 36.
    Liu, D., Yan, S., Hua, X., Zhang, H.: Image retagging using collaborative tag propagation. IEEE Trans. Multimed. 13(4), 702–712 (2011)CrossRefGoogle Scholar
  37. 37.
    Liu, X., Cheng, B., Yan, S., Tang, J., Chua, T., Jin, H.: Label to region by bi-layer sparsity priors. In: Proceedings of ACM MM (2009)Google Scholar
  38. 38.
    Manjunath, B., Ma, W.: Texture features for browsing and retrieval of image data. IEEE Trans. Pattern Anal. Mach. Intell. 18(8), 837–842 (1996)CrossRefGoogle Scholar
  39. 39.
    Manning, C., Raghavan, P., Schutze, H.: An introduction to information retrieval. Cambridge University Press,   (2009)zbMATHGoogle Scholar
  40. 40.
    Metzler, D., Croft, W.: Linear feature-based models for information retrieval. Inf Retr 10(3), 257–274 (2007)CrossRefGoogle Scholar
  41. 41.
    Mu, Y., Shen, J., Yan, S.: Weakly-supervised hashing in kernel space. In: IEEE Proceedings of CVPR (2010)Google Scholar
  42. 42.
    Natsev, A., Haubold, A., Tesic, J., Xie, L., Yan, R.: Semantic concept-based query expansion and re-ranking for multimedia retrieval. In: Proceedings of ACM MM (2007)Google Scholar
  43. 43.
    Nie, L., Wang, M., Zha, Z., Chua, T.: Oracle in image search: A content-based approach to performance prediction. ACM Trans. Inf. Syst. 30(2), 13 (2012)CrossRefGoogle Scholar
  44. 44.
    Nov, O., Ye, C.: Why do people tag?: Motivations for photo tagging. Commun. ACM 53(7), 128–131 (2010)CrossRefGoogle Scholar
  45. 45.
    Parfeni, L.: Flickr boasts 6 billion photo uploads. Softpedia. Retrieved 2012–03-01Google Scholar
  46. 46.
    Park, D., Jeon, Y., Won, C.: Efficient use of local edge histogram descriptor. In: Proceedings of ACM MM (2000)Google Scholar
  47. 47.
    Rao, A., Srihari, R., Zhu, L., Zhang, A.: A method for mearsuring the complexity of image databases. IEEE Trans. Multimed. 4(2), 160–173 (2002)CrossRefGoogle Scholar
  48. 48.
    Rui, Y., Huang, T., Chang, S.: Image retrieval: Current techniques, promising directions, and open issues. J. Vis. Commun. Image Represent 10(1), 39–62 (1999)CrossRefGoogle Scholar
  49. 49.
    Salton, G., Wong, A., Yang, C.: A vector space model for automatic indexing. Commun ACM 18(11), 613–620 (1975)CrossRefzbMATHGoogle Scholar
  50. 50.
    Shannon, C.: Prediction and entropy of printed english. Bell Syst. Tech. J. 30(1), 50–64 (1951)CrossRefzbMATHGoogle Scholar
  51. 51.
    Shaw, J., Fox, E.: Combination of multiple searches. In: The Second Text REtrieval Conference (TREC-2), pp. 243–252 (1994)Google Scholar
  52. 52.
    Shen, J., Cheng, Z.: On effects of visual query complexity. In: The Era of Interactive Media, pp. 531–541. Springer (2013)Google Scholar
  53. 53.
    Shen, J., Wang, M., Yan, S., Hua, X.S.: Multimedia tagging: past, present and future. In: Proceedings of ACM MM Conference (2011)Google Scholar
  54. 54.
    Smeulders, A., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(2), 1349–1380 (2000)CrossRefGoogle Scholar
  55. 55.
    Tamura, H., Mori, S., Yamawaki, T.: Textural features corresponding to visual perception. IEEE Trans. Syst., Man, Cybern. 8(6), 460–473 (1978)CrossRefGoogle Scholar
  56. 56.
    Tamura, H., Yokoya, N.: Image database systems: a survey. Pattern Recognit 17(1), 29–43 (1984)CrossRefGoogle Scholar
  57. 57.
    Tian, X., Lu, Y., Yang, L.: Query difficulty prediction for web image search. IEEE Trans. Multimed. 14(4), 951–962 (2012)CrossRefGoogle Scholar
  58. 58.
    Tsai, D., Jing, Y., Liu, Y., Rowley, H., Ioffe, S., Rehg, J.: Large-scale image annotation using visual synset. In: Proc. of IEEE ICCV (2011)Google Scholar
  59. 59.
    Wang, J., Hua, X.: Interactive image search by color map. ACM Trans. Intell. Syst. Technol. 3(1), Article 12 (2011)Google Scholar
  60. 60.
    Wang, J., Jia, L., Hua, X.: Interactive browsing via diversified visual summarization for image search results. Multimed. Syst. 17(5), 379–391 (2011)CrossRefGoogle Scholar
  61. 61.
    Wang, M., Ni, B., Hua, X., Chua, T.: Assistive tagging: A survey of multimedia tagging with human-computer joint exploration. ACM Comput. Surv. 44(4), Article 25 (2012)Google Scholar
  62. 62.
    Wang, M., Yang, K., Hua, X., Zhang, H.: Towards a relevant and diverse search of social images. IEEE Trans. Multimed. 12(8), 829–842 (2010)CrossRefGoogle Scholar
  63. 63.
    Wilkins, P., Smeaton, A., Ferguson, P.: Properties of optimally weighted data fusion in CBMIR. In: Proceedings of ACM SIGIR (2010)Google Scholar
  64. 64.
    Wu, L., Li, M., Li, Z., Ma, W., Yu, N.: Visual language modeling for image classification. In: Proceedings of ACM MIR (2007)Google Scholar
  65. 65.
    Xing, X., Zhang, Y., Han, M.: Query difficulty prediction for contextual image retrieval. In: Advances in Information Retrieval, pp. 581–585. Springer (2010)Google Scholar
  66. 66.
    Xu, H., Wang, J., Hua, X., Li, S.: Tag refinement by regularized LDA. In: Proceedings of ACM MM (2009)Google Scholar
  67. 67.
    Xu, H., Wang, J., Hua, X., Li, S.: Interactive image search by 2D semantic map. In: Proceedings of WWW (2010)Google Scholar
  68. 68.
    Xu, H., Wang, J., Hua, X., Li, S.: Hybrid image summarization. In: Proceedings of ACM MM (2011)Google Scholar
  69. 69.
    Yang, K., Hua, X., Wang, M., Zhang, H.: Tag tagging: Towards more descriptive keywords of image content. IEEE Trans. Multimed. 13(4), 662–673 (2011)CrossRefGoogle Scholar
  70. 70.
    Yang, Y., Huang, Z., Shen, H., Zhou, X.: Mining multi-tag association for image tagging. World Wide Web 14(2), 133–156 (2011)CrossRefGoogle Scholar
  71. 71.
    Yang, Y., Huang, Z., Yang, Y., Liu, J., Shen, H., Luo, J.: Local image tagging via graph regularized joint group sparsity. Pattern Recognit 46(5), 1358–1368 (2013)CrossRefzbMATHGoogle Scholar
  72. 72.
    Yang, Y., Yang, Y., Huang, Z., Shen, H., Nie, F.: Tag localization with spatial correlations and joint group sparsity. In: Proceedings of IEEE CVPR (2011)Google Scholar
  73. 73.
    Yang, Y., Yang, Y., Shen, H.: Effective transfer tagging from image to video. ACM Trans. Multimed. Comput. Commun. Appl. 9(2), 14 (2013)CrossRefGoogle Scholar
  74. 74.
    Zhou, X., Huang, T.: Relevance feedback in image retrieval: a comprehensive review. Multimed. Syst. 8(6), 536–544 (2003)CrossRefGoogle Scholar
  75. 75.
    Zhu, G., Yan, S., Ma, Y.: Image tag refinement towards low-rank, content-tag prior and error sparsity. In: Proceeding of ACM MM (2010)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  1. 1.School of Information SystemsSingapore Management UniversitySingapore Singapore
  2. 2.A*STARInstitute of High Performance ComputingSingaporeSingapore

Personalised recommendations