Multimedia Tools and Applications

, Volume 62, Issue 3, pp 601–631 | Cite as

Improving image tags by exploiting web search results

  • Xiaoming ZhangEmail author
  • Zhoujun Li
  • Wenhan Chao


Automatic image tagging automatically assigns image with semantic keywords called tags, which significantly facilitates image search and organization. Most of present image tagging approaches are constrained by the training model learned from the training dataset, and moreover they have no exploitation on other type of web resource (e.g., web text documents). In this paper, we proposed a search based image tagging algorithm (CTSTag), in which the result tags are derived from web search result. Specifically, it assigns the query image with a more comprehensive tag set derived from both web images and web text documents. First, a content-based image search technology is used to retrieve a set of visually similar images which are ranked by the semantic consistency values. Then, a set of relevant tags are derived from these top ranked images as the initial tag set. Second, a text-based search is used to retrieve other relevant web resources by using the initial tag set as the query. After the denoising process, the initial tag set is expanded with other tags mined from the text-based search result. Then, an probability flow measure method is proposed to estimate the probabilities of the expanded tags. Finally, all the tags are refined using the Random Walk with Restart (RWR) method and the top ones are assigned to the query images. Experiments on NUS-WIDE dataset show not only the performance of the proposed algorithm but also the advantage of image retrieval and organization based on the result tags.


Image tagging Search based tagging Tag expansion Image retrieval 



This work was supported by the National Natural Science Foundations of China (60973105 and 61003111), and the fund of the State Key Laboratory of Software Development Environment (SKLSDE-2011ZX-03). The authors would like to thank the Editors and the anonymous reviewers 739 for their valuable comments and remarks on the previous versions of this paper.


  1. 1.
    Bailloeul T, Zhu CZ, Xu YH (2008) Automatic image tagging as a random walk with priors on the canonical correlation subspace. In: Proceeding of 9th ACM international conference on multimedia information retrieval, pp 75–82Google Scholar
  2. 2.
    Barnard K, Duygulu P, Forsyth D, de Freitas N, Blei DM, Jordan MI (2003) Matching words and pictures. J Mach Learn Res 3(6):1107–1135zbMATHGoogle Scholar
  3. 3.
    Bruza PD, Song D (2002) Inferring query models by computing information flow. In: Proceedings of CIKM 2002, pp 260–269Google Scholar
  4. 4.
    Cao G, Nie J, Gao J, Robertson S (2008) Selecting good expansion terms for pseudo-relevance feedback. In: Proceedings of the 31st ACM SIGIR conference on research and development in information retrieval. Singapore, pp 243–250Google Scholar
  5. 5.
    Cao L, Pozo AD, Jin X, Luo J, Han J (2010) RankCompete: simultaneous ranking and clustering of web photos. In: Proceedings of the 19th international conference on World Wide WebGoogle Scholar
  6. 6.
    Chang SF, He J, Jiang YG, El Khoury E, Ngo CW, Yanagawa A, Zavesky E (2008) Columbia University/VIREO-CityU/IRIT TRECVID2008 high-level feature extraction and interactive video search. In: Proceedings of TRECVID 2008Google Scholar
  7. 7.
    Chen XY, Mu YD, Yan SC, Chua TS (2010) Efficient large-scale image annotation by probabilistic collaborative multi-label propagation. In: Proceedings of 18th annual ACM international conference on multimedia, pp 35–44Google Scholar
  8. 8.
    Chua T-S, Tang J, Hong R, Li H, Luo Z, Zheng Y-T (2009) NUS-WIDE: a real-world web image database from national University of Singapore. In: ACM international conference on image and video retrieval. Greece, 8–10 Jul 2009Google Scholar
  9. 9.
    Croft WB, Lafferty J (2002) Language models for information retrieval. Kluwer int. series on information retrieval, vol 13. Kluwer Academic PublishersGoogle Scholar
  10. 10.
    Geng B, Yang L, Xu C, Hua X (2008) Collaborative learning for image and video annotation. In: Proceeding of the 1st ACM international conference on multimedia information retrieval, pp 443–450Google Scholar
  11. 11.
    Han J, Kamber M (2001) Data mining: concepts and techniques. Morgan KaufmannGoogle Scholar
  12. 12.
    Heesch D, Yavlinsky A, Ruger S (2006) Nnk: networks and automated annotation for browsing large image collections from the World Wide Web. In: Proceedings of the 14th ACM International Conference on Multimedia, pp 493–494Google Scholar
  13. 13.
    Hong R, Wang M, Xu M, Yan S, Chua T-S (2010) Dynamic caption: video accessibility enhancement for hearing impairment. In: ACM international conference on multimedia (ACM MM)Google Scholar
  14. 14.
    Naphade M, Smith JR, Tesic J, Chang S-F, Hsu W, Kennedy L, Hauptmann A, Curtis J (2006) Large-scale concept ontology for multimedia. IEEE Multimed 13(3):86–91CrossRefGoogle Scholar
  15. 15.
    Jing F, Wang C, Yao Y, Deng K, Zhang L, Ma W (2006) IGroup: web image search results clustering. In: Proceedings of the 14th annual ACM international conference on multimedia, pp 377–384Google Scholar
  16. 16.
    Jones KS, Walker S, Robertson SE (2000) A probabilistic model of information retrieval: development and comparative experiments—part 2. Journal of Information Processing and Management 36(6):809–840CrossRefGoogle Scholar
  17. 17.
    Lei W, Linjun Y, Nenghai Y, Hua XS (2009) Learning to tag. In: Proceedings of the 18th ACM international conference on World Wide Web, pp 20–24Google Scholar
  18. 18.
    Lei W, Steven CH, Rong Jin H, Jianke Z, Nenghai Y (2009) Distance Metric Learning from Uncertain Side Information with Application to Automated Photo Tagging. In: Proceeding of 17th ACM international conference on multimedia, pp 15–24Google Scholar
  19. 19.
    Li J, Wang JZ (2006) Real-time computerized annotation of pictures. In: Proceedings of the 14th annual ACM international conference on multimedia, pp 911–920Google Scholar
  20. 20.
    Li X, Snoek CGM (2009) Visual categorization with negative examples for free. In: Proceedings of the 17th international conference on multimedia, pp 661–664Google Scholar
  21. 21.
    Li X, Snoek CG, Worring M (2009) Learning social tag relevance by neighbor voting. IEEE Trans Multimed 11(7):1310–1322CrossRefGoogle Scholar
  22. 22.
    Li X-R, Snoek CG, Worring M (2009) Annotating images by harnessing worldwide user-tagged photos. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing, pp 3717–3720Google Scholar
  23. 23.
    Liu J, Wang B, Li M, Li Z, Ma W, Lu H, Ma S (2007) Dual cross-media relevance model for image annotation. In: Proceedings of the 15th international conference on multimedia, pp 605–614Google Scholar
  24. 24.
    Liu Y, Zhang D, Lu G, Ma WY (2007) A survey of content-based image retrieval with high-level semantics. Pattern Recogn 40(1):262–282zbMATHCrossRefGoogle Scholar
  25. 25.
    Liu D, Wang M, Hua XS, Zhang HJ (2009) Tag ranking. In: Proceeding of the 18th ACM international conference on World Wide Web, pp 351–340Google Scholar
  26. 26.
    Lu Y, Zhang L, Tian Q, Ma W-Y (2008) What are the high-level concepts with small semantic gaps? In: Proceeding of IEEE 21th conference on computer vision and pattern recognition, pp 1–8Google Scholar
  27. 27.
    Page L, Brin S, Motwani R, Winograd T (1998) The pagerank citationranking: bringing order to theWeb, technical report. Stanford University, StanfordGoogle Scholar
  28. 28.
    Russell BC, Torralba A, Murphy KP, Freeman WT (2008) LabelMe: a database and web-based tool for image annotation. Int J Comput Vis 77(1):157–173CrossRefGoogle Scholar
  29. 29.
    Setz AT, Snoek CGM (2009) Can social tagged images aid concept-based video search? In: Proceedings of ICME, pp 1460–1463Google Scholar
  30. 30.
    Shen Y, Fan JP (2010) Leveraging loosely-tagged images and inter-object correlations for tag recommendation. In: Proceedings of 18th annual ACM international conference on multimedia, pp 5–14Google Scholar
  31. 31.
    Siersdorfer S, San Pedro J, Sanderson M (2009) Automatic video tagging using content redundancy. In: Proceeding of the 32nd ACM international conference on research and development in information retrieval, pp 16–23Google Scholar
  32. 32.
    Tong H, Faloutsos C, Pan J (2006) Fast random walk with restart and its applications. In: Proceedings of the IEEE 6th international conference on data mining, pp 613–622Google Scholar
  33. 33.
    Tsikrika T, Diou C, de Vries AP, Delopoulos A (2010) Reliability and effectiveness of clickthrough data for automatic image annotation. Multimed Tools Appl 55(1):27–52CrossRefGoogle Scholar
  34. 34.
    Turtle HR, Croft WB (1992) A comparison of text retrieval models. Comput J 35(3):279–298zbMATHCrossRefGoogle Scholar
  35. 35.
    Vassilieva NS (2009) Content-based image retrieval methods. Program Comput Softw 35(3):158–180MathSciNetCrossRefGoogle Scholar
  36. 36.
    Wang C, Jing F, Zhang L, Zhang H-J (2006) Image annotation refinement using random walk with restarts. In: Proceedings of 14th ACM international conference on multimedia, pp 647–650Google Scholar
  37. 37.
    Wang X, Zhang L, Jing F, Ma W (2006) AnnoSearch: image auto-annotation by search. In: Proceedings of the 19th IEEE computer society conference on computer vision and pattern recognition, vol 2, pp 1483–1490Google Scholar
  38. 38.
    Wang C, Jing F, Zhang L, Zhang HJ (2008) Scalable search-based image annotation. Multimedia Syst 14(4):205–220CrossRefGoogle Scholar
  39. 39.
    Wang XJ, Zhang L, Li XR, Ma W-Y (2008) Annotating images by mining image search results. IEEE Trans Pattern Anal Mach Intell 30(11):1919–1932CrossRefGoogle Scholar
  40. 40.
    Wang M, Hua X-S, Tang J, Hong R (2009) Beyond distance measurement: constructing neighborhood similarity for video annotation. IEEE Trans Multimedia 11(3):465–476CrossRefGoogle Scholar
  41. 41.
    Yang K, Wang M, Zhang H (2009) Active tagging for image indexing. In: Proceedings of the IEEE international conference on multimedia and expo, pp 1620–1623Google Scholar
  42. 42.
    Zhou X, Wang M, Zhang Q, Zhang J, Shi B (2007) Automatic image annotation by an iterative approach: incorporating keyword correlations and region matching. In: Proceedings of the 6th ACM international conference on image and video retrieval, pp 25–32Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.State Key Laboratory of Software Development EnvironmentBeihang UniversityBeijingChina
  2. 2.School of Computer Science and EngineeringBeihang UniversityBeijingChina
  3. 3.Beijing Key Laboratory of Network TechnologyBeihang UniversityBeijingChina

Personalised recommendations