Combining conceptual query expansion and visual search results exploration for web image retrieval

  • Enamul Hoque
  • Orland Hoeber
  • Grant Strong
  • Minglun Gong
Original Research


Most approaches to image retrieval on the web have their basis in document search techniques. Images are indexed based on the text that is related to the images. Queries are matched to this text to produce a set of search results, which are organized in paged grids that are reminiscent of lists of documents. Due to ambiguity both with user-supplied queries and with the text used to describe the images within the search index, most image searches contain many irrelevant images distributed throughout the search results, and are often focused on the most common interpretation of the query. We propose a method for addressing these problems in which conceptual query expansion is used to generate a diverse range of images, and a multi-resolution extension of a self-organizing map is used to group visually similar images. The resulting interface acts as an intelligent search assistant, automatically diversifying the search results and then allowing the searcher to interactively highlight and filter images based on the concepts, and zoom into an area within the image space to show additional images that are visually similar. Evaluations show that the precision of the image search results increase as a result of concept-based focusing and filtering, as well as visual zooming operations, even for uncommon interpretations of ambiguous queries.


Conceptual query expansion Image search results organization Web image retrieval Interactive exploration 


  1. André P, Cutrell E, Tan DS, Smith G (2009) Designing novel image search interfaces by understanding unique characteristics and usage. In: Proceedings of the IFIP conference on human–computer interaction, pp 340–353Google Scholar
  2. Bentley JL (1975) Multidimensional binary search trees used for associative searching. Commun ACM 18(9):509–517MathSciNetzbMATHCrossRefGoogle Scholar
  3. Bhogal J, Macfarlane A, Smith P (2007) A review of ontology based query expansion. Inf Process Manag 43(4):866–886. ISSN 0306-4573CrossRefGoogle Scholar
  4. Bizer C, Lehmann J, Kobilarov G, Auer S, Becker C, Cyganiak R, Hellmann S (2009) DBpedia—a crystallization point for the web of data. J Web Semant Sci Serv Agents World Wide Web 7(3):154–165CrossRefGoogle Scholar
  5. Cai D, He X, Li Z, Ma W-Y, Wen J-R (2004) Hierarchical clustering of www image search results using visual, textual and link information. In: Proceedings of the annual ACM international conference on multimedia, pp 952–959. ISBN 1-58113-893-8Google Scholar
  6. Chen C, Gagaudakis G, Rosin P (2000) Similarity-based image browsing. In: Proceedings of the IFIP international conference on intelligent information processing, pp 206–213Google Scholar
  7. Datta R, Joshi D, Li J, Wang JZ (2008) Image retrieval: Ideas, influences, and trends of the new age. ACM Comput Surv 40(2):1–60CrossRefGoogle Scholar
  8. Deselaers T, Gass T, Dreuw P, Ney H (2009) Jointy optimising relevance and diversity in image retrieval. In: Proceedings of the ACM international conference on image and video retrieval, pp 1–8Google Scholar
  9. Efthimiadis EN (1996) Query expansion. Annu Rev Inf Syst Technol (ARIST) 31:121–187Google Scholar
  10. Gabrilovich E, Markovitch S (2007) Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In: Proceedings of the international joint conference on artificial intelligence, pp 1606–1611Google Scholar
  11. Heesch D (2008) A survey of browsing models for content based image retrieval. Multimed Tools Appl 42(2): 261–284CrossRefGoogle Scholar
  12. Hoeber O (2008) Web information retrieval support systems: the future of web search. In: Proceedings of the IEEE/WIC/ACM international conference on web intelligence, workshops (international workshop on web information retrieval support systems), pp 29–32Google Scholar
  13. Jansen BJ, Spink A, Pedersen J (2003) An analysis of multimedia searching on AltaVista. In: Proceedings of the ACM SIGMM international workshop on multimedia information retrieval, pp 186–192Google Scholar
  14. Jing Y, Baluja S (2008) VisualRank: Applying PageRank to large-scale image search. IEEE Trans Pattern Anal Mach Intell 30(11):1877–1890CrossRefGoogle Scholar
  15. Joshi D, Datta R, Zhuang Z, Weiss WP, Friedenberg M, Li J, Wang JZ (2006) PARAgrab: a comprehensive architecture for web image management and multimodal querying. In: Proceedings of the international conference on very large databases, pp 1163–1166Google Scholar
  16. Kherfi ML, Ziou D, Bernardi A (2004) Image retrieval from the world wide web: issues, techniques, and systems. ACM Comput Surv 36(1):35–67CrossRefGoogle Scholar
  17. Miller GA (1995) Wordnet: a lexical database for english. Commun ACM 38(11):39–41. ISSN 0001-0782CrossRefGoogle Scholar
  18. Milne D, Witten IH (2008a) An effective, low-cost measure of semantic relatedness obtained from Wikipedia links. In: Proceedings of the AAAI workshop on Wikipedia and artificial intelligence, pp 25–30Google Scholar
  19. Milne D, Witten IH (2008b) Learning to link with Wikipedia. In: Proceedings of the ACM conference on information and knowledge management, pp 509–518Google Scholar
  20. Milne D, Witten IH, Nichols DM (2007) A knowledge-based search engine powered by Wikipedia. In: Proceedings of the ACM conference on information and knowledge management, pp 445–454Google Scholar
  21. Moëllic P-A, Haugeard J-E, Pitel G (2008) Image clustering based on a shared nearest neighbors approach for tagged collections. In: Proceedings of the international conference on content-based image and video retrieval, pp 269–278Google Scholar
  22. Myoupo D, Popescu A, Borgne HL, Moëllic P-A (2009) Multimodal image retrieval over a large database. In: Proceedings of the international conference on cross-language evaluation forum: multimedia experimentsGoogle Scholar
  23. Snavely N, Seitz SM, Szeliski R (2006) Photo tourism: exploring photo collections in 3d. In: Proceedings of the ACM international conference on computer graphics and interactive techniques, pp 835–846Google Scholar
  24. Strong G, Gong M (2008) Browsing a large collection of community photos based on similarity on GPU. In: Proceedings of the international symposium on advances in visual computing, pp 390–399Google Scholar
  25. Strong G, Gong M (2009) Organizing and browsing photos using different feature vectors and their evaluations. In: Proceedings of the international conference on image and video retrieval, pp 1–8Google Scholar
  26. Strong G, Hoeber O, Gong M (2010) Visual image browsing and exploration (Vibe): user evaluations of image search tasks. In: Proceedings of the international conference on active media technology, pp 424–435Google Scholar
  27. Strube M, Ponzetto SP (2006) WikiRelate! computing semantic relatedness using Wikipedia. In: Proceedings of the AAAI conference on artificial intelligence, pp 1419–1424Google Scholar
  28. Torres RS, Silva CG, Medeiros CB, Rocha HV (2003) Visual structures for image browsing. In: Proceedings of the international conference on information and knowledge management, pp 49–55Google Scholar
  29. van Leuken RH, Garcia L, Olivares X, van Zwol R (2009) Visual diversification of image search results. In: Proceedings of the international conference on world wide web, pp 341–350Google Scholar
  30. Wang S, Jing F, He J, Du Q, Zhang L (2007) Igroup: presenting web image search results in semantic clusters. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 587–596. ISBN 978-1-59593-593-9Google Scholar
  31. Yao Y (2002) Information retrieval support systems. In: Proceedings of the 2002 IEEE world congress on computational intelligence, pp 1092–1097Google Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  • Enamul Hoque
    • 1
  • Orland Hoeber
    • 1
  • Grant Strong
    • 1
  • Minglun Gong
    • 1
  1. 1.Department of Computer ScienceMemorial University of Newfoundland St. John’sCanada

Personalised recommendations