Multimedia Tools and Applications

, Volume 66, Issue 3, pp 493–515 | Cite as

Topic modelling of clickthrough data in image search

  • Donn MorrisonEmail author
  • Theodora Tsikrika
  • Vera Hollink
  • Arjen P. de Vries
  • Éric Bruno
  • Stéphane Marchand-Maillet


In this paper we explore the benefits of latent variable modelling of clickthrough data in the domain of image retrieval. Clicks in image search logs are regarded as implicit relevance judgements that express both user intent and important relations between selected documents. We posit that clickthrough data contains hidden topics and can be used to infer a lower dimensional latent space that can be subsequently employed to improve various aspects of the retrieval system. We use a subset of a clickthrough corpus from the image search portal of a news agency to evaluate several popular latent variable models in terms of their ability to model topics underlying queries. We demonstrate that latent variable modelling reveals underlying structure in clickthrough data and our results show that computing document similarities in the latent space improves retrieval effectiveness compared to computing similarities in the original query space. These results are compared with baselines using visual and textual features. We show performance substantially better than the visual baseline, which indicates that content-based image retrieval systems that do not exploit query logs could improve recall and precision by taking this historical data into account.


Image retrieval Latent variable modelling User interaction Clickthrough data 



This research was funded by the Swiss National Science Foundataion (SNF) through IM2 (Interactive Multimedia Information Management) and by EU-FP7-ICT.1.5 NoE PetaMedia. The authors would also like to thank the Belga News Agency for the use of the query logs.


  1. 1.
    Baeza-Yates R, Tiberi A (2007) Extracting semantic relations from query logs. In: Proceedings of ACM KDD’07. ACM, New York, NY, USA, pp 76–85. doi: 10.1145/1281192.1281204 Google Scholar
  2. 2.
    Berry MW, Browne M (2005) Email surveillance using non-negative matrix factorization. Comput Math Organ Theory 11(3):249–264. doi: 10.1007/s10588-005-5380-5. URL: zbMATHCrossRefGoogle Scholar
  3. 3.
    Beyer K, Goldstein J, Ramakrishnan R, Shaft U (1999) When is “nearest neighbor” meaningful? In: Int. conf. on database theory, pp 217–235Google Scholar
  4. 4.
    Bingham E, Kaban A, Fortelius M (2009) The aspect Bernoulli model: multiple causes of presences and absences. Pattern Anal Appl 12(1):55–78. doi: 10.1007/s10044-007-0096-4 MathSciNetCrossRefGoogle Scholar
  5. 5.
    Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. J Mach Learn Res 3:993–1022zbMATHGoogle Scholar
  6. 6.
    Craswell N, Szummer M (2007) Random walks on the click graph. In: Proceedings of ACM SIGIR’07, pp 239–246Google Scholar
  7. 7.
    Deerwester S, Dumais S, Landauer T, Furnas G, Harshman R (1990) Indexing by latent semantic analysis. J Am Soc Inf Sci 4:391–407CrossRefGoogle Scholar
  8. 8.
    Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Ser B Stat Methodol 39(1):1–38. doi: 10.2307/2984875 MathSciNetzbMATHGoogle Scholar
  9. 9.
    Gaussier E, Goutte C (2005) Relation between PLSA and NMF and implications. In: Proceedings of ACM SIGIR’05, pp 601–602. doi: 10.1145/1076034.1076148
  10. 10.
    van Gemert J, Geusebroek JM, Veenman C, Snoek C, Smeulders A (2006) Robust scene categorisation by learning image statistics in context. In: Proceedings of SLAM’06, p 105Google Scholar
  11. 11.
    Griffiths TL, Steyvers M (2004) Finding scientific topics. Proc Natl Acad Sci U S A 101(Suppl 1): 5228–5235. doi: 10.1073/pnas.0307752101 CrossRefGoogle Scholar
  12. 12.
    He X, King O, Ma WY, Li M, Zhang HJ (2003) Learning a semantic space from user’s relevance feedback for image retrieval. IEEE Trans Circuits Syst Video Technol 13(1):39–48. doi: 10.1109/TCSVT.2002.808087 CrossRefGoogle Scholar
  13. 13.
    Heisterkamp D (2002) Building a latent-semantic index of an image database from patterns of relevance feedback. In: Proceedings of the 16th international conference on pattern recognition, pp 134–137.
  14. 14.
    Herlocker JL, Konstan JA, Borchers A, Riedl J (1999) An algorithmic framework for performing collaborative filtering. In: Proceedings of ACM SIGIR’99, pp 230–237. doi: 10.1145/312624.312682
  15. 15.
    Hiemstra D, Rode H, van Os R, Flokstra J (2006) PFTijah: text search in an XML database system. In: Proceedings of OSIR’06, pp 12–17.
  16. 16.
    Hofmann T (1999) Probabilistic latent semantic analysis. In: Proceedings of uncertainty in artificial intelligence.
  17. 17.
    Jansen BJ (2009) Understanding user-web interactions via web analytics. In: Synthesis lectures on information concepts, retrieval, and services, Morgan & ClaypoolGoogle Scholar
  18. 18.
    Jansen BJ, Spink A, Saracevic T (1999) The use of relevance feedback on the web: implications for web IR system design. In: Proceedings of WebNet’99, pp 500–555Google Scholar
  19. 19.
    Joachims T (2003) Evaluating retrieval performance using clickthrough data. In: Franke J, Nakhaeizadeh G, Renz I (eds) Text mining, pp 79–96Google Scholar
  20. 20.
    Joachims T, Granka L, Pang B, Hembrooke H, Gay G (2005) Accurately interpreting clickthrough data as implicit feedback. In: Proceedings of ACM SIGIR’05, pp 154–161Google Scholar
  21. 21.
    Kelly D, Teevan J (2003) Implicit feedback for inferring user preference: a bibliography. SIGIR Forum 37(2):18–28CrossRefGoogle Scholar
  22. 22.
    Koren Y (2009) Collaborative filtering with temporal dynamics. In: Proceedings of ACM SIGKDD’09, pp 447–456. doi: 10.1145/1557019.1557072
  23. 23.
    Kraaij W (2004) Variations on language modeling for information retrieval. PhD thesis, Centre for Telematics and Information Technology, University of TwenteGoogle Scholar
  24. 24.
    Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791. doi: 10.1038/44565 CrossRefGoogle Scholar
  25. 25.
    Lin C, Xue GR, Zeng HJ, Yu Y (2005) Using probabilistic latent semantic analysis for personalized web search. In: Proceedings of the 7th asia-pacific web conference. LNCS, vol 3399, pp 707–717.
  26. 26.
    Macdonald C, Ounis I (2009) Usefulness of quality click-through data for training. In: Proceedings of the workshop on web search click data, pp 75–79. doi: 10.1145/1507509.1507521
  27. 27.
    Müller H, Pun T, Squire D (2004) Learning from user behavior in image retrieval: application of market basket analysis. Int J Comput Vis 56(1–2):65–77CrossRefGoogle Scholar
  28. 28.
    Poblete B, Bustos B, Mendoza M, Barrios JM (2010) Visual-semantic graphs: using queries to reduce the semantic gap in web image retrieval. In: Proceedings of CIKM’10, 26–30 October, Toronto, Canada. ACM Press, New York, NYGoogle Scholar
  29. 29.
    Porteous I, Newman D, Ihler A, Asuncion A, Smyth P, Welling M (2008) Fast collapsed gibbs sampling for latent dirichlet allocation. In: Proceedings of ACM SIGKDD’08. ACM, New York, NY, USA, pp 569–577. doi: 10.1145/1401890.1401960 Google Scholar
  30. 30.
    Smith G, Ashman H (2009) Evaluating implicit judgements from image search interactions. In: Proceedings of WebSci’09: society on-line.
  31. 31.
    Steyvers M, Griffiths T (2005) Probabilistic topic models. Latent Semantic analysis: a road to meaning. Laurence ErlbaumGoogle Scholar
  32. 32.
    Szekely E, Bruno E, Marchand-Maillet S (2010) High-dimensional multimodal distribution embedding. In: IEEE ICDM 2010 workshop on visual analytics and knowledge discovery (VAKD’10), Sydney, AustraliaGoogle Scholar
  33. 33.
    Tsikrika T, Diou C, de Vries AP, Delopoulos A (2009) Image annotation using clickthrough data. In: Proceedings of CIVR’09Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2012

Authors and Affiliations

  • Donn Morrison
    • 1
    Email author
  • Theodora Tsikrika
    • 2
  • Vera Hollink
    • 2
  • Arjen P. de Vries
    • 2
  • Éric Bruno
    • 1
  • Stéphane Marchand-Maillet
    • 1
  1. 1.Computer Vision and Multimedia LaboratoryUniversity of GenevaGenevaSwitzerland
  2. 2.Centrum Wiskunde & InformaticaAmsterdamThe Netherlands

Personalised recommendations