Effects of Position Bias on Click-Based Recommender Evaluation

  • Katja Hofmann
  • Anne Schuth
  • Alejandro Bellogín
  • Maarten de Rijke
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8416)


Measuring the quality of recommendations produced by a recommender system (RS) is challenging. Labels used for evaluation are typically obtained from users of a RS, by asking for explicit feedback, or inferring labels from implicit feedback. Both approaches can introduce significant biases in the evaluation process. We investigate biases that may affect labels inferred from implicit feedback. Implicit feedback is easy to collect but can be prone to biases, such as position bias. We examine this bias using click models, and show how bias following these models would affect the outcomes of RS evaluation. We find that evaluation based on implicit and explicit feedback can agree well, but only when the evaluation metrics are designed to take user behavior and preferences into account, stressing the importance of understanding user behavior in deployed RSs.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bellogín, A., Castells, P., Cantador, I.: Precision-oriented evaluation of recommender systems: an algorithmic comparison. In: RecSys 2011, pp. 333–336 (2011)Google Scholar
  2. 2.
    Craswell, N., Zoeter, O., Taylor, M., Ramsey, B.: An experimental comparison of click position-bias models. In: WSDM 2008, pp. 87–94 (2008)Google Scholar
  3. 3.
    Dupret, G.E., Piwowarski, B.: A user browsing model to predict search engine click data from past observations. In: SIGIR 2008, pp. 331–338 (2008)Google Scholar
  4. 4.
    Guo, F., Liu, C., Wang, Y.M.: Efficient multiple-click models in web search. In: WSDM 2009, pp. 124–131 (2009)Google Scholar
  5. 5.
    Hofmann, K., Whiteson, S., de Rijke, M.: A probabilistic method for inferring preferences from clicks. In: CIKM 2011, pp. 249–258 (2011)Google Scholar
  6. 6.
    Hu, Y., Koren, Y., Volinsky, C.: Collaborative filtering for implicit feedback datasets. In: ICDM 2008, pp. 263–272. IEEE Computer Society (2008)Google Scholar
  7. 7.
    Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: SIGIR 2005, pp. 154–161. ACM (2005)Google Scholar
  8. 8.
    Oard, D., Kim, J.: Implicit feedback for recommender systems. In: AAAI Workshop on Recommender Systems, pp. 81–83 (1998)Google Scholar
  9. 9.
    Ricci, F., Rokach, L., Shapira, B., Kantor, P.B.: Recommender Systems Handbook. Springer (2011)Google Scholar
  10. 10.
    Richardson, M., Dominowska, E., Ragno, R.: Predicting clicks: estimating the click-through rate for new ads. In: WWW 2007, pp. 521–530. ACM (2007)Google Scholar
  11. 11.
    Schuth, A., Hofmann, K., Whiteson, S., de Rijke, M.: Lerot: an Online Learning to Rank Framework. In: LivingLab 2013, pp. 23–26. ACM (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Katja Hofmann
    • 1
  • Anne Schuth
    • 2
  • Alejandro Bellogín
    • 3
  • Maarten de Rijke
    • 2
  1. 1.Microsoft ResearchUK
  2. 2.ISLAUniversity of AmsterdamThe Netherlands
  3. 3.CWIThe Netherlands

Personalised recommendations