How Do Gain and Discount Functions Affect the Correlation between DCG and User Satisfaction?

  • Julián Urbano
  • Mónica Marrero
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9022)

Abstract

We present an empirical analysis of the effect that the gain and discount functions have in the correlation between DCG and user satisfaction. Through a large user study we estimate the relationship between satisfaction and the effectiveness computed with a test collection. In particular, we estimate the probabilities that users find a system satisfactory given a DCG score, and that they agree with a difference in DCG as to which of two systems is more satisfactory. We study this relationship for 36 combinations of gain and discount, and find that a linear gain and a constant discount are best correlated with user satisfaction.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Al-Maskari, A., Sanderson, M., Clough, P.: The Relationship between IR Effectiveness Measures and User Satisfaction. In: ACM SIGIR (2007)Google Scholar
  2. 2.
    Allan, J., Carterette, B., Lewis, J.: When Will Information Retrieval Be ’Good Enough’? In: ACM SIGIR (2005)Google Scholar
  3. 3.
    Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., Hullender, G.: Learning to Rank Using Gradient Descent. In: ICML (2005)Google Scholar
  4. 4.
    Järvelin, K., Kekäläinen, J.: IR Evaluation Methods for Retrieving Highly Relevant Documents. In: ACM SIGIR (2000)Google Scholar
  5. 5.
    Kanoulas, E., Aslam, J.A.: Empirical Justification of the Gain and Discount Function for nDCG. In: ACM CIKM (2009)Google Scholar
  6. 6.
    Lee, J.H.: Crowdsourcing Music Similarity Judgments using Mechanical Turk. In: ISMIR (2010)Google Scholar
  7. 7.
    Sanderson, M., Paramita, M.L., Clough, P., Kanoulas, E.: Do User Preferences and Evaluation Measures Line Up? In: ACM SIGIR (2010)Google Scholar
  8. 8.
    Urbano, J., Downie, J.S., Mcfee, B., Schedl, M.: How Significant is Statistically Significant? The case of Audio Music Similarity and Retrieval. In: ISMIR (2012)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Julián Urbano
    • 1
  • Mónica Marrero
    • 2
  1. 1.Universitat Pompeu FabraBarcelonaSpain
  2. 2.Barcelona Supercomputing CenterSpain

Personalised recommendations