Optimizing nDCG Gains by Minimizing Effect of Label Inconsistency

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7814)


We focus on nDCG choice of gains, and in particular on the fracture between large differences in exponential gains of high relevance labels and the not-so-small confusion, or inconsistency, between these labels in data. We show that better gains can be derived from data by measuring the label inconsistency, to the point that virtually indistinguishable labels correspond to equal gains. Our derived optimal gains make a better nDCG objective for training Learning to Rank algorithms.


Rank Algorithm Gold Silver Topic Expertise Optimal Gain Equal Gain 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bailey, P., Craswell, N., Soboroff, I., Thomas, P., de Vries, A.P., Yilmaz, E.: Relevance assessment: Are judges exchangeable and does it matter? In: SIGIR (2008)Google Scholar
  2. 2.
    Sakai, T.: Evaluating evaluation metrics based on the bootstrap. In: SIGIR (2006)Google Scholar
  3. 3.
    Wu, Q., Burges, C., Svore, K., Gao, J.: Ranking, boosting, and model adaptationGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Northeastern UniversityBostonUSA

Personalised recommendations