Score Transformation in Linear Combination for Multi-criteria Relevance Ranking

  • Shima Gerani
  • ChengXiang Zhai
  • Fabio Crestani
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7224)


In many Information Retrieval (IR) tasks, documents should be ranked based on a combination of multiple criteria. Therefore, we would need to score a document in each criterion aspect of relevance and then combine the criteria scores to generate a final score for each document. Linear combination of these aspect scores has so far been the dominant approach due to its simplicity and effectiveness. However, such a strategy of combination requires that the scores to be combined are “comparable” to each other, an assumption that generally does not hold due to the different ways of scoring each criterion. Thus it is necessary to transform the raw scores for different criteria appropriately to make them more comparable before combination. In this paper we propose a new principled approach to score transformation in linear combination, in which we would learn a separate non-linear transformation function for each relevance criterion based on the Alternating Conditional Expectation (ACE) algorithm and BoxCox Transformation. Experimental results show that the proposed method is effective and is also robust against non-linear perturbations of the original scores.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ounis, I., Macdonald, C., Soboroff, I.: Overview of the TREC-2008 blog track. In: Proceedings of TREC 2008 (2008)Google Scholar
  2. 2.
    Bendersky, M., Croft, W.B., Diao, Y.: Quality-biased ranking of web documents. In: Proceedings of WSDM 2011, pp. 95–104 (2011)Google Scholar
  3. 3.
    Xu, J., Li, H.: Adarank: a boosting algorithm for information retrieval. In: Proceedings of SIGIR 2007, pp. 391–398 (2007)Google Scholar
  4. 4.
    Yue, Y., Finley, T., Radlinski, F., Joachims, T.: A support vector method for optimizing average precision. In: Proceedings of SIGIR 2007, pp. 271–278 (2007)Google Scholar
  5. 5.
    Liu, T.Y., Xu, J., Qin, T., Xiong, W.Y., Li, H.: Letor: Benchmark dataset for research on learning to rank for information retrieval. In: Proceedings of SIGIR 2007 Workshops (2007)Google Scholar
  6. 6.
    Lee, J.H.: Analyses of multiple evidence combination. In: Proceedings of SIGIR 1997, pp. 267–276 (1997)Google Scholar
  7. 7.
    Montague, M., Aslam, J.A.: Relevance score normalization for metasearch. In: Proceedings of CIKM 2001, pp. 427–433 (2001)Google Scholar
  8. 8.
    Manmatha, R., Rath, T., Feng, F.: Modeling score distributions for combining the outputs of search engines. In: Proceedings of SIGIR 2001, pp. 267–275 (2001)Google Scholar
  9. 9.
    Manmatha, R., Sever, H.: A formal approach to score normalization for meta-search. In: Proceedings of HLT 2002, pp. 98–103 (2002)Google Scholar
  10. 10.
    Arampatzis, A., Kamps, J.: A signal-to-noise approach to score normalization. In: Proceedings of CIKM 2009, pp. 797–806 (2009)Google Scholar
  11. 11.
    Fernández, M., Vallet, D., Castells, P.: Using historical data to enhance rank aggregation. In: Proceedings of SIGIR 2006, pp. 643–644 (2006)Google Scholar
  12. 12.
    Sever, H., Tolun, M.R.: Comparison of Normalization Techniques for Metasearch. In: Yakhno, T. (ed.) ADVIS 2002. LNCS, vol. 2457, pp. 133–143. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Arampatzis, A., Robertson, S.: Modeling score distributions in information retrieval. Inf. Retr. 14, 26–46 (2011)CrossRefGoogle Scholar
  14. 14.
    Craswell, N., Robertson, S., Zaragoza, H., Taylor, M.: Relevance weighting for query independent evidence. In: Proceedings of SIGIR 2005, pp. 416–423 (2005)Google Scholar
  15. 15.
    Veaux, R.D.D.: Finding Transformations for Regression Using the ACE Algorithm. Sociological Methods and Research 18(327), 327–359 (1989)CrossRefGoogle Scholar
  16. 16.
    Santos, R.L., He, B., Macdonald, C., Ounis, I.: Integrating Proximity to Subjective Sentences for Blog Opinion Retrieval. In: Boughanem, M., Berrut, C., Mothe, J., Soule-Dupuy, C. (eds.) ECIR 2009. LNCS, vol. 5478, pp. 325–336. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  17. 17.
    Zhang, W., Yu, C., Meng, W.: Opinion retrieval from blogs. In: Proceedings of CIKM 2007, pp. 831–840 (2007)Google Scholar
  18. 18.
    Lee, Y., Na, S.H., Kim, J., Nam, S.H., Jung, H.Y., Lee, J.H.: KLE at TREC 2008 blog track: Blog post and feed retrieval. In: Proceedings of TREC 2008 (2008)Google Scholar
  19. 19.
    Gerani, S., Carman, M.J., Crestani, F.: Proximity-based opinion retrieval. In: Proceedings of SIGIR 2010, pp. 403–410 (2010)Google Scholar
  20. 20.
    Breiman, L., Friedman, J.H.: Estimating Optimal Transformations for Multiple Regression and Correlation. American Statistical Association 80(391) (1985)Google Scholar
  21. 21.
    Box, G.E.P., Cox, D.R.: An Analysis of Transformations. Journal of the Royal Statistical Society B(26), 211–252 (1964)Google Scholar
  22. 22.
    Zhai, C., Lafferty, J.: A study of smoothing methods for language models applied to ad hoc information retrieval. In: Proceedings of SIGIR 2001, pp. 334–342 (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Shima Gerani
    • 1
  • ChengXiang Zhai
    • 2
  • Fabio Crestani
    • 1
  1. 1.Faculty of InformaticsUniversity of LuganoLuganoSwitzerland
  2. 2.Department of Computer ScienceUniversity of Illinois at Urbana-ChampaignUSA

Personalised recommendations