Score Transformation in Linear Combination for Multi-criteria Relevance Ranking

Abstract

In many Information Retrieval (IR) tasks, documents should be ranked based on a combination of multiple criteria. Therefore, we would need to score a document in each criterion aspect of relevance and then combine the criteria scores to generate a final score for each document. Linear combination of these aspect scores has so far been the dominant approach due to its simplicity and effectiveness. However, such a strategy of combination requires that the scores to be combined are “comparable” to each other, an assumption that generally does not hold due to the different ways of scoring each criterion. Thus it is necessary to transform the raw scores for different criteria appropriately to make them more comparable before combination. In this paper we propose a new principled approach to score transformation in linear combination, in which we would learn a separate non-linear transformation function for each relevance criterion based on the Alternating Conditional Expectation (ACE) algorithm and BoxCox Transformation. Experimental results show that the proposed method is effective and is also robust against non-linear perturbations of the original scores.