The precedence function: a numerical evaluation method for multicriteria ranking problems

This paper proposes and characterizes a method to solve multicriteria evaluation problems when individual judgements are categorical and may fail to satisfy both transitivity and completeness. The evaluation function consists of a weighted sum of the average number of times that each alternative precedes some other, in all pairwise comparisons. It provides, therefore, a quantitative assessment which is well-grounded, immediate to compute, and easy to understand.


Introduction
Consider an evaluation problem in which a group of individuals has to provide a comparative assessment of a collection of alternatives with respect to several features or qualities.We focus on those evaluation problems in which the comparison between alternatives is categorical, that is, each individual simply declares if alternative i has more or less of a given quality than alternative j.Moreover, those judgements need not be complete or acyclic.And we look for an evaluation function that transforms those ordinal judgements into quantitative overall assessments.
A familiar evaluation problem of this type is the following.A panel of experts has to evaluate a set of research proposals, in order to allocate a given amount of funds, in some scientific field.Here the alternatives are the research proposals, the individuals correspond to the panel of experts, and each alternative must be evaluated regarding several criteria (e.g., originality, relevance of the topic, budget adequacy, and performance of the research group).Note that, besides finding a global rank of the research projects, it will be most helpful to have some quantitative measure of their relative merit, to provide a guide to allocate funds.
In this paper we propose and characterize an evaluation function that provides a cardinal assessment for problems within this multidimensional scenario.The approach hinges on two complementary ideas: the recourse to pairwise comparisons, and anonymous head-counting evaluation.
Pairwise comparisons permit valuing the relative merit of a set of alternatives in a very general framework, including the case of individual judgements that fail to satisfy the standard requirements of acyclicity and completeness.Head-counting is a natural way of introducing cardinality in the evaluation, regarding categorical judgements, which incorporates a straightforward anonymity principle.Combining these two ideas, which are far from new, our evaluation formula relies on counting how many individuals prefer an alternative to another in each pairwise comparison.As for the way of aggregating that information into an overall judgement, we propose and characterize an evaluation function that consists of a weighted sum of the average number of people who considers that an alternative precedes another, within each dimension, when compared pairwise.The weights correspond to the relative importance of those dimensions in the problem.It is, therefore, a formula applicable to a wide family of problems, easy to compute and easy to understand.
The properties assumed to pinpoint that function are familiar and transparent: independence (the evaluation of each alternative only depends on the comparison of this alternative with the rest), symmetry (two alternatives with the same support will get the same evaluation), uniformity (the change in the evaluation due to a change in a variable is independent on the level of the variable), neutrality (equal changes in two different pairwise comparisons regarding an alternative, produce the same change in the evaluation of that alternative), and scale (the evaluation of each alternative moves between 0, for an alternative that nobody likes, and 1 for an alternative that everybody regards as better than the others, with 1/2 for an alternative which all consider indifferent to the rest).Interestingly, this evaluation protocol admits an equivalent formulation in terms of the differences between "wins and losses", in those pairwise confrontations.Moreover, it can be regarded as a multidimensional version of an extended Borda score.
The paper is organized as follows.Section 2 presents and characterizes the evaluation function.Section 3 concludes with some comments and remarks.

The evaluation protocol
Consider a set of alternatives A = {1, 2, . . ., m} to be evaluated with respect to a collection Q = {1, 2, . . ., q} of qualities or dimensions, by a set N = {1, 2, …, n} of individuals.Those individuals are endowed with binary relations, defined on A, that express their judgements with respect to each dimension.That is, for each ordered pair of alternatives (i, j), i = j, and every k ∈ Q, individual h ∈ N declares whether an alternative precedes the other, or both alternatives are indifferent, or they are not comparable.On those binary relations we only assume that i precedes j implies that j cannot precede i, whereas if i and j are indifferent (resp.non-comparable), then j and i are also indifferent (resp. non-comparable).Moreover, to avoid the trivial case in which there is a dimension where all alternatives are non-comparable, we also assume that for each dimension k ∈ Q, there exists some ordered pair (i, j), i = j, for which some individual h ∈ N considers that i precedes j, or that j precedes i, or that both alternatives are indifferent.We denote by P = (P k ) a profile of preferences, which describes how the agents value the alternatives in the different dimensions.An evaluation problem, or simply a problem, can thus be identified with a profile of preferences P, taking as given the set of alternatives, the dimensions, and the set of agents.Let now denote the set of all problems P of this type.Our target is to find a suitable function φ : → R m that provides a cardinal evaluation of the alternatives in A from the individuals' judgements.Let us formalize this idea.
Given a problem P, let n k i j denote the number of individuals who consider that alternative i precedes alternative j in dimension k, e k i j = e k ji the number of individuals who consider that alternative i matches alternative j, in that dimension, and z k i j = z k ji the number of those unable or unwilling to compare them (don't know/no answer).By construction, n = n k i j + n k ji + e k i j + z k i j , for all (i, j), j = i, ∀k ∈ Q.We now introduce five properties that our evaluation function φ must satisfy: independence, symmetry, uniformity, neutrality, and scale.They convey the idea that all judgements have an equal impact on the evaluation and that all alternatives are treated alike, within each dimension.
The first property, independence, establishes that the evaluation of alternative i only depends on the comparison of this alternative with the rest, across the q dimensions.Note that it involves an anonymity principle, as the evaluation only depends on the number of agents.
Independence: For all i ∈ M and problems P, P ∈ with n k i j = n k i j and = e k i j for all j ∈ M with j = i and all k ∈ Q, we have φ i (P) = φ i P .The second property, symmetry, says that if two alternatives have the same support, in a given problem, then they should get the same evaluation.That is, Symmetry: For all i ∈ M and P ∈ with n k i j = n k r j and e k i j = e k r j for all j ∈ M with j = i, for all r ∈ M with r = j and all k ∈ Q, we have φ i (P) = φ r (P).
The next two properties, uniformity and neutrality, refer to the sensitivity of the evaluation to changes in the agents' opinions.To motivate the property of uniformity, suppose that, after all judgements have been submitted, an individual reconsiders the evaluation of alternative i relative to alternative j in dimension k (e.g., from j precedes i shifts to i precedes j).This change of opinion may produce a change in the evaluation of i. Suppose now that, after that change, another individual also reconsiders and makes exactly the same move.We shall have a new problem and, associated with it, a new evaluation of alternative i.What we require is that the impact of both changes of opinion be the same.In this way, we ensure that the impact on the evaluation of the change of opinion of an individual, does not depend on whether someone else has reconsidered first.
Uniformity: Let P, P , P ∈ .Problem P results from problem P due to a change of opinion of agent h regarding alternatives (i, j) in dimension k.And P 123 results from problem P due to the same change of opinion of another agent h , over the same pair (i, j), regarding dimension k.Then, φ i P −φ i (P) = φ i P −φ i P .
Neutrality introduces the idea that all alternatives count the same in the evaluation, within each dimension.That is, a change of opinion regarding the pair (i, j) in dimension k, has the same impact on the evaluation of i that the same change of opinion with respect to another pair (i, r ), regarding the same dimension.
Neutrality: Let P, P , P ∈ .Problem P results from problem P due to a change of opinion of agent h regarding alternatives (i, j) in dimension k.And P results from problem P due to the same change of opinion of agent h regarding alternatives (i, r ) in dimension k.Then, φ i P − φ i (P) = φ i P − φ i (P).
Finally, the property of scale refers to the case in which one alternative precedes all others, is indifferent to all others, or it is preceded by all others, in all dimensions.In the first case the evaluation of this alternative is 1, in the last case is zero and in the full indifference case ½. Scale: 2 , and n k i j = e k i j = 0, ∀ j = i, ∀k ⇒ φ i (P) = 0.The next result shows that assuming those properties amounts to choose a simple and intuitive evaluation function.
Proposition 1 An evaluation function φ : → R m satisfies the properties of independence, symmetry, uniformity, neutrality, and scale, if and only if there are α = (α k ) q k=1 , with α k ≥ 0, q k=1 α k = 1 such that: Proof Clearly, this function satisfies those properties.Let us consider the converse implication.
First, observe that independence implies that φ i (P) only depends on the numbers of individuals n k i j , e k i j .Moreover, uniformity establishes that the impact of a given change of opinion is the same, no matter how many individuals consider that an alternative precedes or is indifferent to another.This is a linearity feature that allows us to write: For some

Similarly, if e k i j
= e k i j + 1 , and e k ir = e k ir + 1 , it follows that 2 a(k), ∀k, we can rewrite Eq. (2) as follows: (3) Finally, define α k = n(m − 1)a(k), ∀k.It follows that: We shall refer to vector function φ : → R m in Proposition 1 as the Precedence function.To facilitate the interpretation of this result and the ensuing discussion, let us define the precedence score of alternative i with respect to alternative j, for i = j, relative to dimension k, as follows: This is simply the average number of individuals who consider that i precedes j in dimension k, including one-half of those who consider that the quality of both alternatives match (note that 0 ≤ p i j (k) ≤ 1).The average precedence score of alternative i in dimension k is given by: with 0 ≤ p i (k) ≤ 1.Now we can rewrite Eq. ( 1) in a much more compact way, as follows: 123 That is, the evaluation function in Proposition 1 is simply a convex combination of the average precedence scores.
We show next that all the properties in Proposition 1 are independent, so that the characterization is tight.

Proposition 2
The properties of independence, symmetry, uniformity, neutrality, and scale are independent.
Consider the evaluation functions, whose ith component is given by: Function (i) satisfies all properties but independence.Function (ii) satisfies all the properties except symmetry.Function (iii) satisfies all properties but uniformity.Function (iv) satisfies all properties but neutrality.Finally, function (v) satisfies all properties except scale.

Discussion
The ideas presented here can be traced back to the works of Borda ( 1784) and Condorcet (1785), regarding voting procedures.Condorcet's key idea is that the best candidate is the one who beats all others in pairwise confrontations.In terms of our formulation, alternative i is a Condorcet winner whenever p i (k) > p j (k), ∀ j, ∀k.The precedence score of an alternative relative to another, p i j (k), is sometimes called the Condorcet number (Moulin 1988).It is well known that a Condorcet winner may not exist, and that the Condorcet approach permits having weak orderings (indifferences) and admit non-transitive and non-complete preferences.Borda proposed to give a score to each candidate that reflects how much support it accrues, rather than how many supporters back this candidate, as in Condorcet's approach.The "amount of support" of candidate i is given by the total number of candidates that are below i in the rankings of all voters.Besides producing a complete ranking of candidates, the Borda approach also provides a cardinal way of rating them, adding up the number of defeated candidates in each voter's ranking.
In spite of the different principles behind Borda and Condorcet approaches, both methods are closely related.Indeed, the average precedence score, p i (k), is precisely the Borda score relative to dimension k (Moulin 1988).So, the average precedence score of an alternative may be regarded as the normalized Borda score in a more general scenario, and Proposition 1 can be interpreted as a straightforward characterization of the multidimensional Borda score, in a general context. 1he precedence function can also be from two different angles, which provide a richer perspective of this evaluation protocol.One refers to the consideration of wins and losses, and the other to a probabilistic viewpoint.
The average differential score of alternative i is the difference between the average number of times that this alternative precedes some other, and the number of times that some other alternative precedes i.That is, ji , we can write: The Proposition can, therefore, be re-stated as saying that an evaluation function satisfies the properties of independence, symmetry, uniformity, neutrality, and scale, if and only if it is given by: When judgements are complete (i.e., z i (k) = 0, ∀k), the precedence function corresponds to a positive linear transformation of the differences between "wins and losses".This formulation also serves to illustrates the different role played by indifferences and non-comparabilities in the evaluation.Indifferences cancel out in d i (k), whereas non-comparabilities penalize the evaluation.
We can also interpret the precedence score, 2 , as the probability that the alternative i precedes j in dimension k, when choosing randomly an individual h ∈ N .Similarly, the average precedence score, p i (k) = 1 m−1 j =i p i j (k), is the probability that alternative i precedes some other alternative, when facing a pairwise comparison in which both the individual and the other alternative are chosen randomly. 2And, by the same token, d i (k) is the difference between the probability that i beats some other alternative in a pairwise confrontation, and the probability that i be beaten by some other alternative.Consequently, the precedence function tells us the probability that each alternative has of preceding some other, when the individual, the other alternative, and the corresponding dimension are randomly chosen.Note that probabilities exhibit a uniform distribution, regarding individuals and alternatives, whereas the probability of each dimension is given by the corresponding coefficient α k .
Footnote 1 continued setting, like the one presented here.They focus, though, on the Borda ranking rather than on the Borda scores.Besides the general framework, the key element in their work consists of interpreting the Borda rule in terms of the differences between favourable and unfavourable opinions, an idea that we discuss next. 2 This interpretation permits comparing the precedence function with the Borda-Condorcet rule, in Herrero & Villar (2021), in a similar framework.The precedence function associates with each alternative, the oneshot probability of beating some other alternative, in some dimension.The Borda-Condorcet rule applies the same idea in terms of the probability that derives from an indefinite sequence of pairwise encounters, which can be interpreted in terms of the fraction of time that each alternative keeps the floor in this process.
Interestingly, this evaluation formula is decomposable by subgroups, which might be relevant when there is a large number of evaluators with different characteristics.Think for instance of the evaluation of the efficacy of different painkillers in a medical trial.Besides rating globally those painkillers, it might be relevant to know their effects on different groups of patients (e.g., classified by age, gender, clinical record, and occupation).By letting N = G g=1 N g , where N g is population subgroup g with cardinal n g , so that G g=1 n g = n, it is easy to see that we can write: A relevant context in which this decomposability property applies refers to those problems in which different subsets of alternatives in A are evaluated by separate subsets of individuals in N. The evaluation of research projects, mentioned in the Introduction may serve to illustrate this case.Typically, each evaluator receives a subset of the proposals and then a decision is made from those partial evaluations.This can be regarded as a case of different population subgroups, each of which exhibits incomplete evaluations.
Finally, note that Proposition 1 provides an evaluation formula that leaves open the weights that ponder the different dimensions.Yet we might consider the case in which the evaluators are also asked to give weights to those dimensions.A particular way of approaching that problem is to choose the weights consistently with the evaluation process.By that we mean that each dimension is compared pairwise with each other, and the number of individuals who give precedence to one another is computed.Let P 0 the single-dimensional problem in which the set of qualities plays the role of the set of alternatives.Then, φ k P 0 = 1 q−1 j =k p k j , so that we obtain: α k = φ k P 0 q r =1 φ r (P 0 ) .