Selections from ordered sets

Selections are typically made according to a varying blend of objective measures and subjective judgments. A sport coach might base her team choice on the recent performance of individuals in her squad (batting and bowling/pitching averages, tennis rankings, trial results for track and field, and so on), and on her sense of who is the best person for each given role, given the expected conditions. Many universities do not strictly follow SATs results and school grades when choosing whom to admit, but take into account a student’s social background and his potential contribution to desirable characteristics of the student body, like diversity. Applicants for academic jobs might be ranked according to bibliometric measures, but the appointment panel’s judgment often leads to decision that do not map precisely into the ranking.¹ Large and complex procurement contracts often demand the subtle evaluation of complex qualitative elements, and lowest price is seldom the only criterion used to award these contracts.² And so forth and so on.

Observers and decision makers might be interested in some means of comparing the choices of different selectors. If the property of following the measurable dimension is labelled “orderliness”, someone might want to compare the orderliness of different selections, for example to determine which is more “orderly”.

We can think of at least three conceptually distinct situations where this comparison is meaningful.³ First, we may want to compare different selections from the same set. For example, the public may want to know how close the proposed shortlists of the different judges of a book or film prize are to a market-determined (sales, box office earnings) ordering. Second, the comparison might be of selections from altogether different sets. A cricket fan may want to know whether Australia’s selection for the Ashes team is closer to the players’ current rankings than England’s. A university whose admission policies are under scrutiny in court may want to argue that its admission policy is as based on SATs as those of comparable institutions. Or a government minister concerned about corruption in procurement contracts or personnel hiring may want to compare the orderliness of various commissioning boards or hiring panels, to identify and perhaps investigate atypical behaviour. On a broader scale, orderliness is a helpful yardstick when assessing inequality of opportunity: it can serve as a measure of nepotism, telling how close selection into society’s elite is to the pecking order determined by family history, or of plutocracy, measured for example by the closeness of membership of Parliament to a person’s position in the income distribution. The third kind of potential comparisons is for situations where both the selection and the selectors are the same, but there are two or more ways of ordering the elements of a set. In these cases, which are close to the topic of some related literature considered below (Kemeny 1959; Klamler 2008), the focus is on their relative importance in the selection. For example, a rugby analyst may want to know whether weight or speed is more important to be selected as a three-quarter for the Springboks. Or an external funder may want to know which of teaching and research is more important for promotion in a given university.

Comparing selections is straightforward only in the starkest cases. Sure, the selection of the best ranked is unquestionably more orderly than selections that do not. But is a university which, from its 102 applicants, admits as students those ranked according to their SAT scores second to 5th, 8th, 11th, 12th, 24th, 25th, 38th, and 47th more orderly than one that chooses the first 20, the 57th, and those ranked between 170 and 180 out of 500 applicants? Or, in an even simpler example, is picking the second ranked out of ten candidates for a job more orderly than selecting the third ranked out of 19?⁴

This paper takes an axiomatic approach to comparing selections. The axioms are requirements of orderliness: a decision maker who is called to opine which is more orderly of two selections from different sets, that is which selection is in her opinion closer to the order—determined by a given measure—of the set from which it is taken, should only make judgements that obey these axioms. This is conceptually similar to the approach taken by Sen (1976) and Cowell and Kuga (1981) in regard to judgements on which of two groups of individuals poverty or inequality, respectively, is greater.

We begin with three natural axioms, which extend the dominance principle in the comparison of sets (Barberà et al. 2004, BBP in what follows). The first requires that when an element is replaced in the selection with a better one, then the selection is deemed to be more orderly. In the next two axioms we depart from the set-up of the literature reviewed in BBP, and in some of its extensions, such as Rubinstein and Salant (2006). The departure is that we consider elements that are not in the original set, thus allowing the comparison of selections from different sets, as in many of the examples given above. These axioms require that if a new element appears which is better than all the selected (non-selected) ones, then the selection is considered more (less) orderly if the new element is selected. Analogously for the appearance of a worse element. Extending the axioms to different sets implies that we cannnot, like the literature cited, take the rank of the elements as the starting point of the analysis, since a new elements change the ranks of all the elements below it. Instead, we need to make formally explicit the mapping from the measured performance to the rank of the elements ordered by this measure.

These three axioms, which we show to be independent, are not characterising: there are different orderings of selections from sets which satisfies them. In our main result we show that characterisation is achieved by replacing the first axiom with a stronger one, which we label “mirror invariance”. This requires that if a change makes a selection more orderly, then the mirror image of the selection—that is the selection of only the non-selected elements—is made less orderly by the mirror image of the change. This is equivalent to requiring that the importance attributed to changes in the ordering be independent of where they occur: departures from the ranking are as important if they occur among the best or the worst ranked elements. This result is established in Theorem 1, which shows that the orderliness relation characterised by Axioms 2–4 is represented, in close analogy to the representation of a consumer’s preference by a utility function, by an “index of orderliness”, given by a simple function of the number of elements in the set, the number of selected elements, and the sum of the ranks of the selected elements.

The paper continues by showing that our index of orderliness has a simple relation with the concept of distance between rankings originally proposed by Kendall (1938), and given an axiomatic foundation by Kemeny (1959). This distance compares different selections from the same set, and, in this case, Proposition 4 shows that if a selection is more orderly than another selection from the same set, then the ranking of the set naturally induced by the first selection is nearer, in the sense of the Kendall-Tau distance, to the ordering of the set.

The paper is organised as follows: the dominance axioms and some preliminary results are in Sect. 2. The core of the paper is Sect. 3, which presents the index of orderliness, and strengthens one of the axioms to obtain a complete ordering of all selected sets. The relation with existing literature is explored in Sect. 4, and the paper ends with Sect.  5, which shows how the index can be used to assess the evaluation mechanism for promotion to professorship in Italian universities. A brief conclusion follows.

Let \(\mathcal {N}\varsubsetneq \mathbb {R}\) contain \(N\in \mathbb {N}\backslash \left\{ 1\right\} \) distinct elements: \(x,y\in \mathcal {N}\) implies \(x\ne y\) .⁵ The inequality relation in \(\mathbb {R}\) induces a transitive, complete and antisymmetric relation between the members of \(\mathcal {N}\). This relation is naturally defined as the rank in \(\mathcal {N}\), \(\rho _{\mathcal {N}}\), a bijective mapping of the elements of \(\mathcal {N}\) into the set of the first N natural numbers \(\rho _{\mathcal {N}} : \mathcal {N}\longrightarrow \{ 1,2,\ldots ,N\} \), which satisfies \( \rho _{\mathcal {N}}( x) <\rho _{\mathcal {N}} ( y) \) if and only if \(x<y\). We choose this convention, that x has a better⁶ rank in \( \mathcal {N}\) than y if it is smaller, for definiteness. It would refer for example, to a situation where the real numbers are marathon runners’ personal best times; the changes needed to apply the framework to situations where a larger real number is better, such as cricketers’ career runs or scientists’ citation counts, are obvious.

We next define a selection \(\mathcal {K}\) as a proper and non-empty subset of \(\mathcal {N}\), that is \(\mathcal {K}\varsubsetneq \mathcal {N}\), and \(\mathcal {K} \ne \varnothing \). Let \(K\in \left\{ 1,\ldots ,N-1\right\} \) be the number of elements of \(\mathcal {K}\). We define the pair \(\left( \mathcal {N},\mathcal {K} \right) \) a “selected set”.⁷ Let \( \mathscr {S}\) be the family of selected sets.

We single out the situations where the selection follows exactly the ranking.

In words, the selection \(\mathcal {K}\) is perfect if no selected element has a rank worse than a non-selected element, and it is antiperfect if every selected element has a worse rank than every non-selected element.

Let \(\mathscr {S}^{P}\subseteq \mathscr {S}\) be the set of all perfect selections, and conversely, let \(\mathscr {S}^{A}\subseteq \mathscr {S}\) be the set of all antiperfect selections. Note that since \(\mathcal {K}\) is a proper non-empty subset of \(\mathcal {N}\), \(\mathscr {S}^{P}\cap \mathscr {S} ^{A}= \varnothing \): no selection is simultaneously both perfect and antiperfect.

We want to compare selections, that is order the set \(\mathscr {S}\), in the following sense: consider two ranked sets \(\mathcal {N}_{A}\) with \(N_{A}\) elements, and \(\mathcal {N}_{B}\) with \(N_{B}\) elements, and selections \( \mathcal {K}_{A}\) and \(\mathcal {K}_{B}\) from \(\mathcal {N}_{A}\) and \(\mathcal {N }_{B}\), respectively. We want to answer the question: is the selection \( \mathcal {K}_{A}\) from \(\mathcal {N}_{A}\) closer to or farther from the ranking of set \(\mathcal {N}_{A}\) than the selection \(\mathcal {K}_{B}\) from \( \mathcal {N}_{B}\) is to the ranking of set \(\mathcal {N}_{B}\)? In terms of the examples given above, we want to know whether, say, a journalist’s choice of the “all time England cricket test team” is more based on the players’ record than an Italian chemistry academy’s choice of the members of its scientific committee is based on the academics’ citation count.

To formalise this question we define a binary relation \(\mathfrak {M} \subseteq \mathfrak {S}\mathcal {\times \mathfrak {S}}\), which we interpret as “orderliness”: orderliness is the property of being close to the ordering of the set.

We are thus defining equivalence classes in \(\mathscr {S}\): two selected sets \(( \mathcal {N}_{A},\mathcal {K}_{A}) ,( \mathcal {N }_{B},\mathcal {K}_{B}) \in \mathscr {S}\) are in the same equivalence class if and only if \(m ( \mathcal {N}_{A},\mathcal {K}_{A}) =m ( \mathcal {N}_{B},\mathcal {K}_{B}) \), that is if the sets \( \mathcal {N}_{A}\) and \(\mathcal {N}_{B}\) have the same number of elements, and the ranks of the selected elements are the same in the selected sets \( ( \mathcal {N}_{A},\mathcal {K}_{A}) \) and \(( \mathcal {N}_{B}, \mathcal {K}_{B}) \). In particular, the image of \((\{ x_{1},\ldots ,x_{N}\} ,\{ x_{i_{1}},\ldots ,x_{i_{K}}\} ) \) is the same as the image of any set where \(x_{j}\) is replaced by any x in an interval of the real line containing \(x_{j}\) but no other element of \(\left\{ x_{1},\ldots ,x_{N}\right\} \). Thus Definition 2, accordingly, defines orderliness as a relation on the classes of equivalence in \(\mathscr {S}\).

We follow the standard convention used to describe preferences, and write \( ( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{B},\mathcal {K}_{B}) \) when \(( \mathcal {N}_{A}, \mathcal {K}_{A}) \) is at least as orderly as \(( \mathcal {N}_{B}, \mathcal {K}_{B}) \). “Strict orderliness”, \(\succ _{\mathfrak {M}}\), and “equal orderliness”, \(\sim _{\mathfrak {M}} \), are naturally defined: \(( \mathcal {N}_{A},\mathcal {K}_{A}) \) is strictly more orderly than \(( \mathcal {N}_{B},\mathcal {K}_{B}) \) if \(( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M} } ( \mathcal {N}_{B},\mathcal {K}_{B}) \) and not \(( \mathcal {N} _{B},\mathcal {K}_{B}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{A}, \mathcal {K}_{A}) \). And \(( \mathcal {N}_{A},\mathcal {K}_{A}) \) and \(( \mathcal {N}_{B},\mathcal {K}_{B}) \) are equally orderly if \(( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M} } ( \mathcal {N}_{B},\mathcal {K}_{B}) \) and \(( \mathcal {N}_{B}, \mathcal {K}_{B}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{A}, \mathcal {K}_{A}) \).

Even with completeness, without any further restrictions, the orderliness relation can still be vacuous: for example, the relation \(\mathfrak {M}= \mathfrak {S}\mathcal {\times \mathfrak {S}}\), where all selected sets are equally orderly, is transitive, reflexive and complete, and so it can well represent a person’s opinion of the orderliness of selections. In the rest of the paper, therefore, we impose some further requirements. As in BBP, they are natural restrictions on \(\mathfrak {M}\), the relation between the images in \(\mathfrak {S}\) of the sets \(( \mathcal {N}, \mathcal {K}) \). BBP (p 904) begin by imposing the “extension rule”: given \(x,y\in \mathcal {N}\), then \(m ( \mathcal {N}, \{ x \} ) \succsim _{\mathfrak {M}}m ( \mathcal {N}, \{ y \} ) \) if and only if \(x<y\). We strengthen the extension rule, requiring that the orderliness comparison be strict, and that it applies to any set, not just singletons.

In words, Axiom 1 requires that the swap between an element in the selection and an element not in the selection makes the selected set strictly more (less) orderly if the rank of the newly selected element is better (worse) than the rank of the removed element.⁸ Note that Axiom 1 is incompatible with the independence axiom (BBP p 905), which, in our framework, would require that, given selections \(( \mathcal {N},\mathcal {K_{A}}) , ( \mathcal {N},\mathcal {K_{B}} ) \in \mathscr {S}\) and \(x\in \mathcal {N}\backslash ( \mathcal { K_{A}}\cup \mathcal {K_{B}}) \), \(( \mathcal {N},\mathcal {K_{A}} ) \succ _{\mathfrak {M}} ( \mathcal {N},\mathcal {K_{B}}) \) implies \(( \mathcal {N},\mathcal {K_{A}}\cup \{ x \} ) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K_{B}}\cup \{ x \} ) \). As BPP note, this axiom rules out “certain types of complementarities” (p 906), and runs therefore contrary to the motivation of the paper, which views selections in their entirety. In fact, BBP show that independence is a strong requirement in this context, as it prevents comparison between selections, except in very special cases (BBP, pp 910–922).

The next two Axioms we impose are the natural extension of the idea of dominance, which “requires that adding an element which is better (worse) than all elements in a given set A [...] leads to a set that is better (worse) than the original set” (BBP, p 905). In our more complex set-up, we want to compare subsets selected from different sets, and therefore we state the axioms as binary comparisons between sets with different number of elements.

Suppose a new element is added⁹ to the set \(\mathcal {N}\), and this new element has better rank than every selected element. Then Axiom 2 requires that, if this new element is selected, the selection becomes more orderly (Axiom 2.i); if it is not selected, the selection becomes less orderly (Axiom 2.ii).

Axiom 3 is the converse of Axiom 2 at the other end of the ranking of the set \(\mathcal {N}\): if a new element is added to the set \(\mathcal {N}\) which is worse ranked than every selected element, then the selection becomes more (respectively, less) orderly if this new element is not selected, Axiom 3.i (respectively, is selected, Axiom 3.ii).

In the first and fourth rows the selection becomes more orderly, and in the second and third less so. In the first two rows the new element can indifferently be in the first or second position of the ranking; and in the third and fourth row, in any of the bottom four positions.

The selection (1) is neither perfect nor antiperfect, and so all the comparisons between it and the selected sets in the above table are strict.

The three axioms are independent: it is possible to define different orderliness relations such that one Axiom is violated but the other two are satisfied. To see this, consider an orderliness relation, \(\widehat{\mathfrak {M}}\), which, given a selected set \(( \mathcal {N},\mathcal {K})\) and an element \(x\notin \mathcal {N}\), requires \(( \mathcal {N},\mathcal {K}) \succ _{\widehat{\mathfrak {M}}}( \mathcal {N}\cup \{ x \} ,\mathcal {K}\cup \{ x \} ) \) if there exists \(z\in \mathcal {N}\backslash \mathcal {K}\) such that \(x<z<y\) for all \(y\in \mathcal {K}\). In word, in order for a new element to make the selection more orderly, it must be better than at least one non-selected element in \(\mathcal {N}\) which is better than all the selected elements. Clearly \(\widehat{\mathfrak {M}}\) violates Axiom 2, and obeys Axioms 1 and 3. A symmetric argument holds in respect of Axiom 3. Lastly, consider an orderliness relation, \(\widetilde{\mathfrak {M}}\), which, given a selected set \(( \mathcal {N},\mathcal {K}) \) and any two elements \(x\in \mathcal {N}\backslash \mathcal {K}\) and \(y\in \mathcal {K}\), requires \(( \mathcal {N},\mathcal {K}) \succ _{\widetilde{\mathfrak {M}}} ( \mathcal {N},\mathcal {K}\cup \{ x \} \backslash \{ y \} ) \) if there exists \(z\in \mathcal {N}\), such that \(x<z<y\). That is, in order to make a selection more orderly, a swap must be between elements which are separated by at least one element in \(\mathcal {N}\). \(\widetilde{\mathfrak {M}}\) violates Axiom 1, and obeys Axioms 2 and 3.

Axioms 2 and 3 dictate the relative orderliness of two selected sets where the ranked sets, \(\mathcal {N}\) and \(\mathcal {N\cup } \left\{ x\right\} \), differ in size by 1. The dominance axioms, labelled by BBP (p 905) the Gärdenfors principle (1976), impose an ordering on selections from a given ranked set, therefore they apply to sets of the same size. We derive them in our framework as immediate consequences of Axioms 2 and 3 in the following corollary.

Intuitively, if a non-selected element that is better ranked than every selected element were instead selected, the selection would become more orderly (Corollary 1.i), and if the best element in the selection were removed, the selection would become less orderly (Corollary 1.ii). Conversely, consider an element in \(\mathcal {N}\) which is worse ranked than every selected element: if it is removed from the selection, then the selection becomes more orderly (Corollary 1.iii); if it is added to the selection, then the selection becomes less orderly (Corollary 1.iv).

While Corollary 1 is an obvious consequence of Axioms 2 and 3, the next result is less immediate, and shows that, although Axioms 1–3 may seem weak, they do have some bite, as can be inferred from one of their implications on the comparison of “extreme” selections from different sized sets.

In words, Proposition 1 says that all perfect selections are equally orderly, and similarly, that all antiperfect selections are equally orderly. The third statement asserts that every perfect selection is strictly more orderly than every non-perfect selection, and every antiperfect selection is strictly less orderly than every non-antiperfect selection. This is entirely reasonable when the selection is from the same set. Thus, selecting the best ten from a set of one hundred elements is clearly more orderly than selecting the best nine and the eleventh. However, as we want to extend the range of selected sets to be compared, it is possible to think of extreme situations when the size of the sets and of the selections is different, and when the argument that a perfect selection is more orderly is less clear-cut. Consider, for example, the following case: is selecting the better of the two elements of a set unquestionably more orderly than selecting the best 25 and the 27-th ranked out of a set with 10,000 elements? Proposition 1 answers unambiguously yes, but someone might argue in favour of the opposite, on the grounds that the former is more likely than the latter to have been determined by other criteria, which by chance happen to coincide with the rank, whereas the selection of the first 25 and the 27-th elements out of a very large set would almost surely be seen as the consequence of a determinate intention to use the rank as the criterion for selection, with the 26-th element being excluded due to some other criterion.

Note that we require the index to takes values ranging from \(-1\) to 1, in the cases of an antiperfect and a perfect selection, respectively. Of course, a different normalisation would allow us to choose any two end points. However, in analogy with the measures of correlation, having 1 indicating perfect orderliness, and \(-1\) its antithesis, where the selection perversely chooses the worst elements, does seem the natural choice. In analogy with the theory of consumers’ preferences, existence of the index of orderliness is ensured by the additional requirement that the orderliness relation is continuous. The formal proof is identical with the proof of the existence of a utility function; as with consumer theory, a lexicographic relation illustrates well the need of continuity to represent orderliness through an index. Consider the following way of comparing the orderliness of any two selected sets: take first the best ranked element in each: if only one selection includes it, then that selection is strictly more orderly than the other. Otherwise, look next at the second best ranked element, and again, if it is included only in one selection, this is the more orderly one. Again, if they are both or neither selected, go to the next ranked element and so on. If one selection “runs out of elements” before the other, then it is less orderly. In analogy with consumer preferences, this relation is reflexive, complete and transitive, but it is not continuous, and so it cannot be represented by an index of orderliness.¹⁰

To ensure existence of a orderliness index representing a relation \(\mathfrak {M}\), therefore we require that the orderliness relation be continuous.¹¹

The analogous of Proposition 2 for the index (3) holds, but its proof is essentially identical to the above proof and is omitted.

Now consider the following two selected sets, \( \left( \mathcal {N_{A}},\mathcal {K}_{A}\right) \) and \( \left( \mathcal {N_{B}},\mathcal {K}_{B}\right) \) such that \( m\left( \mathcal {N_{A}},\mathcal {K}_{A}\right) = \left( 11 , \left\{ 2,3,5,8,9 \right\} \right) \) and \( m\left( \mathcal {N_{B}},\mathcal {K}_{B}\right) = \left( 11 , \left\{ 1,2,3,8,11 \right\} \right) \). Graphically, these differ only in the selection of the elements in the boxes:

Open image in new window

We have \(M{\left( m \left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \right) } < M{\left( m \left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \right) } \) and \(M_{1}{\left( m \left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \right) }> M_{1}{\left( m \left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \right) }\): if the relation of orderliness is represented by index (2), then the second is more orderly, and vice versa, if orderliness is described by a relation represented by index (3) then the first is more orderly. Thus both indices M and \(M_{1}\) satisfy Axioms 1–3, and yet they give a different answer to the question of the relative orderliness of the two sets: Axioms 1–3 are not characterising.

The restriction required to ensure characterisation is remarkably simple, as the rest of the paper shows. We replace Axiom 1 with the following.

In words, suppose that a new element is added to the set \(\mathcal {N}\), and that this makes the new selected set more orderly. Then it must be the case that the “mirror image” of the new selected set, that is the selection from the same set which includes all the elements which are not selected in the original selection, and excludes all those which were included, is less orderly than the mirror image of the original selected set. Axiom 4 is stated for the case when the new element is selected, but it of course implies the opposite case: \(\big ( \mathcal {N}\cup \left\{ x\right\} , \mathcal {K} \big ) \succsim _{\mathfrak {M}}\left( \mathcal {N},\mathcal {K}\right) \) if and only if \(\left( \mathcal {N} ,\mathcal {N}\backslash \mathcal {K}\right) \succsim _{\mathfrak {M}}\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {N}\backslash \mathcal {K} \cup \left\{ x\right\} \right) \). Axiom 4 is illustrated below, using the representation of a selected set given above:

Open image in new window

If the first selected set is more orderly than the third, which differ from the first as it has a new element, ranked 5, which is selected, then it must be the case that the second is less orderly than the fourth, which is the mirror image of the third, and vice versa.

We can now establish the main result of the paper.

This Lemma implies that a change that decreases the sum of the ranks of the selected elements makes the set more orderly would make the mirror image of the selected set less orderly, as required by Axiom 4.

We can now prove the main theorem, by showing that every index of orderliness representing a relation \(\mathfrak {M}\) satisfying Axioms 2 and 3 which is a decreasing function of the sum of the ranks of the selected elements is a strictly monotonic transformation of \(M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }\).

Obviously any strictly monotonic transformation of the index (2) would represent a relation satisfying Axioms 2–4. The functional form in (2), which is linear in the sum of ranks, has the natural property of being 0, in expectation, if the selection is completely random: to see this, note that the expected rank of K random draws is \(\frac{K(N+1)}{2}\). Substitute this in (2), to obtain 0.

Theorem 1 is the main result of the paper: in words, it shows that the index of orderliness (2) satisfies the dominance Axioms 2 and 3, and the mirror invariance Axiom 4, and, conversely, that any index which satisfies the dominance Axioms 2 and 3, and the mirror invariance Axiom 4, is a monotonic transformation of the index of orderliness (2). Only index (2) satisfies all three axioms, and it is the only one (up to monotonic transformations) to do so: Axioms 2, 3 and 4 together characterise the index of orderliness (2).

The structure of the proof of Theorem 1 allows to replace the mirror invariance Axiom with an equivalent one, which gives a different interpretation to the restriction imposed on the orderliness relation to obtain characterisation.

In words, Axiom 5 considers two subsequent “swaps” between a selected and a non-selected element. The element newly included in the selection and the one removed differ in rank by a. The second swap turns the intermediate selection obtained with the first swap into the final one, by selecting an element not selected and removing another element from the (intermediate) selection which are \(-a\) ranks apart. Thus, for example, if the first swap selects the (originally non-selected) 12-th ranked element and de-selects the 19-th ranked; their difference in rank is \(-7\). Suppose the second swap selects the (originally non-selected) 56-th ranked element and de-selects the 49-th ranked; their difference in rank is 7.¹² Then, by Axiom 5, the initial and the final selection are equally orderly. In shorter, looser words, the effect of a change in the selection depends only on the extent of the change, not on whether it affects the upper or the lower tail of the distribution.¹³ We note that the relative importance of the position in the ranking is at the core of the analysis of distance between preferences in Can (2014). It is straightforward to establish the following.

Note that Theorem 1 and Corollary 2 together imply that Axioms 4, mirror invariance, and Axiom 5, position irrelevance, are equivalent: each holds for a relation if and only if the other does.

The study of metrics on orders, initiated by Kendall (1938) has developed a measure of distance between two rankings of the elements of a given set \(\mathcal {N}\), the Kendall-Tau distance (Kemeny 1959), recently extended to choice functions (Klamler 2008). This is obtained by counting the number of times the two rankings “switch” two elements \(x,y \in \mathcal {N}\). That is, if \(\rho _{1}\) and \(\rho _{2}\) are the two rankings, the distance between them is the number of pairs \(\left( x,y\right) \in \mathcal {N}\times \mathcal {N}\) such that \(\rho _{1}\left( x\right) >\rho _{2}\left( x\right) \) and \(\rho _{1}\left( y\right) <\rho _{2}\left( y\right) \). This count can then be normalised by the maximum possible number of switches.

This idea has been applied to voting mechanism, whereby a ranking is interpreted as a vote, and the outcome of an election as the aggregation of the rankings of different voters.¹⁴ We view the Kendall-Tau distance from a different angle: note in the first place that the index of orderliness (2) applies to a broader range of situations, such as those where the selections being compared are from different sets. Secondly, the orderliness index (2) compares selections, rather than rankings, as the Kendall-Tau distance does. Nevertheless, a selection does rank the elements of a set, albeit in a very coarse manner: the K selected elements are joint first, and the non-selected ones are joint \(\left( K+1\right) \)-th. Recall that the Kendall-Tau distance is defined when there are ties, and so we can measure the Kendall-Tau distance between the ranking determined by a selection and a given ranking \(\rho \) of \(\mathcal {N}\). In this section we study the relation between this Kendall-Tau distance and the index of orderliness (2), \(M{\left( \mathcal {N},\mathcal {K}\right) }\).

Thus, in addition to satisfying natural axioms, the index (2) coincides, in the situations where both can be applied, with an established measure of distance between rankings. This paper therefore provides a micro-foundation of the Kendall-Tau distance, which currently lacks one.

The Kendall-Tau distance and the Spearman coefficient of rank correlation give in general different answers to the question of the comparison between two rankings: this implies that there at least one of the axioms is violated by the Spearman coefficient of correlation between the original ranking and the ordering \(\kappa ^{\mathcal {N}\mathcal {K}}\) derived in (24). The Spearman coefficient, originally intended for situations with no ties in either ordering, has been adapted by Taylor (1964) for the case of ties: the adaptation involves replacing the ties with “fractional ties” and adjust with one correction factor for each set of tied values, again, in either of the two orderings. Given the assumption of no ties in the ordering of the set \(\mathcal {N}\), and given that there are only two ranks in (24), and thus exactly two correction factors will be required, one for K, the other for (N-K) tied elements, the Spearman coefficient of correlation between the original ranking and the ordering \(\kappa ^{\mathcal {N}\mathcal {K}}\) derived in (24) is given by:

$$\begin{aligned} 1-\frac{\frac{6}{N}\sum _{i=1}^{N}\left( \rho _{i}\left( x\right) -\kappa ^{\mathcal {N}\mathcal {K}} \left( x\right) \right) ^{2}+\frac{N^{2}-1-3K\left( N-K\right) }{2}}{N^{2}-1} \text {.} \end{aligned}$$

(28)

We now show one example of a violation of one of the Axioms. Consider a selected set \(\left( \mathcal {N},\mathcal {K}\right) \) represented by 0101110000000. Applying the above formula, we find that the Spearman correlation coefficient is 0.3929. Now consider adding a new element which occupies the seventh place in the new ranking, hence is worse that all of the selected elements, and is itself selected, so the new selected set can be represented by Open image in new window (the new element is the one in the box). By Axiom 3.ii, the new set must be less orderly. However, applying the above formula to the new set gives a higher value of 0.4154, in violation of Axiom 3.ii. Similar counterexamples can be found for the other Axioms, except Axiom 1.

Another measure of correlation is the Kappa correlation. This applies to ranking with very few steps (and coincides with the selection when there are only two categories). This suggests a direction for future research in this area, namely the extension of the index of orderliness to situations where the selection is more nuanced than simply “in” and “out”. An example is a coach selection of the first team and of the second team for a sport tournament.

The orderliness index proposed here finds a natural application in the analysis of promotions in hierarchical organisations, where, at given intervals, individuals from the pool of potential candidates are assessed and some are promoted, some are not. The determinants of promotions may be stated formally or known implicitly: thus for example, academic promotions may be decided by criteria ranging from scientific productivity, to teaching performance, fund-rising ability, seniority, or age; the relative importance of performance along the various criteria may of course vary from institution to institution and from discipline to discipline.

As explained in Checchi et al. (2014), academic careers in Italy are firmly channelled along narrowly defined research fields: every academic is allocated to one and only one of 371 scientific sectors (SSDs), changing sector is relatively unusual, and the members of the promotion panels in each scientific sector are chosen exclusively among academic post holders in that sector.

The dataset assembled by Checchi et al. (2014) allows us to rank all the candidates for promotion to associate professor in each scientific sector in the period from 1995 to 2011 according to two criteria, their record of publication in international journals¹⁵ and their age.

Since with minor exceptions, each panel could select for promotion only candidates from the lower rank within the same sector, it is a simple matter to calculate the orderliness of the selections made by the promotion panel in each scientific sector, in each of the four separate sub-periods which Checchi et al. (2014) suggest to aggregate calendar years to reflect the pattern of the promotion rounds.

Open image in new window

Figure 1 describes the outcome of this process. Each dot corresponds to one of 371 scientific sectors, except the smaller ones, in one 4-year interval. The abscissa of a dot is the orderliness according to scientific productivity, its ordinate the orderliness according to age.¹⁶ Broad scientific areas are colour coded; we have also singled out economics and econometrics among the social science sectors. An analysis of the Italian university sector might use these indices as characteristics of the selection procedures used in promotion and appointments. A cursory visual analysis suggests that overall productivity matters more than age, and also that it matters more in STEM subjects.

Often an agent chooses a number of options from a larger set, the elements of which can be ranked in some objective or generally accepted manner. We propose a way to assess how close this agent’s selection is to the order of the set. Aside from its intrinsic interest in the examples given in the introduction, availability of this measure might address the need reported in the medical literature for an objective evaluation of clinical services (Iverson 1998; Bickman 2012), or help the study of aspects of social mobility, such as the importance of a person’s family position in her access to leadership positions in society.¹⁷ In this paper, we require that the comparison between selections satisfies some natural dominance requirements (Axioms 2 and 3), and the requirement that the mirror image of a change that nears a selection to the ranking must push the mirror image of that selection away from the ranking (Axiom 4). These three simple axioms prove very strong, in that they identify a unique index which provides an unambiguous comparison of the relative closeness of any selection to the order of the set from which it is taken. This index has a very simple expression, which depends only on the sum of the ranks of the selected elements, and the number of elements in the set and in the selection. The paper ends with a specific example of the potential applications of our index to the Italian university system.