Abstract
We study the problem of evaluating whether the selection from a set is close to the ordering of the set determined by an exogenously given measure. Our main result is that three axioms, two naturally capturing “dominance”, and a stronger one imposing a form of symmetry in the comparison of selections, are sufficient to evaluate how close any selection from any set is to the given ordering of the set. This closeness is given by a very simple index, which is a linear function of the sum of the ranks of the selected elements. The paper ends by relating this index to the existing literature on distance between orderings, and also offers a practical application of the index.
1 Introduction
Selections are typically made according to a varying blend of objective measures and subjective judgments. A sport coach might base her team choice on the recent performance of individuals in her squad (batting and bowling/pitching averages, tennis rankings, trial results for track and field, and so on), and on her sense of who is the best person for each given role, given the expected conditions. Many universities do not strictly follow SATs results and school grades when choosing whom to admit, but take into account a student’s social background and his potential contribution to desirable characteristics of the student body, like diversity. Applicants for academic jobs might be ranked according to bibliometric measures, but the appointment panel’s judgment often leads to decision that do not map precisely into the ranking.^{1} Large and complex procurement contracts often demand the subtle evaluation of complex qualitative elements, and lowest price is seldom the only criterion used to award these contracts.^{2} And so forth and so on.
Observers and decision makers might be interested in some means of comparing the choices of different selectors. If the property of following the measurable dimension is labelled “orderliness”, someone might want to compare the orderliness of different selections, for example to determine which is more “orderly”.
We can think of at least three conceptually distinct situations where this comparison is meaningful.^{3} First, we may want to compare different selections from the same set. For example, the public may want to know how close the proposed shortlists of the different judges of a book or film prize are to a marketdetermined (sales, box office earnings) ordering. Second, the comparison might be of selections from altogether different sets. A cricket fan may want to know whether Australia’s selection for the Ashes team is closer to the players’ current rankings than England’s. A university whose admission policies are under scrutiny in court may want to argue that its admission policy is as based on SATs as those of comparable institutions. Or a government minister concerned about corruption in procurement contracts or personnel hiring may want to compare the orderliness of various commissioning boards or hiring panels, to identify and perhaps investigate atypical behaviour. On a broader scale, orderliness is a helpful yardstick when assessing inequality of opportunity: it can serve as a measure of nepotism, telling how close selection into society’s elite is to the pecking order determined by family history, or of plutocracy, measured for example by the closeness of membership of Parliament to a person’s position in the income distribution. The third kind of potential comparisons is for situations where both the selection and the selectors are the same, but there are two or more ways of ordering the elements of a set. In these cases, which are close to the topic of some related literature considered below (Kemeny 1959; Klamler 2008), the focus is on their relative importance in the selection. For example, a rugby analyst may want to know whether weight or speed is more important to be selected as a threequarter for the Springboks. Or an external funder may want to know which of teaching and research is more important for promotion in a given university.
Comparing selections is straightforward only in the starkest cases. Sure, the selection of the best ranked is unquestionably more orderly than selections that do not. But is a university which, from its 102 applicants, admits as students those ranked according to their SAT scores second to 5th, 8th, 11th, 12th, 24th, 25th, 38th, and 47th more orderly than one that chooses the first 20, the 57th, and those ranked between 170 and 180 out of 500 applicants? Or, in an even simpler example, is picking the second ranked out of ten candidates for a job more orderly than selecting the third ranked out of 19?^{4}
This paper takes an axiomatic approach to comparing selections. The axioms are requirements of orderliness: a decision maker who is called to opine which is more orderly of two selections from different sets, that is which selection is in her opinion closer to the order—determined by a given measure—of the set from which it is taken, should only make judgements that obey these axioms. This is conceptually similar to the approach taken by Sen (1976) and Cowell and Kuga (1981) in regard to judgements on which of two groups of individuals poverty or inequality, respectively, is greater.
We begin with three natural axioms, which extend the dominance principle in the comparison of sets (Barberà et al. 2004, BBP in what follows). The first requires that when an element is replaced in the selection with a better one, then the selection is deemed to be more orderly. In the next two axioms we depart from the setup of the literature reviewed in BBP, and in some of its extensions, such as Rubinstein and Salant (2006). The departure is that we consider elements that are not in the original set, thus allowing the comparison of selections from different sets, as in many of the examples given above. These axioms require that if a new element appears which is better than all the selected (nonselected) ones, then the selection is considered more (less) orderly if the new element is selected. Analogously for the appearance of a worse element. Extending the axioms to different sets implies that we cannnot, like the literature cited, take the rank of the elements as the starting point of the analysis, since a new elements change the ranks of all the elements below it. Instead, we need to make formally explicit the mapping from the measured performance to the rank of the elements ordered by this measure.
These three axioms, which we show to be independent, are not characterising: there are different orderings of selections from sets which satisfies them. In our main result we show that characterisation is achieved by replacing the first axiom with a stronger one, which we label “mirror invariance”. This requires that if a change makes a selection more orderly, then the mirror image of the selection—that is the selection of only the nonselected elements—is made less orderly by the mirror image of the change. This is equivalent to requiring that the importance attributed to changes in the ordering be independent of where they occur: departures from the ranking are as important if they occur among the best or the worst ranked elements. This result is established in Theorem 1, which shows that the orderliness relation characterised by Axioms 2–4 is represented, in close analogy to the representation of a consumer’s preference by a utility function, by an “index of orderliness”, given by a simple function of the number of elements in the set, the number of selected elements, and the sum of the ranks of the selected elements.
The paper continues by showing that our index of orderliness has a simple relation with the concept of distance between rankings originally proposed by Kendall (1938), and given an axiomatic foundation by Kemeny (1959). This distance compares different selections from the same set, and, in this case, Proposition 4 shows that if a selection is more orderly than another selection from the same set, then the ranking of the set naturally induced by the first selection is nearer, in the sense of the KendallTau distance, to the ordering of the set.
The paper is organised as follows: the dominance axioms and some preliminary results are in Sect. 2. The core of the paper is Sect. 3, which presents the index of orderliness, and strengthens one of the axioms to obtain a complete ordering of all selected sets. The relation with existing literature is explored in Sect. 4, and the paper ends with Sect. 5, which shows how the index can be used to assess the evaluation mechanism for promotion to professorship in Italian universities. A brief conclusion follows.
2 Axioms of “orderliness”
Let \(\mathcal {N}\varsubsetneq \mathbb {R}\) contain \(N\in \mathbb {N}\backslash \left\{ 1\right\} \) distinct elements: \(x,y\in \mathcal {N}\) implies \(x\ne y\) .^{5} The inequality relation in \(\mathbb {R}\) induces a transitive, complete and antisymmetric relation between the members of \(\mathcal {N}\). This relation is naturally defined as the rank in \(\mathcal {N}\), \(\rho _{\mathcal {N}}\), a bijective mapping of the elements of \(\mathcal {N}\) into the set of the first N natural numbers \(\rho _{\mathcal {N}} : \mathcal {N}\longrightarrow \{ 1,2,\ldots ,N\} \), which satisfies \( \rho _{\mathcal {N}}( x) <\rho _{\mathcal {N}} ( y) \) if and only if \(x<y\). We choose this convention, that x has a better^{6} rank in \( \mathcal {N}\) than y if it is smaller, for definiteness. It would refer for example, to a situation where the real numbers are marathon runners’ personal best times; the changes needed to apply the framework to situations where a larger real number is better, such as cricketers’ career runs or scientists’ citation counts, are obvious.
We next define a selection \(\mathcal {K}\) as a proper and nonempty subset of \(\mathcal {N}\), that is \(\mathcal {K}\varsubsetneq \mathcal {N}\), and \(\mathcal {K} \ne \varnothing \). Let \(K\in \left\{ 1,\ldots ,N1\right\} \) be the number of elements of \(\mathcal {K}\). We define the pair \(\left( \mathcal {N},\mathcal {K} \right) \) a “selected set”.^{7} Let \( \mathscr {S}\) be the family of selected sets.
We single out the situations where the selection follows exactly the ranking.
Definition 1
Given a selected set \(( \mathcal {N},\mathcal {K}) \in \mathscr {S}\) , the selection \(\mathcal {K}\) is “perfect ” if \(x\in \mathcal {K}\) and \(y\in \mathcal {N}\) \(\backslash \mathcal {K}\) implies \(x<y\). The selection \(\mathcal {K}\) is “antiperfect” if \(x\in \mathcal {K}\) and \(y\in \mathcal {N}\) \(\backslash \mathcal {K}\) implies \(y<x\).
In words, the selection \(\mathcal {K}\) is perfect if no selected element has a rank worse than a nonselected element, and it is antiperfect if every selected element has a worse rank than every nonselected element.
Let \(\mathscr {S}^{P}\subseteq \mathscr {S}\) be the set of all perfect selections, and conversely, let \(\mathscr {S}^{A}\subseteq \mathscr {S}\) be the set of all antiperfect selections. Note that since \(\mathcal {K}\) is a proper nonempty subset of \(\mathcal {N}\), \(\mathscr {S}^{P}\cap \mathscr {S} ^{A}= \varnothing \): no selection is simultaneously both perfect and antiperfect.
We want to compare selections, that is order the set \(\mathscr {S}\), in the following sense: consider two ranked sets \(\mathcal {N}_{A}\) with \(N_{A}\) elements, and \(\mathcal {N}_{B}\) with \(N_{B}\) elements, and selections \( \mathcal {K}_{A}\) and \(\mathcal {K}_{B}\) from \(\mathcal {N}_{A}\) and \(\mathcal {N }_{B}\), respectively. We want to answer the question: is the selection \( \mathcal {K}_{A}\) from \(\mathcal {N}_{A}\) closer to or farther from the ranking of set \(\mathcal {N}_{A}\) than the selection \(\mathcal {K}_{B}\) from \( \mathcal {N}_{B}\) is to the ranking of set \(\mathcal {N}_{B}\)? In terms of the examples given above, we want to know whether, say, a journalist’s choice of the “all time England cricket test team” is more based on the players’ record than an Italian chemistry academy’s choice of the members of its scientific committee is based on the academics’ citation count.
To formalise this question we define a binary relation \(\mathfrak {M} \subseteq \mathfrak {S}\mathcal {\times \mathfrak {S}}\), which we interpret as “orderliness”: orderliness is the property of being close to the ordering of the set.
Definition 2
Given two selected sets \(( \mathcal {N}_{A},\mathcal {K} _{A}) , ( \mathcal {N}_{B},\mathcal {K}_{B}) \in \mathscr {S}\), \(( \mathcal {N}_{A},\mathcal {K}_{A}) \) is at least as orderly as \(( \mathcal {N}_{B},\mathcal {K}_{B}) \) if and only if \((m( \mathcal {N}_{A},\mathcal {K}_{A}) ,m( \mathcal {N}_{B},\mathcal {K} _{B}) )\in \mathfrak {M}\).
We are thus defining equivalence classes in \(\mathscr {S}\): two selected sets \(( \mathcal {N}_{A},\mathcal {K}_{A}) ,( \mathcal {N }_{B},\mathcal {K}_{B}) \in \mathscr {S}\) are in the same equivalence class if and only if \(m ( \mathcal {N}_{A},\mathcal {K}_{A}) =m ( \mathcal {N}_{B},\mathcal {K}_{B}) \), that is if the sets \( \mathcal {N}_{A}\) and \(\mathcal {N}_{B}\) have the same number of elements, and the ranks of the selected elements are the same in the selected sets \( ( \mathcal {N}_{A},\mathcal {K}_{A}) \) and \(( \mathcal {N}_{B}, \mathcal {K}_{B}) \). In particular, the image of \((\{ x_{1},\ldots ,x_{N}\} ,\{ x_{i_{1}},\ldots ,x_{i_{K}}\} ) \) is the same as the image of any set where \(x_{j}\) is replaced by any x in an interval of the real line containing \(x_{j}\) but no other element of \(\left\{ x_{1},\ldots ,x_{N}\right\} \). Thus Definition 2, accordingly, defines orderliness as a relation on the classes of equivalence in \(\mathscr {S}\).
We follow the standard convention used to describe preferences, and write \( ( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{B},\mathcal {K}_{B}) \) when \(( \mathcal {N}_{A}, \mathcal {K}_{A}) \) is at least as orderly as \(( \mathcal {N}_{B}, \mathcal {K}_{B}) \). “Strict orderliness”, \(\succ _{\mathfrak {M}}\), and “equal orderliness”, \(\sim _{\mathfrak {M}} \), are naturally defined: \(( \mathcal {N}_{A},\mathcal {K}_{A}) \) is strictly more orderly than \(( \mathcal {N}_{B},\mathcal {K}_{B}) \) if \(( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M} } ( \mathcal {N}_{B},\mathcal {K}_{B}) \) and not \(( \mathcal {N} _{B},\mathcal {K}_{B}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{A}, \mathcal {K}_{A}) \). And \(( \mathcal {N}_{A},\mathcal {K}_{A}) \) and \(( \mathcal {N}_{B},\mathcal {K}_{B}) \) are equally orderly if \(( \mathcal {N}_{A},\mathcal {K}_{A}) \succsim _{\mathfrak {M} } ( \mathcal {N}_{B},\mathcal {K}_{B}) \) and \(( \mathcal {N}_{B}, \mathcal {K}_{B}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{A}, \mathcal {K}_{A}) \).
Even with completeness, without any further restrictions, the orderliness relation can still be vacuous: for example, the relation \(\mathfrak {M}= \mathfrak {S}\mathcal {\times \mathfrak {S}}\), where all selected sets are equally orderly, is transitive, reflexive and complete, and so it can well represent a person’s opinion of the orderliness of selections. In the rest of the paper, therefore, we impose some further requirements. As in BBP, they are natural restrictions on \(\mathfrak {M}\), the relation between the images in \(\mathfrak {S}\) of the sets \(( \mathcal {N}, \mathcal {K}) \). BBP (p 904) begin by imposing the “extension rule”: given \(x,y\in \mathcal {N}\), then \(m ( \mathcal {N}, \{ x \} ) \succsim _{\mathfrak {M}}m ( \mathcal {N}, \{ y \} ) \) if and only if \(x<y\). We strengthen the extension rule, requiring that the orderliness comparison be strict, and that it applies to any set, not just singletons.
Axiom 1
(SwapDominance) For all \(\left( \mathcal {N},\mathcal {K }\right) \in \mathscr {S}\) and \(x\in \mathcal {N}\backslash \mathcal {K}\) and \( y\in \mathcal {K}\), \(\left( \mathcal {N},\mathcal {K\cup }\left\{ x\right\} \backslash \left\{ y\right\} \right) \succ _{\mathfrak {M}}\left( \mathcal {N}, \mathcal {K}\right) \) if and only if \(x<y\).
In words, Axiom 1 requires that the swap between an element in the selection and an element not in the selection makes the selected set strictly more (less) orderly if the rank of the newly selected element is better (worse) than the rank of the removed element.^{8} Note that Axiom 1 is incompatible with the independence axiom (BBP p 905), which, in our framework, would require that, given selections \(( \mathcal {N},\mathcal {K_{A}}) , ( \mathcal {N},\mathcal {K_{B}} ) \in \mathscr {S}\) and \(x\in \mathcal {N}\backslash ( \mathcal { K_{A}}\cup \mathcal {K_{B}}) \), \(( \mathcal {N},\mathcal {K_{A}} ) \succ _{\mathfrak {M}} ( \mathcal {N},\mathcal {K_{B}}) \) implies \(( \mathcal {N},\mathcal {K_{A}}\cup \{ x \} ) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K_{B}}\cup \{ x \} ) \). As BPP note, this axiom rules out “certain types of complementarities” (p 906), and runs therefore contrary to the motivation of the paper, which views selections in their entirety. In fact, BBP show that independence is a strong requirement in this context, as it prevents comparison between selections, except in very special cases (BBP, pp 910–922).
The next two Axioms we impose are the natural extension of the idea of dominance, which “requires that adding an element which is better (worse) than all elements in a given set A [...] leads to a set that is better (worse) than the original set” (BBP, p 905). In our more complex setup, we want to compare subsets selected from different sets, and therefore we state the axioms as binary comparisons between sets with different number of elements.
Axiom 2
 i.
\(( \mathcal {N\cup }\{ x\} , \mathcal {K\cup }\{ x\} ) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K}) \); strictly unless \(( \mathcal {N}, \mathcal {K}) \in \mathscr {S}^{P}\).
 ii.
\(( \mathcal {N},\mathcal {K}) \succsim _{\mathfrak {M}} ( \mathcal {N\cup } \{ x\} ,\mathcal {K} ) \); strictly unless \(( \mathcal {N},\mathcal {K}) \in \mathscr {S}^{A}\).
Suppose a new element is added^{9} to the set \(\mathcal {N}\), and this new element has better rank than every selected element. Then Axiom 2 requires that, if this new element is selected, the selection becomes more orderly (Axiom 2.i); if it is not selected, the selection becomes less orderly (Axiom 2.ii).
Axiom 3
 i.
\(( \mathcal {N\cup }\{ x\} , \mathcal {K}) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K} ) \); strictly unless \(( \mathcal {N},\mathcal {K}) \in \mathscr {S}^{P}\).
 ii.
\(( \mathcal {N},\mathcal {K}) \succsim _{\mathfrak {M}} ( \mathcal {N\cup } \{ x \} ,\mathcal { K\cup } \{ x \} ) \); strictly unless \(( \mathcal {N}, \mathcal {K}) \in \mathscr {S}^{A}\).
Axiom 3 is the converse of Axiom 2 at the other end of the ranking of the set \(\mathcal {N}\): if a new element is added to the set \(\mathcal {N}\) which is worse ranked than every selected element, then the selection becomes more (respectively, less) orderly if this new element is not selected, Axiom 3.i (respectively, is selected, Axiom 3.ii).
In the first and fourth rows the selection becomes more orderly, and in the second and third less so. In the first two rows the new element can indifferently be in the first or second position of the ranking; and in the third and fourth row, in any of the bottom four positions.
The selection (1) is neither perfect nor antiperfect, and so all the comparisons between it and the selected sets in the above table are strict.
The three axioms are independent: it is possible to define different orderliness relations such that one Axiom is violated but the other two are satisfied. To see this, consider an orderliness relation, \(\widehat{\mathfrak {M}}\), which, given a selected set \(( \mathcal {N},\mathcal {K})\) and an element \(x\notin \mathcal {N}\), requires \(( \mathcal {N},\mathcal {K}) \succ _{\widehat{\mathfrak {M}}}( \mathcal {N}\cup \{ x \} ,\mathcal {K}\cup \{ x \} ) \) if there exists \(z\in \mathcal {N}\backslash \mathcal {K}\) such that \(x<z<y\) for all \(y\in \mathcal {K}\). In word, in order for a new element to make the selection more orderly, it must be better than at least one nonselected element in \(\mathcal {N}\) which is better than all the selected elements. Clearly \(\widehat{\mathfrak {M}}\) violates Axiom 2, and obeys Axioms 1 and 3. A symmetric argument holds in respect of Axiom 3. Lastly, consider an orderliness relation, \(\widetilde{\mathfrak {M}}\), which, given a selected set \(( \mathcal {N},\mathcal {K}) \) and any two elements \(x\in \mathcal {N}\backslash \mathcal {K}\) and \(y\in \mathcal {K}\), requires \(( \mathcal {N},\mathcal {K}) \succ _{\widetilde{\mathfrak {M}}} ( \mathcal {N},\mathcal {K}\cup \{ x \} \backslash \{ y \} ) \) if there exists \(z\in \mathcal {N}\), such that \(x<z<y\). That is, in order to make a selection more orderly, a swap must be between elements which are separated by at least one element in \(\mathcal {N}\). \(\widetilde{\mathfrak {M}}\) violates Axiom 1, and obeys Axioms 2 and 3.
Axioms 2 and 3 dictate the relative orderliness of two selected sets where the ranked sets, \(\mathcal {N}\) and \(\mathcal {N\cup } \left\{ x\right\} \), differ in size by 1. The dominance axioms, labelled by BBP (p 905) the Gärdenfors principle (1976), impose an ordering on selections from a given ranked set, therefore they apply to sets of the same size. We derive them in our framework as immediate consequences of Axioms 2 and 3 in the following corollary.
Corollary 1
 i.
For all \(( \mathcal {N},\mathcal {K}) \in \mathscr {S}\) and \(x\in \mathcal {N}\backslash \mathcal {K}\) such that \(x<y\) for all \(y\in \mathcal {K}\), \( ( \mathcal {N},\mathcal {K\cup } \{ x \} ) \succsim _{\mathfrak {M}} ( \mathcal {N},\mathcal {K}) \); strictly unless \(( \mathcal {N},\mathcal {K}) \in \mathscr {S}^{A}\).
 ii.
for all \(\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}\) and \(x\in \mathcal {K}\) such that \(x<y\) for all \(y\in \mathcal {K} \backslash \left\{ x\right\} \), \(\left( \mathcal {N},\mathcal {K}\right) \succsim _{\mathfrak {M}}\left( \mathcal {N},\mathcal {K}\backslash \left\{ x\right\} \right) \); strictly unless \(\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}^{P}\).
 iii.
For all \(\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}\) and \(x\in \mathcal {K}\) such that \(y<x\) for all \(y\in \mathcal {K} \backslash \left\{ x\right\} \), \(\left( \mathcal {N},\mathcal {K} \backslash \left\{ x\right\} \right) \succsim _{\mathfrak {M}}\left( \mathcal {N} ,\mathcal {K}\right) \); strictly unless \(\left( \mathcal {N},\mathcal {K} \right) \in \mathscr {S}^{P}\).
 iv.
for all \(\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}\) and \(x\in \mathcal {N}\backslash \mathcal {K}\) such that \(y<x\) for all \(y\in \mathcal {K}\), \(\left( \mathcal {N},\mathcal {K}\right) \succsim _{ \mathfrak {M}}\left( \mathcal {N},\mathcal {K\cup }\left\{ x\right\} \right) \); strictly unless \(\left( \mathcal {N},\mathcal {K}\right) \in \mathscr {S}^{A}\).
Proof
We only establish the first claim, as the proof of the remainder is essentially identical. Consider a selected set \(\left( \mathcal {N},\mathcal {K }\right) \), and an \(x\in \mathcal {N}\backslash \mathcal {K}\) such that \(x<y\) for all \(y\in \mathcal {K}\). Note that this implies that \(\left( \mathcal {N}, \mathcal {K}\right) \notin \mathscr {S}^{P}\). Take \(z\notin \mathcal {N}\), such that \(z<y\) for all \(y\in \mathcal {K}\), and xRz (that is z is better than every selected element, but worse than x). By Axiom 2.ii, \( \left( \mathcal {N},\mathcal {K}\right) \succ _{\mathfrak {M}}\left( \mathcal {N} \cup \left\{ z\right\} ,\mathcal {K}\right) \). Next note that, by Axiom 3.i, \(\left( \mathcal {N}\cup \left\{ z\right\} ,\mathcal {K} \right) \succ _{\mathfrak {M}}\left( \mathcal {N}\cup \left\{ z\right\} \backslash \left\{ x\right\} ,\mathcal {K}\right) \). Finally, notice that \( m\left( \mathcal {N}\cup \left\{ z\right\} \backslash \left\{ x\right\} , \mathcal {K}\right) =m\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K} \right) \), since an excluded element better than all selected elements is replaced by another, and so \(\left( \mathcal {N},\mathcal {K}\right) \succ _{ \mathfrak {M}}\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {K}\right) \). \(\square \)
Intuitively, if a nonselected element that is better ranked than every selected element were instead selected, the selection would become more orderly (Corollary 1.i), and if the best element in the selection were removed, the selection would become less orderly (Corollary 1.ii). Conversely, consider an element in \(\mathcal {N}\) which is worse ranked than every selected element: if it is removed from the selection, then the selection becomes more orderly (Corollary 1.iii); if it is added to the selection, then the selection becomes less orderly (Corollary 1.iv).
While Corollary 1 is an obvious consequence of Axioms 2 and 3, the next result is less immediate, and shows that, although Axioms 1–3 may seem weak, they do have some bite, as can be inferred from one of their implications on the comparison of “extreme” selections from different sized sets.
Proposition 1
 i.
Let \(( \mathcal {N}_{p_{1}},\mathcal {K} _{p_{1}}) , ( \mathcal {N}_{p_{2}},\mathcal {K}_{p_{2}}) \in \mathscr {S}^{P}\). Then \(( \mathcal {N}_{p_{1}},\mathcal {K} _{p_{1}}) \sim _{\mathfrak {M}} ( \mathcal {N}_{p_{2}},\mathcal {K} _{p_{2}}) \).
 ii.
Similarly, let \( ( \mathcal {N}_{a_{1}},\mathcal {K}_{a_{1}}) ,( \mathcal {N}_{a_{2}},\mathcal {K}_{a_{2}}) \in \mathscr {S}^{A}\). Then \(( \mathcal {N}_{a_{1}},\mathcal {K}_{a_{1}}) \sim _{\mathfrak { M}} ( \mathcal {N}_{a_{2}},\mathcal {K}_{a_{2}}) \).
 iii.
Let \(( \mathcal {N},\mathcal {K}) \in \mathscr {S}\backslash \mathscr {S}^{P}\backslash \mathscr {S}^{A}\); let \(( \mathcal {N}_{p}, \mathcal {K}_{p}) \in \mathscr {S}^{P}\); let \(( \mathcal {N}_{a}, \mathcal {K}_{a}) \in \mathscr {S}^{A}\). Then \(( \mathcal {N}_{p}, \mathcal {K}_{p}) \succ _{\mathfrak {M}}( \mathcal {N},\mathcal {K} ) \) and \(( \mathcal {N},\mathcal {K}) \succ _{\mathfrak {M} }( \mathcal {N}_{a},\mathcal {K}_{a}) \).
Proof
Let us begin with (i). Consider a perfect selection of \( K_{p}\) elements from a set \(( \mathcal {N}_{p},\mathcal {K}_{p}) \), with \(( \mathcal {N}_{p},\mathcal {K}_{p}) \in \mathscr {S}^{P}\). Clearly the best ranked \(K_{p}\) elements are in \(\mathcal {K}_{p}\), the rest in \(\mathcal {N}_{p}\backslash \mathcal {K}_{p}\). Now add to the set and to the selection an element \(z_{1}\) such that \(z_{1}<x\) for all \(x\in \mathcal {K} _{p} \). By Axiom 2.i, \(( \mathcal {N}_{p},\mathcal {\cup }\{ z_{1}\} ,\mathcal {K}_{p}\mathcal {\cup } \{ z_{1}\} ) \succsim _{\mathfrak {M}} ( \mathcal {N}_{p},\mathcal {K}_{p}) \). Next return to the selection \(( \mathcal {N}_{p},\mathcal {K} _{p}) \), and add, again to the set and to the selection, an element \( z_{2}\) such that \(x<z_{2}\) for all \(x\in \mathcal {K}_{p}\), and \(z_{2}<y\) for all \(y\in \mathcal {N}_{p}\backslash \mathcal {K}_{p}\) (that is \(z_{2}\) is worse ranked than every selected element, but better ranked than every nonselected element). By Axiom 3.ii, \(( \mathcal {N} _{p},\mathcal {K}_{p}) \succsim _{\mathfrak {M}} ( \mathcal {N}_{p}, \mathcal {\cup } \{ z_{2} \} ,\mathcal {K}_{p}\mathcal {\cup } \{ z_{2} \} ) \). But now notice that both new selections select only the best ranked \(K_{p}+1\) elements: hence, \(m ( \mathcal {N}_{p},\mathcal { \cup }\{ z_{1}\} ,\mathcal {K}_{p}\mathcal {\cup }\{ z_{1}\} ) =m( \mathcal {N}_{p},\mathcal {\cup }\{ z_{2}\} ,\mathcal {K}_{p}\mathcal {\cup }\{ z_{2}\} ) \) . By transitivity, these both equal \(m ( \mathcal {N}_{p},\mathcal {K} _{p}) \). The process can be repeated to show that all perfect selections are equally orderly. The proof of (ii) is identical. Consider (iii) next. Take a selection \(( \mathcal {N},\mathcal {K}) \) with \( ( \mathcal {N},\mathcal {K}) \in \mathscr {S}\backslash \mathscr {S} ^{P}\backslash \mathscr {S}^{A}\). Let N and K be the number of elements in \(\mathcal {N}\) and \(\mathcal {K}\). Now let \(x_{1}\in \mathcal {K}\) be the best ranked selected element such that there is a nonselected element \(y\in \mathcal {N}\backslash \mathcal {K}\) such that \(y<x_{1}\). Because the selection is not perfect, it is possible to find such a \(x_{1}\). Next let \( y_{1}\in \mathcal {N}\backslash \mathcal {K}\) be the best ranked nonselected element such that there a selected element \(x\in \mathcal {K}\) such that \( y_{1}<x\). Again, because the selection is not antiperfect, it is possible to find such a \(y_{1}\). Now by Axiom 1, \(( \mathcal {N },\mathcal {K\cup }\{ x_{1}\} \backslash \{ y_{1}\} ) \succ _{\mathfrak {M}}( \mathcal {N},\mathcal {K}) \). If the selection \(( \mathcal {N},\mathcal {K\cup }\{ x_{1}\} \backslash \{ y_{1}\} ) \) is perfect, then we are done. If not, we can repeat, until a perfect selection in reached. This happens in at most \(\min \{ K,NK\} \) steps, and establishes that \( ( \mathcal {N}_{p},\mathcal {K}_{p}) \succ _{\mathfrak {M}}( \mathcal {N},\mathcal {K}) \). The demonstration that \(( \mathcal {N}, \mathcal {K}) \succ _{\mathfrak {M}}( \mathcal {N}_{a},\mathcal {K} _{a}) \) is identical. \(\square \)
In words, Proposition 1 says that all perfect selections are equally orderly, and similarly, that all antiperfect selections are equally orderly. The third statement asserts that every perfect selection is strictly more orderly than every nonperfect selection, and every antiperfect selection is strictly less orderly than every nonantiperfect selection. This is entirely reasonable when the selection is from the same set. Thus, selecting the best ten from a set of one hundred elements is clearly more orderly than selecting the best nine and the eleventh. However, as we want to extend the range of selected sets to be compared, it is possible to think of extreme situations when the size of the sets and of the selections is different, and when the argument that a perfect selection is more orderly is less clearcut. Consider, for example, the following case: is selecting the better of the two elements of a set unquestionably more orderly than selecting the best 25 and the 27th ranked out of a set with 10,000 elements? Proposition 1 answers unambiguously yes, but someone might argue in favour of the opposite, on the grounds that the former is more likely than the latter to have been determined by other criteria, which by chance happen to coincide with the rank, whereas the selection of the first 25 and the 27th elements out of a very large set would almost surely be seen as the consequence of a determinate intention to use the rank as the criterion for selection, with the 26th element being excluded due to some other criterion.
3 An index of “orderliness”
Definition 3
An index of orderliness is a function \(M:\mathfrak {S}\longrightarrow \left[ 1 ,1\right] \) such that, given any two selected sets \(\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) ,\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \in \mathscr {S}\), \(\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \succsim _{\mathfrak {M}}\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \) if and only if \(M\left( m\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \right) \geqslant M\left( m\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \right) \).
Note that we require the index to takes values ranging from \(1\) to 1, in the cases of an antiperfect and a perfect selection, respectively. Of course, a different normalisation would allow us to choose any two end points. However, in analogy with the measures of correlation, having 1 indicating perfect orderliness, and \(1\) its antithesis, where the selection perversely chooses the worst elements, does seem the natural choice. In analogy with the theory of consumers’ preferences, existence of the index of orderliness is ensured by the additional requirement that the orderliness relation is continuous. The formal proof is identical with the proof of the existence of a utility function; as with consumer theory, a lexicographic relation illustrates well the need of continuity to represent orderliness through an index. Consider the following way of comparing the orderliness of any two selected sets: take first the best ranked element in each: if only one selection includes it, then that selection is strictly more orderly than the other. Otherwise, look next at the second best ranked element, and again, if it is included only in one selection, this is the more orderly one. Again, if they are both or neither selected, go to the next ranked element and so on. If one selection “runs out of elements” before the other, then it is less orderly. In analogy with consumer preferences, this relation is reflexive, complete and transitive, but it is not continuous, and so it cannot be represented by an index of orderliness.^{10}
To ensure existence of a orderliness index representing a relation \(\mathfrak {M}\), therefore we require that the orderliness relation be continuous.^{11}
Proposition 2
A relation represented by the index \(M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }\) satisfies Axioms 1–3.
Proof
Consider finally Axiom 1, which is straightforward: the swap between an element in the selection and an element not in the selection changes neither K nor N. It only changes r, and so clearly the index M increases if r decreases, that is if a better ranked element takes the place of a worse ranked one in the selection. This establishes that the index (2) satisfies all the Axioms and completes the proof. \(\square \)
Summary of the proof of Proposition 2
\(\forall y\in \mathcal {K}, z \notin \mathcal {N}, zRy\)  \(\forall y\in \mathcal {K},z \notin \mathcal {N},yRz\) 

Axiom 2.i: \(z \in \mathcal {K}\).  Axiom 3.ii: \(z \in \mathcal {K}\). 
r increases by \(\rho _{\mathcal {N}} \left( z \right) +K\),  r increases by \(\rho _{\mathcal {N}} \left( z \right) \), 
K increases by 1.  K increases by 1. 
The orderliness is higher:  The orderliness is lower: 
\(\frac{N+2 \frac{2}{K+1}\left( r+\rho _{\mathcal {N}}\left( z \right) + K\right) }{NK}>M\left( \mathcal {N},\mathcal {K}\right) \)  \(\frac{N+2  \frac{2}{K+1}\left( r+\rho _{\mathcal {N}}\left( z\right) \right) }{NK }<M\left( \mathcal {N},\mathcal {K}\right) \) 
Axiom 2.ii: \(z \notin \mathcal {K}\).  Axiom 3.i: \(z \notin \mathcal {K}\). 
r increases by K,  r does not change, 
K does not change.  K does not change. 
The orderliness is lower:  The orderliness is higher: 
\(\frac{N  \frac{2}{K}r}{N+1K }<M\left( \mathcal {N},\mathcal {K}\right) \)  \(\frac{N+2  \frac{2}{K}r}{ N+1K }>M\left( \mathcal {N},\mathcal {K}\right) \) 
The analogous of Proposition 2 for the index (3) holds, but its proof is essentially identical to the above proof and is omitted.
Proposition 3
A relation represented by the index \(M_{1}{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }\) satisfies Axioms 1–3.
The restriction required to ensure characterisation is remarkably simple, as the rest of the paper shows. We replace Axiom 1 with the following.
Axiom 4
(Mirror invariance) For all \(\left( \mathcal {N} ,\mathcal {K}\right) \in \mathscr {S} \) and \(x\notin \mathcal {N}\), \(\big ( \mathcal {N}\cup \left\{ x\right\} , \mathcal {K}\cup \left\{ x\right\} \big ) \succsim _{\mathfrak {M}}\left( \mathcal {N},\mathcal {K} \right) \) if and only if \(\left( \mathcal {N},\mathcal {N}\backslash \mathcal {K}\right) \succsim _{\mathfrak {M}}\left( \mathcal {N}\cup \left\{ x\right\} ,\mathcal {N}\backslash \mathcal {K}\right) \).
We can now establish the main result of the paper.
Theorem 1
A reflexive transitive, complete and continuous orderliness relation \({\mathfrak {M}}\) satisfies Axioms 2–4 if and only if it can be represented by a monotonic transformation of the index of orderliness \(M{( m ( \mathcal {N},\mathcal {K}) ) }\), given in (2).
Lemma 1
An index of orderliness represents a relation \(\mathfrak {M}\) satisfying Axiom 4 if and only if it is a decreasing function of the sum of the ranks of the selected elements.
Proof
Consider a set \(\left( \mathcal {N},\mathcal {K}\right) \). We show, first, that the index (7), which is decreasing in the sum of the ranks of the selected elements, does indeed satisfy Axiom 4, and, second, that an index which satisfies Axiom 4 must be decreasing in the sum of the ranks of the selected elements.
Next we show that if an orderliness relationship satisfies Axiom 4, then it must be decreasing in the sum of the ranks of the selected elements. And thus there is a monotonic transformation such that it equals index (7).
We proceed from a selected set \(\left( \mathcal {N},\mathcal {K}\right) \), to which we add a new element \(x\notin \mathcal {N}\), which is selected, and then remove a selected element of \(\mathcal {N}\), \(y\in \mathcal {K}\). Thus we start from set \(\left( \mathcal {N},\mathcal {K}\right) \) and arrive at a “final” set \(\left( \mathcal {N} \cup \left\{ x\right\} \backslash \left\{ y\right\} ,\mathcal {K}\cup \left\{ x\right\} \backslash \left\{ y\right\} \right) \), which has the same N and the same K. Suppose that this makes the new set more orderly. We show that the change in the sum of the ranks of the selected elements from the “initial” set and the “final” set has the opposite sign of the corresponding change in their mirror images. Recall that, since this orderliness relation satisfies Axiom 4, then the mirror image of the “new” set is more orderly than the mirror image of the original set \(\left( \mathcal {N},\mathcal {N}\backslash \mathcal {K}\right) \), and so the index is monotonic in the sum of the ranks of the selected elements. Moreover, because it must take values \(1\) and 1 at antiperfect and perfect selections respectively, then it must be monotonically decreasing, and this proves the Lemma.
This Lemma implies that a change that decreases the sum of the ranks of the selected elements makes the set more orderly would make the mirror image of the selected set less orderly, as required by Axiom 4.
We can now prove the main theorem, by showing that every index of orderliness representing a relation \(\mathfrak {M}\) satisfying Axioms 2 and 3 which is a decreasing function of the sum of the ranks of the selected elements is a strictly monotonic transformation of \(M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }\).
Proof of Theorem 1

When the selection is \(\mathcal {K}=\left\{ 1\right\} \), adding a new element to \(\mathcal {N} \) can result in a selected set with any of the following images: \(\left\{ 1,2\right\} \) (Axiom 2.i or Axiom 3.i), \(\left\{ 2\right\} \) (Axiom 2 .ii), \(\left\{ 1\right\} \) (Axiom 3.i) and \(\left\{ 1,3\right\} \) (Axiom 3.ii). Of these \(\left\{ 1,2\right\} \) and \(\left\{ 1\right\} \) are perfect, and so it must be \(a_{2,3} 3b_{2,3} =1\) and \(a_{1,3} b_{1,3} =1\): both of these hold. Conversely, \(\left\{ 2\right\} \) and \(\left\{ 1,3\right\} \) are not perfect, and so \(a_{1,3} 2b_{1,3} \in \left( 1,1\right) \) and \(a_{2,3} 4b_{2,3} \in \left( 1,1\right) \) which hold.

Similarly, when the selection is \(\mathcal {K}=\left\{ 2\right\} \), the add operation can result in any of the following new selected sets: \(\left\{ 1,3\right\} \) (Axiom 2.i) \(\left\{ 2,3\right\} \) (again Axiom 2.i and also Axiom 3 .ii), \(\left\{ 3\right\} \) (Axiom 2.ii), \(\left\{ 2\right\} \) (Axiom 3.i). Selection \(\left\{ 2,3\right\} \) is antiperfect, and so \(a_{2,3} 5b_{2,3} \) must equal \(1\), and selection is also antiperfect \(\left\{ 3\right\} \), which requires \(a_{1,3} 3b_{1,3} =1\). The remaining selections are not antiperfect, and this requires: \(a_{2,3} 4b_{2,3} \in \left( 1,1\right) \), and \(a_{1,3} 2b_{1,3} \in \left( 1,1\right) \). All these hold when \(a_{K,N} \) and \(b_{K,N} \) are given by (18) and (19).
Obviously any strictly monotonic transformation of the index (2) would represent a relation satisfying Axioms 2–4. The functional form in (2), which is linear in the sum of ranks, has the natural property of being 0, in expectation, if the selection is completely random: to see this, note that the expected rank of K random draws is \(\frac{K(N+1)}{2}\). Substitute this in (2), to obtain 0.
Theorem 1 is the main result of the paper: in words, it shows that the index of orderliness (2) satisfies the dominance Axioms 2 and 3, and the mirror invariance Axiom 4, and, conversely, that any index which satisfies the dominance Axioms 2 and 3, and the mirror invariance Axiom 4, is a monotonic transformation of the index of orderliness (2). Only index (2) satisfies all three axioms, and it is the only one (up to monotonic transformations) to do so: Axioms 2, 3 and 4 together characterise the index of orderliness (2).
The structure of the proof of Theorem 1 allows to replace the mirror invariance Axiom with an equivalent one, which gives a different interpretation to the restriction imposed on the orderliness relation to obtain characterisation.
Axiom 5
(Position Irrelevance) For every \(\left( \mathcal {N} ,\mathcal {K}\right) \in \mathscr {S} \) and for every \(y_{1}\in \mathcal {N} \backslash \mathcal {K}\) and \(x_{1}\in \mathcal {K}\), satisfying \(\rho \left( x_{1}\right) =\rho \left( y_{1}\right) +a\), and any \(y_{2}\in \mathcal {N} \backslash \mathcal {K\cup }\left\{ x_{1}\right\} \) and \(x_{2}\in \mathcal {K\cup }\left\{ y_{1}\right\} \), satisfying \(\rho \left( x_{2}\right) =\rho \left( y_{2}\right) a\), where a is any integer such that \(\rho \left( y_{1}\right) +a, \rho \left( y_{2}\right) a\in \left\{ 1,\ldots ,N\right\} \), then \(\left( \mathcal {N},\mathcal {K}\right) \sim _{\mathfrak {M}}\left( \mathcal {N},\mathcal {K\cup }\left\{ y_{1} ,y_{2}\right\} \backslash \left\{ x_{1},x_{2}\right\} \right) \).
In words, Axiom 5 considers two subsequent “swaps” between a selected and a nonselected element. The element newly included in the selection and the one removed differ in rank by a. The second swap turns the intermediate selection obtained with the first swap into the final one, by selecting an element not selected and removing another element from the (intermediate) selection which are \(a\) ranks apart. Thus, for example, if the first swap selects the (originally nonselected) 12th ranked element and deselects the 19th ranked; their difference in rank is \(7\). Suppose the second swap selects the (originally nonselected) 56th ranked element and deselects the 49th ranked; their difference in rank is 7.^{12} Then, by Axiom 5, the initial and the final selection are equally orderly. In shorter, looser words, the effect of a change in the selection depends only on the extent of the change, not on whether it affects the upper or the lower tail of the distribution.^{13} We note that the relative importance of the position in the ranking is at the core of the analysis of distance between preferences in Can (2014). It is straightforward to establish the following.
Corollary 2
A reflexive transitive, complete and continuous orderliness relation \({\mathfrak {M}}\) satisfies Axioms 2, 3, and 5 if and only if it can be represented by a monotonic transformation of the index of orderliness \(M{\left( m \left( \mathcal {N},\mathcal {K}\right) \right) }\), given in (2).
Proof
This is simply a consequence of Lemma 1 together with the obvious observation that the sum of the ranks is left unchanged by a swap that satisfies Axiom 5. \(\square \)
Note that Theorem 1 and Corollary 2 together imply that Axioms 4, mirror invariance, and Axiom 5, position irrelevance, are equivalent: each holds for a relation if and only if the other does.
4 The index M and the KendallTau distance
The study of metrics on orders, initiated by Kendall (1938) has developed a measure of distance between two rankings of the elements of a given set \(\mathcal {N}\), the KendallTau distance (Kemeny 1959), recently extended to choice functions (Klamler 2008). This is obtained by counting the number of times the two rankings “switch” two elements \(x,y \in \mathcal {N}\). That is, if \(\rho _{1}\) and \(\rho _{2}\) are the two rankings, the distance between them is the number of pairs \(\left( x,y\right) \in \mathcal {N}\times \mathcal {N}\) such that \(\rho _{1}\left( x\right) >\rho _{2}\left( x\right) \) and \(\rho _{1}\left( y\right) <\rho _{2}\left( y\right) \). This count can then be normalised by the maximum possible number of switches.
This idea has been applied to voting mechanism, whereby a ranking is interpreted as a vote, and the outcome of an election as the aggregation of the rankings of different voters.^{14} We view the KendallTau distance from a different angle: note in the first place that the index of orderliness (2) applies to a broader range of situations, such as those where the selections being compared are from different sets. Secondly, the orderliness index (2) compares selections, rather than rankings, as the KendallTau distance does. Nevertheless, a selection does rank the elements of a set, albeit in a very coarse manner: the K selected elements are joint first, and the nonselected ones are joint \(\left( K+1\right) \)th. Recall that the KendallTau distance is defined when there are ties, and so we can measure the KendallTau distance between the ranking determined by a selection and a given ranking \(\rho \) of \(\mathcal {N}\). In this section we study the relation between this KendallTau distance and the index of orderliness (2), \(M{\left( \mathcal {N},\mathcal {K}\right) }\).
Proposition 4
\(M{\left( m \left( \mathcal {N} ,\mathcal {K}\right) \right) }=12\hat{\tau }{\left( \mathcal {N} ,\mathcal {K}\right) }\).
Proof
Thus, in addition to satisfying natural axioms, the index (2) coincides, in the situations where both can be applied, with an established measure of distance between rankings. This paper therefore provides a microfoundation of the KendallTau distance, which currently lacks one.
Another measure of correlation is the Kappa correlation. This applies to ranking with very few steps (and coincides with the selection when there are only two categories). This suggests a direction for future research in this area, namely the extension of the index of orderliness to situations where the selection is more nuanced than simply “in” and “out”. An example is a coach selection of the first team and of the second team for a sport tournament.
5 Example: orderliness in Italian universities
The orderliness index proposed here finds a natural application in the analysis of promotions in hierarchical organisations, where, at given intervals, individuals from the pool of potential candidates are assessed and some are promoted, some are not. The determinants of promotions may be stated formally or known implicitly: thus for example, academic promotions may be decided by criteria ranging from scientific productivity, to teaching performance, fundrising ability, seniority, or age; the relative importance of performance along the various criteria may of course vary from institution to institution and from discipline to discipline.
As explained in Checchi et al. (2014), academic careers in Italy are firmly channelled along narrowly defined research fields: every academic is allocated to one and only one of 371 scientific sectors (SSDs), changing sector is relatively unusual, and the members of the promotion panels in each scientific sector are chosen exclusively among academic post holders in that sector.
The dataset assembled by Checchi et al. (2014) allows us to rank all the candidates for promotion to associate professor in each scientific sector in the period from 1995 to 2011 according to two criteria, their record of publication in international journals^{15} and their age.
Figure 1 describes the outcome of this process. Each dot corresponds to one of 371 scientific sectors, except the smaller ones, in one 4year interval. The abscissa of a dot is the orderliness according to scientific productivity, its ordinate the orderliness according to age.^{16} Broad scientific areas are colour coded; we have also singled out economics and econometrics among the social science sectors. An analysis of the Italian university sector might use these indices as characteristics of the selection procedures used in promotion and appointments. A cursory visual analysis suggests that overall productivity matters more than age, and also that it matters more in STEM subjects.
6 Concluding remarks
Often an agent chooses a number of options from a larger set, the elements of which can be ranked in some objective or generally accepted manner. We propose a way to assess how close this agent’s selection is to the order of the set. Aside from its intrinsic interest in the examples given in the introduction, availability of this measure might address the need reported in the medical literature for an objective evaluation of clinical services (Iverson 1998; Bickman 2012), or help the study of aspects of social mobility, such as the importance of a person’s family position in her access to leadership positions in society.^{17} In this paper, we require that the comparison between selections satisfies some natural dominance requirements (Axioms 2 and 3), and the requirement that the mirror image of a change that nears a selection to the ranking must push the mirror image of that selection away from the ranking (Axiom 4). These three simple axioms prove very strong, in that they identify a unique index which provides an unambiguous comparison of the relative closeness of any selection to the order of the set from which it is taken. This index has a very simple expression, which depends only on the sum of the ranks of the selected elements, and the number of elements in the set and in the selection. The paper ends with a specific example of the potential applications of our index to the Italian university system.
Footnotes
 1.
An example from recent implementation of policy which some readers will be familiar with is the extent by which bibliometric criteria should be used in the evaluation of university research departments. Unlike in Italy, the UK funding body was persuaded to allow panels not to adhere strictly to bibliometric measures of departmental output, but allow the latitude afforded by peer review. We reprise this theme in Sect. 5.
 2.
As, for instance, the 1991 auctions for the 16 regional television franchises in the UK, when only half the franchises were awarded to the highest bidder; see Cabizza and De Fraja (1998), especially Table 1, pp 11–12.
 3.
Selection is of course a broader concept than full ordering: choosing the alltime best 20 in a list of 1000 footballers is different and less demanding than naming the best 20. The relation between selection and ordering is explored in Sect. 4.
 4.
According to our paper, it turns out that these four selections are equally orderly.
 5.
We therefore rule out ties between elements, consideration of which is important for practical purposes, but not straightforward. We remark briefly in Sect. 5 below on possible approaches when some elements of the set are ranked equally, otherwise we restrict our attention to antisymmetric relations.
 6.
We use the terminology “best” and “worst” ranked element, rather than highest and lowest, given the potential linguistic ambiguity due to the lowest number being attached to the highest ranked element.
 7.
Selecting members of a set is related but different from the administration of a test. In the former, the number of available places is usually fixed, or at least within a given range; whereas the important feature of a test, such as a school leaving exam, or the rejection of potentially faulty items from a production line, is the minimisation of errors, possibly weighted by the relative importance of type I and type II errors. Loosely speaking, one can think of selection according to a rank and testing as inverse operations: in the former, the number of slots is fixed, and the distribution in the population of the metric on which the ranks are based determines the threshold for selection; in the latter, the pass/fail threshold is fixed, and the distribution determines the number of successful elements.
 8.
One could make an analogy with the Dalton–Pigou principle (Dalton 1920, p 351); a transfer of a resource (being selected in our case, or income in Dalton’s) from a worse ranked/richer to a better ranked/poorer element/person, so long as that transfer does not reverse the ranking of the two, will result in greater orderliness/equity.
 9.
Note that the statement of Axioms 2 and 3 requires that x can be measured against the existing elements of \(\mathcal {N}\). Thus if \(\mathcal {N}\) is the set of English test cricketers, x is the record of a newly eligible player; if \(\mathcal {N}\) is the set of Italian chemistry professors, x is the citation count of a newly appointed one.
 10.
The relation given in the text can be formalised as follows.
Given any two selected sets \(\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) ,\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \in \mathscr {S} \), take \(x_{1}\in \mathcal {N}_{A}\) with \(\rho \left( x_{1}\right) =1\) and \(y_{1} \in \mathcal {N}_{B}\) with \(\rho \left( y_{1}\right) =1\). If \(x_{1} \in \mathcal {K}_{A}\) and \(y_{1}\notin \mathcal {K}_{B}\) then \(\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \succ _{\mathfrak {M}}\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \), and vice versa if \(x_{1}\notin \mathcal {K}_{A}\) and \(y_{1}\in \mathcal {K}_{B}\) then \(\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \succ _{\mathfrak {M}}\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \). Otherwise, that is if either (\(x_{1}\in \mathcal {K}_{A}\) and \(y_{1}\in \mathcal {K}_{B}\)) or (\(x_{1} \notin \mathcal {K}_{A}\) and \(y_{1}\notin \mathcal {K}_{B}\)), then take \(x_{2} \in \mathcal {N}_{A}\) with \(\rho \left( x_{2}\right) =2\) and \(y_{2} \in \mathcal {N}_{B}\) with \(\rho \left( y_{2}\right) =2\). If \(x_{2} \in \mathcal {K}_{A}\) and \(y_{2}\notin \mathcal {K}_{B}\) then \(\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \succ _{\mathfrak {M}}\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \), and vice versa if \(x_{2}\notin \mathcal {K}_{A}\) and \(y_{2}\in \mathcal {K}_{B}\) then \(\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \succ _{\mathfrak {M}}\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \).
For \(i\geqslant 2\): if (\(x_{i}\in \mathcal {K}_{A}\) and \(y_{i}\in \mathcal {K}_{B}\)) or if (\(x_{i} \notin \mathcal {K}_{A}\) and \(y_{i}\notin \mathcal {K}_{B}\)), then: if \(K_{A}=i\) and \(K_{B}=i\), then \(\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \sim _{\mathfrak {M}}\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \); if \(K_{A}>i\) and \(K_{B}=i\), then \(\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \succ _{\mathfrak {M}}\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \), and vice versa if \(K_{A}=i\) and \(K_{B}>i\), then \(\left( \mathcal {N} _{B},\mathcal {K}_{B}\right) \succ _{\mathfrak {M}}\left( \mathcal {N} _{A},\mathcal {K}_{A}\right) \); if \(K_{A}>i\) and \(K_{B}>i\), then take \(x_{i+1}\in \mathcal {N}_{A}\) with \(\rho ( x_{i+1}) =i+1\) and \(y_{i+1}\in \mathcal {N}_{B}\) with \(\rho ( y_{i+1}) =i+1\). If \(x_{i+1}\in \mathcal {K}_{A}\) and \(y_{i+1}\notin \mathcal {K}_{B}\) then \(\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \succ _{\mathfrak {M}}\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \), and vice versa if \(x_{i+1}\notin \mathcal {K}_{A}\) and \(y_{i+1}\in \mathcal {K}_{B}\) then \(\left( \mathcal {N}_{B},\mathcal {K}_{B}\right) \succ _{\mathfrak {M}}\left( \mathcal {N}_{A},\mathcal {K}_{A}\right) \).
 11.
That is, the inverse image \(\mathfrak {M}\) of any open subset of \(\mathfrak {S}\) is itself open. Given a set \(\mathfrak {s} \subseteq \mathfrak {S}\), its inverse image is the set \(\mathfrak {M}^{1} \left( \mathfrak {s} \right) = \left\{ s \in \mathfrak {S}  \exists s_{1} \in \mathfrak {s}: s\mathfrak {M} s_{1} \right\} \). Because the cardinality of the set \(\mathfrak {S}\) is the same as that of the set of real numbers (Lucas 1990, pp 134–135), open sets in \(\mathfrak {S}\) are those that are put in correspondence with open intervals in \(\mathbb {R}\) by a surjection.
 12.
The element swapped in the second swap need not be different from the elements swapped in the first.
 13.
A simple example may illustrate this idea: in some sports, a team’s success is determined by the performance of its best athletes. This is typically the case, for example, in “Grand Tour” cycling, where a team’s objective is for the team leader to win the race. In rowing (and in team pursuit cycling), on the other hand, everyone must push at the same rate, and the team’s result is heavily influenced by the performance of its weakest members. Ranking of selections would, in these sports, violate Axiom 5. Conversely, relays in track and field (especially the \(4\times 400\)) approximately satisfy Axiom 5.
 14.
 15.
Details are again in Checchi et al. (2014): from the Thomson Reuters Web of Knowledge dataset, we construct for each candidate a score given by a combination of research output and impact: the former measured by the number of publications, and the latter by the individual hindex.
 16.
Occasionally, the rankings we constructed in this way display ties. The analysis of this paper applies to antisymmetric relations on the set \(\mathcal {N}\), and thus it excludes ties at the outset. To break the ties in the construction of the orderliness index displayed in Fig. 1, we have followed a randomisation approach, by bootstrapping the orderliness index (2) over many repetitions of the procedure, whenever a scientific sector’s ranking has ties.
 17.
Orderliness is at the basis of the popular book and film Moneyball (Lewis 2004), which tell the story of a baseball team with relatively scarce financial resources which was able systematically to outperform its much wealthier rivals by basing all its selections on the players’ ranks; Hakes and Sauer (2006) confirm econometrically the book’s intuition. Similarly, “artists and repertoire” talent spotters are being replaced in the music industry by detailed analyses of big data harvested from social media (Mukerji 2015). Anderson and Sally (2013) argue that the nature of the sport makes the method less applicable to soccer.
References
 Anderson C, Sally D (2013) The numbers game: why everything you know about football is wrong. Penguin Books, LondonGoogle Scholar
 Barberà S, Bossert W, Pattanaik PK (2004) Ranking sets of objects. In: Barberà S, Hammond PJ, Seidl C (eds) Handbook of utility theory. Springer, Boston, pp 893–977CrossRefGoogle Scholar
 Betzler N, Dorn B (2010) Towards a dichotomy for the possible winner problem in elections based on scoring rules. J Comput Syst Sci 76:812–836CrossRefGoogle Scholar
 Bickman L (2012) Why can’t mental health services be more like modern baseball? Adm Policy Ment Health Ment Health Serv Res 39:1–2CrossRefGoogle Scholar
 Cabizza MM, De Fraja G (1998) Quality considerations in auctions for television franchises. Inf Econ Policy Spec Issue Media Multimed Econ 10:9–22Google Scholar
 Can B (2014) Weighted distances between preferences. J Math Econ 51:109–115CrossRefGoogle Scholar
 Checchi D, De Fraja G, Verzillo S (2014) Publish or perish? An analysis of the academic job market in Italy, 1991–2012. In: Discussion Paper 10084. CEPR Discussion Paper, LondonGoogle Scholar
 Cowell FA, Kuga K (1981) Inequality measurement: an axiomatic approach. Eur Econ Rev 15:287–305CrossRefGoogle Scholar
 Dalton H (1920) The measurement of the inequality of incomes. Econ J 30:348–461CrossRefGoogle Scholar
 Davenport A, Kalagnanam J (2004) A computational study of the Kemeny rule for preference aggregation. In: Proceedings of the 19th national conference on artifical intelligence, San Jose, CaliforniaGoogle Scholar
 Gärdenfors P (1976) Manipulation of social choice functions. J Econ Theory 13:217–228CrossRefGoogle Scholar
 Hakes JK, Sauer RD (2006) An economic evaluation of the moneyball hypothesis. J Econ Perspect 20(3):173–185CrossRefGoogle Scholar
 Iverson DJ (1998) Meritocracy in graduate medical education? Some suggestions for creating a report card. Acad Med 73:1223–1225CrossRefGoogle Scholar
 Kemeny JG (1959) Mathematics without numbers. Daedalus 88:577–591Google Scholar
 Kendall MG (1938) A new measure of rank correlation. Biometrika 30:81–93CrossRefGoogle Scholar
 Klamler C (2008) A distance measure for choice functions. Soc Choice Welf 30:419–425CrossRefGoogle Scholar
 Lewis M (2004) Moneyball: the art of winning an unfair game. Norton, New YorkGoogle Scholar
 Lucas JF (1990) Introduction to abstract mathematics, 2nd edn. Ardsley House, LanhamGoogle Scholar
 Mukerji P (2015) Moneyballing music. MusicTank Publishing, University of Westminster, LondonGoogle Scholar
 Rubinstein A, Salant Y (2006) A model of choice from lists. Theor Econ 1:3–17Google Scholar
 Taylor WL (1964) Correcting the average rank correlation coefficient for ties in rankings. J Am Stat Assoc 59:872–876CrossRefGoogle Scholar
 Sen A (1976) Poverty: an ordinal approach to measurement. Econometrica 44:219–231CrossRefGoogle Scholar
 Young P (1995) Optimal voting rules. J Econ Perspect 9:51–64CrossRefGoogle Scholar
Copyright information
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.