Evaluating competing theories via a common language of qualitative verdicts

Abstract

Kuhn (The essential tension—Selected studies in scientific tradition and change, 1977) claimed that several algorithms can be defended to select the best theory based on epistemic values such as simplicity, accuracy, and fruitfulness. In a recent paper, Okasha (Mind 129(477):83–115, 2011) argued that no theory choice algorithm exists which satisfies a set of intuitively compelling conditions that Arrow (Social choice and individual values, 1963) had proposed for a consistent aggregation of individual preference orderings. In this paper, we put forward a solution to avoid this impossibility result. Based on previous work by Gaertner and Xu (Mathematical Social Sciences 63:193–196, 2012), we suggest to view the theory choice problem in a cardinal context and to use a general scoring function defined over a set of qualitative verdicts for every epistemic value. This aggregation method yields a complete and transitive ranking and the rule satisfies all Arrovian conditions appropriately reformulated within a cardinal setting. We also propose methods that capture the aggregation across different scientists.

Appendix

To formally characterise the E-based scoring function, we need to introduce three concepts: (i, j)-variance, a monotonicity condition, and the property of cancellation independence.¹¹

For any $s,\, s'\in S$, any $i, j \in N$, and any $x,y \in X$, we say that s and $s'$ are (i, j)-variant with respect to (x, y) if $s_k(x)=s_k' (x)$ and $s_k (y) =s_k' (y)$ for all $k\in N\backslash \left\{ {i,j}\right\}$.

Let us now introduce two properties of an aggregation rule f.

Monotonicity (M). For all $s \in S$ and all $x, y\in X$, if $s_i (x) \ge s_i (y)$ for all

$i\in N$, then $x\succsim y$ and if $s_i (x) \ge s_i (y)$ for all $i \in N$ and $s_j (x)> s_j (y)$ for some $j\in N$, then $x \succ y$.

Condition M is a simple vector dominance condition. It requires that, in ranking two theories x and y, if the score assigned to x by each criterion $\hbox {i} \in \hbox {N}$ is at least as great as the score assigned to y by the same criterion i, then x must be ranked at least as high as y by the evaluating scientist, and if in addition, some criterion assigns a higher score to x than to y, then x must be ranked higher than y.

Cancellation Independence (CI). For all $s,\, s'\in S$, all $x,y \in X$ and all $i,j\in N$, if s and $s'$ are (i, j)-variant with respect to $(x,y),\, s_i (x)-s_i (y)=a,\, s_j (y)-s_j (x)=b, \,s'_{i}(x)=s_{i}(x),\, s'_{j}(y)=s_{j}(y),\, s'_{i}(y)=s_{i}(y)+\gamma$ and $s'_{j}(x)=s_{j}(x)+\gamma$ where $\gamma =min(a,b)$ when $a\ge 0$ and $b\ge 0$ and $\gamma =max(a,b)$ when $a<0$ and $b<0$, then $x\succsim y\leftrightarrow x \succsim ' y$, where $\succsim =f(s)$ and $\succsim ' =f(s')$.

Condition CI makes use of the fact that for any pair of alternatives, rank differences of opposite sign can be reduced without changing the aggregate outcome of the ranking procedure. This reduction procedure is performed in a stepwise fashion, starting with any two theories x and y, let us say, and picking any two criteria whose rank differences for x and y are of opposite sign. The “net” rank difference between x and y for this pair of criteria is determined. Then another criterion is picked whose rank difference for x and y is opposite in sign to the net rank difference of the first two criteria. The new net rank difference for x and y is calculated and the next criterion is picked whose rank difference again is opposite in sign to the just determined net rank difference with respect to x and y, if there is still one such criterion, and so on.

In Condition CI, vectors s and $s'$ define scoring profiles that are aggregate-rank equivalent with respect to any pair of scientific theories. We call $s'$ an s-reduced scoring profile. Condition CI therefore requires that f(s) and $f(s')$ order any x and y in exactly the same way. Note that Condition CI makes an implicit assumption about an inter-criterion comparison of scores for which a common language is required.

Theorem. $f=f_E$ if and only if f satisfies the properties of Monotonicity and Cancellation Independence.

A proof of this result can be found in Gaertner and Xu (2012).

The theorem above establishes, for each evaluating person, an ordering over the set of alternative theories if that person has consented to a common language.