Evaluating competing theories via a common language of qualitative verdicts


Kuhn (The essential tension—Selected studies in scientific tradition and change, 1977) claimed that several algorithms can be defended to select the best theory based on epistemic values such as simplicity, accuracy, and fruitfulness. In a recent paper, Okasha (Mind 129(477):83–115, 2011) argued that no theory choice algorithm exists which satisfies a set of intuitively compelling conditions that Arrow (Social choice and individual values, 1963) had proposed for a consistent aggregation of individual preference orderings. In this paper, we put forward a solution to avoid this impossibility result. Based on previous work by Gaertner and Xu (Mathematical Social Sciences 63:193–196, 2012), we suggest to view the theory choice problem in a cardinal context and to use a general scoring function defined over a set of qualitative verdicts for every epistemic value. This aggregation method yields a complete and transitive ranking and the rule satisfies all Arrovian conditions appropriately reformulated within a cardinal setting. We also propose methods that capture the aggregation across different scientists.

This is a preview of subscription content, access via your institution.


  1. 1.

    We take it to be the case that Okasha (2011) as well as the replies in the literature, which will be discussed below, share this focus. It is worthwhile to note that thereby Kuhn’s discussion of methodological incommensurability is left aside.

  2. 2.

    Note that Weber (2011) offers an alternative interpretation of Kuhn. He views Kuhn as a social epistemologist (ibid., 3) who treats the scientific community level as the relevant decision making entity (ibid., 7). Accordingly, he would reject our two step approach to the choice problem.

  3. 3.

    We will say more about the details of this scale. For the moment, the reader should not be irritated by the facts that we specify an exact number of grades and choose a particular formulation of the qualitative verdicts.

  4. 4.

    Let us add two further clarifications to our motivation for a move from the ordinal to the cardinal world. First, note that the informational content of the assessment context (i.e. the broad epistemic projects and its features) need to be distinct from the epistemic values. If this informational content were identical to the Kuhnian epistemic values or could be rephrased in terms of additional epistemic values, then, given our problem set-up, it would only provide ordinal information and, hence, we would not be able to justify the move to a cardinal scale in this way. Secondly, let us emphasize that we do not presuppose that the epistemic values directly provide cardinal information. Rather, we assume an additional and mentally challenging step by the scientist which involves the careful consideration of the assessment context of a theory choice problem.

  5. 5.

    See Pivato (2014, p. 50) for a similar discussion of the possibility to impose cardinality on a ranking of alternatives.

  6. 6.

    Notice that we do not treat the qualitative verdict ‘insufficient’ as an eliminative verdict in the sense that whenever with respect to one epistemic value an alternative gets an ‘insufficient’ verdict, this alternative is eliminated from the choice set. The reason for this is that we, in line with Kuhn, do not share the intuition that one of the epistemic values should be treated as a killer criterion. Accordingly, even if a theory is insufficiently simple, let us say, but receives the best qualitative verdicts with respect to all other criteria such that the aggregate rank score is the highest of all alternatives, the low grade in terms of simplicity has to be seen in relation to the high grades obtained from the other criteria.

  7. 7.

    In general, the required minimal level of graininess depends on the particular theory choice problem at hand. However, if one criterion (i) ranks all alternative theories in a strict order, then due to \(xP_i y\leftrightarrow s_i (x)>s_i (y)\), the minimal level of graininess is the number of alternative theories.

  8. 8.

    This E-based scoring function can be characterised in a fairly simple way. Please see the Appendix for a formal characterisation.

  9. 9.

    Our approach can allow different weights of the epistemic values. A natural way to account for this difference is to divide a criterion (e.g. ‘fruitfulness’) into two sub-criteria (e.g. ‘fruitfulness with respect to the discipline’ and ‘fruitfulness with respect to neighbouring disciplines’). In this way, the initial criterion gets, quite naturally, a higher weight in our summation procedure.

  10. 10.

    A discussion of the reformulated version of the independence requirement in the context of the utilitarian rule can be found in Gaertner (2013, pp. 125–126).

  11. 11.

    See Pivato (2014) for an alternative axiomatic characterisation of this scoring function.


  1. Arrow, K. J. 1963 [1951]. Social choice and individual values, 2nd edn. New York: John Wiley.

  2. Balinski, M., & Laraki, R. (2007). A theory of measuring, electing, and ranking. Proceedings of the National Academy of Sciences, 104, 8720–8725.

    Article  Google Scholar 

  3. Balinski, M., & Laraki, R. (2010). Majority Judgment: measuring, ranking and electing. Cambridge, MA: The MIT Press.

    Google Scholar 

  4. Brams, S. J., & Fishburn, P. C. (1983). Approval voting. Boston: Birkhäuser.

    Google Scholar 

  5. Gaertner, W. (2013). A primer in social choice theory. Oxford: Oxford University Press.

    Google Scholar 

  6. Gaertner, W., & Xu, Y. (2012). A general scoring rule. Mathematical Social Sciences, 63, 193–196.

  7. Hoyningen-Huene, P. (1993). Reconstructing scientific revolutions: Thomas S. Kuhn’s philosophy of science (Transl. by A. T. Levine). With a foreword by Thomas S. Kuhn. Chicago: University of Chicago Press.

  8. Kuhn, T. S. (1977). Objectivity, value judgment, and theory choice. In T. S. Kuhn (Ed.), The essential tension—Selected studies in scientific tradition and change (pp. 320–339). Chicago: The University of Chicago Press.

    Google Scholar 

  9. Kuhn, T. S. 1996 [1962]. The structure of scientific revolutions. Chicago: The University of Chicago Press.

  10. Morreau, M. (2014). Mr. Fit, Mr. simplicity and Mr. scope: From social choice to theory choice. Erkenntnis, 79, 1253–1268.

    Article  Google Scholar 

  11. Morreau, M. (2015). Theory choice and social choice: Kuhn vindicated. Mind, 124(493), 239–262.

    Article  Google Scholar 

  12. Okasha, S. (2011). Theory choice and social choice: Kuhn vs. Arrow. Mind, 129(477), 83–115.

    Article  Google Scholar 

  13. Pivato, M. (2014). Formal utilitarianism and range voting. Mathematical Social Sciences, 67, 50–56.

    Article  Google Scholar 

  14. Rizza, D. (2014). Arrow’s theorem and theory choice. Synthese, 191, 1847–1856.

    Article  Google Scholar 

  15. Sen, A. (1977). On weights and measures: Informational constraints in social welfare analysis. Econometrica, 45, 1539–1572.

    Article  Google Scholar 

  16. Sen, A. (1986). Social choice theory. In K. J. Arrow & M. D. Intriligator (Eds.), Handbook of mathematical economics (Vol. 3, pp. 1073–1181). Amsterdam: North-Holland.

    Google Scholar 

  17. Stegenga, J. (2015). Theory choice and social choice: Okasha versus Sen. Mind, 124(493), 263–277.

    Article  Google Scholar 

  18. Weber, M. (2011). Experimentation versus theory choice: A social-epistemological approach. In H. B. Schmidt, D. Sirtes, & M. Weber (Eds.), Collective epistemology (pp. 1–26). Frankfurt: Ontos Verlag.

    Google Scholar 

  19. Young, P. H. (1974). An axiomatization of Borda’s rule. Journal of Economic Theory, 9, 43–52.

    Article  Google Scholar 

Download references


We would like to thank an anonymous reviewer for helpful comments regarding the presentation of our overall argument. Furthermore, Claus Beisbart, Georg Brun, Kamilla Buchter, Gregory Fried, Stephan Güttinger, Paul Hoyingen-Huene, Jurgis Karpus, Simon Lohse, Alex Marcoci, James Nguyen, and Mantas Radzvilas provided fruitful feedback on earlier versions of this paper.

Conflict of interest

The authors declare that they have no conflict of interest.

Author information



Corresponding author

Correspondence to Nicolas Wüthrich.



To formally characterise the E-based scoring function, we need to introduce three concepts: (ij)-variance, a monotonicity condition, and the property of cancellation independence.Footnote 11

For any \(s,\, s'\in S\), any \(i, j \in N\), and any \(x,y \in X\), we say that s and \(s'\) are (ij)-variant with respect to (xy) if \(s_k(x)=s_k' (x)\) and \(s_k (y) =s_k' (y)\) for all \(k\in N\backslash \left\{ {i,j}\right\} \).

Let us now introduce two properties of an aggregation rule f.

Monotonicity (M). For all \(s \in S\) and all \(x, y\in X\), if \(s_i (x) \ge s_i (y)\) for all

\(i\in N\), then \(x\succsim y\) and if \(s_i (x) \ge s_i (y)\) for all \(i \in N\) and \(s_j (x)> s_j (y)\) for some \(j\in N\), then \(x \succ y\).

Condition M is a simple vector dominance condition. It requires that, in ranking two theories x and y, if the score assigned to x by each criterion \(\hbox {i} \in \hbox {N}\) is at least as great as the score assigned to y by the same criterion i, then x must be ranked at least as high as y by the evaluating scientist, and if in addition, some criterion assigns a higher score to x than to y, then x must be ranked higher than y.

Cancellation Independence (CI). For all \(s,\, s'\in S\), all \(x,y \in X\) and all \(i,j\in N\), if s and \(s'\) are (ij)-variant with respect to \((x,y),\, s_i (x)-s_i (y)=a,\, s_j (y)-s_j (x)=b, \,s'_{i}(x)=s_{i}(x),\, s'_{j}(y)=s_{j}(y),\, s'_{i}(y)=s_{i}(y)+\gamma \) and \(s'_{j}(x)=s_{j}(x)+\gamma \) where \(\gamma =min(a,b)\) when \(a\ge 0\) and \(b\ge 0\) and \(\gamma =max(a,b)\) when \(a<0\) and \(b<0\), then \(x\succsim y\leftrightarrow x \succsim ' y\), where \(\succsim =f(s)\) and \(\succsim ' =f(s')\).

Condition CI makes use of the fact that for any pair of alternatives, rank differences of opposite sign can be reduced without changing the aggregate outcome of the ranking procedure. This reduction procedure is performed in a stepwise fashion, starting with any two theories x and y, let us say, and picking any two criteria whose rank differences for x and y are of opposite sign. The “net” rank difference between x and y for this pair of criteria is determined. Then another criterion is picked whose rank difference for x and y is opposite in sign to the net rank difference of the first two criteria. The new net rank difference for x and y is calculated and the next criterion is picked whose rank difference again is opposite in sign to the just determined net rank difference with respect to x and y, if there is still one such criterion, and so on.

In Condition CI, vectors s and \(s'\) define scoring profiles that are aggregate-rank equivalent with respect to any pair of scientific theories. We call \(s'\) an s-reduced scoring profile. Condition CI therefore requires that f(s) and \(f(s')\) order any x and y in exactly the same way. Note that Condition CI makes an implicit assumption about an inter-criterion comparison of scores for which a common language is required.

Theorem. \(f=f_E\) if and only if f satisfies the properties of Monotonicity and Cancellation Independence.

A proof of this result can be found in Gaertner and Xu (2012).

The theorem above establishes, for each evaluating person, an ordering over the set of alternative theories if that person has consented to a common language.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Gaertner, W., Wüthrich, N. Evaluating competing theories via a common language of qualitative verdicts. Synthese 193, 3293–3309 (2016). https://doi.org/10.1007/s11229-015-0929-4

Download citation


  • Theory choice
  • Social choice theory
  • Scoring rules
  • Thomas S. Kuhn
  • Epistemic values