Skip to main content
Log in

Statistical evaluation of voting rules

Social Choice and Welfare Aims and scope Submit manuscript

Cite this article


We generate synthetic elections using two sources of survey data, two spatial models, and two standard models from the voting literature, IAC and IC. For each election that we generate, we test whether each of 54 voting rules is (1) non-manipulable, and (2) efficient in the sense of maximizing summed utilities. We find that Hare and Condorcet–Hare are the most strategy-resistant non-dictatorial rules. Most rules have very similar efficiency scores, apart from a few poor performers such as random dictator, plurality and anti-plurality. Our results are highly robust across data-generating processes. In addition to presenting our numerical results, we explore analytically the effects of adding a Condorcet provision to a base rule and show that, for all but a few base rules, this modification cannot introduce a possibility of manipulation where none existed before. Our analysis provides support for the Condorcet–Hare rule, which has not been prominent in the literature.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others


  1. For summaries see Richelson (1978), Tideman (2006), and Felsenthal (2012).

  2. Other dimensions investigated include the frequency of electing the Condorcet winner, the frequency of avoiding the election of the Condorcet loser, the frequency of avoiding situations where monotonicity or participation failures might arise, etc. Some voting rules perform perfectly in some of these dimensions, but no voting rule performs perfectly in all of them.

  3. For example, take an election in which 20 voters have preferences \({\text {A}} \succ {\text {B}} \succ {\text {C}}\), 35 voters have preferences \( {\text {B}} \succ {\text {A}} \succ {\text {C}} \), and 45 voters have preferences \( {\text {C}} \succ {\text {B}} \succ {\text {A}} \). In plurality, the first group of voters can change the winner from C to B by voting for B.

  4. Green-Armytage (2014) defines compromising and burying strategies, drawing from a framework developed in unpublished work by Blake Cretney.

  5. For example, take an election in which 45 voters have preferences \({\text {A}} \succ {\text {B}} \succ {\text {C}}\), 40 voters have preferences \({\text {B}} \succ {\text {A}} \succ {\text {C}}\), and 15 voters have preferences \({\text {C}} \succ {\text {B}} \succ {\text {A}}\). In Borda, the first group of voters can change the winner from B to A by voting \({\text {A}} \succ {\text {C}} \succ {\text {B}}\). The same can be said of minimax, and of several other rules that are equivalent to minimax in the three-candidate case.

  6. See Green-Armytage (2014).

  7. Voters rate the candidates on a closed scale (e.g. 0–100), and the ratings are summed to determine the winner.

  8. One ballot is selected at random, and the candidate listed first on this ballot is elected.

  9. See, for example, Nitzan (1985), Smith (1999), and Aleskerov and Kurbanov (1999).

  10. See, for example, Chamberlin (1985), Lepelley and Mbih (1994), Kim and Roush (1996), Lepelley and Valognes (2003), Favardin and Lepelley (2006), Pritchard and Wilson (2007), and Green-Armytage (2014).

  11. Chamberlin (1985), Kim and Roush (1996), Lepelley and Valognes (2003), Pritchard and Wilson (2007) and Green-Armytage (2014) use the IC process to measure voting rules’ resistance to coalitional manipulation.

  12. Lepelley and Mbih (1994), Favardin et al. (2002), Lepelley and Valognes (2003), Favardin and Lepelley (2006), and Pritchard and Wilson (2007) use the IAC process to measure voting rules’ resistance to coalitional manipulation.

  13. Chamberlin (1985) and Green-Armytage (2014) use spatial models to measure voting rules’ resistance to coalitional manipulation.

  14. As noted e.g. by Tsetlin et al. (2003).

  15. For example, this measure is employed by Chamberlin (1985), Lepelley and Mbih (1994), Kim and Roush (1996), Lepelley and Valognes (2003), Favardin and Lepelley (2006), Pritchard and Wilson (2007), and Green-Armytage (2014), as we note in Sect. 3.1 above.

  16. For example, suppose that there is an election in which 49 voters have preferences \({\text {A}} \succ {\text {B}} \succ {\text {C}}\), 48 voters have preferences \({\text {B}} \succ {\text {A}} \succ {\text {C}}\), and 3 voters have preferences \({\text {C}} \succ {\text {B}} \succ {\text {A}}\). Sincere voting under plurality chooses A, but those who prefer B to A can succeed in electing B by voting for B. Sincere voting under the Borda count chooses B, but those who prefer A to B can succeed in electing A by voting \({\text {A}} \succ {\text {C}} \succ {\text {B}}\). Sincere voting under Hare chooses B, and neither the voters who prefer A to B nor the voters who prefer C to B can do anything to elect their preferred candidate. Thus, this election provides an instance in which plurality and Borda are manipulable while Hare is not.

  17. Descriptions of our algorithms, and the codes themselves, are available on request.

  18. See Borda (1784).

  19. This rule is also known as the alternative vote, instant runoff voting, and ranked choice voting. Thomas Hare proposed transferring votes from plurality losers as a refinement of the single transferable vote procedure of proportional representation. See Hoag and Hallett (1926, 162–95).

  20. See Baldwin (1926).

  21. See Coombs (1964). Some versions of Coombs include a provision to automatically elect a candidate who holds a majority of first choice votes, but we do not use this provision here.

  22. Black (1958) creates and advocates this rule.

  23. Green-Armytage (2011) examines four Condorcet–Hare hybrid rules (proposed separately by different voting theorists) that can occasionally yield different results when there are four or more candidates; these include ‘alternative Smith’ as defined in Tideman (2006). However, since they always yield the same result when there are three or fewer candidates, we can treat them as one rule for present purposes.

  24. As Nanson (1882) explains, a Condorcet winner always has an above-average Borda score. Therefore the Baldwin rule can never eliminate such a candidate, which means that a Condorcet winner must also be a Baldwin winner.

  25. See Brams and Fishburn (1978).

  26. Here, ‘average utility’ means the average of the individual voter’s utilities from the three candidates. There is no general agreement in the literature on what it means to sincerely ‘disapprove of’ a candidate, but the idea that it means liking the candidate less than average seems as straightforward as any.

  27. See Green-Armytage (2004).

  28. Black (1958, p. 175) develops the minimax method as a possible interpretation of Condorcet’s proposal. It has also been variously referred to as ‘Simpson’, ‘Simpson–Kramer’, ‘successive reversal’, and ‘maximin’.

  29. For discussion of these rules see Tideman (2006) pp. 182–90, 199–201, 217–223 and 228–232.

  30. See Nanson (1882). Like Baldwin, this rule is Condorcet-consistent, so ‘Condorcet–Nanson’ would be redundant.

  31. This probability depends on the data-generating processed used, the number of candidates and voters, etc. In most of the models that we employ here, it is above \(90~\%\); the two exceptions to this are our implementation of the IAC and IC models.

  32. A core equilibrium with respect to voting is a situation in which no group of voters can achieve a mutually preferable result by changing their votes in cooperation with one another.

  33. The question (translated from German) is phrased as, “Please tell me, again with the thermometer of \(+5\) to \(-5\), what you think of some political leaders. \(+5\) means that you think a lot of the politician; \(-5\) means that you think nothing at all of him. If you do not know a politician, you naturally do not need to grade him. What do you think of...” The data are available via

  34. The question is phrased as, “I’d like to get your feelings toward some of our political leaders and other people who are in the news these days. I’ll read the name of a person and I’d like you to rate that person using something we call the feeling thermometer. Ratings between \(50^{\circ }\) and \(100^{\circ }\) mean that you feel favorable and warm toward the person. Ratings between \(0^{\circ }\) and \(50^{\circ }\) mean that you don’t feel favorable toward the person and that you don’t care too much for that person. You would rate the person at the \(50^{\circ }\) mark if you don’t feel particularly warm or cold toward the person. If we come to a person whose name you don’t recognize, you don’t need to rate that person. Just tell me and we’ll move on to the next one.” The data are available via

  35. As a robustness check, we also ran simulations in which 99 respondents were randomly chosen (with replacement) from each possible three-candidate subset of each survey, thus creating synthetic elections of equal size to the other four data-generating processes described below. For the Politbaometer data, the resistance to strategy results from this alternative approach had a correlation of 0.9993 with our main approach, and the utilitarian efficiency results had a correlation of 0.9972. For the ANES data (averaged over ten repetitions to increase the number of elections from 847 to 8470), the analogous correlations were 0.9987 and 0.9848. Therefore, we consider this robustness check to be successful.

  36. In our IAC, IC, and spatial models, utilities are converted to sincere ratings on the allowed [0, 1] ballot range in a linear manner, such that the highest utility of any voter for any candidate is mapped to a rating of 1, and the lowest utility of any voter for any candidate is mapped to a rating of 0.

  37. Chamberlin and Cohen (1978) and Green-Armytage (2014) use similar models.

  38. We’ve also gathered R and E statistics with larger electorate sizes for selected voting rules, using each of our six data-generating processes, including the randomized version of the survey-based processes as described in footnote 34. Preliminary results suggest that increasing the electorate size from 99 has only a very modest impact on the results, but we lack the space here to develop an authoritative treatment of the subject.

  39. While increasing the dimensionality of the spatial model beyond this point continues to have an impact on manipulability scores, the marginal impact diminishes as the number of dimensions increases. Thus we choose \(S=1\) and \(S=8\) to represent the extreme case of a strictly one-dimensional space as well as an example of higher-dimensional spaces in general. Although it is true that any spatial example with more than \(C-1\) dimensions can be represented as a \((C-1)\)-dimensional example, when a generating process with random uncorrelated coordinates for candidates is used, increasing the dimensionality of the space beyond \(C-1\) has an impact on manipulability scores.

  40. This is consistent with Feld and Grofman (1992), who analyze a data set of private elections conducted using STV, and find that fewer than of all possible three-candidate subsets create majority-rule cycles.

  41. Green-Armytage (2011; 2014) shows another potentially important advantage: Hare is more likely than Condorcet–Hare to provide incentives for candidates to strategically exit races, which can have the effect of depressing the number of candidates, and reducing the competitiveness of elections. To see why this is the case, consider that if there is a Condorcet winner, and the voting rule used is Condorcet-consistent, the removal of any non-winning candidate from the ballot cannot change the outcome.

  42. It is worth noting that the IC model with smaller numbers of voters leads to R scores that correlate more closely to the other data-generating processes. For example, with 9 voters, the IC model has an average correlation of \(92.4~\%\) with the four most consonant data-generating processes. However, we present the results with 99 voters here with the idea that it would be inconsistent to suddenly shift our focus to such small elections.

  43. This result also holds when we measure utilitarian efficiency as the average sum of utilities from the winning candidate, rather than the share of trials in which the candidate who maximizes the sum of utilities is chosen. This shows that theorem 3.1 from Apesteguia et al. (2011) does not extend to all data-generating processes.

  44. For example, Chamberlin (1985), Lepelley and Valognes (2003), Favardin and Lepelley (2006), Tideman (2006), and Green-Armytage (2014) all agree on these points.

  45. For example, Weber (1978), Bordley (1983), and Merrill (1984) all agree on these points.

  46. The only examples that we are aware of are Tideman (2006) and Green-Armytage (2011), which provide measures of resistance to strategy, but not utilitarian efficiency.


  • Aleskerov F, Kurbanov E (1999) Degree of manipulability of social choice procedures. In: Alkan A (ed) Current trends in economics. Springer, Berlin

    Google Scholar 

  • Apesteguia J, Ballester M, Ferrer R (2011) On the justice of decision rules. Rev Econ Stud 78:1–16

    Article  Google Scholar 

  • Baldwin J (1926) The technique of the Nanson preferential majority system of election. Trans Proc R Soc Vic 39:42–52

    Google Scholar 

  • Berg S (1985) Paradox of voting under an urn model: the effect of homogeneity. Public Choice 47:377–387

    Article  Google Scholar 

  • Black D (1958) The theory of committees and elections. Cambridge University Press, London

  • Bordley R (1983) A pragmatic method for evaluating election schemes through simulation. Am Polit Sci Rev 77(1):123–141

    Article  Google Scholar 

  • Brams S, Fishburn P (1978) Approval voting. Am Polit Sci Rev 72(3):831–847

    Article  Google Scholar 

  • Chamberlin J (1985) An investigation into the relative manipulability of four voting systems. Behav Sci 30(4):195–203

    Article  Google Scholar 

  • Chamberlin J, Cohen M (1978) Toward applicable social choice theory: a comparison of social choice functions under spatial model assumptions. Am Polit Sci Rev 72(4):1341–1356

    Article  Google Scholar 

  • Coombs C (1964) A theory of data. Wiley, New York

    Google Scholar 

  • de Borda J-C (1784) Mémoire sur les élections au scrutin. Histoire de l’Academie Royale des Sciences, Paris

    Google Scholar 

  • Favardin P, Lepelley D (2006) Some further results on the manipulability of social choice rules. Soc Choice Welf 26:485–509

    Article  Google Scholar 

  • Favardin P, Lepelley D, Serais J (2002) Borda rule, Copeland method and strategic manipulation. Rev Econ Des 7:213–228

    Google Scholar 

  • Feld S, Grofman B (1992) Who’s afraid of the big bad cycle? Evidence from 36 elections. J Theor Polit 4(2):231–237

    Article  Google Scholar 

  • Felsenthal D (2012) Review of paradoxes afflicting procedures for electing a single candidate. In: Felsenthal D, Machover M (eds) Electoral systems: paradoxes, assumptions, and procedures. Springer, Heidelberg, pp 19–91

  • Gibbard A (1973) Manipulation of voting schemes: a general result. Econometrica 41:587–601

    Article  Google Scholar 

  • Green-Armytage J (2004) Cardinal-weighted pairwise comparison. Voting Matters 19:6–13

    Google Scholar 

  • Green-Armytage J (2011) Four Condorcet-Hare hybrid methods for single-winner elections. Voting Matters 29:1–14

    Google Scholar 

  • Green-Armytage J (2014) Strategic voting and nomination. Soc Choice Welf 42(1):111–138

    Article  Google Scholar 

  • Hoag C, Hallett G (1926) Proportional representation. MacMillian, New York

    Google Scholar 

  • Kim KH, Roush FW (1996) Statistical manipulability of social choice functions. Group Decis Negot 5:263–282

    Article  Google Scholar 

  • Lepelley D, Mbih B (1994) The vulnerability of four social choice functions to coalitional manipulation of preferences. Soc Choice Welf 11(3):253–265

    Article  Google Scholar 

  • Lepelley D, Valognes F (2003) Voting rules, manipulability and social homogeneity. Public Choice 116:165–184

    Article  Google Scholar 

  • Merrill S (1984) A comparison of efficiency of multicandidate electoral systems. Am J Polit Sci 28(1):23–48

    Article  Google Scholar 

  • Merrill S, Tideman TN (1991) The relative efficiency of approval and Condorcet voting procedures. Ration Soc 3:65–77

    Article  Google Scholar 

  • Nanson E (1882) Methods of election. Trans Proc R Soc Vic 18(954):197–240

    Google Scholar 

  • Nitzan S (1985) The vulnerability of point-voting schemes to preference variation and strategic manipulation. Public Choice 47:349–370

    Article  Google Scholar 

  • Pritchard G, Wilson M (2007) Exact results on manipulability of positional voting rules. Soc Choice Welf 29:487–513

    Article  Google Scholar 

  • Richelson J (1978) A comparative analysis of social choice functions, III. Behav Sci 23:169–176

    Article  Google Scholar 

  • Saari D (1994) Geometry of voting. Springer, Berlin

    Book  Google Scholar 

  • Satterthwaite M (1975) Strategy-proofness and arrow’s conditions: existence and correspondence theorems for voting procedures and social welfare functions. J Econ Theory 10:187–217

    Article  Google Scholar 

  • Smith D (1999) Manipulability measures of common social choice functions. Soc Choice Welf 16:639–661

    Article  Google Scholar 

  • Tideman TN (2006) Collective decisions and voting: the potential for public choice. Ashgate, Aldershot

  • Tsetlin I, Regenwetter M, Grofman B (2003) The impartial culture maximizes the probability of majority rule cycles. Soc Choice Welf 21:387–398

    Article  Google Scholar 

  • Weber R (1978) Comparison of voting systems. Manuscript.

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to James Green-Armytage.

Additional information

We are grateful to Florenz Plassmann for supplying a copy of the German Politbarometer candidate data that he downloaded and organized. We are also grateful to those who commented on earlier drafts of this paper at the 2013 Public Choice Society conference and the 2013 Southern Economic Association conference. Finally, we are grateful to Marc Fleurbaey and two anonymous referees.

Appendix: Proofs of propositions 1–3

Appendix: Proofs of propositions 13

Proof of proposition 1, part 1

Positional rules with \(p>1/2\) lack CMD. \(\square \)

Here we show that for any \(p>1/2\), there is an election that a specified majority cannot win. Given a value of \(p>1/2\), let n be an integer greater than \(\left( {2-p} \right) /\left( {2p-1} \right) \).

Consider an election in which there are 2n voters in the minority and \(2n+2\) voters in the majority. Specify further that those in the minority have cast n votes for \(\hbox {B}\succ \hbox {C}\succ \hbox {A}\) and n votes for \(C\succ B\succ A\). We have assumed that every vote must list a second choice as well as a first choice. Thus those who want A to win must give p points to either B or C for each point that they give to A. If it is possible at all for A to win, the hurdle to be overcome will be lowest if the A voters give the same number of points to B and C, so that B and C will have the same number of points after the election. This can be accomplished by casting \(n+1\) votes for \(\hbox {A}\succ \hbox {B}\succ \hbox {C}\) and \(n+1\) votes for \(A\succ C\succ B\). Then the result of the election is that A has \(2n+2\) points, whereas B and C each have \(n+2np+p\) points. Thus A wins if and only if

$$\begin{aligned}&n+2np+p<2n+2\\&n<\left( {2-p} \right) /\left( {2p-1} \right) \end{aligned}$$

But n has been assumed to be greater than this. Thus the specified majority cannot elect A.

Proof of proposition 1, part 2

Positional rules with \(p\le 1/2\) possess CMD. \(\square \)

Assume that the majority want A. Let the majority appoint one of their own voters to counter each voter who ranks B or C first, in the following manner: The counter-voter ranks A first followed by B and C in the opposite of the order used by the voter being countered. After such countering, no pair (voter plus counter) adds more to B or C than to A. (For example, if a minority voter ranks \(\hbox {B}\succ \hbox {C}\succ \hbox {A}\) and is countered with \(\hbox {A}\succ \hbox {C}\succ \hbox {B}\), then the pair add 1, 2p, and 1 to the scores of A, B, and C respectively. If a minority voter ranks \(\hbox {B}\succ \hbox {A}\succ \hbox {C}\) and is countered with \(\hbox {A}\succ \hbox {C}\succ \hbox {B}\), the pair adds \(1+p\), 1, and p.) Majority voters who are not needed for countering rank A first, and the remaining candidates in either order. Thus the majority can achieve the election of A no matter how the minority votes.

Proof of proposition 2

  1. 1.

    Let A be the winner when voting rule X is applied to preference profile \(\Pi \). Since sincere voting is a core equilibrium, there is no candidate \(\hbox {B}\ne \hbox {A}\) such that those who prefer B to A can change their votes so as to make B the winner under voting rule X.

  2. 2.

    Since, by assumption, voting rule X possesses CMD, point 1 implies that there is no candidate \(\hbox {B}\ne \hbox {A}\) such that a majority of voters prefer B to A. (That is, if a majority preferred B to A, this majority would be able to make B the winner, since voting rule X possesses CMD.)

  3. 3.

    Point 2 implies that no candidate other than A could possibly be a sincere Condorcet winner for profile \(\Pi \). Therefore, since A is the only candidate who could be a Condorcet winner for profile \(\Pi \), A must be the sincere winner for Condorcet-X as well as X.

  4. 4.

    Point 2 implies that there is no candidate B \(\ne \) A such that those who prefer B to A can change their votes so as to make B a Condorcet winner. That is, from point 2, there is not a majority of voters in favor of B over A, and since those who prefer B to A already rank B above A (because \(\Pi \) is a profile of sincere preferences), there is nothing that those who prefer B to A can do to give B a majority over A.

  5. 5.

    Combining points 1, 3, and 4: A is the sincere Condorcet-X winner, and for any candidate \(\hbox {B} \ne \hbox {A}\), voters preferring B to A cannot change their votes either so as to make B the winner according to the voting rule X, or so as to make B the Condorcet winner. Therefore, they cannot make B the winner according to the voting rule Condorcet-X. \(\square \)

Proof of proposition 3, case 1

Candidate A is a sincere Condorcet winner.

Case 1-1: A is the sincere elimination- p winner as well as a sincere Condorcet winner. In this case, we want to show that if elimination-p is manipulable, Condorcet-elimination-p is also manipulable. So, suppose that the voters who prefer B to A can make B the elimination-p winner. This implies that they can vote in such a way as to create a situation in which A is the positional-p loser, and B beats C pairwise. The question is whether this necessarily implies that they can arrange for C to beat or tie A pairwise at the same time, so that there is no Condorcet winner and thus the elimination-p rule decides the Condorcet-elimination-p winner.

To address this question, define a, b, and c as the numbers of sincere first-choice votes for A, B, and C, respectively; and define \(V\equiv a+b+c\). To provide the most critical test, we want to explore the case in which it is easiest for strategizers to make A the positional-p loser without being able to make C pairwise-defeat A, so we suppose that there are no voters with the sincere ordering \(\hbox {C}\succ \hbox {A}\succ \hbox {B}\), and that the strategizers are able to vote so that first-round points are evenly divided between B and C (that is, that they can use their own ballots to balance out the difference in points between B and C from the ballots of non-strategizers). Thus, in the strategic voting situation that we are examining, the A voters’ ballots give a points to A, and ap points that are divided between B and C, while the B and C voters’ ballots give \(b+bp+c+cp\) points that are divided between B and C. Since A is to be made the positional-p loser, we know that

$$\begin{aligned} a< & {} \frac{1}{2}\left( {b+c+ap+bp+cp} \right) \\ a< & {} \frac{1}{3}\left( {1+p} \right) V \end{aligned}$$

In this situation, we know that with the strategic votes, A pairwise-defeats B, B pairwise-defeats C, and A is the positional-p loser. To guarantee that B is the Condorcet-elimination-p winner, all that remains is to show that strategizers can cause C to pairwise-defeat (or pairwise-tie) A at the same time, causing a cycle that will be resolved in B’s favor. Since all voters other than those whose sincere first choice is A rank C above A in the situation that is optimal for the strategizers, we can represent this requirement by \(a\le V/2\). From the expression above, this is true if \(\left( {1/3} \right) \left( {1+p} \right) V\le V/2\), and thus it is true if \(p\le 1/2\).

Case 1-2: While A is a Condorcet winner, another candidate, B, is the sincere elimination- p winner. In this case, the elimination-p rule is certainly manipulable; the majority who prefer A to B just need to rank A first and allocate their second choices so that neither B nor C has a higher positional score than A. The algebra employed in case 1-1 can be used again to show that if \(p\le 1/2\), those who prefer B to A can make B the Condorcet-X winner. \(\square \)

Proof of proposition 3case 2 There is no sincere Condorcet winner. Let A be the sincere elimination-p winner.

Case 2-1: There is another candidate B who is preferred by a majority to A. In this case, both elimination-p and Condorcet-elimination-p must be manipulable, because both possess CMD.

Case 2-2: There is not another candidate B who is preferred by a majority to A. (That is, candidate A has a pairwise tie with one or both other candidates.) A is the sincere winner in both rules, and the algebra employed in case 1-1 can be used again to show that if \(p\le 1/2\), the conditions necessary for those who prefer B to make B the elimination-p winner are also sufficient to ensure that they can make B the Condorcet-elimination-p winner. \(\square \)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Green-Armytage, J., Tideman, T.N. & Cosman, R. Statistical evaluation of voting rules. Soc Choice Welf 46, 183–212 (2016).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: