Composite Index Ranking of Economic Well-Being in U.S. Metropolitan Areas: How Prevalent are Rank Anomalies?

Abstract

Composite indicators have the advantage of summarizing complex multi-dimensional concepts in a single measurement. They also suffer from disadvantages such as subjectivity in choice of indicators, weighting, and aggregation methods. In this paper, we update Medcalfe’s (Social Indicators Research 139(3), 1147–1167. https://doi.org/10.1007/s11205-017-1755-5. 2018) Economic Well-Being (EWB) index of US Metropolitan Statistical Areas with the latest (2017) available data. Using this index of EWB, we investigate two social choice violations that have been understudied in the composite indicators literature. We provide theoretical and empirical evidence of cycles and violations of independence of irrelevant alternatives. Depending on the number of cities and ranking components, incidence of these social choice violations can be large, creating ambiguity in a set of rankings. In general, having more ranking components reduces the expected and, for the most part, realized, incidence of social choice violations. Further, the results suggest that the EWB index rankings should potentially be interpreted in terms of rank tiers rather than in terms of individual rankings.

Appendix 1: Essential Code Discussion

All of the code for both the simulation and empirical analysis is freely available online at the following Github Repository: https://github.com/Syracuse-University-Sport-Analytics/BordaCount, and was written in R 3.6.1. This Appendix 1 summarizes the key functions that is provided within the lib/rankSum.R script, which is the analytical library that both the simulation script (R/rankSumSimulationTripletsOnlyParallel.R) and the empirical analysis script (R/rankSumFullAnalysisParallelTripletsOnly.R) extensively use for calculating IIA violations and cycles. These individual scripts make use of the doSNOW package to parallelize the code across 24 cores, which is required to compute the results in days instead of months.

The simulation script simulates every possible ranking for n number of columns (categories) for three hypothetical cities. The empirical script selects every possible triplet among the 40 cities, and then selects every n possible columns out of a total 10 empirical columns. The script runs both the simulation and empirical analysis from 3 to 10 columns.

For each of the empirical or simulated cities and each of the 3–10 possible columns, the checkForIndependenceViolationsAndCycles function is used to determine if an IIA violation or a cycle exists. The code for this function is listed in Fig. 

Fig. 5

Function to find a cycle or an IIA violation

5. This function returns a vector with containing both a IIA violation indicator and a cycle indicator. This computations for finding IIA violations are located in the independenceViolationWithinSet (Fig. 

Fig. 6

Function to find an IIA violations

6) function and the computations for detecting cycles are found in the cycleViolation (Fig. 

Fig. 7

Function to find a cycle

7) function.

The independenceViolationWithinSet function (Fig. 6) computes the presence of a IIA violation. This is accomplished by first finding the rank of each city globally by calling the createIndependentRankColumn function on the entire triplet. Then each possible pair is ranked independently by using the createIndependentRankColumn function on each pair. The rank function is called on each pair’s global ranking to find the relative global rankings, which is then compared to the independent ranking, which only uses the pair. For strong violations, which is the subject of this paper, the ranking order needs to be reversed for when comparing a pair’s independent ordering with the pair’s global ordering. The containsTies function is used to determine if a tie exists, which implies a strong IIA violation did not occur. For weak IIA violations, ties are considered and if a tie exists in the independent ordering but not the global ordering (and the reverse), a weak violation is reported.

The cycleViolation function (A-3) computes the presence of a cycle. To do this each of the cities within the triplet are compared to each other and the number of wins and losses are calculated. The paricipant1BeatsParticipant2 function determines if a team beats another team by comparing the sum of the ranks. A cycle occurs if and only if all three cities records losses. If any of the cities have zero losses, then a cycle is impossible, otherwise a cycle occurred.