Skip to main content

The Balanced Worth: A Procedure to Evaluate Performance in Terms of Ordered Attributes

Abstract

There are many problems in the social sciences that refer to the evaluation of the relative performance of some populations when their members’ achievements are described by a distribution of outcomes over a set of ordered categories. A new method for the evaluation of this type of problems is presented here. That method, called balanced worth, offers a cardinal, complete and transitive evaluation that is based on the likelihood of getting better results. The evaluation of each society is based on the probability of obtaining better results with respect to the others. The balanced worth is a refinement of “the worth” (Herrero and Villar in PLoS ONE 8(12):e84784, 2013. https://doi.org/10.1371/journal.pone.0084784) that overcomes its excessive sensitivity to the differences, due to the presence of ties. We also discuss how this method can be applied for the case of heterogeneous populations and show how it can be applied in different contexts. An empirical example, regarding life satisfaction in Spain is used to illustrate the working of this method.

This is a preview of subscription content, access via your institution.

Notes

  1. 1.

    This happens, for instance, when analysing the citation impact of research articles regarding different disciplines. As the number of citations is rather idiosyncratic, one usually takes the shares of the papers in the different percentiles of each discipline to make an analysis independent of the differences in the mean and the variance of the citations distributions.

  2. 2.

    Note that, for a given problem, the probability of ties will depend on the number of admissible categories defined. The difference between the balanced worth and the worth will thus be smaller the finer the grid of possible outcomes and vanishes for continuous distributions.

  3. 3.

    We say that group k is fully dominated when \( p_{kj} = 0\,,\,\,\forall \,\,j \ne k \).

  4. 4.

    Let us recall here that inequality measures typically give more weight to the realizations in the lower part of the distribution. This makes sense when heterogeneity is bad but this is not always the case. For instance when comparing years of schooling across generations in a given country, one would typically like to find that the young generation has higher values than the old one, so that perfect equality is not the desideratum.

  5. 5.

    In the empirical application regarding life satisfaction individual answers have been grouped into a rougher set of categories. One may reasonably wonder what the purpose is of losing information by grouping those data into broader categories when we have all the individual numerical responses. The main reason is that when dealing with subjective evaluations in terms of numerical scales, there is no guarantee that numbers mean the same for different people (your 7 and my 7 may well represent very different things). Moreover, individual scales need not be linear (i.e. an evaluation 8 need not be twice one of 4, even for a single individual). Grouping numerical answers into categories may thus helps illuminate some structural features of the groups, enhance robustness and reduce the comparability assumptions required.

References

  1. Albarrán, P., Herrero, C., Ruiz-Castillo, J., & Villar, A. (2017). The Herrero-Villar approach to citation impact. Journal of Informetrics, 11(2), 625–640.

    Article  Google Scholar 

  2. Bellù, L. G., & Liberati, P. (2005). Social welfare analysis of income distributions ranking income distributions with crossing generalised Lorenz curves. Rome: Food and Agriculture Organization of the United Nations.

    Google Scholar 

  3. Border, K. C. (1989). Fixed point theorems with applications to economics and game theory. Cambridge: Cambridge University Press.

    Google Scholar 

  4. Bourguignon, F., Ferreira, F. H. G., & Leite, P. G. (2008). Beyond Oaxaca–Blinder: Accounting for differences in household income distributions. Journal of Economic Inequality, 6(2), 117–148.

    Article  Google Scholar 

  5. Chakravarty, S. R., & Silber, J. (2007). A generalized index of employment segregation. Mathematical Social Sciences, 53(2), 185–195.

    Article  Google Scholar 

  6. Crespo, J. A., Li, Y., & Ruiz-Castillo, J. (2013). The measurement of the effect on citation inequality of differences in citation practices across scientific fields. PLoS ONE, 8(3), e58727. https://doi.org/10.1371/journal.pone.0058727.

    Article  Google Scholar 

  7. Cuhadaroglu, T. (2013). My group beats your group: Evaluating non-income inequalities. St Andrews: W.P. School of Economics and Finances, University of St Andrews.

    Google Scholar 

  8. Echenique, F., & Fryer, R. G. (2005). On the measurement of segregation (Vol. w11258). Cambridge: National Bureau of Economic Research.

    Book  Google Scholar 

  9. Eurostat. (2015). Quality of life. Facts and views. Luxemburg: Publication Office of the European Union.

    Google Scholar 

  10. Frankel, D. M., & Volij, O. (2011). Measuring school segregation. Journal of Economic Theory, 146(1), 1–38.

    Article  Google Scholar 

  11. Gallén, M. L., & Peraita, C. (2015). A comparison of corporate social responsibility engagement in the OECD countries with categorical data. Applied Economics Letters, 22(12), 1005–1009.

    Article  Google Scholar 

  12. Gonzalez-Diaz, J., Hendrichx, R., & Lohmann, E. (2014). Paired comparison analysis: An axiomatic approach to ranking methods. Social Choice and Welfare, 42(1), 139–169.

    Article  Google Scholar 

  13. Grannis, R. (2002). Segregation indices and their functional inputs. Sociological Methodology, 32(1), 69–84.

    Article  Google Scholar 

  14. Herrero, C., Méndez, I., & Villar, A. (2014). Analysis of group performance with categorical data when agents are heterogeneous: The evaluation of scholastic performance in the OECD through PISA. Economics of Education Review, 40, 140–151.

    Article  Google Scholar 

  15. Herrero, C., & Villar, A. (2013). On the comparison of group performance with categorical data. PLoS ONE, 8(12), e84784. https://doi.org/10.1371/journal.pone.0084784.

    Article  Google Scholar 

  16. Laband, D. N., & Piette, M. J. (1994). The relative impacts of economics journals: 1970–1990. Journal of Economic Literature, 32(2), 640–666.

    Google Scholar 

  17. Laslier, J. (1997). Tournament solutions and majority voting. Berlin: Springer.

    Book  Google Scholar 

  18. Li, F., Yi, K., & Jestes, J. (2009). Ranking distributed probabilistic data. SIGMOD’09, June 29–July 2.

  19. Lieberson, S. (1976). Rank-sum comparisons between groups. Sociological Methodology, 7, 276–291. https://doi.org/10.2307/270713.

    Article  Google Scholar 

  20. Martínez-Mekler, G., Martínez, R. A., del Río, M. B., Mansilla, R., Miramontes, P., et al. (2009). Universality of rank-ordering distributions in the arts and sciences. PLoS ONE, 4(3), e4791. https://doi.org/10.1371/journal.pone.0004791.

    Article  Google Scholar 

  21. Palacios-Huerta, I., & Volij, O. (2004). The measurement of intellectual influence. Econometrica, 72(3), 963–977.

    Article  Google Scholar 

  22. Pavot, W., & Diener, E. (2008). The satisfaction with life scale and the emerging construct of life satisfaction. The Journal of Positive Psychology, 3, 137–152.

    Article  Google Scholar 

  23. Pinski, G., & Narin, F. (1976). Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing and Management, 12(5), 297–312.

    Article  Google Scholar 

  24. Reardon, S. F., & Firebaugh, G. (2002). Measures of multi-group segregation. Sociological Methodology, 32, 33–76.

    Article  Google Scholar 

  25. Rosvall, M., & Bergstrom, C. T. (2007). An information-theoretic framework for resolving community structure in complex networks. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.0611034104.

    Article  Google Scholar 

  26. Sheriff, G., & Maguire, K. (2013). Ranking distributions of environmental outcomes across population groups. National Center for Environmental Economics. August.

  27. Shorrocks, A. F. (1983). Ranking income distributions. Economica, 50(197), 3–17.

    Article  Google Scholar 

  28. Slutzki, G., & Volij, O. (2006). Scoring of web pages and tournaments. Social Choice and Welfare, 26(1), 75–92.

    Article  Google Scholar 

  29. Torregrosa, R. (2015). Medición y evolución del sentimiento autonómico en España. Investigaciones Regionales- Journal of Regional Research, 33, 53–70.

    Google Scholar 

  30. Veenhoven, R. (1991). Is happiness relative? Social Indicators Research, 24, 1–34.

    Article  Google Scholar 

  31. Villar, A. (2014). Education and cognitive skills in the Spanish adult population. Intergenerational comparison of mathematical knowledge form PIAAC data. Advances in Social Sciences Research Journal, 1(1), 72–88.

    Google Scholar 

  32. Yalonetzky, G. (2012). A dissimilarity index of multidimensional inequality of opportunity. The Journal of Economic Inequality, 10(3), 343–373.

    Article  Google Scholar 

Download references

Acknowledgements

Thanks are also due to Héctor García Peris, for his help in developing the algorithm that computes the evaluation, and to an anonymous referee for very helpful comments and suggestions. Funding was provided by the Spanish Ministerio de Economía y Competitividad (Grant Nos. ECO2015-65408-R, ECO2015-65820-P).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Antonio Villar.

Appendix: Existence and uniqueness of the balanced worth

Appendix: Existence and uniqueness of the balanced worth

Here we prove that the balanced worth always exists and that, under very general conditions, it is unique and strictly positive.

Definition

Given a problem A, a group j is fully dominated in A, if for all i ≠ j it happens that \( p_{ij} = 1 \).

When a group j is fully dominated, \( p_{ji} = e_{ji} = 0,\;\forall i \ne j \). That is, all individuals in group j belong to a lower level than any other individual in the rest of the groups. Note that, in practice, the probability of finding a fully dominated group is zero.

Let us formally state that a solution to the system of g equations with g unknowns (3), always exists.

Theorem 1

Let A be an evaluation problem. Then:

  1. (i)

    A vector \( {\mathbf{w}} \in {\mathbb{R}}_{ + }^{g} \) exists that solves equation system (3). That is, a vector w* such that:

    $$ w_{i}^{*} = \frac{{\sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)w_{j}^{*} } }}{{\sum\nolimits_{j \ne i} {\left( {p_{ji} + \frac{{e_{ji} }}{2}} \right)} }},\quad i = 1,2, \ldots ,g $$
  2. (ii)

    If no group is fully dominated, then the solution is unique (up to a scalar multiplication) and strictly positive.

Proof

(i) Let \( W = \left\{ {{\mathbf{w}} \in {\mathbb{R}}_{ + }^{g} /\sum\nolimits_{i = 1}^{g} {w_{i} = g} } \right\} \) and consider the function \( \varphi :W \to {\mathbb{R}} \), given by:

$$ \varphi_{i} \left( {\mathbf{w}} \right) = w_{i} - \frac{1}{g - 1}\left( {w_{i} \sum\nolimits_{j \ne i} {\left( {p_{ji} + \frac{{e_{ji} }}{2}} \right) - \sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)w_{j} } } } \right) $$

As \( \sum\nolimits_{j \ne i} {\left( {p_{ji} + \frac{{e_{ij} }}{2}} \right) \le g - 1} \), we have:

$$ \varphi_{i} \left( {\mathbf{w}} \right) \ge w_{i} - w_{i} + \frac{1}{g - 1}\sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)w_{j} } \ge 0 $$

Moreover,

$$ \sum\nolimits_{i = 1}^{g} {\varphi_{i} \left( {\mathbf{w}} \right)} = g - \frac{1}{g - 1}\left( {\sum\nolimits_{i = 1}^{g} {w_{i} } \sum\nolimits_{j \ne i} {\left( {p_{ji} + \frac{{e_{ji} }}{2}} \right) - \sum\nolimits_{i = 1}^{g} {\sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)w_{j} } } } } \right) $$

Note that, by construction,

$$ \sum\nolimits_{i = 1}^{g} {w_{i} } \sum\nolimits_{j \ne i} {\left( {p_{ji} + \frac{{e_{ji} }}{2}} \right) = \sum\nolimits_{i = 1}^{g} {\sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)w_{j} } } } $$

which means that \( \sum\nolimits_{i = 1}^{g} {\varphi_{i} \left( {\mathbf{w}} \right)} = g \) and hence that function φ maps W into itself. As it is a continuous function and W is a compact convex set, Brouwer’s Theorem (e.g. Border 1989), ensures the existence of a fixpoint, \( {\mathbf{w}}^{*} = \varphi \left( {{\mathbf{w}}^{*}} \right) \). That is,

$$ w_{i}^{*} = w_{i}^{*} - \frac{1}{g - 1}\left( {w_{i}^{*} \sum\nolimits_{j \ne i} {\left( {p_{ji} + \frac{{e_{ji} }}{2}} \right) - \sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)w_{j}^{*} } } } \right) $$

and, therefore,

$$ w_{i}^{*} = \frac{{\sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)w_{j}^{*} } }}{{\sum\nolimits_{j \ne i} {\left( {p_{ji} + \frac{{e_{ji} }}{2}} \right)} }},\quad i = 1,2, \ldots ,g $$

(ii) Assume now that there is no fully dominated group, that is, \( (p_{ij} + \frac{{e_{ij} }}{2}) > 0\;\forall i,j \). Then, the solutions must be strictly positive. This is so because both numerator and denominator are strictly positive. To prove uniqueness, suppose there are two strictly positive vectors, w, y, that solve the equation system (3). Then, we can write:

$$ \sum\nolimits_{j \ne i} {\left( {p_{ji} + \frac{{e_{ij} }}{2}} \right) = \frac{{\sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)} w_{j} }}{{w_{i} }} = \frac{{\sum\nolimits_{j \ne i} {\left( {p_{ij} + \frac{{e_{ij} }}{2}} \right)y_{j} } }}{{y_{i} }},\quad i = 1,2, \ldots ,g} $$

For a given i, this expression can be rewritten as:

$$ A = \sum\limits_{i = 1}^{g - 1} {B_{i} x_{i} = \sum\limits_{i = 1}^{g - 1} {B_{i} z_{i} } } $$

where all terms are strictly positive, with \( x_{j} = w_{j} /w_{i} ,\;z_{j} = y_{j} /y_{i} \). But this is the equation of a hyperplane with a given normal, which means that vectors x and z are to be proportional. That is, the solution is unique up to the choice of units.□

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Herrero, C., Villar, A. The Balanced Worth: A Procedure to Evaluate Performance in Terms of Ordered Attributes. Soc Indic Res 140, 1279–1300 (2018). https://doi.org/10.1007/s11205-017-1818-7

Download citation

Keywords

  • Evaluation method
  • Categorical variables
  • Relative group performance