Abstract
Considering the impact composite indicators can have on public opinion and policy development, the need for their frameworks to be methodologically sound and statistically verified is growing daily. One of the issues in the process of composite indicator construction which has generated much debate is how to choose the weighting scheme. To address this slippery step, we propose an optimization approach based on the enhanced Scatter Search (eSS) metaheuristic. In this paper, the eSS algorithm is applied to obtain a weighting scheme which will increase the stability of the composite indicator. The objective function is based on the relative contributions of indicators, while the problem constraints rely on the bootstrap Composite I-distance Indicator (CIDI) approach which is also proposed herein. The eSS-CIDI approach combines the exploration capability of eSS and the data-driven constraints devised from the bootstrap CIDI. This novel weighting approach was tested on two acknowledged composite indicators: the Academic Ranking of World Universities (ARWU) and the Networked Readiness Index (NRI). Results indicate that the composite indicators created using the eSS-CIDI weighting approach are more stable than the official ones.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The obtained CIDI weighting scheme is: Alumni 17.6%, Award 16.7%, HiCi 16.9%, N&S 19.4%, PUB 13.4%, and PCP 16.0%.
The CIDI weighting scheme is: Environment 24.8%, Readiness 22.5%, Usage 26.5%, and Impacts 26.2%.
References
Ács, Z. J., Autio, E., & Szerb, L. (2014). National systems of entrepreneurship: Measurement issues and policy implications. Research Policy, 43(3), 476–494. https://doi.org/10.1016/j.respol.2013.08.016.
Amado, C. A. F., São José, J. M. S., & Santos, S. P. (2016). Measuring active ageing: A data envelopment analysis approach. European Journal of Operational Research, 255, 207–223. https://doi.org/10.1016/j.ejor.2016.04.048.
Angilella, S., Catalfo, P., Corrente, S., Giarlotta, A., Greco, S., & Rizzo, M. (2018). Robust sustainable development assessment with composite indices aggregating interacting dimensions: The hierarchical-SMAA-Choquet integral approach. Knowledge-Based Systems, 158, 136–153. https://doi.org/10.1016/j.knosys.2018.05.041.
Antony, J., Swarnkar, R., Kumar, M., & Tiwari, M. K. (2006). Design of synchronised supply chain: A genetic algorithm based six sigma constrained approach. International Journal of Logistics Systems and Management, 2, 120–140.
Arcones, M. A., & Gine, E. (1989). The bootstrap of the mean with arbitrary bootstrap sample size. Annales de l’I.H.P. Probabilités et Statistiques, 25(4), 457–481.
Austin, P. C., & Small, D. S. (2014). The use of bootstrapping when using propensity-score matching without replacement: A simulation study. Statistics in Medicine, 33(24), 4306–4319. https://doi.org/10.1002/sim.6276.
Banerjee, A. K. (2018). Multidimensional indices with data-driven dimensional weights: A multidimensional coefficient of variation. Arthaniti: Journal of Economic Theory and Practice. https://doi.org/10.1177/0976747918792644.
Becker, W., Saisana, M., Paruolo, P., & Vandecasteele, I. (2017). Weights and importance in composite indicators: Closing the gap. Ecological Indicators, 80, 12–22. https://doi.org/10.1016/j.ecolind.2017.03.056.
Beran, R., & Ducharme, G. (1991). Asymptotic theory for bootstrap methods in statistics. Montréal: Centre de Recherches Mathematiques.
Bickel, P. J., Gotze, F., & van Zwet, W. R. (1997). Resampling fewer than n observations: Gains, losses and remedies for losses. Statistica Sinica, 7, 1–31.
Bickel, P. J., & Sakov, A. (2008). On the choice of m in the m out of n bootstrap and its application to confidence bounds for extreme percentiles. Statistica Sinica, 18, 967–985.
Bilbao-Osorio, B., Dutta, S., & Lanvin, B. (2014). The global information technology report 2014. Rewards and risks of big data.
Birkmann, J. (2007). Risk and vulnerability indicators at different scales: Applicability, usefulness and policy implications. Environmental Hazards, 7(1), 20–31. https://doi.org/10.1016/j.envhaz.2007.04.002.
Blancas, F. J., Contreras, I., & Ramírez-Hurtado, J. M. (2013). Constructing a composite indicator with multiplicative aggregation under the objective of ranking alternatives. Journal of the Operational Research Society, 64(5), 668–678. https://doi.org/10.1057/jors.2012.90.
Blum, C., & Roli, A. (2003). Metaheuristics in combinatorial optimization: Overview and conceptual comparison. ACM Computing Surveys, 35(3), 189–213. https://doi.org/10.1007/s10479-005-3971-7.
Bojórquez-Tapia, L. A., Sánchez-Colon, S., & Florez, A. (2005). Building consensus in environmental impact assessment through multicriteria modeling and sensitivity analysis. Environmental Management, 36(3), 469–481. https://doi.org/10.1007/s00267-004-0127-5.
Booysen, F. (2002). An overview and evaluation of composite indices of development. Social Indicators Research, 59(2), 115–151. https://doi.org/10.1023/A:1016275505152.
Braga-Neto, U. M., & Dougherty, E. R. (2004). Is cross-validation valid for small-sample microarray classification? Bioinformatics, 20(3), 374–380. https://doi.org/10.1093/bioinformatics/btg419.
Bretagnolle, J. (1983). Lois limites du bootstrap de certaines fonctionnelles. Annales de l’IHP Probabilités et Statistiques, 19(3), 281–296.
Burgass, M. J., Halpern, B. S., Nicholson, E., & Milner-Gulland, E. J. (2017). Navigating uncertainty in environmental composite indicators. Ecological Indicators, 75, 268–278. https://doi.org/10.1016/j.ecolind.2016.12.034.
Casadio Tarabusi, E., & Guarini, G. (2013). An unbalance adjustment method for development indicators. Social Indicators Research, 112(1), 19–45. https://doi.org/10.1007/s11205-012-0070-4.
Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429–444. https://doi.org/10.1016/0377-2217(78)90138-8.
Cherchye, L., Moesen, W., Rogge, N., & Van Puyenbroeck, T. (2007). An introduction to “benefit of the doubt” composite indicators. Social Indicators Research, 82(1), 111–145. https://doi.org/10.1007/s11205-006-9029-7.
Cherchye, L., Moesen, W., Rogge, N., Van Puyenbroeck, T., Saisana, M., Saltelli, A., et al. (2008a). Creating composite indicators with DEA and robustness analysis: The case of the Technology Achievement Index. Journal of the Operational Research Society, 59(2), 239–251. https://doi.org/10.1057/palgrave.jors.2602445.
Cherchye, L., Ooghe, E., & Van Puyenbroeck, T. (2008b). Robust human development rankings. The Journal of Economic Inequality, 6(4), 287–321. https://doi.org/10.1007/s10888-007-9058-8.
Choquet, G. (1953). Theory of capacities. Annales de l’Institut Fourier, 5(54), 131–295.
Chowdhury, S., Moral, R. J., & Dulikravich, G. S. (2008). Predator-prey evolutionary multi-objective optimization algorithm: Performance and improvements. In Proceedings of 7th ASMO-UK/ISSMO international conference on engineering design optimization (pp. 1–10). Bath, UK. Retrieved from https://maidroc.fiu.edu/wp-content/uploads/2012/05/CP16pcp30.pdf.
Cole, T. J. (2015a). Setting number of decimal places for reporting risk ratios: Rule of four. The BMJ, 350, h1845. https://doi.org/10.1136/BMJ.H1845.
Cole, T. J. (2015b). Too many digits: The presentation of numerical data. Archives of Disease in Childhood, 100(7), 608–609. https://doi.org/10.1136/archdischild-2014-307149.
Corrente, S., Greco, S., Nicotra, M., Romano, M., & Schillaci, C. E. (2018). Evaluating and comparing entrepreneurial ecosystems using SMAA and SMAA-S. The Journal of Technology Transfer. https://doi.org/10.1007/s10961-018-9684-2.
Cox, D. R., Fitzpatrick, R., Fletcher, A. E., Gore, S. M., Spiegelhalter, D. J., & Jones, D. R. (1992). Quality-of-life assessment: Can we keep it simple? Journal of the Royal Statistical Society. Series A, 155(3), 353–393. https://doi.org/10.2307/2982889.
Daraio, C., & Bonaccorsi, A. (2017). Beyond university rankings? Generating new indicators on universities by linking data in open platforms. Journal of the Association for Information Science and Technology, 68(2), 508–529. https://doi.org/10.1002/asi.23679.
Davison, A. C., Hinkley, D. V., & Young, G. A. (2003). Recent developments in bootstrap methodology. Statistical Science, 18(2), 141–157. https://doi.org/10.1214/ss/1063994969.
De Bin, R., Janitza, S., Sauerbrei, W., & Boulesteix, A. L. (2016). Subsampling versus bootstrapping in resampling-based model selection for multivariable regression. Biometrics, 72(1), 272–280. https://doi.org/10.1111/biom.12381.
Decancq, K., & Lugo, M. A. (2013). Weights in multidimensional indices of wellbeing: An overview. Econometric Reviews, 32(1), 7–34. https://doi.org/10.1080/07474938.2012.690641.
Dehon, C., McCathie, A., & Verardi, V. (2010). Uncovering excellence in academic rankings: a closer look at the Shanghai ranking. Scientometrics, 83(2), 515–524. https://doi.org/10.1007/s11192-009-0076-0.
Despotis, D. K. (2005). A reassessment of the human development index via data envelopment analysis. Journal of the Operational Research Society, 56(8), 969–980. https://doi.org/10.1057/palgrave.jors.2601927.
Dialga, I., & Thi Hang Giang, L. (2017). Highlighting methodological limitations in the steps of composite indicators construction. Social Indicators Research, 131(2), 441–465. https://doi.org/10.1007/s11205-016-1263-z.
Dijkstra, L., Annoni, P., & Kozovska, K. (2011). A series of short papers on regional research and indicators produced by the Directorate-General for Regional Policy.
Dobbie, M. J., & Dail, D. (2013). Robustness and sensitivity of weighting and aggregation in constructing composite indices. Ecological Indicators, 29, 270–277. https://doi.org/10.1016/j.ecolind.2012.12.025.
Dobrota, M., Bulajic, M., Bornmann, L., & Jeremic, V. (2016). A new approach to the QS university ranking using the composite I-distance indicator: Uncertainty and sensitivity analyses. Journal of the Association for Information Science and Technology, 67(1), 200–211. https://doi.org/10.1002/asi.23355.
Dobrota, M., & Dobrota, M. (2016). ARWU ranking uncertainty and sensitivity: What if the award factor was excluded? Journal of the Association for Information Science and Technology, 67(2), 480–482. https://doi.org/10.1002/asi.23527.
Dobrota, M., & Jeremic, V. (2017). Shedding the light on the stability of university rankings in the ICT field. IETE Technical Review, 34(1), 75–82. https://doi.org/10.1080/02564602.2016.1144487.
Dobrota, M., Martic, M., Bulajic, M., & Jeremic, V. (2015). Two-phased composite I-distance indicator approach for evaluation of countries’ information development. Telecommunications Policy, 39(5), 406–420. https://doi.org/10.1016/j.telpol.2015.03.003.
Docampo, D., & Cram, L. (2015). On the effects of institutional size in university classifications: The case of the Shanghai ranking. Scientometrics, 102(2), 1325–1346. https://doi.org/10.1007/s11192-014-1488-z.
Dubey, H. M., Pandit, M., & Panigrahi, B. K. (2016). Hydro-thermal-wind scheduling employing novel ant lion optimization technique with composite ranking index. Renewable Energy, 99, 18–34. https://doi.org/10.1016/j.renene.2016.06.039.
Ebert, U., & Welsch, H. (2004). Meaningful environmental indices: A social choice approach. Journal of Environmental Economics and Management, 47(2), 270–283. https://doi.org/10.1016/j.jeem.2003.09.001.
Edirisinghe, N. C. P., & Zhang, X. (2010). Input/output selection in DEA under expert information, with application to financial markets. European Journal of Operational Research, 207(3), 1669–1678. https://doi.org/10.1016/j.ejor.2010.06.027.
Efron, B. (1979). Bootstrap methods: Another look at the jacknife. The Annals of Statistics, 7(1), 1–26. https://doi.org/10.1214/aoms/1177692541.
Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. Boca Raton: Chapman & Hall/CRC.
Efron, B., & Tibshirani, R. (1997). Improvements on cross-validation: The 632 + bootstrap method. Journal of the American Statistical Association, 92(438), 548–560. https://doi.org/10.1080/01621459.1997.10474007.
Egea, J. A., Balsa-Canto, E., García, M.-S. G., & Banga, J. R. (2009). Dynamic optimization of nonlinear processes with an enhanced scatter search method. Industrial and Engineering Chemistry Research, 48(9), 4388–4401. https://doi.org/10.1021/ie801717t.
Egea, J. A., Henriques, D., Cokelaer, T., Villaverde, A., MacNamara, A., Danciu, D.-P., et al. (2014). MEIGO: An open-source software suite based on metaheuristics for global optimization in systems biology and bioinformatics. BMC Bioinformatics, 15(1), 136. https://doi.org/10.1186/1471-2105-15-136.
Ferrier, G. D., & Hirschberg, J. G. (1999). Can we bootstrap DEA scores? Journal of Productivity Analysis, 11(1), 81–92. https://doi.org/10.1023/A:1007787406099.
Foster, J. E., McGillivray, M., & Seth, S. (2013). Composite indices: Rank robustness, statistical association, and redundancy. Econometric Reviews, 32(1), 35–56. https://doi.org/10.1080/07474938.2012.690647.
Freudenberg, M. (2003). Composite indicators of country performance: A critical assessment. OECD Science, Technology and Industry Working Papers, 16, 35. https://doi.org/10.1787/405566708255.
Fusco, E. (2015). Enhancing non-compensatory composite indicators: A directional proposal. European Journal of Operational Research, 242(2), 620–630. https://doi.org/10.1016/j.ejor.2014.10.017.
Gen, M., & Cheng, R. (2000). Genetic algorithms and engineering optimization. Hoboken: Wiley.
Gendreau, M., & Potvin, J. Y. (2010). Handbook of metaheuristics (Vol. 146). Berlin: Springer. https://doi.org/10.1007/978-1-4419-1665-5.
Geyer, C. (2013). 5601 Notes: The subsampling bootstrap. Retrieved February 10, 2018 from http://www.stat.umn.edu/geyer/5601/notes/sub.pdf.
Giambona, F., & Vassallo, E. (2013). Composite indicator of financial development in a benefit-of-doubt approach. Economic Notes, 42(2), 171–202. https://doi.org/10.1111/j.1468-0300.2013.12005.x.
Giannetti, B. F., Bonilla, S. H., Silva, C. C., & Almeida, C. M. V. B. (2009). The reliability of experts’ opinions in constructing a composite environmental index: The case of ESI 2005. Journal of Environmental Management, 90(8), 2448–2459. https://doi.org/10.1016/j.jenvman.2008.12.018.
Glover, F. (1977). Heuristics for integer programming using surrogate constraints. Decision Sciences, 8(1), 156–166. https://doi.org/10.1111/j.1540-5915.1977.tb01074.x.
Glover, F., & Kochenberger, G. A. (2003). Handbook of metaheuristics. Dordrecht: Kluwer Academic Publishers.
Glover, F., Laguna, M., & Martí, R. (2000). Fundamentals of scatter search and path relinking. Control and Cybernetics, 29(3), 652–684. https://doi.org/10.1007/0-306-48056-5_1.
Greco, S., Ishizaka, A., Matarazzo, B., & Torrisi, G. (2018a). Stochastic multi-attribute acceptability analysis (SMAA): An application to the ranking of Italian regions. Regional Studies, 52(4), 585–600. https://doi.org/10.1080/00343404.2017.1347612.
Greco, S., Ishizaka, A., Tasiou, M., & Torrisi, G. (2018b). On the methodological framework of composite indices: A review of the issues of weighting, aggregation, and robustness. Social Indicators Research. https://doi.org/10.1007/s11205-017-1832-9.
Grupp, H., & Mogee, M. E. (2004). Indicators for national science and technology policy: How robust are composite indicators? Research Policy, 33(9), 1373–1384. https://doi.org/10.1016/j.respol.2004.09.007.
Grupp, H., & Schubert, T. (2010). Review and new evidence on composite innovation indicators for evaluating national performance. Research Policy, 39(1), 67–78. https://doi.org/10.1016/j.respol.2009.10.002.
Gupta, J. N., & Sexton, R. S. (1999). Comparing backpropagation with a genetic algorithm for neural network training. Omega. https://doi.org/10.1016/S0305-0483(99)00027-4.
Hatefi, S. M., & Torabi, S. A. (2010). A common weight MCDA-DEA approach to construct composite indicators. Ecological Economics, 70(1), 114–120. https://doi.org/10.1016/j.ecolecon.2010.08.014.
Hazelkorn, E. (2007). How do rankings impact on higher education? OECD Institutional Management in Higher Education. December 2007, 1–4. Available at http://www.oecd.org/education/imhe/39802910.pdf. Accessed 21 Mar 2018.
Hedges, S. (1992). The number of replications needed for accurate estimation of the bootstrap P value in phylogenetic studies. Molecular Biology and Evolution, 9(2), 366–369. https://doi.org/10.1093/oxfordjournals.molbev.a040725.
Hellwig, Z. (1969). On the problem of weighting in international comparisons. Paris: UNESCO, Methods and Analysis Unit, Department of Social Sciences.
Ivanovic, B. (1977). Teorija klasifikacije. Beograd: Institut za ekonomiku industrije.
Jain, V., Kumar, A., Kumar, S., & Chandra, C. (2015). Weight restrictions in data envelopment analysis: A comprehensive genetic algorithm based approach for incorporating value judgments. Expert Systems with Applications, 42(3), 1503–1512. https://doi.org/10.1016/j.eswa.2014.09.034.
Jeremic, V., Bulajic, M., Martic, M., & Radojicic, Z. (2011). A fresh approach to evaluating the academic ranking of world universities. Scientometrics, 87(3), 587–596. https://doi.org/10.1007/s11192-011-0361-6.
Jeremic, V., Jovanovic Milenkovic, M., Radojicic, Z., & Martic, M. (2013). Excellence with leadership: The crown indicator of Scimago Institutions Rankings Iber report. El Profesional de La Información, 22, 474–480. https://doi.org/10.3145/epi.2013.sep.13.
Jiang, W., & Simon, R. (2007). A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification. Statistics in Medicine, 26(29), 5320–5334. https://doi.org/10.1002/sim.2968.
Joro, T., & Viitala, E.-J. (2004). Weight-restricted DEA in action: From expert opinions to mathematical models. Journal of the Operational Research Society, 55(8), 814–821. https://doi.org/10.1057/palgrave.jors.2601752.
Jovanovic, M., Jeremic, V., Savic, G., Bulajic, M., & Martic, M. (2012). How does the normalization of data affect the ARWU ranking? Scientometrics, 93(2), 319–327. https://doi.org/10.1007/s11192-012-0674-0.
Karaboga, D., Akay, B., & Ozturk, C. (2007). Artificial Bee Colony (ABC) optimization algorithm for training feed-forward neural networks. Modeling Decisions for Artificial Intelligence, 4617, 318–329. https://doi.org/10.1007/978-3-540-73729-2_30.
Kawaguchi, S., & Nishii, R. (2007). Hyperspectral image classification by bootstrap adaboost with random decision stumps. IEEE Transactions on Geoscience and Remote Sensing, 45(11), 3845–3851. https://doi.org/10.1109/TGRS.2007.903708.
Kim, D. Y., Han, S. H., & Kim, H. (2008). Discriminant analysis for predicting ranges of cost variance in international construction projects. Journal of Construction Engineering and Management, 134(6), 398–410. https://doi.org/10.1061/(ASCE)0733-9364(2008)134:6(398).
Kim, K., & Han, I. (2000). Genetic algorithms approach to feature discretization in artificial neural networks for the prediction of stock price index. Expert Systems with Applications, 19(2), 125–132. https://doi.org/10.1016/S0957-4174(00)00027-0.
Kirkman, S., Osorio, A., & Sachs, D. (2002). The networked readiness index: Measuring the preparedness of nations for the networked world. In Kirkman (Ed.), The global information technology report 2001–2002 readiness for the networked world (pp. 10–29). New York: Oxford University Press.
Kleiner, A., Talwalkar, A., Sarkar, P., & Jordan, M. I. (2014). A scalable bootstrap for massive data. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 76(4), 795–816. https://doi.org/10.1111/rssb.12050.
Kline, R. B. (2005). Principles and practice of structural equation modeling. Principles and practice of strucutral equation modeling (2nd ed., Vol. 156). New York: The Guliford press. https://doi.org/10.1038/156278a0.
Koch, P. N., Yang, R. J., & Gu, L. (2004). Design for six sigma through robust optimization. Structural and Multidisciplinary Optimization, 26(3–4), 235–248. https://doi.org/10.1007/s00158-003-0337-0.
Konietschke, F., Bathke, A. C., Harrar, S. W., & Pauly, M. (2015). Parametric and nonparametric bootstrap methods for general MANOVA. Journal of Multivariate Analysis, 140, 291–301. https://doi.org/10.1016/j.jmva.2015.05.001.
Lagarias, J. C., Reeds, J. A., Wright, M. H., & Wright, P. E. (1998). Convergence properties of the Nelder–Mead simplex method in low dimensions. SIAM Journal on Optimization, 9(1), 112–147. https://doi.org/10.1137/S1052623496303470.
Lahdelma, R., Hokkanen, J., & Salminen, P. (1998). SMAA—Stochastic multiobjective acceptability analysis. European Journal of Operational Research, 106(1), 137–143. https://doi.org/10.1016/S0377-2217(97)00163-X.
Ligmann-Zielinska, A., & Jankowski, P. (2008). A framework for sensitivity analysis in spatial multiple criteria evaluation. In T. J. Cova, H. J. Miller, K. Beard, A. U. Frank & M. F. Goodchild (Eds.), Geographic information science (pp. 217–233). Berlin: Springer. https://doi.org/10.1007/978-3-540-87473-7_14.
Liu, N. C., & Cheng, Y. (2005). The academic ranking of world universities. Higher Education in Europe, 30(2), 127–136. https://doi.org/10.1080/03797720500260116.
Macedo, L. L., Godinho, P., & Alves, M. J. (2017). Mean-semivariance portfolio optimization with multiobjective evolutionary algorithms and technical analysis rules. Expert Systems with Applications, 79, 33–43. https://doi.org/10.1016/j.eswa.2017.02.033.
Macharis, C., Springael, J., De Brucker, K., & Verbeke, A. (2004). PROMETHEE and AHP: The design of operational synergies in multicriteria analysis. European Journal of Operational Research, 153(2), 307–317. https://doi.org/10.1016/S0377-2217(03)00153-X.
Maggino, F. (2009). Towards more participative methods in the construction of social indicators: Survey techniques aimed at determining importance weights. In 62nd Annual conference of the world association for public opinion research on public opinion and survey research in a changing world. Lausanne, Switzerland. Retrieved from https://flore.unifi.it/retrieve/handle/2158/370603/11441/Maggino_-_paper.pdf.
Mammen, E. (1992). When does bootstrap work? (Vol. 77). New York, NY: Springer. https://doi.org/10.1007/978-1-4612-2950-6.
Mariano, E. B., Sobreiro, V. A., & do Nascimento Rebelatto, D. A. (2015). Human development and data envelopment analysis: A structured literature review. Omega, 54, 33–49. https://doi.org/10.1016/j.omega.2015.01.002.
Maricic, M., Bulajic, M., Martic, M., & Dobrota, M. (2015). Measuring the ict development: The fusion of biased and objective approach. Naval Academy Scientific Bulletin, 18(2), 326–334.
Maricic, M., Bulajic, M., Radojicic, Z., & Jeremic, V. (2016). Multivariate approach to imposing additional constraints on the benefit-of-the-doubt model: The case of QS world university rankings by subject. Croatian Review of Economic, Business and Social Statistics, 2(1), 1–14. https://doi.org/10.1515/crebss-2016-0005.
Maricic, M., & Kostic-Stankovic, M. (2016). Towards an impartial Responsible Competitiveness Index: A twofold multivariate I-distance approach. Quality & Quantity, 50(1), 103–120. https://doi.org/10.1007/s11135-014-0139-z.
Maricic, M., Zornic, N., Pilcevic, I., & Dacic-Pilcevic, A. (2017). ARWU vs. alternative ARWU ranking: What are the consequences for lower ranked universities? Management: Journal of Sustainable Business and Management Solutions in Emerging Economies, 22(1), 1–14. https://doi.org/10.7595/management.fon.2017.0002.
Marković, M., Zdravković, S., Mitrović, M., & Radojičić, A. (2016). An iterative multivariate post hoc I-distance approach in evaluating OECD Better Life Index. Social Indicators Research, 126(1), 1–19. https://doi.org/10.1007/s11205-015-0879-8.
Mazziotta, M., & Pareto, A. (2007). Un indicatore sintetico di dotazione infrastrutturale: il metodo delle penalità per coefciente di variazione. In Lo sviluppo regionale nell’Unione Europea-Obiettivi, strategie, politiche. Atti della XXVIII Conferenza Italiana di Scienze Regionali. Bolzano.
Mazziotta, M., & Pareto, A. (2012). Methods for constructing composite indices: One for all or all for one. Rivista Italiana Di Economia Demografia e Statistica, 67(2), 67–80.
Melyn, W., & Moesen, W. (1991). Towards a synthetic indicator of macroeconomic performance: Unequal weighting when limited information is available. Leuven University, Working paper ID 26175691.
Milenkovic, M. J., Brajovic, B., Milenkovic, D., Vukmirovic, D., & Jeremic, V. (2016). Beyond the equal-weight framework of the Networked Readiness Index: A multilevel I-distance methodology. Information Development, 32(4), 1120–1136. https://doi.org/10.1177/0266666915593136.
Mirjalili, S., Mohd Hashim, S. Z., & Moradian Sardroudi, H. (2012). Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm. Applied Mathematics and Computation, 218(22), 11125–11137. https://doi.org/10.1016/j.amc.2012.04.069.
Moed, H. F. (2017). A critical comparative analysis of five world university rankings. Scientometrics, 110(2), 967–990. https://doi.org/10.1007/s11192-016-2212-y.
Mousavi-Nasab, S. H., & Sotoudeh-Anvari, A. (2018). A new multi-criteria decision making approach for sustainable material selection problem: A critical study on rank reversal problem. Journal of Cleaner Production, 182, 466–484. https://doi.org/10.1016/j.jclepro.2018.02.062.
Munda, G. (2005a). “Measuring sustainability”: A multi-criterion framework. Environment, Development and Sustainability, 7(1), 117–134. https://doi.org/10.1007/s10668-003-4713-0.
Munda, G. (2005b). Multiple criteria decision analysis and sustainable development. In S. Greco, M. Ehrgott, & J. Figueira (Eds.), Multiple criteria decision analysis: State of the art surveys (pp. 953–986). New York: Springer.
Munda, G., & Nardo, M. (2009). Noncompensatory/nonlinear composite indicators for ranking countries: A defensible setting. Applied Economics, 41(12), 1513–1523. https://doi.org/10.1080/00036840601019364.
Murias, P., de Miguel, J. C., & Rodríguez, D. (2008). A composite indicator for university quality assesment: The case of Spanish higher education system. Social Indicators Research, 89(1), 129–146. https://doi.org/10.1007/s11205-007-9226-z.
Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffman, A., & Giovannini, E. (2005). Handbook on constructing composite indicators. OECD Statistics Working Papers. https://doi.org/10.1787/533411815016.
Nevitt, J., & Hancock, G. R. (2001). Performance of bootstrapping approaches to model test statistics and parameter standard error estimation in structural equation modeling—Google Académico. Structural Equation Modeling: A Multidisciplinary Journal, 8(3), 353–377. https://doi.org/10.1207/S15328007SEM0803_2.
OECD. (2004). The OECD-JRC handbook on practices for developing composite indicators. Paris: OECD.
Otero-Muras, I., & Banga, J. R. (2014). Multicriteria global optimization for biocircuit design. BMC Systems Biology, 8(1), 113. https://doi.org/10.1186/s12918-014-0113-3.
Paracchini, M. L., Pacini, C., Calvo, S., & Vogt, J. (2008). Weighting and aggregation of indicators for sustainability impact assessment in the SENSOR context. In K. Helming, M. Pérez-Soba & P. Tabbush (Eds.), Sustainability impact assessment of land use changes (pp. 349–372). Berlin: Springer. https://doi.org/10.1007/978-3-540-78648-1_18.
Paruolo, P., Saisana, M., & Saltelli, A. (2013). Ratings and rankings: Voodoo or science? Journal of the Royal Statistical Society: Series A (Statistics in Society), 176(3), 609–634. https://doi.org/10.1111/j.1467-985X.2012.01059.x.
Pattengale, N. D., Alipour, M., Bininda-Emonds, O. R. P., Moret, B. M. E., & Stamatakis, A. (2009). How many bootstrap replicates are necessary? In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, C. Pandu Rangan, B. Steffen, D. Terzopoulos & D. Tygar (Eds.), Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Vol. 5541 LNBI, pp. 184–200). https://doi.org/10.1007/978-3-642-02008-7_13.
Penas, D. R., González, P., Egea, J. A., Banga, J. R., & Doallo, R. (2015). Parallel metaheuristics in computational biology: An asynchronous cooperative enhanced scatter search method. Procedia Computer Science, 51, 630–639. https://doi.org/10.1016/j.procs.2015.05.331.
Penas, D. R., González, P., Egea, J. A., Doallo, R., & Banga, J. R. (2017). Parameter estimation in large-scale systems biology models: A parallel and self-adaptive cooperative strategy. BMC Bioinformatics, 18(1), 52. https://doi.org/10.1186/s12859-016-1452-4.
Pérez, V., Guerrero, F., González, M., Pérez, F., & Caballero, R. (2013). Composite indicator for the assessment of sustainability: The case of Cuban nature-based tourism destinations. Ecological Indicators, 29, 316–324. https://doi.org/10.1016/j.ecolind.2012.12.027.
Permanyer, I. (2011). Assessing the robustness of composite indices rankings. Review of Income and Wealth, 57(2), 306–326. https://doi.org/10.1111/j.1475-4991.2011.00442.x.
Politis, D. N., & Romano, J. P. (1994). Large sample confidence regions based on subsamples under minimal assumptions. The Annals of Statistics, 22(4), 2031–2050. https://doi.org/10.1214/aos/1176325770.
Politis, D. N., Romano, J. P., & Wolf, M. (1999). Subsampling. New York, NY: Springer. https://doi.org/10.1007/978-1-4612-1554-7.
Pollesch, N. L., & Dale, V. H. (2016). Normalization in sustainability assessment: Methods and implications. Ecological Economics, 130, 195–208. https://doi.org/10.1016/j.ecolecon.2016.06.018.
Radojicic, M., Savic, G., & Jeremic, V. (2018). Measuring the efficiency of banks: The bootstrapped I-distance gar dea approach. Technological and Economic Development of Economy, 24(4), 1581–1605. https://doi.org/10.3846/tede.2018.3699.
Radojicic, M., Savic, G., Radovanovic, S., & Jeremic, V. (2015). A novel bootstrap dba-dea approach in evaluating efficiency of banks. Naval Academy Scientific Bulletin, 18(2), 375–384.
Ramón, N., Ruiz, J. L., & Sirvent, I. (2012). Common sets of weights as summaries of DEA profiles of weights: With an application to the ranking of professional tennis players. Expert Systems with Applications, 39(5), 4882–4889. https://doi.org/10.1016/j.eswa.2011.10.004.
Ray, A. K. (2007). Measurement of social development: An international comparison. Social Indicators Research, 86(1), 1–46. https://doi.org/10.1007/s11205-007-9097-3.
Reggi, L., Arduini, D., Biagetti, M., & Zanfei, A. (2014). How advanced are Italian regions in terms of public e-services? The construction of a composite indicator to analyze patterns of innovation diffusion in the public sector. Telecommunications Policy, 38(5–6), 514–529. https://doi.org/10.1016/j.telpol.2013.12.005.
Remli, M. A., Deris, S., Mohamad, M. S., Omatu, S., & Corchado, J. M. (2017). An enhanced scatter search with combined opposition-based learning for parameter estimation in large-scale kinetic models of biochemical systems. Engineering Applications of Artificial Intelligence, 62, 164–180. https://doi.org/10.1016/j.engappai.2017.04.004.
Renzi, G., Canfora, L., Salvati, L., & Benedetti, A. (2017). Validation of the soil Biological Fertility Index (BFI) using a multidimensional statistical approach: A country-scale exercise. CATENA, 149, 294–299. https://doi.org/10.1016/j.catena.2016.10.002.
Rodriguez-Fernandez, M., Egea, J. A., & Banga, J. R. (2006). Novel metaheuristic for parameter estimation in nonlinear dynamic biological systems. BMC Bioinformatics, 7(1), 483. https://doi.org/10.1186/1471-2105-7-483.
Rogge, N. (2012). Undesirable specialization in the construction of composite policy indicators: The Environmental Performance Index. Ecological Indicators, 23, 143–154. https://doi.org/10.1016/j.ecolind.2012.03.020.
Rogge, N. (2018a). Composite indicators as generalized benefit-of-the-doubt weighted averages. European Journal of Operational Research, 267(1), 381–392. https://doi.org/10.1016/j.ejor.2017.11.048.
Rogge, N. (2018b). On aggregating benefit of the doubt composite indicators. European Journal of Operational Research, 264(1), 364–369. https://doi.org/10.1016/j.ejor.2017.06.035.
Saisana, M., & D’Hombres, B. (2008). Higher education rankings: Robustness issues and critical assessment. How much confidence can we have in higher education rankings? JRC. https://doi.org/10.2788/92295.
Saisana, M., D’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university rankings and policy implications. Research Policy, 40(1), 165–177. https://doi.org/10.1016/j.respol.2010.09.003.
Saisana, M., & Saltelli, A. (2014). JCR statistical audit of the WJP Rule of Law index 2014. In World justice project: The world justice project Rule of Law index 2014 (pp. 188–197). Washington, DC: World Justice Project.
Saisana, M., Saltelli, A., & Tarantola, S. (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Society: Series A (Statistics in Society), 168(2), 307–323. https://doi.org/10.1111/j.1467-985X.2005.00350.x.
Saisana, M., & Tarantola, S. (2002). State-of-the-art re port on current methodologies and practices for composite indicator development. Retrieved December 20, 2017 from http://bookshop.europa.eu/en/state-of-the-art-report-on-current-methodologies-and-practices-for-composite-indicator-development-pbEUNA20408/%2520.
Saltelli, A. (2007). Composite indicators between analysis and advocacy. Social Indicators Research, 81(1), 65–77. https://doi.org/10.1007/s11205-006-0024-9.
Saltelli, A., Chan, K., & Scott, E. M. (2000). Sensitivity analysis. Wiley series in probability and statistics. Hoboken: Wiley.
Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., et al. (2007). Global sensitivity analysis. The primer. Chichester: Wiley. https://doi.org/10.1002/9780470725184.
Savic, D., Jeremic, V., & Petrovic, N. (2016). Rebuilding the pillars of sustainable society index: A multivariate post hoc I-distance approach. Problemy Ekorozwoju – Problems of Sustainable Development, 12(1), 125–134.
Sayed, H., Hamed, R., Hosny, S. H., & Abdelhamid, A. H. (2018). Avoiding ranking contradictions in human development index using goal programming. Social Indicators Research, 138(2), 405–442. https://doi.org/10.1007/s11205-017-1663-8.
Sayed, H., Hamed, R., Ramadan, M. A.-G., & Hosny, S. (2015). Using meta-goal programming for a new human development indicator with distinguishable country ranks. Social Indicators Research, 123(1), 1–27. https://doi.org/10.1007/s11205-014-0723-6.
Sébastien, L., Bauler, T., & Lehtonen, M. (2014). Can indicators bridge the gap between science and policy? An exploration into the (non)use and (non)influence of indicators in EU and UK policy making. Nature and Culture. https://doi.org/10.3167/nc.2014.090305.
Sevigny, E. L., & Saisana, M. (2016). Measuring interstate variations in the consequences of illegal drugs: A composite indicator approach. Social Indicators Research, 128(2), 501–529. https://doi.org/10.1007/s11205-015-1042-2.
ShanghaiRanking. (2017). ARWU world university rankings 2017. Academic ranking of world universities 2017. Top 500 universities. Shanghai ranking—2017. Retrieved February 14, 2018, from http://www.shanghairanking.com/ARWU2017.html.
ShanghaiRanking. (2018). Academic ranking of world universities 2018. Retreived December 24, 2018 from http://www.shanhairanking.com/ARWU2018.html.
Shattock, M. (2017). The ‘world class’ university and international ranking systems: what are the policy implications for governments and institutions? Policy Reviews in Higher Education, 1(1), 4–21. https://doi.org/10.1080/23322969.2016.1236669.
Singh, R. K., Murty, H. R., Gupta, S. K., & Dikshit, A. K. (2007). Development of composite sustainability performance index for steel industry. Ecological Indicators, 7, 565–588. https://doi.org/10.1016/j.ecolind.2006.06.004.
Socha, K., & Blum, C. (2007). An ant colony optimization algorithm for continuous optimization: Application to feed-forward neural network training. Neural Computing and Applications, 16(3), 235–247. https://doi.org/10.1007/s00521-007-0084-z.
Sörensen, K., Sevaux, M., & Glover, F. (2018). A history of metaheuristics. In R. Martí, P. Panos & M. G. C. Resende (Eds.), Handbook of heuristics (pp. 1–18). Cham: Springer. https://doi.org/10.1007/978-3-319-07153-4_4-1.
Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8, 25.
Taghdisian, H., Pishvaie, M. R., & Farhadi, F. (2015). Multi-objective optimization approach for green design of methanol plant based on CO2-efficeincy indicator. Journal of Cleaner Production, 103, 640–650. https://doi.org/10.1016/j.jclepro.2014.05.032.
Talukder, B., Hipel, K., & VanLoon, G. (2017). Developing composite indicators for agricultural sustainability assessment: Effect of normalization and aggregation techniques. Resources, 6(4), 66. https://doi.org/10.3390/resources6040066.
Van Puyenbroeck, T., & Rogge, N. (2017). Geometric mean quantity index numbers with benefit-of-the-doubt weights. European Journal of Operational Research, 256(3), 1004–1014. https://doi.org/10.1016/j.ejor.2016.07.038.
Verbunt, P., & Rogge, N. (2018). Geometric composite indicators with compromise benefit-of-the-doubt weights. European Journal of Operational Research, 264(1), 388–401. https://doi.org/10.1016/j.ejor.2017.06.061.
Villaverde, A., Egea, J. A., & Banga, J. R. (2012). A cooperative strategy for parameter estimation in large scale systems biology models. BMC Systems Biology, 6(1), 75. https://doi.org/10.1186/1752-0509-6-75.
Wang, J. J., Jing, Y. Y., Zhang, C. F., & Zhao, J. H. (2009). Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renewable and Sustainable Energy Reviews. https://doi.org/10.1016/j.rser.2009.06.021.
Wang, X., & Triantaphyllou, E. (2008). Ranking irregularities when evaluating alternatives by using some ELECTRE methods. Omega, 36(1), 45–63. https://doi.org/10.1016/j.omega.2005.12.003.
Wang, Y.-M., & Luo, Y. (2009). On rank reversal in decision analysis. Mathematical and Computer Modelling, 49(5–6), 1221–1229. https://doi.org/10.1016/j.mcm.2008.06.019.
World Economic Forum. (2016a). Global information technology report 2016—Reports—World economic forum. Retrieved December 23, 2017, from http://reports.weforum.org/global-information-technology-report-2016/.
World Economic Forum. (2016b). Structure and methodology. Retrieved September 10, 2017, from http://reports.weforum.org/global-information-technology-report-2015/structure-and-methodology/.
Xu, L. W., Yang, F. Q., Abula, A., & Qin, S. (2013). A parametric bootstrap approach for two-way ANOVA in presence of possible interactions with unequal variances. Journal of Multivariate Analysis, 115, 172–180. https://doi.org/10.1016/j.jmva.2012.10.008.
Yang, W., Mckinnon, M. C., & Turner, W. R. (2015). Quantifying human well-being for sustainability research and policy. Ecosystem Health and Sustainability, 1(4), 1–13. https://doi.org/10.1890/EHS15-0004.1.
Zanakis, S. H., Newburry, W., & Taras, V. (2016). Global social tolerance index and multi-method country rankings sensitivity. Journal of International Business Studies, 47(4), 480–497. https://doi.org/10.1057/jibs.2016.5.
Zanella, A., Camanho, A. S., & Dias, T. G. (2015). Undesirable outputs and weighting schemes in composite indicators based on data envelopment analysis. European Journal of Operational Research, 245(2), 517–530. https://doi.org/10.1016/j.ejor.2015.03.036.
Zhou, P., Ang, B. W., & Poh, K. L. (2006). Comparing aggregating methods for constructing the composite environmental index: An objective measure. Ecological Economics, 59(3), 305–311. https://doi.org/10.1016/j.ecolecon.2005.10.018.
Zhou, P., Ang, B. W., & Zhou, D. Q. (2010). Weighting and aggregation in composite indicator construction: A multiplicative optimization approach. Social Indicators Research, 96(1), 169–181. https://doi.org/10.1007/s11205-009-9472-3.
Zornic, N., Bornmann, L., Maricic, M., Markovic, A., Martic, M., & Jeremic, V. (2015). Ranking institutions within a university based on their scientific performance: A percentile-based approach. El Profesional de La Información, 24(5), 551–566. https://doi.org/10.3145/epi.2015.sep.05.
Zuur, A. F., Ieno, E. N., & Elphick, C. S. (2010). A protocol for data exploration to avoid common statistical problems. Methods in Ecology and Evolution, 1(1), 3–14. https://doi.org/10.1111/j.2041-210X.2009.00001.x.
Acknowledgements
The authors thank the anonymous Reviewers and the Editor for their insightful comments and suggestions which helped us enhance the original version of the manuscript.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Maricic, M., Egea, J.A. & Jeremic, V. A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators. Soc Indic Res 144, 497–537 (2019). https://doi.org/10.1007/s11205-018-02056-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11205-018-02056-x