A Hybrid Enhanced Scatter Search—Composite I-Distance Indicator (eSS-CIDI) Optimization Approach for Determining Weights Within Composite Indicators

  • Milica Maricic
  • Jose A. Egea
  • Veljko Jeremic
Original Research


Considering the impact composite indicators can have on public opinion and policy development, the need for their frameworks to be methodologically sound and statistically verified is growing daily. One of the issues in the process of composite indicator construction which has generated much debate is how to choose the weighting scheme. To address this slippery step, we propose an optimization approach based on the enhanced Scatter Search (eSS) metaheuristic. In this paper, the eSS algorithm is applied to obtain a weighting scheme which will increase the stability of the composite indicator. The objective function is based on the relative contributions of indicators, while the problem constraints rely on the bootstrap Composite I-distance Indicator (CIDI) approach which is also proposed herein. The eSS-CIDI approach combines the exploration capability of eSS and the data-driven constraints devised from the bootstrap CIDI. This novel weighting approach was tested on two acknowledged composite indicators: the Academic Ranking of World Universities (ARWU) and the Networked Readiness Index (NRI). Results indicate that the composite indicators created using the eSS-CIDI weighting approach are more stable than the official ones.


Composite indicators Optimization Weighting scheme Enhanced Scatter Search Bootstrap Composite I-distance indicator approach 



The authors thank the anonymous Reviewers and the Editor for their insightful comments and suggestions which helped us enhance the original version of the manuscript.


  1. Ács, Z. J., Autio, E., & Szerb, L. (2014). National systems of entrepreneurship: Measurement issues and policy implications. Research Policy, 43(3), 476–494. Scholar
  2. Amado, C. A. F., São José, J. M. S., & Santos, S. P. (2016). Measuring active ageing: A data envelopment analysis approach. European Journal of Operational Research, 255, 207–223. Scholar
  3. Angilella, S., Catalfo, P., Corrente, S., Giarlotta, A., Greco, S., & Rizzo, M. (2018). Robust sustainable development assessment with composite indices aggregating interacting dimensions: The hierarchical-SMAA-Choquet integral approach. Knowledge-Based Systems, 158, 136–153. Scholar
  4. Antony, J., Swarnkar, R., Kumar, M., & Tiwari, M. K. (2006). Design of synchronised supply chain: A genetic algorithm based six sigma constrained approach. International Journal of Logistics Systems and Management, 2, 120–140.CrossRefGoogle Scholar
  5. Arcones, M. A., & Gine, E. (1989). The bootstrap of the mean with arbitrary bootstrap sample size. Annales de l’I.H.P. Probabilités et Statistiques, 25(4), 457–481.Google Scholar
  6. Austin, P. C., & Small, D. S. (2014). The use of bootstrapping when using propensity-score matching without replacement: A simulation study. Statistics in Medicine, 33(24), 4306–4319. Scholar
  7. Banerjee, A. K. (2018). Multidimensional indices with data-driven dimensional weights: A multidimensional coefficient of variation. Arthaniti: Journal of Economic Theory and Practice. Scholar
  8. Becker, W., Saisana, M., Paruolo, P., & Vandecasteele, I. (2017). Weights and importance in composite indicators: Closing the gap. Ecological Indicators, 80, 12–22. Scholar
  9. Beran, R., & Ducharme, G. (1991). Asymptotic theory for bootstrap methods in statistics. Montréal: Centre de Recherches Mathematiques.Google Scholar
  10. Bickel, P. J., Gotze, F., & van Zwet, W. R. (1997). Resampling fewer than n observations: Gains, losses and remedies for losses. Statistica Sinica, 7, 1–31.Google Scholar
  11. Bickel, P. J., & Sakov, A. (2008). On the choice of m in the m out of n bootstrap and its application to confidence bounds for extreme percentiles. Statistica Sinica, 18, 967–985.Google Scholar
  12. Bilbao-Osorio, B., Dutta, S., & Lanvin, B. (2014). The global information technology report 2014. Rewards and risks of big data.Google Scholar
  13. Birkmann, J. (2007). Risk and vulnerability indicators at different scales: Applicability, usefulness and policy implications. Environmental Hazards, 7(1), 20–31. Scholar
  14. Blancas, F. J., Contreras, I., & Ramírez-Hurtado, J. M. (2013). Constructing a composite indicator with multiplicative aggregation under the objective of ranking alternatives. Journal of the Operational Research Society, 64(5), 668–678. Scholar
  15. Blum, C., & Roli, A. (2003). Metaheuristics in combinatorial optimization: Overview and conceptual comparison. ACM Computing Surveys, 35(3), 189–213. Scholar
  16. Bojórquez-Tapia, L. A., Sánchez-Colon, S., & Florez, A. (2005). Building consensus in environmental impact assessment through multicriteria modeling and sensitivity analysis. Environmental Management, 36(3), 469–481. Scholar
  17. Booysen, F. (2002). An overview and evaluation of composite indices of development. Social Indicators Research, 59(2), 115–151. Scholar
  18. Braga-Neto, U. M., & Dougherty, E. R. (2004). Is cross-validation valid for small-sample microarray classification? Bioinformatics, 20(3), 374–380. Scholar
  19. Bretagnolle, J. (1983). Lois limites du bootstrap de certaines fonctionnelles. Annales de l’IHP Probabilités et Statistiques, 19(3), 281–296.Google Scholar
  20. Burgass, M. J., Halpern, B. S., Nicholson, E., & Milner-Gulland, E. J. (2017). Navigating uncertainty in environmental composite indicators. Ecological Indicators, 75, 268–278. Scholar
  21. Casadio Tarabusi, E., & Guarini, G. (2013). An unbalance adjustment method for development indicators. Social Indicators Research, 112(1), 19–45. Scholar
  22. Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429–444. Scholar
  23. Cherchye, L., Moesen, W., Rogge, N., & Van Puyenbroeck, T. (2007). An introduction to “benefit of the doubt” composite indicators. Social Indicators Research, 82(1), 111–145. Scholar
  24. Cherchye, L., Moesen, W., Rogge, N., Van Puyenbroeck, T., Saisana, M., Saltelli, A., et al. (2008a). Creating composite indicators with DEA and robustness analysis: The case of the Technology Achievement Index. Journal of the Operational Research Society, 59(2), 239–251. Scholar
  25. Cherchye, L., Ooghe, E., & Van Puyenbroeck, T. (2008b). Robust human development rankings. The Journal of Economic Inequality, 6(4), 287–321. Scholar
  26. Choquet, G. (1953). Theory of capacities. Annales de l’Institut Fourier, 5(54), 131–295.Google Scholar
  27. Chowdhury, S., Moral, R. J., & Dulikravich, G. S. (2008). Predator-prey evolutionary multi-objective optimization algorithm: Performance and improvements. In Proceedings of 7th ASMO-UK/ISSMO international conference on engineering design optimization (pp. 1–10). Bath, UK. Retrieved from
  28. Cole, T. J. (2015a). Setting number of decimal places for reporting risk ratios: Rule of four. The BMJ, 350, h1845. Scholar
  29. Cole, T. J. (2015b). Too many digits: The presentation of numerical data. Archives of Disease in Childhood, 100(7), 608–609. Scholar
  30. Corrente, S., Greco, S., Nicotra, M., Romano, M., & Schillaci, C. E. (2018). Evaluating and comparing entrepreneurial ecosystems using SMAA and SMAA-S. The Journal of Technology Transfer. Scholar
  31. Cox, D. R., Fitzpatrick, R., Fletcher, A. E., Gore, S. M., Spiegelhalter, D. J., & Jones, D. R. (1992). Quality-of-life assessment: Can we keep it simple? Journal of the Royal Statistical Society. Series A, 155(3), 353–393. Scholar
  32. Daraio, C., & Bonaccorsi, A. (2017). Beyond university rankings? Generating new indicators on universities by linking data in open platforms. Journal of the Association for Information Science and Technology, 68(2), 508–529. Scholar
  33. Davison, A. C., Hinkley, D. V., & Young, G. A. (2003). Recent developments in bootstrap methodology. Statistical Science, 18(2), 141–157. Scholar
  34. De Bin, R., Janitza, S., Sauerbrei, W., & Boulesteix, A. L. (2016). Subsampling versus bootstrapping in resampling-based model selection for multivariable regression. Biometrics, 72(1), 272–280. Scholar
  35. Decancq, K., & Lugo, M. A. (2013). Weights in multidimensional indices of wellbeing: An overview. Econometric Reviews, 32(1), 7–34. Scholar
  36. Dehon, C., McCathie, A., & Verardi, V. (2010). Uncovering excellence in academic rankings: a closer look at the Shanghai ranking. Scientometrics, 83(2), 515–524. Scholar
  37. Despotis, D. K. (2005). A reassessment of the human development index via data envelopment analysis. Journal of the Operational Research Society, 56(8), 969–980. Scholar
  38. Dialga, I., & Thi Hang Giang, L. (2017). Highlighting methodological limitations in the steps of composite indicators construction. Social Indicators Research, 131(2), 441–465. Scholar
  39. Dijkstra, L., Annoni, P., & Kozovska, K. (2011). A series of short papers on regional research and indicators produced by the Directorate-General for Regional Policy.Google Scholar
  40. Dobbie, M. J., & Dail, D. (2013). Robustness and sensitivity of weighting and aggregation in constructing composite indices. Ecological Indicators, 29, 270–277. Scholar
  41. Dobrota, M., Bulajic, M., Bornmann, L., & Jeremic, V. (2016). A new approach to the QS university ranking using the composite I-distance indicator: Uncertainty and sensitivity analyses. Journal of the Association for Information Science and Technology, 67(1), 200–211. Scholar
  42. Dobrota, M., & Dobrota, M. (2016). ARWU ranking uncertainty and sensitivity: What if the award factor was excluded? Journal of the Association for Information Science and Technology, 67(2), 480–482. Scholar
  43. Dobrota, M., & Jeremic, V. (2017). Shedding the light on the stability of university rankings in the ICT field. IETE Technical Review, 34(1), 75–82. Scholar
  44. Dobrota, M., Martic, M., Bulajic, M., & Jeremic, V. (2015). Two-phased composite I-distance indicator approach for evaluation of countries’ information development. Telecommunications Policy, 39(5), 406–420. Scholar
  45. Docampo, D., & Cram, L. (2015). On the effects of institutional size in university classifications: The case of the Shanghai ranking. Scientometrics, 102(2), 1325–1346. Scholar
  46. Dubey, H. M., Pandit, M., & Panigrahi, B. K. (2016). Hydro-thermal-wind scheduling employing novel ant lion optimization technique with composite ranking index. Renewable Energy, 99, 18–34. Scholar
  47. Ebert, U., & Welsch, H. (2004). Meaningful environmental indices: A social choice approach. Journal of Environmental Economics and Management, 47(2), 270–283. Scholar
  48. Edirisinghe, N. C. P., & Zhang, X. (2010). Input/output selection in DEA under expert information, with application to financial markets. European Journal of Operational Research, 207(3), 1669–1678. Scholar
  49. Efron, B. (1979). Bootstrap methods: Another look at the jacknife. The Annals of Statistics, 7(1), 1–26. Scholar
  50. Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. Boca Raton: Chapman & Hall/CRC.CrossRefGoogle Scholar
  51. Efron, B., & Tibshirani, R. (1997). Improvements on cross-validation: The 632 + bootstrap method. Journal of the American Statistical Association, 92(438), 548–560. Scholar
  52. Egea, J. A., Balsa-Canto, E., García, M.-S. G., & Banga, J. R. (2009). Dynamic optimization of nonlinear processes with an enhanced scatter search method. Industrial and Engineering Chemistry Research, 48(9), 4388–4401. Scholar
  53. Egea, J. A., Henriques, D., Cokelaer, T., Villaverde, A., MacNamara, A., Danciu, D.-P., et al. (2014). MEIGO: An open-source software suite based on metaheuristics for global optimization in systems biology and bioinformatics. BMC Bioinformatics, 15(1), 136. Scholar
  54. Ferrier, G. D., & Hirschberg, J. G. (1999). Can we bootstrap DEA scores? Journal of Productivity Analysis, 11(1), 81–92. Scholar
  55. Foster, J. E., McGillivray, M., & Seth, S. (2013). Composite indices: Rank robustness, statistical association, and redundancy. Econometric Reviews, 32(1), 35–56. Scholar
  56. Freudenberg, M. (2003). Composite indicators of country performance: A critical assessment. OECD Science, Technology and Industry Working Papers, 16, 35. Scholar
  57. Fusco, E. (2015). Enhancing non-compensatory composite indicators: A directional proposal. European Journal of Operational Research, 242(2), 620–630. Scholar
  58. Gen, M., & Cheng, R. (2000). Genetic algorithms and engineering optimization. Hoboken: Wiley.Google Scholar
  59. Gendreau, M., & Potvin, J. Y. (2010). Handbook of metaheuristics (Vol. 146). Berlin: Springer. Scholar
  60. Geyer, C. (2013). 5601 Notes: The subsampling bootstrap. Retrieved February 10, 2018 from
  61. Giambona, F., & Vassallo, E. (2013). Composite indicator of financial development in a benefit-of-doubt approach. Economic Notes, 42(2), 171–202. Scholar
  62. Giannetti, B. F., Bonilla, S. H., Silva, C. C., & Almeida, C. M. V. B. (2009). The reliability of experts’ opinions in constructing a composite environmental index: The case of ESI 2005. Journal of Environmental Management, 90(8), 2448–2459. Scholar
  63. Glover, F. (1977). Heuristics for integer programming using surrogate constraints. Decision Sciences, 8(1), 156–166. Scholar
  64. Glover, F., & Kochenberger, G. A. (2003). Handbook of metaheuristics. Dordrecht: Kluwer Academic Publishers.Google Scholar
  65. Glover, F., Laguna, M., & Martí, R. (2000). Fundamentals of scatter search and path relinking. Control and Cybernetics, 29(3), 652–684. Scholar
  66. Greco, S., Ishizaka, A., Matarazzo, B., & Torrisi, G. (2018a). Stochastic multi-attribute acceptability analysis (SMAA): An application to the ranking of Italian regions. Regional Studies, 52(4), 585–600. Scholar
  67. Greco, S., Ishizaka, A., Tasiou, M., & Torrisi, G. (2018b). On the methodological framework of composite indices: A review of the issues of weighting, aggregation, and robustness. Social Indicators Research. Scholar
  68. Grupp, H., & Mogee, M. E. (2004). Indicators for national science and technology policy: How robust are composite indicators? Research Policy, 33(9), 1373–1384. Scholar
  69. Grupp, H., & Schubert, T. (2010). Review and new evidence on composite innovation indicators for evaluating national performance. Research Policy, 39(1), 67–78. Scholar
  70. Gupta, J. N., & Sexton, R. S. (1999). Comparing backpropagation with a genetic algorithm for neural network training. Omega. Scholar
  71. Hatefi, S. M., & Torabi, S. A. (2010). A common weight MCDA-DEA approach to construct composite indicators. Ecological Economics, 70(1), 114–120. Scholar
  72. Hazelkorn, E. (2007). How do rankings impact on higher education? OECD Institutional Management in Higher Education. December 2007, 1–4. Available at Accessed 21 Mar 2018.
  73. Hedges, S. (1992). The number of replications needed for accurate estimation of the bootstrap P value in phylogenetic studies. Molecular Biology and Evolution, 9(2), 366–369. Scholar
  74. Hellwig, Z. (1969). On the problem of weighting in international comparisons. Paris: UNESCO, Methods and Analysis Unit, Department of Social Sciences.Google Scholar
  75. Ivanovic, B. (1977). Teorija klasifikacije. Beograd: Institut za ekonomiku industrije.Google Scholar
  76. Jain, V., Kumar, A., Kumar, S., & Chandra, C. (2015). Weight restrictions in data envelopment analysis: A comprehensive genetic algorithm based approach for incorporating value judgments. Expert Systems with Applications, 42(3), 1503–1512. Scholar
  77. Jeremic, V., Bulajic, M., Martic, M., & Radojicic, Z. (2011). A fresh approach to evaluating the academic ranking of world universities. Scientometrics, 87(3), 587–596. Scholar
  78. Jeremic, V., Jovanovic Milenkovic, M., Radojicic, Z., & Martic, M. (2013). Excellence with leadership: The crown indicator of Scimago Institutions Rankings Iber report. El Profesional de La Información, 22, 474–480. Scholar
  79. Jiang, W., & Simon, R. (2007). A comparison of bootstrap methods and an adjusted bootstrap approach for estimating the prediction error in microarray classification. Statistics in Medicine, 26(29), 5320–5334. Scholar
  80. Joro, T., & Viitala, E.-J. (2004). Weight-restricted DEA in action: From expert opinions to mathematical models. Journal of the Operational Research Society, 55(8), 814–821. Scholar
  81. Jovanovic, M., Jeremic, V., Savic, G., Bulajic, M., & Martic, M. (2012). How does the normalization of data affect the ARWU ranking? Scientometrics, 93(2), 319–327. Scholar
  82. Karaboga, D., Akay, B., & Ozturk, C. (2007). Artificial Bee Colony (ABC) optimization algorithm for training feed-forward neural networks. Modeling Decisions for Artificial Intelligence, 4617, 318–329. Scholar
  83. Kawaguchi, S., & Nishii, R. (2007). Hyperspectral image classification by bootstrap adaboost with random decision stumps. IEEE Transactions on Geoscience and Remote Sensing, 45(11), 3845–3851. Scholar
  84. Kim, D. Y., Han, S. H., & Kim, H. (2008). Discriminant analysis for predicting ranges of cost variance in international construction projects. Journal of Construction Engineering and Management, 134(6), 398–410. Scholar
  85. Kim, K., & Han, I. (2000). Genetic algorithms approach to feature discretization in artificial neural networks for the prediction of stock price index. Expert Systems with Applications, 19(2), 125–132. Scholar
  86. Kirkman, S., Osorio, A., & Sachs, D. (2002). The networked readiness index: Measuring the preparedness of nations for the networked world. In Kirkman (Ed.), The global information technology report 20012002 readiness for the networked world (pp. 10–29). New York: Oxford University Press.Google Scholar
  87. Kleiner, A., Talwalkar, A., Sarkar, P., & Jordan, M. I. (2014). A scalable bootstrap for massive data. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 76(4), 795–816. Scholar
  88. Kline, R. B. (2005). Principles and practice of structural equation modeling. Principles and practice of strucutral equation modeling (2nd ed., Vol. 156). New York: The Guliford press.
  89. Koch, P. N., Yang, R. J., & Gu, L. (2004). Design for six sigma through robust optimization. Structural and Multidisciplinary Optimization, 26(3–4), 235–248. Scholar
  90. Konietschke, F., Bathke, A. C., Harrar, S. W., & Pauly, M. (2015). Parametric and nonparametric bootstrap methods for general MANOVA. Journal of Multivariate Analysis, 140, 291–301. Scholar
  91. Lagarias, J. C., Reeds, J. A., Wright, M. H., & Wright, P. E. (1998). Convergence properties of the Nelder–Mead simplex method in low dimensions. SIAM Journal on Optimization, 9(1), 112–147. Scholar
  92. Lahdelma, R., Hokkanen, J., & Salminen, P. (1998). SMAA—Stochastic multiobjective acceptability analysis. European Journal of Operational Research, 106(1), 137–143. Scholar
  93. Ligmann-Zielinska, A., & Jankowski, P. (2008). A framework for sensitivity analysis in spatial multiple criteria evaluation. In T. J. Cova, H. J. Miller, K. Beard, A. U. Frank & M. F. Goodchild (Eds.), Geographic information science (pp. 217–233). Berlin: Springer. Scholar
  94. Liu, N. C., & Cheng, Y. (2005). The academic ranking of world universities. Higher Education in Europe, 30(2), 127–136. Scholar
  95. Macedo, L. L., Godinho, P., & Alves, M. J. (2017). Mean-semivariance portfolio optimization with multiobjective evolutionary algorithms and technical analysis rules. Expert Systems with Applications, 79, 33–43. Scholar
  96. Macharis, C., Springael, J., De Brucker, K., & Verbeke, A. (2004). PROMETHEE and AHP: The design of operational synergies in multicriteria analysis. European Journal of Operational Research, 153(2), 307–317. Scholar
  97. Maggino, F. (2009). Towards more participative methods in the construction of social indicators: Survey techniques aimed at determining importance weights. In 62nd Annual conference of the world association for public opinion research on public opinion and survey research in a changing world. Lausanne, Switzerland. Retrieved from
  98. Mammen, E. (1992). When does bootstrap work? (Vol. 77). New York, NY: Springer. Scholar
  99. Mariano, E. B., Sobreiro, V. A., & do Nascimento Rebelatto, D. A. (2015). Human development and data envelopment analysis: A structured literature review. Omega, 54, 33–49. Scholar
  100. Maricic, M., Bulajic, M., Martic, M., & Dobrota, M. (2015). Measuring the ict development: The fusion of biased and objective approach. Naval Academy Scientific Bulletin, 18(2), 326–334.Google Scholar
  101. Maricic, M., Bulajic, M., Radojicic, Z., & Jeremic, V. (2016). Multivariate approach to imposing additional constraints on the benefit-of-the-doubt model: The case of QS world university rankings by subject. Croatian Review of Economic, Business and Social Statistics, 2(1), 1–14. Scholar
  102. Maricic, M., & Kostic-Stankovic, M. (2016). Towards an impartial Responsible Competitiveness Index: A twofold multivariate I-distance approach. Quality & Quantity, 50(1), 103–120. Scholar
  103. Maricic, M., Zornic, N., Pilcevic, I., & Dacic-Pilcevic, A. (2017). ARWU vs. alternative ARWU ranking: What are the consequences for lower ranked universities? Management: Journal of Sustainable Business and Management Solutions in Emerging Economies, 22(1), 1–14. Scholar
  104. Marković, M., Zdravković, S., Mitrović, M., & Radojičić, A. (2016). An iterative multivariate post hoc I-distance approach in evaluating OECD Better Life Index. Social Indicators Research, 126(1), 1–19. Scholar
  105. Mazziotta, M., & Pareto, A. (2007). Un indicatore sintetico di dotazione infrastrutturale: il metodo delle penalità per coefciente di variazione. In Lo sviluppo regionale nell’Unione Europea-Obiettivi, strategie, politiche. Atti della XXVIII Conferenza Italiana di Scienze Regionali. Bolzano.Google Scholar
  106. Mazziotta, M., & Pareto, A. (2012). Methods for constructing composite indices: One for all or all for one. Rivista Italiana Di Economia Demografia e Statistica, 67(2), 67–80.Google Scholar
  107. Melyn, W., & Moesen, W. (1991). Towards a synthetic indicator of macroeconomic performance: Unequal weighting when limited information is available. Leuven University, Working paper ID 26175691.Google Scholar
  108. Milenkovic, M. J., Brajovic, B., Milenkovic, D., Vukmirovic, D., & Jeremic, V. (2016). Beyond the equal-weight framework of the Networked Readiness Index: A multilevel I-distance methodology. Information Development, 32(4), 1120–1136. Scholar
  109. Mirjalili, S., Mohd Hashim, S. Z., & Moradian Sardroudi, H. (2012). Training feedforward neural networks using hybrid particle swarm optimization and gravitational search algorithm. Applied Mathematics and Computation, 218(22), 11125–11137. Scholar
  110. Moed, H. F. (2017). A critical comparative analysis of five world university rankings. Scientometrics, 110(2), 967–990. Scholar
  111. Mousavi-Nasab, S. H., & Sotoudeh-Anvari, A. (2018). A new multi-criteria decision making approach for sustainable material selection problem: A critical study on rank reversal problem. Journal of Cleaner Production, 182, 466–484. Scholar
  112. Munda, G. (2005a). “Measuring sustainability”: A multi-criterion framework. Environment, Development and Sustainability, 7(1), 117–134. Scholar
  113. Munda, G. (2005b). Multiple criteria decision analysis and sustainable development. In S. Greco, M. Ehrgott, & J. Figueira (Eds.), Multiple criteria decision analysis: State of the art surveys (pp. 953–986). New York: Springer.CrossRefGoogle Scholar
  114. Munda, G., & Nardo, M. (2009). Noncompensatory/nonlinear composite indicators for ranking countries: A defensible setting. Applied Economics, 41(12), 1513–1523. Scholar
  115. Murias, P., de Miguel, J. C., & Rodríguez, D. (2008). A composite indicator for university quality assesment: The case of Spanish higher education system. Social Indicators Research, 89(1), 129–146. Scholar
  116. Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffman, A., & Giovannini, E. (2005). Handbook on constructing composite indicators. OECD Statistics Working Papers. Scholar
  117. Nevitt, J., & Hancock, G. R. (2001). Performance of bootstrapping approaches to model test statistics and parameter standard error estimation in structural equation modeling—Google Académico. Structural Equation Modeling: A Multidisciplinary Journal, 8(3), 353–377. Scholar
  118. OECD. (2004). The OECD-JRC handbook on practices for developing composite indicators. Paris: OECD.Google Scholar
  119. Otero-Muras, I., & Banga, J. R. (2014). Multicriteria global optimization for biocircuit design. BMC Systems Biology, 8(1), 113. Scholar
  120. Paracchini, M. L., Pacini, C., Calvo, S., & Vogt, J. (2008). Weighting and aggregation of indicators for sustainability impact assessment in the SENSOR context. In K. Helming, M. Pérez-Soba & P. Tabbush (Eds.), Sustainability impact assessment of land use changes (pp. 349–372). Berlin: Springer. Scholar
  121. Paruolo, P., Saisana, M., & Saltelli, A. (2013). Ratings and rankings: Voodoo or science? Journal of the Royal Statistical Society: Series A (Statistics in Society), 176(3), 609–634. Scholar
  122. Pattengale, N. D., Alipour, M., Bininda-Emonds, O. R. P., Moret, B. M. E., & Stamatakis, A. (2009). How many bootstrap replicates are necessary? In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, C. Pandu Rangan, B. Steffen, D. Terzopoulos & D. Tygar (Eds.), Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Vol. 5541 LNBI, pp. 184–200).
  123. Penas, D. R., González, P., Egea, J. A., Banga, J. R., & Doallo, R. (2015). Parallel metaheuristics in computational biology: An asynchronous cooperative enhanced scatter search method. Procedia Computer Science, 51, 630–639. Scholar
  124. Penas, D. R., González, P., Egea, J. A., Doallo, R., & Banga, J. R. (2017). Parameter estimation in large-scale systems biology models: A parallel and self-adaptive cooperative strategy. BMC Bioinformatics, 18(1), 52. Scholar
  125. Pérez, V., Guerrero, F., González, M., Pérez, F., & Caballero, R. (2013). Composite indicator for the assessment of sustainability: The case of Cuban nature-based tourism destinations. Ecological Indicators, 29, 316–324. Scholar
  126. Permanyer, I. (2011). Assessing the robustness of composite indices rankings. Review of Income and Wealth, 57(2), 306–326. Scholar
  127. Politis, D. N., & Romano, J. P. (1994). Large sample confidence regions based on subsamples under minimal assumptions. The Annals of Statistics, 22(4), 2031–2050. Scholar
  128. Politis, D. N., Romano, J. P., & Wolf, M. (1999). Subsampling. New York, NY: Springer. Scholar
  129. Pollesch, N. L., & Dale, V. H. (2016). Normalization in sustainability assessment: Methods and implications. Ecological Economics, 130, 195–208. Scholar
  130. Radojicic, M., Savic, G., & Jeremic, V. (2018). Measuring the efficiency of banks: The bootstrapped I-distance gar dea approach. Technological and Economic Development of Economy, 24(4), 1581–1605. Scholar
  131. Radojicic, M., Savic, G., Radovanovic, S., & Jeremic, V. (2015). A novel bootstrap dba-dea approach in evaluating efficiency of banks. Naval Academy Scientific Bulletin, 18(2), 375–384.Google Scholar
  132. Ramón, N., Ruiz, J. L., & Sirvent, I. (2012). Common sets of weights as summaries of DEA profiles of weights: With an application to the ranking of professional tennis players. Expert Systems with Applications, 39(5), 4882–4889. Scholar
  133. Ray, A. K. (2007). Measurement of social development: An international comparison. Social Indicators Research, 86(1), 1–46. Scholar
  134. Reggi, L., Arduini, D., Biagetti, M., & Zanfei, A. (2014). How advanced are Italian regions in terms of public e-services? The construction of a composite indicator to analyze patterns of innovation diffusion in the public sector. Telecommunications Policy, 38(5–6), 514–529. Scholar
  135. Remli, M. A., Deris, S., Mohamad, M. S., Omatu, S., & Corchado, J. M. (2017). An enhanced scatter search with combined opposition-based learning for parameter estimation in large-scale kinetic models of biochemical systems. Engineering Applications of Artificial Intelligence, 62, 164–180. Scholar
  136. Renzi, G., Canfora, L., Salvati, L., & Benedetti, A. (2017). Validation of the soil Biological Fertility Index (BFI) using a multidimensional statistical approach: A country-scale exercise. CATENA, 149, 294–299. Scholar
  137. Rodriguez-Fernandez, M., Egea, J. A., & Banga, J. R. (2006). Novel metaheuristic for parameter estimation in nonlinear dynamic biological systems. BMC Bioinformatics, 7(1), 483. Scholar
  138. Rogge, N. (2012). Undesirable specialization in the construction of composite policy indicators: The Environmental Performance Index. Ecological Indicators, 23, 143–154. Scholar
  139. Rogge, N. (2018a). Composite indicators as generalized benefit-of-the-doubt weighted averages. European Journal of Operational Research, 267(1), 381–392. Scholar
  140. Rogge, N. (2018b). On aggregating benefit of the doubt composite indicators. European Journal of Operational Research, 264(1), 364–369. Scholar
  141. Saisana, M., & D’Hombres, B. (2008). Higher education rankings: Robustness issues and critical assessment. How much confidence can we have in higher education rankings? JRC.
  142. Saisana, M., D’Hombres, B., & Saltelli, A. (2011). Rickety numbers: Volatility of university rankings and policy implications. Research Policy, 40(1), 165–177. Scholar
  143. Saisana, M., & Saltelli, A. (2014). JCR statistical audit of the WJP Rule of Law index 2014. In World justice project: The world justice project Rule of Law index 2014 (pp. 188–197). Washington, DC: World Justice Project.Google Scholar
  144. Saisana, M., Saltelli, A., & Tarantola, S. (2005). Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. Journal of the Royal Statistical Society: Series A (Statistics in Society), 168(2), 307–323. Scholar
  145. Saisana, M., & Tarantola, S. (2002). State-of-the-art re port on current methodologies and practices for composite indicator development. Retrieved December 20, 2017 from
  146. Saltelli, A. (2007). Composite indicators between analysis and advocacy. Social Indicators Research, 81(1), 65–77. Scholar
  147. Saltelli, A., Chan, K., & Scott, E. M. (2000). Sensitivity analysis. Wiley series in probability and statistics. Hoboken: Wiley.Google Scholar
  148. Saltelli, A., Ratto, M., Andres, T., Campolongo, F., Cariboni, J., Gatelli, D., et al. (2007). Global sensitivity analysis. The primer. Chichester: Wiley. Scholar
  149. Savic, D., Jeremic, V., & Petrovic, N. (2016). Rebuilding the pillars of sustainable society index: A multivariate post hoc I-distance approach. Problemy Ekorozwoju – Problems of Sustainable Development, 12(1), 125–134.Google Scholar
  150. Sayed, H., Hamed, R., Hosny, S. H., & Abdelhamid, A. H. (2018). Avoiding ranking contradictions in human development index using goal programming. Social Indicators Research, 138(2), 405–442. Scholar
  151. Sayed, H., Hamed, R., Ramadan, M. A.-G., & Hosny, S. (2015). Using meta-goal programming for a new human development indicator with distinguishable country ranks. Social Indicators Research, 123(1), 1–27. Scholar
  152. Sébastien, L., Bauler, T., & Lehtonen, M. (2014). Can indicators bridge the gap between science and policy? An exploration into the (non)use and (non)influence of indicators in EU and UK policy making. Nature and Culture. Scholar
  153. Sevigny, E. L., & Saisana, M. (2016). Measuring interstate variations in the consequences of illegal drugs: A composite indicator approach. Social Indicators Research, 128(2), 501–529. Scholar
  154. ShanghaiRanking. (2017). ARWU world university rankings 2017. Academic ranking of world universities 2017. Top 500 universities. Shanghai ranking—2017. Retrieved February 14, 2018, from
  155. ShanghaiRanking. (2018). Academic ranking of world universities 2018. Retreived December 24, 2018 from
  156. Shattock, M. (2017). The ‘world class’ university and international ranking systems: what are the policy implications for governments and institutions? Policy Reviews in Higher Education, 1(1), 4–21. Scholar
  157. Singh, R. K., Murty, H. R., Gupta, S. K., & Dikshit, A. K. (2007). Development of composite sustainability performance index for steel industry. Ecological Indicators, 7, 565–588. Scholar
  158. Socha, K., & Blum, C. (2007). An ant colony optimization algorithm for continuous optimization: Application to feed-forward neural network training. Neural Computing and Applications, 16(3), 235–247. Scholar
  159. Sörensen, K., Sevaux, M., & Glover, F. (2018). A history of metaheuristics. In R. Martí, P. Panos & M. G. C. Resende (Eds.), Handbook of heuristics (pp. 1–18). Cham: Springer. Scholar
  160. Strobl, C., Boulesteix, A.-L., Zeileis, A., & Hothorn, T. (2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8, 25.CrossRefGoogle Scholar
  161. Taghdisian, H., Pishvaie, M. R., & Farhadi, F. (2015). Multi-objective optimization approach for green design of methanol plant based on CO2-efficeincy indicator. Journal of Cleaner Production, 103, 640–650. Scholar
  162. Talukder, B., Hipel, K., & VanLoon, G. (2017). Developing composite indicators for agricultural sustainability assessment: Effect of normalization and aggregation techniques. Resources, 6(4), 66. Scholar
  163. Van Puyenbroeck, T., & Rogge, N. (2017). Geometric mean quantity index numbers with benefit-of-the-doubt weights. European Journal of Operational Research, 256(3), 1004–1014. Scholar
  164. Verbunt, P., & Rogge, N. (2018). Geometric composite indicators with compromise benefit-of-the-doubt weights. European Journal of Operational Research, 264(1), 388–401. Scholar
  165. Villaverde, A., Egea, J. A., & Banga, J. R. (2012). A cooperative strategy for parameter estimation in large scale systems biology models. BMC Systems Biology, 6(1), 75. Scholar
  166. Wang, J. J., Jing, Y. Y., Zhang, C. F., & Zhao, J. H. (2009). Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renewable and Sustainable Energy Reviews. Scholar
  167. Wang, X., & Triantaphyllou, E. (2008). Ranking irregularities when evaluating alternatives by using some ELECTRE methods. Omega, 36(1), 45–63. Scholar
  168. Wang, Y.-M., & Luo, Y. (2009). On rank reversal in decision analysis. Mathematical and Computer Modelling, 49(5–6), 1221–1229. Scholar
  169. World Economic Forum. (2016a). Global information technology report 2016—Reports—World economic forum. Retrieved December 23, 2017, from
  170. World Economic Forum. (2016b). Structure and methodology. Retrieved September 10, 2017, from
  171. Xu, L. W., Yang, F. Q., Abula, A., & Qin, S. (2013). A parametric bootstrap approach for two-way ANOVA in presence of possible interactions with unequal variances. Journal of Multivariate Analysis, 115, 172–180. Scholar
  172. Yang, W., Mckinnon, M. C., & Turner, W. R. (2015). Quantifying human well-being for sustainability research and policy. Ecosystem Health and Sustainability, 1(4), 1–13. Scholar
  173. Zanakis, S. H., Newburry, W., & Taras, V. (2016). Global social tolerance index and multi-method country rankings sensitivity. Journal of International Business Studies, 47(4), 480–497. Scholar
  174. Zanella, A., Camanho, A. S., & Dias, T. G. (2015). Undesirable outputs and weighting schemes in composite indicators based on data envelopment analysis. European Journal of Operational Research, 245(2), 517–530. Scholar
  175. Zhou, P., Ang, B. W., & Poh, K. L. (2006). Comparing aggregating methods for constructing the composite environmental index: An objective measure. Ecological Economics, 59(3), 305–311. Scholar
  176. Zhou, P., Ang, B. W., & Zhou, D. Q. (2010). Weighting and aggregation in composite indicator construction: A multiplicative optimization approach. Social Indicators Research, 96(1), 169–181. Scholar
  177. Zornic, N., Bornmann, L., Maricic, M., Markovic, A., Martic, M., & Jeremic, V. (2015). Ranking institutions within a university based on their scientific performance: A percentile-based approach. El Profesional de La Información, 24(5), 551–566. Scholar
  178. Zuur, A. F., Ieno, E. N., & Elphick, C. S. (2010). A protocol for data exploration to avoid common statistical problems. Methods in Ecology and Evolution, 1(1), 3–14. Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Faculty of Organizational SciencesUniversity of BelgradeBelgradeSerbia
  2. 2.Department of Applied Mathematics and StatisticsTechnical University of CartagenaCartagenaSpain

Personalised recommendations