Abstract
As a typical multi-criteria group decision making (MCGDM) problem, research and development (R&D) project selection involves multiple decision criteria which are formulated by different frames of discernment, and multiple experts who are associated with different weights and reliabilities. The evidential reasoning (ER) rule is a rational and rigorous approach to deal with such MCGDM problems and can generate comprehensive distributed evaluation outcomes for each R&D project. In this paper, an ER rule based model taking into consideration experts’ weights and reliabilities is proposed for R&D project selection. In the proposed approach, a utility based information transformation technique is applied to handle qualitative evaluation criteria with different evaluation grades, and both adaptive weights of criteria and utilities assigned to evaluation grades are introduced to the ER rule based model. A nonlinear optimisation model is developed for the training of weights and utilities. A case study with the National Science Foundation of China is conducted to demonstrate how the proposed method can be used to support R&D project selection. Validation data show that the evaluation results become more reliable and consistent with reality by using the trained weights and utilities from historical data.
Similar content being viewed by others
References
Agarski, B., Budak, I., Kosec, B., & Hodolic, J. (2012). An approach to multi-criteria environmental evaluation with multiple weight assignment. Environmental Modeling & Assessment, 17(3), 255–266.
Amiri, M. P. (2010). Project selection for oil-fields development by using the AHP and fuzzy TOPSIS methods. Expert Systems with Applications, 37(9), 6218–6224.
Baker, N., & Freeland, J. (1975). Recent advances in R&D benefit measurement and project selection methods. Management Science, 21(10), 1164–1175.
Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45(1), 197–245.
Chiang, T. A., & Che, Z. H. (2010). A fuzzy robust evaluation model for selecting and ranking NPD projects using Bayesian belief network and weight-restricted DEA. Expert Systems with Applications, 37(11), 7408–7418.
Collan, M., & Luukka, P. (2014). Evaluating R&D projects as investments by using an overall ranking from four new fuzzy similarity measure-based TOPSIS variants. IEEE Transactions on Fuzzy Systems, 22(3), 505–515.
Dempster, A. P. (1967). Upper and lower probabilities induced by a multivalued mapping. The Annals of Mathematical Statistics, 38, 325–339.
Feng, B., Ma, J., & Fan, Z. P. (2011). An integrated method for collaborative R&D project selection: Supporting innovative research teams. Expert Systems with Applications, 38(5), 5532–5543.
Heidenberger, K., & Stummer, C. (1999). Research and development project selection and resource allocation: A review of quantitative modelling approaches. International Journal of Management Reviews, 1(2), 197–224.
Henriksen, A. D., & Traynor, A. J. (1999). A practical R&D project-selection scoring tool. IEEE Transactions on Engineering Management, 46(2), 158–170.
Hsu, Y. G., Tzeng, G. H., & Shyu, J. Z. (2003). Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects. R&D Management, 33(5), 539–551.
Huang, C. C., Chu, P. Y., & Chiang, Y. H. (2008). A fuzzy AHP application in government-sponsored R&D project selection. Omega, 36(6), 1038–1052.
Jackson, B. (1983). Decision methods for selecting a portfolio of R&D projects. Research Management, 26(5), 21–26.
Jung, U., & Seo, D. W. (2010). An ANP approach for R&D project evaluation based on interdependencies between research objectives and evaluation criteria. Decision Support Systems, 49(3), 335–342.
Južnič, P., Pečlin, S., Žaucer, M., Mandelj, T., Pušnik, M., & Demšar, F. (2010). Scientometric indicators: Peer-review, bibliometric methods and conflict of interests. Scientometrics, 85(2), 429–441.
Lee, C. J., Sugimoto, C. R., Zhang, G., & Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17.
Liu, W., Hu, G., Tang, L., & Wang, Y. (2015a). China's global growth in social science research: Uncovering evidence from bibliometric analyses of SSCI publications (1978–2013). Journal of Informetrics, 9(3), 555–569.
Liu, W., Tang, L., Gu, M., & Hu, G. (2015b). Feature report on China: A bibliometric analysis of China-related articles. Scientometrics, 102(1), 503–517.
Liu, X. B., Pei, F., Yang, J. B., & Yang, S. L. (2010). An MAGDM approach combining numerical values with uncertain linguistic information and its application in evaluation of R&D projects. International Journal of Computational Intelligence Systems, 3(5), 575–589.
Mahmoodzadeh, S., Shahrabi, J., Pariazar, M., & Zaeri, M. S. (2007). Project selection by using fuzzy AHP and TOPSIS technique. International Journal of Human and social sciences, 1(3), 135–140.
Meade, L. M., & Presley, A. (2002). R&D project selection using the analytic network process. IEEE Transactions on Engineering Management, 49(1), 59–66.
Poh, K. L., Ang, B. W., & Bai, F. (2001). A comparative analysis of R&D project evaluation methods. R&D Management, 31(1), 63–75.
Provost, F., & Kohavi, R. (1998). Guest editors’ introduction: On applied research in machine learning. Machine Learning, 30(2), 127–132.
Saaty, T. L. (1980). The analytic hierarchy process: Planning, priority setting, resources allocation. New York: McGraw.
Schmidt, R. L., & Freeland, J. R. (1992). Recent progress in modeling R&D project-selection processes. IEEE Transactions on Engineering Management, 39(2), 189–201.
Shafer, G. (1976). A mathematical theory of evidence (Vol. 1). Princeton: Princeton University Press.
Smarandache, F., Dezert, J., & Tacnet, J. M. (2010). Fusion of sources of evidence with different importances and reliabilities. In 2010 13th Conference on information fusion (FUSION) (pp. 1–8). IEEE.
Taroun, A., & Yang, J. B. (2011). Dempster–Shafer theory of evidence: Potential usage for decision making and risk analysis in construction project management. The Built & Human Environment Review, 4(Special Issue 1), 155–166.
Tian, Q., Ma, J., Liang, J., Kwok, R. C., & Liu, O. (2005). An organizational decision support system for effective R&D project selection. Decision Support Systems, 39(3), 403–413.
van Raan, A. (1996). Advanced bibliometric methods as quantitative core of peer review based evaluation and foresight exercises. Scientometrics, 36(3), 397–420.
Wessely, S. (1998). Peer review of grant applications: What do we know? The Lancet, 352, 301–305.
Xiao, Z., Yang, X., Pang, Y., & Dang, X. (2012). The prediction for listed companies’ financial distress by using multiple prediction methods with rough set and Dempster–Shafer evidence theory. Knowledge-Based Systems, 26, 196–206.
Xu, D. L. (2012). An introduction and survey of the evidential reasoning approach for multiple criteria decision analysis. Annals of Operations Research, 195(1), 163–187.
Yang, J. B. (2001). Rule and utility based evidential reasoning approach for multiattribute decision analysis under uncertainties. European Journal of Operational Research, 131(1), 31–61.
Yang, J. B., & Singh, M. G. (1994). An evidential reasoning approach for multiple-attribute decision making with uncertainty. IEEE Transactions on Systems, Man, and Cybernetics, 24(1), 1–18.
Yang, J. B., & Xu, D. L. (2002). On the evidential reasoning algorithm for multiple attribute decision analysis under uncertainty. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 32(3), 289–304.
Yang, J. B., & Xu, D. L. (2013). Evidential reasoning rule for evidence combination. Artificial Intelligence, 205, 1–29.
Yang, J. B., & Xu, D. L. (2014). A study on generalising Bayesian inference to evidential reasoning. In International conference on belief functions (pp. 180–189). Springer.
Zhu, W. D., Liu, F., Chen, Y. W., Yang, J. B., Xu, D. L., & Wang, D. P. (2015). Research project evaluation and selection: An evidential reasoning rule-based method for aggregating peer review information with reliabilities. Scientometrics, 105(3), 1469–1490.
Acknowledgements
This research is partially supported by the National Natural Science Foundation of China under Grant No. 71071048 and 71601060 and the Scholarship from China Scholarship Council under Grant No. 201306230047.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Liu, F., Zhu, Wd., Chen, Yw. et al. Evaluation, ranking and selection of R&D projects by multiple experts: an evidential reasoning rule based approach. Scientometrics 111, 1501–1519 (2017). https://doi.org/10.1007/s11192-017-2278-1
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-017-2278-1