, Volume 105, Issue 3, pp 2109–2135 | Cite as

Methodi Ordinatio: a proposed methodology to select and rank relevant scientific papers encompassing the impact factor, number of citation, and year of publication

  • Regina Negri PaganiEmail author
  • João Luiz Kovaleski
  • Luis Mauricio Resende


An increase in the number of scientific publications in the last few years, which is directly proportional to the appearance of new journals, has made the researchers’ job increasingly complex and extensive regarding the selection of bibliographic material to support their research. Not only is it a time consuming task, it also requires suitable criteria, since the researchers need to elect systematically the most relevant literature works. Thus the objective of this paper is to propose the methodology called Methodi Ordinatio, which presents criteria to select scientific articles. This methodology employs an adaptation of the ProKnow-C for selection of publications and the InOrdinatio, which is an index to rank by relevance the works selected. This index crosses the three main factors under evaluation in a paper: impact factor, year of publication and number of citations. When applying the equation, the researchers identify among the works selected the most relevant ones to be in their bibliographic portfolio. As a practical application, it is provided a research sample on the theme technology transfer models comprising papers from 1990 to 2015. The results indicated that the methodology is efficient regarding the objectives proposed, and the most relevant papers on technology transfer models are presented.


Bibliographic portfolio Research methodology Methodi Ordinatio InOrdinatio Technology transfer models 



We thank the Brazilian Government, the Ministry of Education, and UTFPR that supported this research.


  1. Afonso, M. H. F., de Souza, J. V., Ensslin, S. R., & Ensslin, L. (2012). Como construir conhecimento sobre o tema de pesquisa? Aplicação do processo proknow-c na busca de literatura sobre avaliação do desenvolvimento sustentável. Revista de Gestão Social E Ambiental,. doi: 10.5773/rgsa.v5i2.424.Google Scholar
  2. Antelman, K. (2004). Do open-access articles have a greater research impact? College & Research Libraries, 65(5), 372–382. doi: 10.5860/crl.65.5.372.CrossRefGoogle Scholar
  3. Barham, B. L., Foltz, J. D., & Prager, D. L. (2014). Making time for science. Research Policy, 43(1), 21–31. doi: 10.1016/j.respol.2013.08.007.CrossRefGoogle Scholar
  4. Bhupatiraju, S., et al. (2012). Knowledge flows: analyzing the core literature of innovation, entrepreneurship and science and technology studies. Research Policy, 41, 1205–1218. doi: 10.1016/j.respol.2012.03.011.CrossRefGoogle Scholar
  5. Bornmann, L. (2010). Towards an ideal method of measuring research performance: Some comments to the Opthof and Leydesdorff (2010) paper. Journal of Informetrics, 4(3), 441–443. doi: 10.1016/j.joi.2010.04.004.CrossRefGoogle Scholar
  6. Bruggemann, R., & Carlsen, L. (2012). Multi-criteria decision analyses. Viewing MCDA in terms of both process and aggregation methods: Some thoughts, motivated by the paper of Huang, Keisler and Linkov. Science of the Total Environment, 425, 293–295.CrossRefGoogle Scholar
  7. Cho, K. T. (2003). Multicriteria decision methods: An attempt to evaluate and unify. Mathematical and Computer Modelling, 37(9–10), 1099–1119. doi: 10.1016/S0895-7177(03)00122-5.MathSciNetCrossRefzbMATHGoogle Scholar
  8. Cinelli, M., Coles, S. R., & Kirwan, K. (2014). Analysis of the potentials of multicriteria decision analysis methods to conduct sustainability assessment. Ecological Indicators, 46, 138–148.CrossRefGoogle Scholar
  9. De Greve, J. P., & Frijdal, A. (1989). Evaluation of scientific research profile analysis: a mixed method. Higher Education Management, 1, 83–90.Google Scholar
  10. Dieks, D., & Chang, H. (1976). Differences in impact of scientific publications: Some indices derived from a citation analysis. Social Studies of Science, 6, 247–267. doi: 10.1177/030631277600600204.CrossRefGoogle Scholar
  11. Dyer, J. S., Fishburn, P. C., Steuer, R. E., Wallenius, J., & Zionts, S. (1992). Multiple criteria decision making, multiattribute utility theory: The next ten years. Management Science, 38(5), 645–654.CrossRefzbMATHGoogle Scholar
  12. Figueira, J., Mousseau, V., & Roy, B. (2005). Electre methods. In Multiple criteria decision analysis: State of the art surveys. International series in operations research and management science (Vol. 78, pp. 133–153). United States: Springer.Google Scholar
  13. Greco, S., Matarazzo, B., & Slowinski, R. (2001a). Rough sets theory for multicriteria decision analysis. European Journal of Operational Research, 129, 1–47.MathSciNetCrossRefzbMATHGoogle Scholar
  14. Greco, S., Matarazzo, B., Slowinski, R. & Stefanowski, J. (2001b). Variable consistency model of dominance-based rough sets approach. In Rough sets and current trends in computing. Lecture notes in computer science, 2005, 170–18.Google Scholar
  15. Haeussler, C., Jiang, L., Thursby, J., & Thursby, M. (2014). Specific and general information sharing among competing academic researchers. Research Policy, 43(3), 465–475. doi: 10.1016/j.respol.2013.08.017.CrossRefGoogle Scholar
  16. Higgins, J.P.T. & Green, S. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. The Cochrane Collaboration, Retrieved July 8, 2015, from
  17. Irvine, J., & Martin, B. R. (1983). Assessing basic research: The case of the Isaac Newton Telescope. Social Studies of Science, 13, 49–86. doi: 10.1177/030631283013001004.CrossRefGoogle Scholar
  18. Keeney, R. L., & Raiffa, H. (1993). Decisions with multiple objectives: Preferences and value trade-offs. Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  19. Lacerda, R. T. O., Ensslin, L. & Ensslin, S. R. (2012). A bibliometric analysis of strategy and performance measurement. Gestão & Produção, 19(1), 59–78. Retrieved July 7, 2015, from
  20. Lacerda, R. T. O., Ensslin, L., & Ensslin, S. R. (2015). Research methods and success meaning in project management. In B. Pasian (Ed.), Designs, methods and practices for research of project management. England: Gower Publishing Ltd.Google Scholar
  21. Martin, B. R. (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36(3), 343–362. doi: 10.1007/BF02129599.CrossRefGoogle Scholar
  22. Nightingale, A. (2009). A guide to systematic literature reviews. Surgery (Oxford), 27(9), 381–384.CrossRefGoogle Scholar
  23. Pohekar, S. D., & Ramachandran, M. (2004). Application of multi-criteria decision making to sustainable energy planning: A review. Renewable and Sustainable Energy Reviews, 8, 365–381.CrossRefGoogle Scholar
  24. Roy, B. (1993). Decision science or decision-aid science? European Journal of Operational Research, 66, 184–203.CrossRefGoogle Scholar
  25. Roy, B. (2005). Paradigms and challenges. In J. Figueira, S. Greco, & M. Ehrgott (Eds.), Multiple criteria decision analysis: State of the art surveys. Berlin: Springer.Google Scholar
  26. Saaty, T. L. (1990). How to make a decision? The analytic hierarchy process. European Journal of Operantional Research, 48, 9–26.CrossRefzbMATHGoogle Scholar
  27. Small, H., Boyack, K. W., & Klavans, R. (2014). Identifying emerging topics in science and technology. Research Policy, 43(8), 1450–1467. doi: 10.1016/j.respol.2014.02.005.CrossRefGoogle Scholar
  28. Taylor, P. J. (2002). A partial order scalogram analysis of communication behavior in crisis negotiation with the prediction of outcome. The International Journal of Conflict Management, 13(1), 4–37.CrossRefGoogle Scholar
  29. Ul Haq, M. (2003). The birth of the Human Development Index. In A. Kumar (Ed.), Readings in human development (pp. 127–137). Oxford: Oxford University Press.Google Scholar
  30. UNDP (2015). United Nations Development Program. Human Development Reports. Retrieved July 9, 2015, from
  31. van Raan, A. F. J. (2004). Measuring science. Capita selecta of current main issues. In H. F. Moed, W. Glänzel, & U. Schmoch (Eds.), Handbook of quantitative science and technology research. The use of publication and patent statistics in studies of S&T systems (pp. 19–50). Kluwer Academic Publishers: Dordrecht.Google Scholar
  32. Vaz, C. R., Tasca, J. E., Ensslin, L., Ensslin, S. R., & Selig, P. M. (2013). Avaliação de desempenho na gestão estratégica organizacional: seleção de um referencial teórico de pesquisa e análise bibliométrica. Revista Gestão Industrial,. doi: 10.3895/S1808-04482012000400008.Google Scholar
  33. Vinkler, P. (1986a). Management system for a scientific research institute based on the assessment of scientific publications. Research Policy, 15(2), 77–87. doi: 10.1016/0048-7333(86)90003-X.CrossRefGoogle Scholar
  34. Vinkler, P. (1986b). Evaluation of some methods for the relative assessment of scientific publications. Scientometrics, 10, 157–177. doi: 10.1007/BF02026039.CrossRefGoogle Scholar
  35. Vinkler, P. (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36(3), 343–362. doi: 10.1007/BF02129599.CrossRefGoogle Scholar
  36. Vinkler, P. (2004). Characterization of the impact of sets of scientific papers: The Garfield (Impact) Factor. Journal of the American Society for Information Science and Technology, 55, 431–435. doi: 10.1002/asi.10391.CrossRefGoogle Scholar
  37. Vinkler, P. (2009). πv-index: A new indicator for assessing scientific impact. Journal of Information Science, 35, 602–612. doi: 10.1177/0165551509103601.CrossRefGoogle Scholar
  38. Vinkler, P. (2010). The πv-index: A new indicator to characterize the impact of journals. Scientometrics, 82, 461–475. doi: 10.1007/s11192-010-0182-z.CrossRefGoogle Scholar
  39. Vinkler, P. (2012). The case of scientometricians with the “absolute relative” impact Indicator. Journal of Informetrics, 6, 254–264. doi: 10.1016/j.joi.2011.12.004.CrossRefGoogle Scholar
  40. Zopounidis, C., & Doumpos, M. (2002). Multicriteria classification and sorting methods: A literature review. European Journal of Operational Research, 138, 229–246.MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2015

Authors and Affiliations

  1. 1.Department of Industrial EngineeringFederal University of Technology - Paraná (UTFPR) Câmpus Ponta GrossaPonta GrossaBrazil
  2. 2.Department of Post-graduation in Production EngineeringFederal University of Technology - Paraná (UTFPR) Câmpus Ponta GrossaPonta GrossaBrazil

Personalised recommendations