, Volume 112, Issue 2, pp 963–1006 | Cite as

Quantity versus impact of software engineering papers: a quantitative study

  • Vahid Garousi
  • João M. Fernandes


According to the data from the Scopus publication database, as analyzed in several recent studies, more than 70,000 papers have been published in the area of Software Engineering (SE) since late 1960’s. According to our recent work, 43% of those papers have received no citations at all. Since citations are the most commonly used metric for measuring research (academic) impact, these figures raise questions (doubts) about the (non-existing) impact of such a large set of papers. It is a reality that typical academic reward systems encourage researchers to publish more papers and do not place a major emphasis on research impact. To shed light on the issue of volume (quantity) versus citation-based impact of SE research papers, we conduct and report in this paper a quantitative bibliometrics assessment in four aspects: (1) quantity versus impact of different paper types (e.g., conference versus journal papers), (2) ratios of uncited (non-impactful) papers, (3) quantity versus impact of papers originating from different countries, and (4) quantity versus impact of papers by each of the top-10 authors (in terms of number of papers). To achieve the above objective, we conducted a quantitative exploratory bibliometrics assessment, comprised of four research questions, to assess quantity versus impact of SE papers with respect to the aspects discussed above. We extracted the data through a systematic, automated and repeatable process from the Scopus paper database, which we also used in two previous papers. Our results show that the distribution of SE publications has a major inequality in terms of impact overall, and also when categorized in terms of the above four aspects. The situation in the SE literature is similar to the other areas of science as studied by previous bibliometrics studies. Also, among our results is the fact that journal articles and conference papers have been cited 12.6 and 3.6 times on average, confirming the expectation that journal articles have more impact, in general, than conference papers. Also, papers originated from English-speaking countries have in general more visibility and impact (and consequently citations) when compared to papers originated from non-English-speaking countries. Our results have implications for improvement of academic reward systems, which nowadays mainly encourage researchers to publish more papers and usually neglect research impact. Also, our results can help researchers in non-English-speaking countries to consider improvements to increase their research impact of their upcoming papers.


Bibliometrics Software engineering Research impact Countries Authors Exploratory study 



Vahid Garousi was partially supported by several internal grants provided by the Hacettepe University and the Scientific and Technological Research Council of Turkey (TÜBİTAK). João M. Fernandes was supported by FCT - Fundação para a Ciência e Tecnologia within the Project Scope UID/CEC/00319/2013.


  1. Abramo, G., Cicero, T., & D’Angelo, C. A. (2014). Are the authors of highly cited articles also the most productive ones? Journal of Informetrics, 8, 89–97.CrossRefGoogle Scholar
  2. Abrizah, A., Zainab, A. N., Kiran, K., & Raj, R. G. (2013). LIS journals scientific impact and subject categorization: a comparison between Web of Science and Scopus. Scientometrics, 94, 721–740.CrossRefGoogle Scholar
  3. Aksnes, D. W. (2003). Characteristics of highly cited papers. Research Evaluation, 12, 159–170.CrossRefGoogle Scholar
  4. Allison, P. D., & Stewart, J. A. (1974). Productivity differences among scientists: Evidence for accumulative advantage. American Sociological Review, 39, 596–606.CrossRefGoogle Scholar
  5. Altmetric LLP, “Bookmarklet for Researchers,” in Last Accessed May 2017.
  6. Antonakis, J., Bastardoz, N., Liu, Y. H., & Schriesheim, C. A. (2014). What makes articles highly cited? Leadership Quarterly, 25, 152–179.CrossRefGoogle Scholar
  7. Archambault, É., Campbell, D., Gingras, Y., & Larivière, V. (2009). Comparing bibliometric statistics obtained from the web of science and scopus. Journal of American Society for Information Science, 60, 1320–1326.CrossRefGoogle Scholar
  8. Aversa, E. (1985). Citation patterns of highly cited papers and their relationship to literature aging: A study of the working literature. Scientometrics, 7, 383–389.CrossRefGoogle Scholar
  9. Basili, V. R. (1992) Software modeling and measurement: The Goal/Question/Metric paradigm. In: Technical Report, University of Maryland at College Park.Google Scholar
  10. Belcher, D. D. (2007). Seeking acceptance in an English-only research world. Journal of Second Language Writing, 16, 1–22.CrossRefGoogle Scholar
  11. Biswas, A. K. and Kirchherr, J. (2016) Prof, no one is reading you. Last Accessed May 2017.
  12. Bornmann, L. (2014). How are excellent (highly cited) papers defined in bibliometrics? A quantitative analysis of the literature. Research Evaluation, 23, 166–173.CrossRefGoogle Scholar
  13. Bornmann, L., de Moya Anegón, F., & Leydesdorff, L. (2010). Do scientific advancements lean on the shoulders of giants? A bibliometric investigation of the ortega hypothesis. PLoS ONE, 5, e13327.CrossRefGoogle Scholar
  14. Broad, W. (1981). The publishing game: Getting more for less. Science, 211, 1137–1139.MathSciNetCrossRefGoogle Scholar
  15. Buttliere, B., & Buder, J. (2017). Personalizing papers using Altmetrics: Comparing paper ‘Quality’ or ‘Impact’ to person ‘Intelligence’ or ‘Personality’. Scientometrics, 111, 219–239.CrossRefGoogle Scholar
  16. Chadegani, A. A., Salehi, H., Yunus, M. M., Farhadi, H., Fooladi, M., Farhadi, M., et al. (2013). A comparison between two main academic literature collections: Web of science and scopus databases. Asian Social Science, 9, 18–26.CrossRefGoogle Scholar
  17. Chidamber, S. R., & Kemerer, C. F. (1994). A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 20, 476–493.CrossRefGoogle Scholar
  18. Cole, S., Cole, J. R., & Dietrich, L. (1978). Measuring the cognitive state of scientific disciplines. In Y. Elkana, J. Lederberg, R. K. Merton, A. Thackray, & H. Zuckerman (Eds.), Toward a Metric of Science: The Advent of Science Indicators (pp. 209–251). Hoboken: Wiley.Google Scholar
  19. Corby, Z. (2010). To be the best, cite the best. Nature News. doi: 10.1038/news.2010.539.Google Scholar
  20. Danell, R. (2011). Can the quality of scientific work be predicted using information on the author’s track record? Journal of the American Society for Information Science and Technology, 62, 50–60.CrossRefGoogle Scholar
  21. de Freitas, F., & de Souza, J. (2011). Ten years of search based software engineering: A bibliometric analysis. In M. Cohen & M. Ó. Cinnéide (Eds.), Search Based Software Engineering (Vol. 6956, pp. 18–32). Berlin Heidelberg: Springer.CrossRefGoogle Scholar
  22. Easterbrook, S., Singer, J., Storey, M.-A., & Damian, D. (2008). Selecting empirical methods for software engineering research. In F. Shull, J. Singer, & D. K. Sjøberg (Eds.), Guide to advanced empirical software engineering (pp. 285–311). London: Springer.Google Scholar
  23. Eaton, D. J. (2014). Highly cited papers in medical physics. Medical Physics, 41, 43–44.CrossRefGoogle Scholar
  24. Egger, M., & Smith, G. D. (1998). Bias in location and selection of studies. BMJ British Medical Journal, 316, 61–66.CrossRefGoogle Scholar
  25. Eric, W. W., Tse, T. H., Glass, R. L., Basili, V. R., & Chen, T. Y. (2011). An assessment of systems and software engineering scholars and institutions (2003–2007 and 2004–2008). Journal of Systems and Software, 84, 162–168.CrossRefGoogle Scholar
  26. Falagas, M. E., Pitsouni, E. I., Malietzis, G. A., & Pappas, G. (2008). Comparison of PUBMED, scopus, web of science, and google scholar: Strengths and weaknesses. The FASEB Journal, 22, 338–342.CrossRefGoogle Scholar
  27. Farhoodi, R., Garousi, V., Pfahl, D., & Sillito, J. P. (2013). Development of scientific software: A systematic mapping, bibliometrics study and a paper repository. International Journal of Software Engineering and Knowledge Engineering, 23, 463–506.CrossRefGoogle Scholar
  28. Fernandes, J. M. (2014). Authorship trends in software engineering. Scientometrics, 101, 257–271.CrossRefGoogle Scholar
  29. Flowerdew, J. (1999). Problems in writing for scholarly publication in English: The case of Hong Kong. Journal of Second Language Writing, 8, 243–264.CrossRefGoogle Scholar
  30. Foster, J. G., Rzhetsky, A., & Evans, J. A. (2015). Tradition and innovation in scientists’ research strategies. American Sociological Review, 80, 875–908.CrossRefGoogle Scholar
  31. Freyne, J., Coyle, L., Smyth, B., & Cunningham, P. (2010). Relative status of journal and conference publications in computer science. Communications of the ACM, 53, 124–132.CrossRefGoogle Scholar
  32. Garfield, E. (1955). Citation indexes for science: A new dimension in documentation through association of ideas. Science, 122, 108–111.CrossRefGoogle Scholar
  33. Garousi, V. (2015). A bibliometric analysis of the Turkish software engineering research community. Springer Journal on Scientometrics, 105, 23–49.CrossRefGoogle Scholar
  34. Garousi, V., & Fernandes, J. M. (2016a). Highly-cited papers in software engineering: The top-100. Information and Software Technology, 71, 108–128.CrossRefGoogle Scholar
  35. Garousi, V. and Fernandes, J. M. (2016) All source data for quantity versus quality: A bibliometric assessment of the number versus impact of software engineering research papers. Last Accessed May 2017.
  36. Garousi, V., and Mäntylä, M. V. (2015) All source data for bibliometrics study of the software engineering community in Last Accessed May 2017.
  37. Garousi, V., & Mäntylä, M. V. (2016). Citations, research topics and active countries in software engineering: A bibliometrics study. Elsevier Computer Science Review, 19, 56–77.MathSciNetCrossRefGoogle Scholar
  38. Garousi, V., Petersen, K., & Özkan, B. (2016). Challenges and best practices in industry-academia collaborations in software engineering: A systematic literature review. Information and Software Technology, 79, 106–127.CrossRefGoogle Scholar
  39. Garousi, V., & Ruhe, G. (2013). A bibliometric/geographic assessment of 40 years of software engineering research (1969–2009). International Journal of Software Engineering and Knowledge Engineering, 23, 1343–1366.CrossRefGoogle Scholar
  40. Garousi, V., & Varma, T. (2010). A bibliometric assessment of canadian software engineering scholars and institutions (1996–2006). Canadian Journal on Computer and Information Science, 3, 19–29.Google Scholar
  41. Génova, G., Astudillo, H., & Fraga, A. (2016). The scientometric bubble considered harmful. Science and Engineering Ethics, 22, 227–235.CrossRefGoogle Scholar
  42. Ghosh, A., Chattopadhyay, N., & Chakrabarti, B. K. (2014). Inequality in societies, academic institutions and science journals: Gini and k-indices. Physica A: Statistical Mechanics and its Applications, 410, 30–34.CrossRefGoogle Scholar
  43. Grigore, R. (2007) David Parnas is wrong!. Last Accessed May 2017.
  44. Halffman, W., & Leydesdorff, L. (2010). Is inequality among universities increasing? Gini coefficients and the elusive rise of elite universities. Minerva, 48, 55–72.CrossRefGoogle Scholar
  45. Hamrick, T. A., Fricker, R. D., & Brown, G. G. (2010). Assessing what distinguishes highly cited from less-cited papers published in interfaces. Interfaces, 40, 454–464.CrossRefGoogle Scholar
  46. Harzing, A.-W., & Alakangas, S. (2016). Google scholar, scopus and the web of science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106, 787–804.CrossRefGoogle Scholar
  47. Ioannidis, J. P. A., Boyack, K. W., Small, H., Sorensen, A. A., & Klavan, R. (2014). Is your most cited work your best? Nature, 514, 561–562.CrossRefGoogle Scholar
  48. King, D. A. (2004). The scientific impact of nations. Nature, 430, 311–316.CrossRefGoogle Scholar
  49. Latour, B., & Woolgar, S. (1979). Laboratory Life: The Construction of Scientific Facts. Los Angeles: Sage.Google Scholar
  50. Lawrence, P. A. (2003). The politics of publication. Nature, 422, 259–261.CrossRefGoogle Scholar
  51. LibGuides at Duke University Medical Center, “Altmetrics: a primer,” in Last Accessed May 2017.
  52. Lotka, A. J. (1926). The frequency distribution of scientific productivity. Journal of the Washington Academy of Sciences, 16, 317–323.Google Scholar
  53. Merton, R. K. (1968). The Matthew effect in science: The reward and communication systems of science are considered. Science, 159, 56–63.CrossRefGoogle Scholar
  54. Mingers, J., & Lipitakis, E. A. E. C. G. (2010). Counting the citations: a comparison of Web of Science and Google Scholar in the field of business and management. Scientometrics, 85, 613–625.CrossRefGoogle Scholar
  55. Miyairi, N., & Chang, H.-W. (2012). Bibliometric characteristics of highly cited papers from Taiwan, 2000–2009. Scientometrics, 92, 197–205.CrossRefGoogle Scholar
  56. Moed, H. F. (2006). Citation analysis in research evaluation. Berlin: Springer.Google Scholar
  57. Moed, H. F. and Visser, M. S. (2007) Developing bibliometric indicators of research performance in computer science: An exploratory study. In: Research Report, Centre for Science and Technology Studies (CWTS), Leiden University, the Netherlands.Google Scholar
  58. Newman, M. E. J. (2014). Prediction of highly cited papers. Europhysics letters (EPL), 105, 6.CrossRefGoogle Scholar
  59. Nieminen, P., Carpenter, J., Rucker, G., & Schumacher, M. (2006). The relationship between quality of research and citation frequency. BMC Medical Research Methodology, 6, 1–8.CrossRefGoogle Scholar
  60. Noorden, R. V., Maher, B., & Nuzzo, R. (2014). The top 100 papers. Nature, 514, 550–553.CrossRefGoogle Scholar
  61. Nuseibeh, B. (2011). Editorial: What makes a publication archival? Software Engineering IEEE Transactions on, 37, 145.CrossRefGoogle Scholar
  62. Osterweil, L. J., Ghezzi, C., Kramer, J., & Wolf, A. L. (2008). Determining the impact of software engineering research on practice. Computer, 41, 39–49.CrossRefGoogle Scholar
  63. Paris, G., Leo, G. D., Menozzi, P., & Gatto, M. (1998). Region-based citation bias in science. Nature, 396, 210.CrossRefGoogle Scholar
  64. Parnas, D. L. (2007). Stop the numbers game. Communications of the ACM, 50, 19–21.CrossRefGoogle Scholar
  65. Patterson, D. A. (2004). The health of research conferences and the dearth of big idea papers. Communication of the ACM, 47, 23–24.CrossRefGoogle Scholar
  66. Persson, O. (2010). Are highly cited papers more international? Scientometrics, 83, 397–401.CrossRefGoogle Scholar
  67. Piwowar, H. (2013). Altmetrics: Value all research products. Nature, 493, 159.Google Scholar
  68. Ponce, F. A., & Lozano, A. M. (2010). Highly cited works in neurosurgery. Part I: The 100 top-cited papers in neurosurgical journals. Journal of Neurosurgery, 112, 223–232.CrossRefGoogle Scholar
  69. Priem, J., Taraborelli, D., Groth, P, Neylon C. (2016) Altmetrics: A manifesto, in, 2010. Last Accessed May 2017.
  70. Pyšek, P., Richardson, D. M., & Jarošík, V. (2006). Who cites who in the invasion zoo: insights from an analysis of the most highly cited papers in invasion ecology. Preslia, 78, 437–468.Google Scholar
  71. Rahm, E. (2008). Comparing the scientific impact of conference and journal publications in computer science. Inf. Serv. Use, 28, 127–128.Google Scholar
  72. Reuters, T. (2016) Highly cited researchers. Last Accessed May 2017.
  73. Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91, 42–61.Google Scholar
  74. Sandström, U., & van den Besselaar, P. (2016). Quantity and/or quality? The importance of publishing many papers. PLoS ONE, 11, e0166149.CrossRefGoogle Scholar
  75. Sarewitz, D. (2016). The pressure to publish pushes down quality. Nature, 533, 147.CrossRefGoogle Scholar
  76. Schliesser, E. (2016) On ‘Me 2 research’ and English-as-Second-language citation rates. Last Accessed May 2017.
  77. Siler, K., Lee, K., & Bero, L. (2015). Measuring the effectiveness of scientific gatekeeping. Proceedings of the National Academy of Sciences, 112, 360–365.CrossRefGoogle Scholar
  78. Tijssen, R., Visser, M., & van Leeuwen, T. (2002). Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics, 54, 381–397.CrossRefGoogle Scholar
  79. Tongai, I. (2013) Incentives for researchers drive up publication output. Last Accessed May 2017.
  80. Uzuner, S. (2008). Multilingual scholars’ participation in core/global academic communities: A literature review. Journal of English for Academic Purposes, 7, 250–263.CrossRefGoogle Scholar
  81. Vardi, M. Y. (2009). Conferences vs. journals in computing research. Communication of the ACM, 52, 5.Google Scholar
  82. Various authors. (2016). Why is it said that judging a paper by citation count is a bad idea?. Last Accessed May 2017.
  83. Vasconcelos, S. M. R., Sorenson, M. M., & Leta, J. (2007). Scientist-friendly policies for non-native English-speaking authors: timely and welcome. Brazilian Journal of Medical and Biological Research, 40, 743–747.CrossRefGoogle Scholar
  84. Vrettas, G., & Sanderson, M. (2015). Conferences versus journals in computer science. Journal of the Association for Information Science and Technology, 66, 2674–2684.CrossRefGoogle Scholar
  85. Wang, M., Yu, G., & Yu, D. (2011). Mining typical features for highly cited papers. Scientometrics, 87, 695–706.CrossRefGoogle Scholar
  86. West, R., & Stenius, K. (2009). Use and abuse of citations. In T. F. Babor, K. Stenius, S. Savva, & J. O’Reill (Eds.), Publishing addiction science: A guide for the perplexed. Multi-Science Publishing Co.Google Scholar
  87. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2000). Experimentation in Software Engineering: An Introduction. Berlin: Kluwer Academic Publishers.CrossRefzbMATHGoogle Scholar
  88. Wong, B. B. M., & Kokko, H. (2005). Is science as global as we think? Trends in Ecology & Evolution, 20, 475–476.CrossRefGoogle Scholar
  89. Wong, W. E., Tse, T. H., Glass, R. L., Basili, V. R., & Chen, T. Y. (2008). An assessment of systems and software engineering scholars and institutions (2001–2005). Journal of Systems and Software, 81, 1059–1062.CrossRefGoogle Scholar
  90. Wong, W. E., Tse, T. H., Glass, R. L., Basili, V. R., & Chen, T. Y. (2009). An assessment of systems and software engineering scholars and institutions (2002–2006). Journal of Systems and Software, 82, 1370–1373.CrossRefGoogle Scholar
  91. Zeng, L., Benatallah, B., Ngu, A. H. H., Dumas, M., Kalagnanam, J., & Chang, H. (2004). QoS-aware middleware for Web services composition. IEEE Transactions on Software Engineering, 30, 311–327.CrossRefGoogle Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2017

Authors and Affiliations

  1. 1.SnT centerUniversity of LuxembourgLuxembourgLuxembourg
  2. 2.Software Engineering Research Group, Department of Computer EngineeringHacettepe UniversityAnkaraTurkey
  3. 3.Department of Informatics/Centro ALGORITMI, School of EngineeringUniversity of MinhoBragaPortugal

Personalised recommendations