Skip to main content
SpringerLink
Log in

An empirical approach to compare the performance of heterogeneous academic fields

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In this paper, we propose a ‘scaling’ approach to compare the scientific performance of Italian heterogeneous academic disciplines. This method is based on the idea that, after eliminating the percentages of ‘silent’ researchers, the distribution of bibliometric parameters of the different academic fields can be superimposed and collapse into a unique master curve by a single scaling parameter. By using data on the scientific production of around 2,500 scholars of the university of Rome ‘La Sapienza’ from the Web of Science from 2004 to 2008, we (i) demonstrate the existence of a master curve, (ii) determine the scaling factors that work like rates of substitution to compare the scientific production across different academic fields on a common ground, (iii) show that the master bibliometric distribution follows a log-normal law and (iv) illustrate the relevance of the proposed approach for research assessment and allocation of competitive funding at the university level.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. For a comparative analysis on European universities microdata, see Daraio et al. (2011).

  2. For a macro bibliometric analysis of Italian science with respect to the main European countries over the period 1980–2009 and its implications in terms of funding, see Daraio and Moed (2011).

  3. In particular, Article 6 paragraph 7 of Law n. 240/2010 states that it is the ‘exclusive competence of the university to evaluate positively or negatively the activity of individual researchers’, and Article 5, paragraph 1 of Law n. 240/2010 establishes the ‘enhancement of quality and the efficiency of universities and consequent introduction of incentive schemes in the distribution of public resources on the basis of criteria defined ex-ante’.

  4. For a general presentation and rich empirical evidence on universities as autonomous decision-making units within their institutional contexts and Italian university performance within the European landscape, see Bonaccorsi and Daraio (2007a, b) and Daraio et al. (2011).

  5. Given the large number of scholars considered in this analysis, we consider that its representativeness is reasonable. A discussion on this issue is reported in “Data” section.

  6. We investigate the SSDs for which at least 10 observations were available.

  7. For a comprehensive presentation of bibliometric and informetric distributions, see Egghe and Rousseau (1990). Simon (1955), Laherrére and Sornette (1998) and Stock (2006) are useful references for a general overview on skew distributions. For a presentation of the mechanisms for generating power laws and the methods to detect them, see Newman (2005).

  8. When the data are noisy (because we have a small sample and consequently small numbers) the calculation of cumulative distribution, by definition (Grimmett and Stirzaker 2001), will yield larger numbers and reduce the percentage error.

  9. Where P stands for publications (PUB),  C stands for citations (CIT) and IF for Impact factor.

  10. The indicators have been obtained by Sapienza university under a commercial agreement from Research Value S.r.l., which elaborated the data under license from WoS of Thomson Reuters. We did not have access to the database on which the indicators have been calculated; therefore, we were unable to calculate additional or different bibliometric indicators than those provided by Research Value.

  11. Moreover, this analysis has been performed by Research Value S.r.l. under a commercial agreement with Sapienza University.

  12. We conducted an analysis on comparable data from the university of Naples Federico II, which increased our sample representativeness from approximately 7 % to around 12 %, and obtained identical results (within their statistical uncertainties) for the available SSDs.

  13. It is important to note that being ‘silent’ does not mean being “inactive”; for example, it is possible that a ‘silent’ researcher in our analysis has published many papers in journals not covered by WoS, or he/she has been a promotor of a big and challenging research project that will radically change a discipline, but the outputs of the project are not yet codified in scientific articles, or even that the researcher has played an important role in advising governmental body on strategic issues.

  14. Note that B(s) will be subtracted in the following steps, see Eq. (2).

  15. A similar distribution was found in van Raan (2001).

  16. More detailed results are not shown to save space but are available from the authors upon request.

  17. This is beyond the scope of the present paper and is a potential topic for future research.

  18. In Spain, the research activity of professors is assessed on a voluntarily individual base by ‘la Comision Nacional Evaluadora de la Actividad Investigadora (CNEAI)’ performed on a time frame of six years (sexenios de investigacion). If a professor gets a positive assessment, he/she receives a productivity bonus incentive. The evaluation is conducted by different panels, according to the specific SSD considered.

  19. Therefore, the literature refers to ‘scaling’ relationships to describe the correlation between CIT and PUB. Undoubtedly, the the meaning of ‘scaling’ in this context is completely different from the scaling approach described in the previous sections, which was used to search for a master curve (and its related scaling factors) to compare heterogeneous academic fields.

  20. This statement is confirmed by descriptive analysis, which has not been shown to save space. Interestingly, the estimated scaling factors \((\gamma_{\varepsilon,s})\) are useful for calculating mean, variance and median (of PUB,  CIT and IF by SSD)—without requiring access to the whole underlying database—by simply multiplying \(\gamma_{\varepsilon,s}\) with \(\frac{3}{2}\rm{ln}(3)\) (for the average), 1.59 (for the variance) and ln(4) (for the median), respectively.

  21. For a general overview on log-normal distributions, see Limpert et al. (2001).

References

  • Albarran, P., Crespo, J. A., Ortugno, I., & Ruiz-Castillo, J. (2011). The skewness of science in 219 sub-fields and a number of aggregates. Scientometrics, 88, 385–397.

    Article  Google Scholar 

  • Bak, P., Tang, C., & Wiesenfeld, K. (1987). Self-organized criticality: An explanation of 1/f noise. Physical Review Letters, 59(4), 381–384.

    Article  MathSciNet  Google Scholar 

  • Bonaccorsi, A., & Daraio, C. (2007a). Theoretical perspectives on university strategy. In A. Bonaccorsi & C. Daraio (Eds.), Universities and strategic knowledge creation. Specialization and performance in Europe (pp. 3–30). Cheltenham: Edward Elgar Publisher.

    Chapter  Google Scholar 

  • Bonaccorsi, A., & Daraio, C. (2007b). Universities as strategic knowledge creators. Some preliminary evidence. In A. Bonaccorsi & C. Daraio (Eds.), Universities and strategic knowledge creation. Specialization and performance in Europe (pp. 31–84). Cheltenham: Edward Elgar Publisher.

    Chapter  Google Scholar 

  • Costas, R., van Leeuwen, T., & Bordons, M. (2010). A bibliometric classificatory approach for the study and assessment of research performance at the individual level: The effects of age on productivity and impact. Journal of the American Society for Information Science and Technology, 61(8), 1564–1581.

    Google Scholar 

  • D’Angelo, C. A., Giuffrida, C., & Abramo, G. (2010). A heuristic approach to author name disambiguation in large-scale bibliometric databases. Journal of the American Society for Information Science and Technology, 62(2), 257–269.

    Article  Google Scholar 

  • Daraio, C., & Moed, H. F. (2011). Is Italian science declining?. Research Policy, 40(10), 1380–1392.

    Article  Google Scholar 

  • Daraio, C. et al. (2011). The European University landscape: A micro characterization based on evidence from the Aquameth project. Research Policy, 40, 148–164.

    Article  Google Scholar 

  • Deiaco, E., Holmén, M., & McKelvey, M. (2009). What does it mean conceptually that universities compete? In M. McKelvey, M. Holmén (eds.), Learning to compete in European Universities: From social institution to knowledge business (pp. 300–328). Cheltenham: Edward Elgar.

    Google Scholar 

  • Deiaco, E., Hughes, A., & McKelvey, M. (2012). Universities as strategic actors in the knowledge economy. Cambridge Journal of Economics, 36, 525–541.

    Article  Google Scholar 

  • Egghe, L., & Rousseau, R. (1990). Introduction to informetrics. Quantitative methods in library, documentation and information science. Amsterdam: Elsevier.

    Google Scholar 

  • Evans, T. S., Hopkins, N., & Kaube, B. S. (2012). Universality of performance indicators based on citation and reference counts. Scientometrics, 93, 473–495.

    Google Scholar 

  • Glanzel, W. (2010). The role of the h-index and the characteristic scores and scales in testing the tail properties of scientometric distributions. Scientometrics, 83, 697–709.

    Article  Google Scholar 

  • Grimmett, G., & Stirzaker, D. (2001). Probability and random processes (3rd ed.). Oxford: Oxford University Press.

    Google Scholar 

  • Katz, J. S. (1999). The self-similar science system. Research Policy, 28, 501–517.

    Article  Google Scholar 

  • Katz, J. S. (2000). Scale independent indicators and research evaluation. Science and Public Policy, 27(1), 23–36.

    Article  Google Scholar 

  • Knorr-Cetina, K. (1995). Laboratory studies: The cultural approach to the study of science. In S. Jasanoff, G. E. Markle, J. C. Petersen, & T. Pinch (Eds.), Handbook of science and technology studies (pp. 140–166). London: Sage.

    Chapter  Google Scholar 

  • Laherrére, J., Sornette, D. (1998). Stretched exponential distributions in nature and economy: “fat tails” with characteristic scales. European Physical Journal B, 2, 525–539.

    Article  Google Scholar 

  • Langton, C. G. (1990). Computation at the edge of chaos: Phase transitions and emergent computation. Physica D, 42, 1–3, 12–37.

    Google Scholar 

  • Laredo, P., & Mustar, P. (Eds.). (2001). Research and innovation policies in the new global economy: An international comparative analysis. Cheltenham/Northampton, MA: Edward Elgar.

    Google Scholar 

  • LERU (2012). Research universities and research assessment. Position paper, May 2012.

  • Limpert, E., Stahel, W. A., & Abbt, M. (2001). Log-normal distributions across the sciences: Keys and clues. BioScience, 51(5), 341–352.

    Article  Google Scholar 

  • Lotka, A. J. (1926). The frequency distribution of scientific productivity. Journal of the Washington Academy of Sciences, 16, 317–323.

    Google Scholar 

  • Ma, S.-K. (1976). Modern theory of critical phenomena. New York: Benjamin. ISBN: 0-8053-6671-7.

  • Merton, R. K. (1968). The Matthew effect in science. Science, 159(3810), 56–63.

    Article  Google Scholar 

  • Naranan, S. (1971). Power law relations in science bibliography—a self consistent interpretation. Journal of Documentation, 27(2), 83–97.

    Article  Google Scholar 

  • Newman, M. E. J. (2005). Power laws, Pareto distributions and Zipfs law. Contemporary Physics, 46, 323–351.

    Article  Google Scholar 

  • Press, W. H., Teukolsky, S. A., Vetterling, W. T., Flannery, B. P. (2007). Numerical recipes 3rd edition: The art of scientific computing. New York: Cambridge University Press.

    Google Scholar 

  • Price, D. D. S. (1976). A general theory of bibliometric and other cumulative advantage processes. Journal of the American Society for Information Science, 27(5), 292–306.

    Article  Google Scholar 

  • Radicchi, F., Fortunato, S., Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences of the United States of America, 105, 17268–17272.

    Article  Google Scholar 

  • Redner, S. (1998). How popular is your paper? An empirical study of the citation distribution. European Physical Journal B, 4, 131–134.

    Article  Google Scholar 

  • Seglen, P. (1992). The skewness of science. Journal of the American Society for Information Science, 43, 628–638.

    Article  Google Scholar 

  • Simon, H. (1955). On a class of skew distribution functions. Biometrika, 42(3-4), 425–440.

    Article  MathSciNet  MATH  Google Scholar 

  • Stock, W. G. (2006). On relevance distributions. Journal of the American Society for Information Science and Technology, 57(8), 1126–1129.

    Article  Google Scholar 

  • van Raan, A. F. J. (2001). Competition amongst scientists for publication status: Toward a model of scientific publication and citation distributions. Scientometrics, 51, 347–357.

    Article  Google Scholar 

  • van Raan, A. F. J. (2006). Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics, 67(3), 491–502.

    Google Scholar 

  • van Raan, A. F. J. (2008). Scaling rules in the science system: Influence of field-specific citation characteristics on the impact of research groups. Journal of the American Society for Information Science and Technology, 59(4), 565–576.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., van Raan, A. F. J. (2012). Universality of citation distributions revisited. Journal of the American Society for Information Science and Technology, 63(1), 72–77.

    Article  Google Scholar 

Download references

Acknowledgments

This paper was presented during a MORE@DIAG Seminar, 19 September 2012, Sapienza University of Rome. We thank Henk F. Moed, Gianluca Imbriani and Francesco Giannino as well as the seminar participants for their valuable comments and discussions. We express our gratitude to two anonymous referees for their comments, which have enabled us to considerably improve our work. The usual disclaimers apply.

Author information

Authors and Affiliations

  1. Department of Physics, University of Rome ‘La Sapienza’ and Center for Life NanoScience@LaSapienza, IIT, University of Rome ‘La Sapienza’, Viale Regina Elena 295, Rome, Italy

    Giancarlo Ruocco

  2. Department of Computer, Control and Management Engineering Antonio Ruberti (DIAG), University of Rome ‘La Sapienza’, via Ariosto, 25, 00185, Rome, Italy

    Cinzia Daraio

Authors
  1. Giancarlo Ruocco
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cinzia Daraio
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cinzia Daraio.

Appendix

Appendix

Detailed results

In this section, we present the detailed results obtained for all the SSDs with at least 10 observations, grouped by disciplinary area. ξ P is the number of publications required per year to reach the 50-th percentile of the specific SSD; ξ IF is the average Impact factor of the journals (normalized at the subject category level) to reach the 50-th percentile of the specific SSD; ξ C is the average number of citations of a scholar’s publications normalized on the Italian median (See Fig. 11

Fig. 11
figure 11

Top panel: Distributions of IF for selected SSDs without silent researchers plotted as a function of the scaled variable IF/ξ IF . Bottom panel: Distributions of C for selected SSDs without silent researchers plotted as a function of the scaled variable CIT/ξ C . Each point represents a scholar

Full size image

; Tables 2, 3, 4, 5, 6, 7, 8, 9).

Table 2 Results: Area 01—mathematics and informatics
Full size table
Table 3 Results: Area 02—physics
Full size table
Table 4 Results: Area 03—chemistry
Full size table
Table 5 Results: Area 04—earth sciences
Full size table
Table 6 Results: Area 05—biology
Full size table
Table 7 Results: Area 06—medicine
Full size table
Table 8 Results: Area 09—industrial and information engineering
Full size table
Table 9 Area 11—history, philosophy, pedagogy and psychology
Full size table

Rights and permissions

Reprints and permissions

About this article

Cite this article

Ruocco, G., Daraio, C. An empirical approach to compare the performance of heterogeneous academic fields. Scientometrics 97, 601–625 (2013). https://doi.org/10.1007/s11192-013-1038-0

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-013-1038-0

Keywords

Search

Navigation