Abstract
The use of quantitative performance measures to evaluate the productivity, impact and quality of research has spread to almost all parts of public R&D systems, including Big Science where traditional measures of technical reliability of instruments and user oversubscription have been joined by publication counts to assess scientific productivity. But such performance assessment has been shown to lead to absurdities, as the calculated average cost of single journal publications easily may reach hundreds of millions of dollars. In this article, the issue of productivity and impact is therefore further qualified by the use of additional measures such as the immediacy index as well as network analysis to evaluate qualitative aspects of the impact of contemporary Big Science labs. Connecting to previous work within what has been called “facilitymetrics”, the article continues the search for relevant bibliometric measures of the performance of Big Science labs with the use of a case study of a recently opened facility that is advertised as contributing to “breakthrough” research, by using several more measures and thus qualifying the topic of performance evaluation in contemporary Big Science beyond simple counts of publications, citations, and costs.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The number of citations going to articles published in 2012 was estimated for the year 2014, since by the time of writing the article this period has not finished. The estimation was made by extrapolating on basis of the growth rate from the second to the third year from the citation curve of article published one year earlier.
References
Birgeneau, B. & Shen, Z.-X. (1997). Report of the Basic Energy Sciences Advisory Committee Panel on D.O.E. synchrotron radiation sources and science. Washington, D.C.: U.S. Department of Energy’s Office of Science’s Office of Basic Energy Sciences.
Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10), P10008.
Doing, P. (2009). Velvet revolution at the synchrotron: Biology, physics, and change in science. Cambridge, MA: MIT Press.
Elzinga, A. (2012). Features of the current science policy regime: Viewed in historical perspective. Science and Public Policy, 39, 416–428.
Galison, P., & Hevly, B. (Eds.). (1992). Big science—The growth of large-scale research. Stanford, CA: Stanford University Press.
Garfield, E. (1955). Citation indexes to science: A new dimension in documentation through association of ideas. Science, 122, 108–111.
Gowanlock, M. G., & Gazan, R. (2013). Assessing researcher interdisciplinarity: A case study of the University of Hawaii NASA Astrobiology Institute. Scientometrics, 94(1), 133–161.
Greenberg, D. S. (1999/1967). The politics of pure science, 2nd edn. Chicago: The University of Chicago Press.
Greenberg, D. S. (2001). Science, money and politics: Political triumph and ethical erosion. Chicago: The University of Chicago Press.
Hallonsten, O. (2013). Introducing facilitymetrics: A first review and analysis of commonly used measures of scientific leadership among synchrotron radiation facilities worldwide. Scientometrics, 96(2), 497–513.
Hallonsten, O. (2014). How expensive is Big Science? Consequences of using simple publication counts in performance assessment of large scientific facilities. Scientometrics, 100(2), 483–496.
Hallonsten, O., & Heinze, T. (2012). Institutional persistence through gradual adaptation: Analysis of national laboratories in the USA and Germany. Science and Public Policy, 39, 450–463.
Hallonsten, O., & Heinze, T. (2013). From particle physics to photon science: Multidimensional and multilevel renewal at DESY and SLAC. Science and Public Policy, 40(5), 591–603.
Hazelkorn, E. (2011). Rankings and the reshaping of higher education. The battle for world-class excellence. Hampshire: Palgrave Macmillan.
Heinze, T., Heidler, R., Heiberger, H., & Riebling, R. (2013). New patterns of scientific growth. How research expanded after the invention of scanning tunneling microscopy and the discovery of Buckminsterfullerenes. Journal of the American Society for Information Science and Technology, 64, 829–843.
Kevles, D. J. (1997). Big Science and big politics in the United States: Reflections on the death of the SSC and the life of the Human Genome Project. Historical studies in the physical and biological sciences, 27(2), 269–297.
LCLS. (1998). Linac coherent light source (LCLS) design study report. SLAC-R-521, Rev. December 1998, UC-414.
LCLS. (2000). LCLS. The first experiments. Report, SLAC, September 2000.
LCLS. (2002). Linac coherent light source (LCLS) conceptual design report. SLAC–R–593, April 2002, UC-414.
Leone, S. R. (1999). Report of the basic energy sciences advisory committee panel on novel coherent light sources. Washington, D.C.: U.S. Department of Energy’s Office of Science’s Office of Basic Energy Sciences.
Münch, R. (2013). Academic capitalism. London: Routledge.
Porter, A., & Rafols, I. (2009). Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics, 81(3), 719–745.
Small, H. (2010). Maps of science as interdisciplinary discourse: Co-citation contexts and the role of analogy. Scientometrics, 83(3), 835–849.
Stevens, H. (2003). Fundamental physics and its justifications, 1945–1993. Historical Studies in the Physical and Biological Sciences, 34(1), 151–197.
Westfall, C. (2008). Surviving the squeeze: National laboratories in the 1970s and 1980s. Historical Studies in the Natural Sciences, 38(4), 475–478.
Westfall, C. (2012). Institutional persistence and the material transformation of the US national labs: the curious story of the advent of the advanced photon source. Science and Public Policy, 39(4), 439–449.
Westwick, P. J. (2003). The national labs: Science in an American system 1947–1974. Cambridge, MA: Harvard University Press.
Wildavsky, B. (2010). The great brain race: How global universities are reshaping the world. Princeton: Princeton University Press.
Woods, H. R. (2006). New life for a linac. Symmetry, 3(7), 10–15.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Heidler, R., Hallonsten, O. Qualifying the performance evaluation of Big Science beyond productivity, impact and costs. Scientometrics 104, 295–312 (2015). https://doi.org/10.1007/s11192-015-1577-7
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-015-1577-7