Skip to main content

Qualifying the performance evaluation of Big Science beyond productivity, impact and costs

Scientometrics volume 104, pages 295–312 (2015)Cite this article

Abstract

The use of quantitative performance measures to evaluate the productivity, impact and quality of research has spread to almost all parts of public R&D systems, including Big Science where traditional measures of technical reliability of instruments and user oversubscription have been joined by publication counts to assess scientific productivity. But such performance assessment has been shown to lead to absurdities, as the calculated average cost of single journal publications easily may reach hundreds of millions of dollars. In this article, the issue of productivity and impact is therefore further qualified by the use of additional measures such as the immediacy index as well as network analysis to evaluate qualitative aspects of the impact of contemporary Big Science labs. Connecting to previous work within what has been called “facilitymetrics”, the article continues the search for relevant bibliometric measures of the performance of Big Science labs with the use of a case study of a recently opened facility that is advertised as contributing to “breakthrough” research, by using several more measures and thus qualifying the topic of performance evaluation in contemporary Big Science beyond simple counts of publications, citations, and costs.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Notes

  1. http://portal.slac.stanford.edu/sites/lcls_public/Pages/Default.aspx.

  2. https://portal.slac.stanford.edu/sites/lcls_public/Pages/Publications.aspx.

  3. The number of citations going to articles published in 2012 was estimated for the year 2014, since by the time of writing the article this period has not finished. The estimation was made by extrapolating on basis of the growth rate from the second to the third year from the citation curve of article published one year earlier.

References

  • Birgeneau, B. & Shen, Z.-X. (1997). Report of the Basic Energy Sciences Advisory Committee Panel on D.O.E. synchrotron radiation sources and science. Washington, D.C.: U.S. Department of Energy’s Office of Science’s Office of Basic Energy Sciences.

  • Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10), P10008.

    Article  Google Scholar 

  • Doing, P. (2009). Velvet revolution at the synchrotron: Biology, physics, and change in science. Cambridge, MA: MIT Press.

    Google Scholar 

  • Elzinga, A. (2012). Features of the current science policy regime: Viewed in historical perspective. Science and Public Policy, 39, 416–428.

    Article  Google Scholar 

  • Galison, P., & Hevly, B. (Eds.). (1992). Big science—The growth of large-scale research. Stanford, CA: Stanford University Press.

    Google Scholar 

  • Garfield, E. (1955). Citation indexes to science: A new dimension in documentation through association of ideas. Science, 122, 108–111.

    Article  Google Scholar 

  • Gowanlock, M. G., & Gazan, R. (2013). Assessing researcher interdisciplinarity: A case study of the University of Hawaii NASA Astrobiology Institute. Scientometrics, 94(1), 133–161.

    Article  Google Scholar 

  • Greenberg, D. S. (1999/1967). The politics of pure science, 2nd edn. Chicago: The University of Chicago Press.

  • Greenberg, D. S. (2001). Science, money and politics: Political triumph and ethical erosion. Chicago: The University of Chicago Press.

    Google Scholar 

  • Hallonsten, O. (2013). Introducing facilitymetrics: A first review and analysis of commonly used measures of scientific leadership among synchrotron radiation facilities worldwide. Scientometrics, 96(2), 497–513.

    Article  Google Scholar 

  • Hallonsten, O. (2014). How expensive is Big Science? Consequences of using simple publication counts in performance assessment of large scientific facilities. Scientometrics, 100(2), 483–496.

    Article  Google Scholar 

  • Hallonsten, O., & Heinze, T. (2012). Institutional persistence through gradual adaptation: Analysis of national laboratories in the USA and Germany. Science and Public Policy, 39, 450–463.

    Article  Google Scholar 

  • Hallonsten, O., & Heinze, T. (2013). From particle physics to photon science: Multidimensional and multilevel renewal at DESY and SLAC. Science and Public Policy, 40(5), 591–603.

    Article  Google Scholar 

  • Hazelkorn, E. (2011). Rankings and the reshaping of higher education. The battle for world-class excellence. Hampshire: Palgrave Macmillan.

    Book  Google Scholar 

  • Heinze, T., Heidler, R., Heiberger, H., & Riebling, R. (2013). New patterns of scientific growth. How research expanded after the invention of scanning tunneling microscopy and the discovery of Buckminsterfullerenes. Journal of the American Society for Information Science and Technology, 64, 829–843.

    Article  Google Scholar 

  • Kevles, D. J. (1997). Big Science and big politics in the United States: Reflections on the death of the SSC and the life of the Human Genome Project. Historical studies in the physical and biological sciences, 27(2), 269–297.

    Article  Google Scholar 

  • LCLS. (1998). Linac coherent light source (LCLS) design study report. SLAC-R-521, Rev. December 1998, UC-414.

  • LCLS. (2000). LCLS. The first experiments. Report, SLAC, September 2000.

  • LCLS. (2002). Linac coherent light source (LCLS) conceptual design report. SLAC–R–593, April 2002, UC-414.

  • Leone, S. R. (1999). Report of the basic energy sciences advisory committee panel on novel coherent light sources. Washington, D.C.: U.S. Department of Energy’s Office of Science’s Office of Basic Energy Sciences.

  • Münch, R. (2013). Academic capitalism. London: Routledge.

    Google Scholar 

  • Porter, A., & Rafols, I. (2009). Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics, 81(3), 719–745.

    Article  Google Scholar 

  • Small, H. (2010). Maps of science as interdisciplinary discourse: Co-citation contexts and the role of analogy. Scientometrics, 83(3), 835–849.

    MathSciNet  Article  Google Scholar 

  • Stevens, H. (2003). Fundamental physics and its justifications, 1945–1993. Historical Studies in the Physical and Biological Sciences, 34(1), 151–197.

    Article  Google Scholar 

  • Westfall, C. (2008). Surviving the squeeze: National laboratories in the 1970s and 1980s. Historical Studies in the Natural Sciences, 38(4), 475–478.

    MathSciNet  Article  Google Scholar 

  • Westfall, C. (2012). Institutional persistence and the material transformation of the US national labs: the curious story of the advent of the advanced photon source. Science and Public Policy, 39(4), 439–449.

    Article  Google Scholar 

  • Westwick, P. J. (2003). The national labs: Science in an American system 1947–1974. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Wildavsky, B. (2010). The great brain race: How global universities are reshaping the world. Princeton: Princeton University Press.

    Google Scholar 

  • Woods, H. R. (2006). New life for a linac. Symmetry, 3(7), 10–15.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Education and Social Sciences, Wuppertal University, Gaussstr. 20, 42119, Wuppertal, Germany

    Richard Heidler

  2. Department of Business Administration, Lund University, P.O. Box 7080, 220 07, Lund, Sweden

    Olof Hallonsten

Authors
  1. Richard Heidler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Olof Hallonsten
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Olof Hallonsten.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Heidler, R., Hallonsten, O. Qualifying the performance evaluation of Big Science beyond productivity, impact and costs. Scientometrics 104, 295–312 (2015). https://doi.org/10.1007/s11192-015-1577-7

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-015-1577-7

Keywords

  • Big Science
  • Quality assessment
  • Performance assessment
  • Productivity
  • Network analysis