, Volume 104, Issue 1, pp 295–312 | Cite as

Qualifying the performance evaluation of Big Science beyond productivity, impact and costs

  • Richard Heidler
  • Olof Hallonsten


The use of quantitative performance measures to evaluate the productivity, impact and quality of research has spread to almost all parts of public R&D systems, including Big Science where traditional measures of technical reliability of instruments and user oversubscription have been joined by publication counts to assess scientific productivity. But such performance assessment has been shown to lead to absurdities, as the calculated average cost of single journal publications easily may reach hundreds of millions of dollars. In this article, the issue of productivity and impact is therefore further qualified by the use of additional measures such as the immediacy index as well as network analysis to evaluate qualitative aspects of the impact of contemporary Big Science labs. Connecting to previous work within what has been called “facilitymetrics”, the article continues the search for relevant bibliometric measures of the performance of Big Science labs with the use of a case study of a recently opened facility that is advertised as contributing to “breakthrough” research, by using several more measures and thus qualifying the topic of performance evaluation in contemporary Big Science beyond simple counts of publications, citations, and costs.


Big Science Quality assessment Performance assessment Productivity Network analysis 


  1. Birgeneau, B. & Shen, Z.-X. (1997). Report of the Basic Energy Sciences Advisory Committee Panel on D.O.E. synchrotron radiation sources and science. Washington, D.C.: U.S. Department of Energy’s Office of Science’s Office of Basic Energy Sciences.Google Scholar
  2. Blondel, V. D., Guillaume, J.-L., Lambiotte, R., & Lefebvre, E. (2008). Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment, 2008(10), P10008.CrossRefGoogle Scholar
  3. Doing, P. (2009). Velvet revolution at the synchrotron: Biology, physics, and change in science. Cambridge, MA: MIT Press.Google Scholar
  4. Elzinga, A. (2012). Features of the current science policy regime: Viewed in historical perspective. Science and Public Policy, 39, 416–428.CrossRefGoogle Scholar
  5. Galison, P., & Hevly, B. (Eds.). (1992). Big science—The growth of large-scale research. Stanford, CA: Stanford University Press.Google Scholar
  6. Garfield, E. (1955). Citation indexes to science: A new dimension in documentation through association of ideas. Science, 122, 108–111.CrossRefGoogle Scholar
  7. Gowanlock, M. G., & Gazan, R. (2013). Assessing researcher interdisciplinarity: A case study of the University of Hawaii NASA Astrobiology Institute. Scientometrics, 94(1), 133–161.CrossRefGoogle Scholar
  8. Greenberg, D. S. (1999/1967). The politics of pure science, 2nd edn. Chicago: The University of Chicago Press.Google Scholar
  9. Greenberg, D. S. (2001). Science, money and politics: Political triumph and ethical erosion. Chicago: The University of Chicago Press.Google Scholar
  10. Hallonsten, O. (2013). Introducing facilitymetrics: A first review and analysis of commonly used measures of scientific leadership among synchrotron radiation facilities worldwide. Scientometrics, 96(2), 497–513.CrossRefGoogle Scholar
  11. Hallonsten, O. (2014). How expensive is Big Science? Consequences of using simple publication counts in performance assessment of large scientific facilities. Scientometrics, 100(2), 483–496.CrossRefGoogle Scholar
  12. Hallonsten, O., & Heinze, T. (2012). Institutional persistence through gradual adaptation: Analysis of national laboratories in the USA and Germany. Science and Public Policy, 39, 450–463.CrossRefGoogle Scholar
  13. Hallonsten, O., & Heinze, T. (2013). From particle physics to photon science: Multidimensional and multilevel renewal at DESY and SLAC. Science and Public Policy, 40(5), 591–603.CrossRefGoogle Scholar
  14. Hazelkorn, E. (2011). Rankings and the reshaping of higher education. The battle for world-class excellence. Hampshire: Palgrave Macmillan.CrossRefGoogle Scholar
  15. Heinze, T., Heidler, R., Heiberger, H., & Riebling, R. (2013). New patterns of scientific growth. How research expanded after the invention of scanning tunneling microscopy and the discovery of Buckminsterfullerenes. Journal of the American Society for Information Science and Technology, 64, 829–843.CrossRefGoogle Scholar
  16. Kevles, D. J. (1997). Big Science and big politics in the United States: Reflections on the death of the SSC and the life of the Human Genome Project. Historical studies in the physical and biological sciences, 27(2), 269–297.CrossRefGoogle Scholar
  17. LCLS. (1998). Linac coherent light source (LCLS) design study report. SLAC-R-521, Rev. December 1998, UC-414.Google Scholar
  18. LCLS. (2000). LCLS. The first experiments. Report, SLAC, September 2000.Google Scholar
  19. LCLS. (2002). Linac coherent light source (LCLS) conceptual design report. SLAC–R–593, April 2002, UC-414.Google Scholar
  20. Leone, S. R. (1999). Report of the basic energy sciences advisory committee panel on novel coherent light sources. Washington, D.C.: U.S. Department of Energy’s Office of Science’s Office of Basic Energy Sciences.Google Scholar
  21. Münch, R. (2013). Academic capitalism. London: Routledge.Google Scholar
  22. Porter, A., & Rafols, I. (2009). Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics, 81(3), 719–745.CrossRefGoogle Scholar
  23. Small, H. (2010). Maps of science as interdisciplinary discourse: Co-citation contexts and the role of analogy. Scientometrics, 83(3), 835–849.MathSciNetCrossRefGoogle Scholar
  24. Stevens, H. (2003). Fundamental physics and its justifications, 1945–1993. Historical Studies in the Physical and Biological Sciences, 34(1), 151–197.CrossRefGoogle Scholar
  25. Westfall, C. (2008). Surviving the squeeze: National laboratories in the 1970s and 1980s. Historical Studies in the Natural Sciences, 38(4), 475–478.MathSciNetCrossRefGoogle Scholar
  26. Westfall, C. (2012). Institutional persistence and the material transformation of the US national labs: the curious story of the advent of the advanced photon source. Science and Public Policy, 39(4), 439–449.CrossRefGoogle Scholar
  27. Westwick, P. J. (2003). The national labs: Science in an American system 1947–1974. Cambridge, MA: Harvard University Press.Google Scholar
  28. Wildavsky, B. (2010). The great brain race: How global universities are reshaping the world. Princeton: Princeton University Press.Google Scholar
  29. Woods, H. R. (2006). New life for a linac. Symmetry, 3(7), 10–15.Google Scholar

Copyright information

© Akadémiai Kiadó, Budapest, Hungary 2015

Authors and Affiliations

  1. 1.Department of Education and Social SciencesWuppertal UniversityWuppertalGermany
  2. 2.Department of Business AdministrationLund UniversityLundSweden

Personalised recommendations