Abstract
Although the nuclear era and the Cold War superpower competition have long since passed, governments are still investing in Big Science, although these large facilities are nowadays mostly geared towards areas of use closer to utility. Investments in Big Science are also motivated not only by promises of scientific breakthroughs but also by expectations (and demands) of measurable impact, and with an emerging global market of competing user-oriented Big Science facilities, quantitative measures of productivity and quality have become mainstream. Among these are rather simple and one-sided publication counts. This article uses publication counts and figures of expenditure for three cases that are disparate but all represent the state-of-the-art of Big Science of their times, discussing at depth the problems of using simple publication counts as a measure of performance in science. Showing, quite trivially, that Big Science is very expensive, the article also shows the absurd consequences of consistently using simple publication counts to display productivity and quality of Big Science, and concludes that such measures should be deemed irrelevant for analyses on the level of organizations in science and replaced by qualitative assessment of the content of the science produced.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
See, e.g., the ‘facts and figues’ section of the annual reports of the European Synchrotron Radiation Facility (the ‘ESRF Highlights’), available at http://www.esrf.eu/UsersAndScience/Publications/Highlights or the diagrams at the Publications page for LCLS, available at https://portal.slac.stanford.edu/sites/lcls_public/Pages/Publications.aspx.
Although, obviously, the size of the teams doing such data taking have grown from a handful of scientists and engineers in the 50s and 60s to thousands today (e.g. Bodnarczuk and Hoddeson 2008). In this sense, the CERN facilities of the 1950s and 60s are clearly more apt for comparison with the ESRF and LCLS than today’s CERN, where team size routinely exceeds a thousand people.
The figures for numbers of publications were obtained by manual counting. The figures for expenditure were recalculated to 2012 US dollars (USD) to ensure comparability. This recalculation proceeded in the following way. First, expenditures for CERN are originally reported in Swiss franc (CHF), 1958 prices, and we converted them to yearly nominal values by the use of inflation figures for Switzerland for the years 1958–1965 obtained from the Federal Statistics Office of Switzerland (www.bfs.ch). Each year’s nominal values were converted to US-dollars (USD) with the aid of the year by year exchange rates CHF-USD as available through MeasuringWorth (www.measuringworth.com/exchangeglobal). It shall be noted that construction costs and operations costs are not separated in the data source for CERN. Second, construction and operations expenditures for ESRF are originally reported in current prices French franc (FF) for every year except expenditures for construction for the years 1988–1994, which were given in constant 1987 prices. The 1988–1994 expenditures were hence converted to yearly nominal values by the use of inflation rates for France 1988–1994 (obtained from www.inflation.eu). Thereafter, the nominal figures for every year were converted to USD with the aid of the year by year exchange rates FF-USD as available through MeasuringWorth (www.measuringworth.com/exchangeglobal). Third, construction and operations expenditures for LCLS are originally reported in USD, nominal values for every year. Having calculated all figures for all three cases to nominal USD, these were adjusted to 2012 prices by the use of the official Consumer Price Index data for the years 1954–1965, 1998–1997, and 2002–2012 from the US bureau of labor statistics (www.bls.gov).
An extension of the analysis conducted here, to citations and journal impacts, would not change this. Citation cultures are also severely different now than fifty years ago, and journal impacts are similarly impossible to compare.
These figures have also been obtained from annual reports (ESRF Highlights) for the relevant years.
The figures are averages for the years 2004–2010. What precludes the data from Hallonsten (2013a) to be used in this article is the lack of comprehensive figures on expenditures from the first instance of construction of the facilities and on. The difference between the ESRF and APS is erased when accounting for the fact that the ESRF has ten more experimental stations than does the APS, but in a similar exercise the SPring-8 comes out even worse, running twenty beamlines more than the ESRF. A thorough discussion on these differences is found in Hallonsten (2013a: 508–512).
References
Agrell, W. (2012). Framing prospects and risk in the public promotion of ESS-Scandinavia. Science and Public Policy, 39, 429–438.
Bodnarczuk, M., & Hoddeson, L. (2008). Megascience in particle physics: The birth of an experiment string at fermilab. Historical Studies in the Natural Sciences, 38(4), 508–534.
Eisler, M. (2013). The Ennobling Unity of Science and Technology: Materials Sciences and Engineering, the Department of Energy, and the Nanotechnology Enigma. Minerva, 51, 225–251.
Elzinga, A. (2012). Features of the current science policy regime: Viewed in historical perspective. Science and Public Policy, 39, 416–428.
Giudice, G. F. (2012). Big Science and the Large Hadron Collider. Physics in Perspective, 14, 95–112.
Guston, D. H. (1999). Stabilizing the boundary between US politics and science: The role of the office of technology transfer as a boundary organization. Social Studies of Science, 29(1), 87–111.
Hallonsten, O. (2013a). Introducing facilitymetrics: A first review and analysis of commonly used measures of scientific leadership among synchrotron radiation facilities worldwide. Scientometrics, 96(2), 497–513.
Hallonsten, O. (2013b). Myths and realities of the ESS project: A systematic scrutiny of readily accepted truths. In T. Kaiserfeld & T. O’Dell (Eds.), Legitimizing ESS: Big Science as a collaboration across boundaries. Lund: Nordic Academic Press.
Hallonsten, O. (2014). The politics of European collaboration in big science. In M. Mayer, M. Carpes, & R. Knoblich (Eds.), International relations and the global politics of science and technology. Dordrecht: Springer.
Hallonsten, O. (submitted). The parasites: Synchrotron radiation at SLAC, 1972–1992.
Hallonsten, O., & Heinze, T. (2012). Institutional persistence through gradual adaptation: Analysis of national laboratories in the USA and Germany. Science and Public Policy, 39, 450–463.
Hallonsten, O., & Heinze, T. (2013). From particle physics to photon science: Multidimensional and multilevel renewal at DESY and SLAC. Science and Public Policy, 40, 591–603.
Hallonsten, O. & Heinze, T. (submitted). Formation and expansion of a new organizational field in experimental science: Synchrotron Radiation Labs in Europe and the United States, 1974–2012.
Hazelkorn, E. (2011). Rankings and the reshaping of higher education. The battle for world-class excellence. Hampshire: Palgrave Macmillan.
Irvine, J., & Martin, B. (1984). CERN: Past performance and future prospects. I. CERN’s position in world high-energy physics. Research Policy, 13(4), 183–210.
Jacob, M., & Hallonsten, O. (2012). The persistence of big science and megascience in research and innovation policy. Science and Public Policy, 39, 411–415.
Johnson, A. (2004). The end of pure science: science policy from Bayh–Dole to the NNI. In D. Baird, A. Nordmann, & J. Schummer (Eds.), Discovering the nanoscale. Amsterdam: IOS Press.
Kevles, D. J. (1997). Big Science and big politics in the United States: Reflections on the death of the SSC and the life of the Human Genome Project. Historical Studies in the Physical and Biological Sciences, 27(2), 269–297.
Krige, J. (2003). The politics of european scientific collaboration. In J. Krige & D. Pestre (Eds.), Companion to science in the twentieth century. London: Routledge.
Krige, J. (2006). American hegemony and the postwar reconstruction of science in Europe. Cambridge: MIT Press.
Lohrmann, E., & Söding, P. (2009). Von schnellen Teilchen und hellem Licht: 50 Jahre Deutsches Elektronen-Synchrotron DESY. New York: Wiley.
McCray, W. P. (2000). Large telescopes and the moral economy of recent astronomy. Social Studies of Science, 30(5), 685–711.
Pestre, D., & Krige, J. (1992). Some thoughts on the early history of CERN. In P. Galison & B. Hevly (Eds.), Big Science—The growth of large-scale research. Stanford: Stanford University Press.
Westfall, C. (2010). Surviving to tell the tale: Argonne’s intense pulsed neutron source from an ecosystem perspective. Historical Studies in the Natural Sciences, 40(3), 350–398.
Westfall, C. (2012). Institutional persistence and the material transformation of the US National Labs: The curious story of the advent of the advanced photon source. Science and Public Policy, 39, 439–449.
Wildavsky, B. (2010). The Great Brain race: How global universities are reshaping the world. Princeton: Princeton University Press.
Woods, H. R. (2006). New life for a linac. Symmetry, 3(7), 10–15.
Acknowledgments
The author is grateful to Richard Heidler and Heiko Heiberger for their assistance in processing the quantitative material used in the article.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hallonsten, O. How expensive is Big Science? Consequences of using simple publication counts in performance assessment of large scientific facilities. Scientometrics 100, 483–496 (2014). https://doi.org/10.1007/s11192-014-1249-z
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-014-1249-z