Skip to main content
SpringerLink
Log in
Book cover

Incentives and Performance pp 307–321Cite as

Double-Blind Peer Review: How to Slaughter a Sacred Cow

  • Chapter
  • First Online:

Abstract

The performance evaluation system in academia has been much criticized during the last years. But there are few suggestions how to improve it. In particular double blind pre-publication peer review has become a sacred cow that has not been touched. We analyze the flaws of the present system and discuss open post-publication peer review as a promising alternative.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD   109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    Whether peer reviews are really independent is questionable as long as they are part of research communities that share assumptions.

  2. 2.

    According to Wilhite and Fong (2012) journals published by commercial companies show greater use of coercive tactics. These authors also find that highly ranked journals are more likely to coerce, but the direction of causality is unclear.

  3. 3.

    According to Popper (2005) falsification of hypotheses is even the only avenue to scientific progress.

  4. 4.

    OE = Open Evaluation.

  5. 5.

    The “desired impact factor” Kriegeskorte mentions is different from a journal impact factor. Therefore, the problem of free-riding on a small number of highly cited articles in high-impact-journals is avoided with Kriegeskorte’s approach. However, when certain PEFs will have gained prominence, counting exercises are likely to set in.

References

  • Abramo GD, Angelo CA, Caprasecca A (2009) Allocative efficiency in public research funding: can bibliometrics help? Res Policy 38:206–215

    Article  Google Scholar 

  • Adler NJ, Harzing A-W (2009) When knowledge wins: transcending the sense and nonsense of academic rankings. Acad Manag Learn Edu 8:72–95

    Article  Google Scholar 

  • Adler R, Ewing J, Taylor P (2008) Citation statistics. Report from the International Mathematical Union (IMU) in cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS). Stat Sci 24:1–14

    Article  Google Scholar 

  • Alberts B (2013) Editorial: impact factor distortions. Science 340(6134):787

    Article  Google Scholar 

  • Alvesson M, Sandberg J (2013) Has management studies lost its way? Ideas for more imaginative and innovative research. J Manag Stud 50(1):128–152

    Article  Google Scholar 

  • Archambault É, Larivière V (2009) History of the journal impact factor: contingencies and consequences. Scientometrics 79(3):639–653

    Article  Google Scholar 

  • Baum JAC (2011) Free-riding on power laws: questioning the validity of the impact factor as a measure of research quality in organization studies. Organization 18:449–466

    Article  Google Scholar 

  • Bedeian AG (2003) The manuscript review process: the proper roles of authors, referees, and editors. J Manag Inq 12:331–338

    Google Scholar 

  • Bedeian AG (2004) Peer review and the social construction of knowledge in the management discipline. Acad Manag Learn Educ 3:198–216

    Article  Google Scholar 

  • Bedeian AG, Taylor SG, Miller AN (2010) Management science on the credibility bubble: cardinal sins and various misdemeanors. Acad Manag Learn Educ 9(4):715–725

    Article  Google Scholar 

  • Bhattacharjee Y (2011) Saudi universities offer cash in exchange for academic prestige. Science 334(6061):1344–1345

    Article  Google Scholar 

  • Bornmann L, Daniel HD (2009) The luck of the referee draw: the effect of exchanging reviews. Learned Publish 22(2):117–125

    Article  Google Scholar 

  • Bornmann L, Mutz R, Neuhaus C, Daniel D (2008) Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results. Ethics Sci Environ Politics 8:93–102

    Article  Google Scholar 

  • Bush V (1945) Science: the endless frontier: report to the president by Vannevar Bush, director of the office of scientific research and development. United States Government Printing Office, Washington, DC

    Google Scholar 

  • Campanario JM (1996) Using citation classics to study the incidence of serendipity in scientific discovery. Scientometrics 37:3–24

    Article  Google Scholar 

  • Campanario JM (1998a) Peer review for journals as it stands today, part 1. Sci Commun 19(3):181–211

    Article  Google Scholar 

  • Campanario JM (1998b) Peer review for journals as it stands today, part 2. Sci Commun 19(4):277–306

    Article  Google Scholar 

  • Campbell P (2008) Escape from the impact factor. Ethics Sci Environ Politics 8:5–7

    Article  Google Scholar 

  • Dumas M, Garcia L, Kisselite K, Kungas P, Trepowski CP (2011) Homophily-weighted citation measures. Interdisciplines. http://www.interdisciplines.org/paper.php?paperID=111. Accessed 24 Mar 2014

  • Dunbar RLM, Bresser RK (2014) Knowledge generation and governance in management research. J Bus Econ 84:129–144

    Article  Google Scholar 

  • Eisenhart M (2002) The paradox of peer review: admitting too much of allowing too little. Res Sci Educ 32:241–255

    Article  Google Scholar 

  • Flickinger M, Tuschke A, Grueber-Muecke T, Fiedler M (2014) In search of rigor, relevance, and legitimacy: what drives the impact of publications? J Bus Econ 84:99–128

    Article  Google Scholar 

  • Frey BS (2003) Publishing as prostitution? – choosing between one’s own ideas and academic success. Public Choice 116:205–223

    Article  Google Scholar 

  • Frey BS (2009) Economists in the PITS. Int Rev Econ 56(4):335–346

    Article  Google Scholar 

  • Frey BS, Rost K (2010) Do rankings reflect research quality? J Appl Econ 13(1):1–38

    Article  Google Scholar 

  • Frost J, Brockmann J (2014) When quality is equated with quantitative productivity – scholars caught in a performance paradox. Zeitschrift für Erziehungswissenschaft (in press)

    Google Scholar 

  • Gans JS, Shepherd GB (1994) How are the mighty fallen: rejected classic articles by leading economists. J Econ Perspect 8:165–179

    Article  Google Scholar 

  • Garfield E (1997) Editors are justified in asking authors to cite equivalent references from same journal. Br Med J 314:1765

    Article  Google Scholar 

  • Gillies D (2005) Hempelian and Kuhnian approaches in the philosophy of medicine: the Semmelweis case. Stud Hist Phil Biol Biomed Sci 36:159–181

    Article  Google Scholar 

  • Gillies D (2008) How should research be organised? College Publication King’s College, London

    Google Scholar 

  • Goodhart C (1975) Monetary relationships: a new form of Threadneedle street. Papers in Monetary Economics 1, Reserve Bank of Australia

    Google Scholar 

  • Gottfredson SD (1978) Evaluating psychological research reports: dimensions, reliability, and correlates of quality judgments. Am Psychol 33(10):920–934

    Article  Google Scholar 

  • Heintz B (2010) Numerische Differenz. Überlegungen zu einer Soziologie des (quantiativen) Vergleichs. Z Soziol 39(3):162–181

    Google Scholar 

  • Helbing D, Balietti S (2011) How to create an innovation accelerator. http://arxiv.org/abs/1011.3794v3. Accessed 24 Mar 2014

  • Hudson J (2013) Ranking journals. Econ J 123:F202–F222

    Article  Google Scholar 

  • Jarwal SD, Brion AM, King ML (2009) Measuring research quality using the journal impact factor, citations and ‘Ranked Journals’: blunt instruments or inspired metrics? J High Educ Pol Manag 31(4):289–300

    Article  Google Scholar 

  • Judge TA, Cable DM, Colbert AE, Rynes SL (2007) What causes a management article to be cited – article, author, or journal? Acad Manag J 50(3):489–508

    Article  Google Scholar 

  • Kieser A (2012) JOURQUAL – der Gebrauch, nicht der Missbrauch, ist das Problem. Oder: Warum Wirtschaftsinformatik die beste deutschsprachige betriebswirtschaftliche Zeitschrift ist. Die Betriebswirtschaft 72:93–110

    Google Scholar 

  • Kriegeskorte N (2012) Open evaluation: a vision for entirely transparent post-publication peer review and rating for science. Front Comput Neurosci 6:1–18

    Google Scholar 

  • Kriegeskorte N, Deca D (eds) (2012) Beyond open access: visions for open evaluation of scientific papers by post-publication peer review. Special topic image. Frontiers, Lausanne

    Google Scholar 

  • Laband DN (2013) On the use and abuse of economics journal rankings. Econ J 123:F223–F254

    Article  Google Scholar 

  • Laband DN, Tollison RD (2003) Dry holes in economic research. Kyklos 56:161–174

    Article  Google Scholar 

  • Lalo F, Mosseri R (2009) Bibliometric evaluation of individual researchers: not even right… not even wrong! Europhys News 40(5):26–29

    Article  Google Scholar 

  • Latour B (1988) Drawing things together. In: Lynch M, Woolgar S (eds) Representation in scientific practice. MIT Press, Cambridge, pp 19–68

    Google Scholar 

  • Lawrence PA (2003) The politics of publication – authors, reviewers, and editors must act to protect the quality of research. Nature 422:259–261

    Article  Google Scholar 

  • Lawrence PA (2008) Lost in publication: how measurement harms science. Ethics Sci Environ Politics 8(1):9–11

    Article  Google Scholar 

  • Lucas RE (1976) Econometric policy evaluation: a critique. In: Brunner K, Meltzer AH (eds) Carnegie-Rochester conference series on public policy. The Phillips curve and labor markets. North Holland, New York, pp 19–46

    Google Scholar 

  • Macdonald S, Kam J (2007) Ring a ring o’ roses: quality journals and gamesmanship in management studies. J Manag Stud 44:640–655

    Article  Google Scholar 

  • March JG, Sutton RI (1997) Organizational performance as a dependent variable. Organ Sci 8(6):698–706

    Article  Google Scholar 

  • Merton RK (1973) The sociology of science: theoretical and empirical investigation. University of Chicago Press, Chicago

    Google Scholar 

  • Meyer MW, Gupta V (1994) The performance paradox. Res Organ Behav 16:309–369

    Google Scholar 

  • Monastersky R (2005) The number that’s devouring science. Chron High Educ 52(8):A12

    Google Scholar 

  • Nelson R (2004) The market economy, and the scientific commons. Res Policy 33:455–471

    Article  Google Scholar 

  • Nkomo SM (2009) The seductive power of academic journal rankings: challenges of searching for the otherwise. Acad Manag Learn Educ 8:106–112

    Article  Google Scholar 

  • Osterloh M (2010) Governance by numbers. Does it really work in research? Analyse Kritik 32(2):267–283

    Google Scholar 

  • Osterloh M, Frey BS (2014) Ranking games. Eval Rev (in press)

    Google Scholar 

  • Oswald AJ (2007) An examination of the reliability of prestigious scholarly journals: evidence and implications for decision-makers. Economica 74:21–31

    Article  Google Scholar 

  • Ouchi WG (1979) A conceptual framework for the design of organizational control mechanisms. Manag Sci 25:833–848

    Article  Google Scholar 

  • Polanyi M (1962) The republic of science: its political and economic theory. Minerva 1:54–73. Reprinted in Polanyi M (1969) From knowing and being. University of Chicago Press, Chicago, pp 49–72. Re-reprinted in Mirowski P, Sent EM (eds) (2002) Science bought and sold. Essays in the economics of science. The University of Chicago Press, Chicago, pp 465–485

    Google Scholar 

  • Popper K (2005) The logic of scientific discovery. Routledge/Taylor & Francis e-Library, London

    Google Scholar 

  • Power M (2004) Counting, control and calculation: reflections on measuring and management. Hum Relat 57:765–783

    Article  Google Scholar 

  • Rothwell PM, Martyn CN (2000) Reproducibility of peer review in clinical neuroscience. Is agreement between reviewers any greater than would be expected by chance alone? Brain 123:1964–1969

    Article  Google Scholar 

  • San Francisco Declaration on Research Assessment (DORA) December 16, 2012. http://am.ascb.org/dora/files/SFDeclarationFINAL.pdf. Accessed 24 Mar 2014

  • Sauder M, Espeland WN (2009) The discipline of rankings: tight coupling and organizational change. Am Sociol Rev 74:63–82

    Article  Google Scholar 

  • Simkin MV, Roychowdhury VP (2005) Do copied citations create renowned papers? Ann Improb Res 11(1):24–27

    Article  Google Scholar 

  • Singh G, Haddad KM, Chow S (2007) Are articles in “top” management journals necessarily of higher quality? J Manag Inq 16:319–331

    Google Scholar 

  • Smith R (1997) Journal accused of manipulating impact factor. Br Med J 314:463

    Google Scholar 

  • Starbuck WH (2005) How much better are the most prestigious journals? The statistics of academic publication. Organ Sci 16:180–200

    Article  Google Scholar 

  • Starbuck WH (2006) The production of knowledge. The challenge of social science research. Oxford University Press, Oxford

    Google Scholar 

  • Starbuck WH (2009) The constant causes of never-ending faddishness in the behavioural and social sciences. Scand J Manag 25:225–227

    Article  Google Scholar 

  • Starbuck WH (2015) Issues and trends in publishing behavioral science: a quarrelsome crew struggling with a disintegrating boat on a stormy sea. In: Welpe IM, Wollersheim J, Ringelhan S, Osterloh M (eds) Incentives and performance: governance of research organizations. Springer, Cham

    Google Scholar 

  • Stephan PE (1996) The economics of science. J Econ Lit 34:1199–1235

    Google Scholar 

  • The Economist (2013a) How science goes wrong. October 19th–25th 2013, p 11

    Google Scholar 

  • The Economist (2013b) Trouble at the lab. Scientists like to think of science as self-correcting. To an alarming degree, it is not. October 19th–25th 2013, pp 21–24

    Google Scholar 

  • Tsang EWK, Frey BS (2007) The as-is journal review process: let authors own their ideas. Acad Manag Learn Educ 6:128–136

    Article  Google Scholar 

  • Ursprung HW, Zimmer M (2006) Who is the “Platz–Hirsch” of the German economics profession? A citation analysis. Jahrbücher für Nationalökonomie und Statistik 227:187–202

    Google Scholar 

  • Walsh J (2011) Embracing the sacred in our secular scholarly world. 2010 presidential address. Acad Manag Rev 36(2):215–234

    Article  Google Scholar 

  • Weingart P (2005) Impact of bibliometrics upon the science system: inadvertent consequences? Scientometrics 62:117–131

    Article  Google Scholar 

  • Wilhite AW, Fong EA (2012) Coercive citing in academic publishing. Science 335:542–543

    Article  Google Scholar 

  • Willmott H (2011) Journal list fetishism and the perversion of scholarship: reactivity and the ABS list. Organization 18:429–442

    Article  Google Scholar 

  • Woelert P (2013) The “Economy of Memory”: publications, citations, and the paradox of effective research governance. Minerva 51:341–362

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Zeppelin University, Friedrichshafen, Germany

    Margit Osterloh & Alfred Kieser

Authors
  1. Margit Osterloh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alfred Kieser
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Margit Osterloh .

Editor information

Editors and Affiliations

  1. TUM School of Management, Technische Universität München, Munich, Germany

    Isabell M. Welpe

  2. TUM School of Management, Technische Universität München, Munich, Germany

    Jutta Wollersheim

  3. TUM School of Management, Technische Universität München, Munich, Germany

    Stefanie Ringelhan

  4. Zeppelin University, Friedrichshafen, Germany, and University of Zurich, Zurich, Switzerland

    Margit Osterloh

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Osterloh, M., Kieser, A. (2015). Double-Blind Peer Review: How to Slaughter a Sacred Cow. In: Welpe, I., Wollersheim, J., Ringelhan, S., Osterloh, M. (eds) Incentives and Performance. Springer, Cham. https://doi.org/10.1007/978-3-319-09785-5_19

Download citation

Publish with us

Policies and ethics

Search

Navigation