Metrika

, Volume 79, Issue 1, pp 37–57

A statistical approach to calibrating the scores of biased reviewers of scientific papers

  • Wiltrud Kuhlisch
  • Magnus Roos
  • Jörg Rothe
  • Joachim Rudolph
  • Björn Scheuermann
  • Dietrich Stoyan
Article

DOI: 10.1007/s00184-015-0542-z

Cite this article as:
Kuhlisch, W., Roos, M., Rothe, J. et al. Metrika (2016) 79: 37. doi:10.1007/s00184-015-0542-z

Abstract

Peer reviewing is the key ingredient of evaluating the quality of scientific work. Based on the review scores assigned by individual reviewers to papers, program committees of conferences and journal editors decide which papers to accept for publication and which to reject. A similar procedure is part of the selection process of grant applications and, among other fields, in sports. It is well known that the reviewing process suffers from measurement errors due to a lack of agreement among multiple reviewers of the same paper. And if not all papers are reviewed by all reviewers, the naive approach of averaging the scores is biased. Several statistical methods are proposed for aggregating review scores, which all can be realized by standard statistical software. The simplest method uses the well-known fixed-effects two-way classification with identical variances, while a more advanced method assumes different variances. As alternatives a mixed linear model and a generalized linear model are employed. The application of these methods implies an evaluation of the reviewers, which may help to improve reviewing processes. An application example with real conference data shows the potential of these statistical methods.

Keywords

Analysis of variance Model with main effects Design matrix  Peer reviewing Review scores 

Mathematics Subject Classification

62J10 91B82 

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Wiltrud Kuhlisch
    • 1
  • Magnus Roos
    • 2
  • Jörg Rothe
    • 3
  • Joachim Rudolph
    • 4
  • Björn Scheuermann
    • 5
  • Dietrich Stoyan
    • 6
  1. 1.Institut für Mathematische StochastikTU DresdenDresdenGermany
  2. 2.Institut für Sprache und InformationHeinrich-Heine-Universität DüsseldorfDüsseldorfGermany
  3. 3.Institut für InformatikHeinrich-Heine-Universität DüsseldorfDüsseldorfGermany
  4. 4.Institut für SozialwissenschaftenHumboldt-Universität zu BerlinBerlinGermany
  5. 5.Institut für InformatikHumboldt-Universität zu BerlinBerlinGermany
  6. 6.Institut für StochastikTU Bergakademie FreibergFreibergGermany