Advertisement

(Why) Do you trust your reviewers? Influence behaviors, trustworthiness, and commitment to peer review

  • Fabian Hattke
  • Isabel Bögner
  • Rick Vogel
Originalartikel

Abstract

Peer review in academic publishing relies on the voluntary engagement of scholars who are, at best, committed to that practice. Current debates on peer review suggest that this commitment is diminishing. Conceptualizing peer review as an instance of social exchange allows us to propose a conceptual model of commitment to peer review and test it by means of a structural equation analysis. Our empirical study is based on survey data from the social sciences (n = 359). Results show that authors are more committed to the practice of peer review if reviewers base their recommendations on rational arguments so that authors can trust them for their competence. By contrast, benevolent reviewers who try to collaborate with authors are not effective in fostering trust and commitment. Within the limitations of our data and with regard to reviewers’ behaviors and characteristics, we cannot support sweeping criticisms of the operational reliability of academic journals.

Keywords

Academic journals Trust Peer control Reviewing Science studies 

(Warum) Trauen Sie Ihren Gutachtern? Einflussnahme, Vertrauenswürdigkeit und Commitment gegenüber Peer Review

Zusammenfassung

Peer Review im akademischen Publizieren basiert auf dem freiwilligen Engagement von Wissenschaftlerinnen und Wissenschaftlern, die, im besten Fall, ein hohes Commitment zu dieser Praxis aufweisen. Aktuelle Debatten über den Peer-Review-Prozess deuten darauf hin, dass dieses Commitment abnimmt. Basierend auf der Social-Exchange-Theorie entwickeln wir ein konzeptionelles Modell, das wir mithilfe eines Strukturgleichungsmodells testen. Unsere empirische Studie basiert auf einer Umfrage in den Sozialwissenschaften (n = 359). Die Ergebnisse zeigen, dass Autoren ein höheres Commitment zur Praxis des Peer Review aufweisen, wenn Gutachter ihre Kommentare auf rationalen Argumenten gründen und die Autoren ihnen aufgrund dieser Kompetenz vertrauen können. Sind Gutachter allerdings zu wohlwollend oder versuchen sie mit den Autoren zu kollaborieren, erwecken sie kein Vertrauen und vermindern das Commitment der Autoren. Im Hinblick auf die Eigenschaften und Verhaltensweisen von Gutachtern und unter Berücksichtigung der Limitationen unserer empirischen Studie können wir die Kritik an der Funktionsweise des Peer Reviews in wissenschaftlichen Zeitschriften nicht bestätigen.

Schlüsselwörter

Wissenschaftliche Zeitschriften Vertrauen Peer Control Begutachtungsprozesse Wissenschaftsforschung 

Notes

Acknowledgements

This research is supported by the German Federal Ministry of Education and Research (Grant No. 01PY13014).

References

  1. Aguinis H, Shapiro D, Antonacopoulou EP, Cummings TG (2014) Scholarly impact: a pluralist conceptualization. Acad Manage Learn Educ 13:1–17CrossRefGoogle Scholar
  2. Antonakis J, Bendahan S, Jacquart P, Lalive R (2010) On making causal claims: a review and recommendations. Leadersh Q 21:1086–1120CrossRefGoogle Scholar
  3. Aryee S, Budhwar, Chen ZX (2002) Trust as a mediator of the relationship between organizational justice and work outcomes: test of a social exchange model. J Organ Behav 23:267–286CrossRefGoogle Scholar
  4. Bandura A (1977) Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev 84:191–215CrossRefGoogle Scholar
  5. Bedeian AG (2003) The manuscript review process: the proper roles of authors, referees, and editors. J Manag Inq 12:331–338CrossRefGoogle Scholar
  6. Bedeian AG (2004) Peer review and the social construction of knowledge in the management discipline. Acad Manag Learn Educ 3:198–216CrossRefGoogle Scholar
  7. Berkenkotter C (1995) The power and the perils of peer review. Rhetor Rev 13:245–248Google Scholar
  8. Blau PM (1964) Exchange and power in social life. John Wiley, New YorkGoogle Scholar
  9. Bollen K (1990) Overall fit in covariance structure models: two types of sample size effects. Psychol Bull 107:256–259CrossRefGoogle Scholar
  10. Bornmann L, Wolf M, Daniel H‑D (2011) Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics 91:843–856CrossRefGoogle Scholar
  11. Bornmann L, Herich H, Joos H, Daniel H‑D (2012) In public peer review of submitted manuscripts, how do reviewer comments differ from comments written by interested members of the scientific community? A content analysis of comments written for atmospheric chemistry and physics. Scientometrics 93:915–929CrossRefGoogle Scholar
  12. Bradley JV (1982) Editorial overkill. Bull Psychon Soc 19:271–274CrossRefGoogle Scholar
  13. Campanario JM (1998a) Peer review for journals as it stands today—part 1. Sci Commun 19:181–211CrossRefGoogle Scholar
  14. Campanario JM (1998b) Peer review for journals as it stands today—part 2. Sci Commun 19:277–306CrossRefGoogle Scholar
  15. Cardy RL, Dobbins GH (1986) Affect and appraisal accuracy: liking as an integral dimension in evaluating performance. J Appl Psychol 71:672–678CrossRefGoogle Scholar
  16. Chubin DE, Hackett EJ (1990) Peerless science: peer review and U.S. science policy. Sunny Press, New YorkGoogle Scholar
  17. Clark T, Wright M (2007) Reviewing journal rankings and revisiting peer reviews: editorial perspectives. J Manag Stud 44:612–621CrossRefGoogle Scholar
  18. Colquitt JA, Scott BA, LePine JA (2007) Trust, trustworthiness, and trust propensity: a meta-analytic test of their unique relationships with risk taking and job performance. J Appl Psychol 92:909–927CrossRefGoogle Scholar
  19. Cropanzano R, Mitchell MS (2005) Social exchange theory: an interdisciplinary review. J Manage 31:874–900Google Scholar
  20. Crossman A, Lee-Kelley L (2004) Trust, commitment and team working: the paradox of virtual organizations. Glob Netw (Oxf) 4:375–390CrossRefGoogle Scholar
  21. Cullen JB, Johnson JL, Sakano T (2000) Success through commitment and trust: the soft side of strategic alliance management. J World Bus 35:223–240CrossRefGoogle Scholar
  22. D’Andrea R, O’Dwyer JP (2017) Can editors save peer review from peer reviewers? PLoS ONE 12:18–20CrossRefGoogle Scholar
  23. Dulac T, Coyle-Shapiro JA, Henderson DJ, Wayne SJ (2008) Not all responses to breach are the same: the interconnection of social exchange and psychological contract processes in organizations. Acad Manage J 51:1079–1098CrossRefGoogle Scholar
  24. Eisenhart M (2002) The paradox of peer review: admitting too much or allowing too little? Res Sci Educ 32:241–255CrossRefGoogle Scholar
  25. Elsbach KD (2004) Managing images of trustworthiness in organizations. In: Kramer R, Cook K (eds) Trust and distrust in organizations. SAGE, New York, pp 275–292Google Scholar
  26. Emerson RM (1976) Social exchange theory. Annu Rev Sociol 2:335–362CrossRefGoogle Scholar
  27. Frey BS (2003) Publishing as prostitution? Choosing between one’s own ideas and academic success. Public Choice 116:205–223CrossRefGoogle Scholar
  28. Fu P, Yukl G (2000) Perceived effectiveness of influence tactics in the United States and China. Leadersh Q 11:251–266CrossRefGoogle Scholar
  29. Giebe T, Gürtler O (2012) Optimal contracts for lenient supervisors. J Econ Behav Organ 81:403–420CrossRefGoogle Scholar
  30. Glogoff S (1988) Reviewing the gatekeepers: a survey of referees of library journals. J Am Soc Inf Sci 39:400–407CrossRefGoogle Scholar
  31. Gould-Williams J, Davies F (2005) Using social exchange theory to predict the effects of HRM practice on employee outcomes: an analysis of public sector workers. Public Manag Rev 7:1–24CrossRefGoogle Scholar
  32. Graf C, Wager E, Bowman A, Fiack S, Scott-Lichter D, Robinson A (2007) Best practice guide-lines on publication ethics: a publisher’s perspective. Int J Clin Pract 61:1–26CrossRefGoogle Scholar
  33. Hausman A, Johnston WJ (2010) The impact of coercive and non-coercive forms of influence on trust, commitment, and compliance in supply chains. Ind Mark Manag 39:519–526CrossRefGoogle Scholar
  34. Hemlin S, Rasmussen S (2006) The shift in academic quality control. Sci Technol Hum Values 31:173–198CrossRefGoogle Scholar
  35. Iacobucci D (2010) Structural equations modeling: fit indices, sample size, and advanced topics. J Consum Psychol 20:90–98CrossRefGoogle Scholar
  36. Kingshott RPJ, Pecotich A (2007) The impact of psychological contracts on trust and commitment in supplier-distributor relationships. Eur J Mark 41:1053–1072CrossRefGoogle Scholar
  37. Klein HJ, Molloy JC, Brinsfield CT (2012) Reconceptualizing workplace commitment to redress a stretched construct: revisiting assumptions and removing confounds. Acad Manage Rev 37:130–151Google Scholar
  38. Klein HJ, Cooper JT, Molloy JC, Swanson JA (2014) The assessment of commitment: advantages of a unidimensional, target-free approach. J Appl Psychol 99:222–238CrossRefGoogle Scholar
  39. Korsgaard MA, Brower HH, Lester SW (2015) It isn’t always mutual: a critical review of dyadic trust. J Manage 41:47–70Google Scholar
  40. Lance CE (2006) The sources of four commonly reported cutoff criteria: what did they really say? Organ Res Methods 9:202–220CrossRefGoogle Scholar
  41. Langfred CW (2004) Too much of a good thing? Negative effects of high trust and individual autonomy in self-managing teams. Acad Manage J 47:385–399Google Scholar
  42. van Maanen J, Sørensen JB, Mitchell TR (2007) The interplay between theory and method. Acad Manage Rev 32:1145–1154CrossRefGoogle Scholar
  43. MacKinnon DP, Fairchild AJ, Fritz MS (2007) Mediation analysis. Annu Rev Psychol 58:593–614CrossRefGoogle Scholar
  44. Mahoney MJ (1977) Publication prejudices: an experimental study of confirmation bias in the peer review system. Cognit Ther Res 1:161–175CrossRefGoogle Scholar
  45. Malay DS (2009) Peer review and scientific misconduct: bad authors and trusting reviewers. J Foot Ankle Surg 48:283–284CrossRefGoogle Scholar
  46. Maloney P, Grawitch MJ, Barber LK (2011) Strategic item selection to reduce survey length: reduction in validity? Consult Psychol J Pract Res 63:162–175CrossRefGoogle Scholar
  47. Mayer R, Davis J (1999) The effect of the performance appraisal system on trust for management: a field quasi-experiment. J Appl Psychol 84:123–136CrossRefGoogle Scholar
  48. Mayer R, Davis J, Schoorman F (1995) An integrative model of organizational trust. Acad Manage Rev 20:709–734CrossRefGoogle Scholar
  49. Meade AW, Watson AM, Kroustalis CM (2007) Assessing common methods bias in organizational research. Annual Meeting of the Society for Industrial and Organizational Psychology. Society for Industrial and Organizational Psychology, New YorkGoogle Scholar
  50. Merton R (1973) Recognition and excellence: instructive ambiguities. In: Merton R (ed) The sociology of science. Theoretical and empirical investigations. University of Chicago Press, Chicago, pp 419–437Google Scholar
  51. Meyer JP, Allen NJ (1984) Testing the “side-bet theory” of organizational commitment: some methodological considerations. J Appl Psychol 69:372–378CrossRefGoogle Scholar
  52. Meyer JP, Stanley DJ, Herscovitch L, Topolnytsky L (2002) Affective, continuance, and normative commitment to the organization: a meta-analysis of antecedents, correlates, and consequences. J Vocat Behav 61:20–52CrossRefGoogle Scholar
  53. Miller CC (2006) Peer review in the organizational and management sciences: prevalence and effects of reviewer hostility, bias, and dissensus. Acad Manage J 49:425–431CrossRefGoogle Scholar
  54. Miller CC, Van de Ven AH (2015) Peer review, root canals, and other amazing life events. Acad Manag Discov 1:117–123CrossRefGoogle Scholar
  55. Miner JB (2003) Commentary on Arthur Bedeian’s “The manuscript review process: the proper roles of authors, referees, and editors.”. J Manag Inq 12:339–343CrossRefGoogle Scholar
  56. Molm LD, Takahashi N, Peterson G (2000) Risk and trust in social exchange: an experi-mental test of a classical proposition. Am J Sociol 105:1396–1427CrossRefGoogle Scholar
  57. Morgan R, Hunt S (1994) The commitment-trust theory of relationship marketing. J Mark 58:20–38CrossRefGoogle Scholar
  58. Mowday RT, Porter LW, Steers RM (1982) Employee-organization linkages: the psychology of commitment, absenteeism, and turnover. Academic Press, New YorkGoogle Scholar
  59. Mulligan A, Hall L, Raphael E (2013) Peer review in a changing world: an international study measuring the attitudes of researchers. J Am Soc Inf Sci Technol 64:132–161CrossRefGoogle Scholar
  60. Murnighan JK (2002) The effects of contracts on interpersonal trust. Adm Sci Q 47:534–559CrossRefGoogle Scholar
  61. Newton DP (2010) Quality and peer review of research: an adjudicating role for editors. Account Res 17:130–145CrossRefGoogle Scholar
  62. Osterloh M (2010) Governance by numbers. Does it really work in research? Anal Krit 32:267–283Google Scholar
  63. Osterloh M, Kieser A (2015) Double-blind peer review: how to slaughter a sacred cow. In: Welpe IM, Wollersheim J, Ringelhan S, Osterloh M (eds) Incentives and performance. Springer, Cham, pp 307–321Google Scholar
  64. Petersen J, Hattke F, Vogel R (2017) Editorial governance and journal impact: a study of management and business journals. Scientometrics 112:1593–1614CrossRefGoogle Scholar
  65. Pillai R, Scandura TA, Williams EA (1999) Leadership and organizational justice: similarities and differences across cultures. J Int Bus Stud 30:763–779CrossRefGoogle Scholar
  66. Placier M (1995) But I have to have an A: probing the cultural meanings and ethical dilemmas of grades in teacher education. Teach Educ Q 22:45–62Google Scholar
  67. Podsakoff PM, MacKenzie SB, Lee J‑Y, Podsakoff NP (2003) Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol 88:879–903CrossRefGoogle Scholar
  68. Prendergast C (1999) The provision of incentives in firms. J Econ Lit 37:7–63CrossRefGoogle Scholar
  69. Prendergast C, Topel RH (1996) Favoritism in organizations. J Polit Econ 104:958–978CrossRefGoogle Scholar
  70. Rabine JL (1999) Putting your trust in reviews: the ethics of book reviewing. Libr Collect Acquis Tech Serv 23:202–203CrossRefGoogle Scholar
  71. Raja U, Johns G, Ntalianis F (2004) The impact of personality on psychological contracts. Acad Manage J 47:350–367Google Scholar
  72. Ramarajan L (2014) Past, present and future research on multiple identities: toward an intrapersonal network approach. Acad Manag Ann 8:589–659CrossRefGoogle Scholar
  73. Robinson SL (1996) Trust and breach of the psychological contract. Adm Sci Q 41:574–599CrossRefGoogle Scholar
  74. Roccas S, Brewer MB (2002) Social identity complexity. Pers Soc Psychol Rev 6:88–106CrossRefGoogle Scholar
  75. Rousseau DM (2001) Schema, promise and mutuality: the building blocks of the psychological contract. J Occup Organ Psychol 74:511–541CrossRefGoogle Scholar
  76. Rousseau DM, Tijoriwala SA (1998) Assessing psychological contracts: issues, alternatives and measures. J Organ Behav 19:679–695CrossRefGoogle Scholar
  77. Schoorman F, Mayer R, Davis J (2007) An integrative model of organizational trust: past, present, and future. Acad Manage Rev 32:344–354CrossRefGoogle Scholar
  78. Seibert S (2006) Anatomy of an R&R (Or, reviewers are an author’s best friends). Acad Manage J 49:203–207CrossRefGoogle Scholar
  79. Settoon RP, Bennett N, Liden RC (1996) Social exchange in organizations: perceived organizational support, leader-member exchange, and employee reciprocity. J Appl Psychol 81:219–227CrossRefGoogle Scholar
  80. Shapiro SP (1987) The social control of impersonal trust. Am J Sociol 93:623–658CrossRefGoogle Scholar
  81. Smith R (2006) Peer review: a flawed process at the heart of science and journals. J R Soc Med 99:178–182CrossRefGoogle Scholar
  82. Spector PE (2006) Method variance in organizational research. Truth or urban legend? Organ Res Methods 9:221–232CrossRefGoogle Scholar
  83. Spector PE, Brannick MT (2011) Methodological urban legends: the misuse of statistical control variables. Organ Res Methods 14:287–305CrossRefGoogle Scholar
  84. Squazzoni F, Bravo G, Takács K (2013) Does incentive provision increase the quality of peer review? An experimental study. Res Policy 42:287–294CrossRefGoogle Scholar
  85. Stanton JM, Sinar EF, Balzer WK, Smith PC (2002) Issues and strategies for reducing the length of self-report scales. Pers Psychol 55:167–194CrossRefGoogle Scholar
  86. Starbuck WH (2003) Turning lemons into lemonade: where is the value in peer reviews? J Manag Inq 12:344–351CrossRefGoogle Scholar
  87. Van der Stede WA, Young SM, Chen CX (2005) Assessing the quality of evidence in empirical management accounting research: the case of survey studies. Account Organ Soc 30:655–684CrossRefGoogle Scholar
  88. Teixeira da Silva JA, Dobránszki J (2015) Problems with traditional science publishing and finding a wider niche for post-publication peer review. Account Res 22:22–40CrossRefGoogle Scholar
  89. Tett RP, Meyer JP (1993) Job satisfaction, organizational commitment, turnover intention, and turnover: path analyses based on meta-analytic findings. Pers Psychol 46:259–293CrossRefGoogle Scholar
  90. Tsang E (2014) Ensuring manuscript quality and preserving authorial voice: the balancing act of editors. Manag Organ Rev 10:191–197CrossRefGoogle Scholar
  91. Tsang EWK, Frey BS (2007) The as-is journal review process: let authors own their ideas. Acad Manag Learn Educ 6:128–136CrossRefGoogle Scholar
  92. Vogel R, Hattke F (2018) How is the use of performance information related to performance of public sector professionals? Evidence from the field of academic research. Public Perfom Manag Rev 41(2):390–414CrossRefGoogle Scholar
  93. Vogel R, Hattke F, Petersen J (2017) Journal rankings in management and business studies: what rules do we play by? Res Policy 46:1707–1722CrossRefGoogle Scholar
  94. Wheeler B (2011) The ontology of the scholarly journal and the place of peer review. J Sch Publ 42:307–322CrossRefGoogle Scholar
  95. Whitley R (1984a) The intellectual and social organization of the sciences. Oxford University Press, OxfordGoogle Scholar
  96. Whitley R (1984b) The fragmented state of management studies: Reasons and consequences. J Manag Stud 21:331–348CrossRefGoogle Scholar
  97. Whyte G, Saks AM, Hook S (1997) When success breeds failure: the role of self-efficacy in escalating commitment to a losing course of action. J Organ Behav 18:415–432CrossRefGoogle Scholar
  98. Yukl G, Tracey JB (1992) Consequences of influence tactics used with subordinates, peers, and the boss. J Appl Psychol 77:525–535CrossRefGoogle Scholar
  99. Yukl G, Lepsinger R, Lucia T (1992) Preliminary report on development and validation of the influence behavior questionnaire. In: Clark KE, Clark MB, Campbell DP (eds) Impact of leadership. Center for creative leadership, Greensboro, pp 417–427Google Scholar
  100. Yukl G, Chavez C, Seifert CF (2005) Assessing the construct validity and utility of two new influence tactics. J Organ Behav 26:705–725CrossRefGoogle Scholar
  101. Yukl G, Seifert CF, Chavez C (2008) Validation of the extended influence behavior questionnaire. Leadersh Q 19:609–621CrossRefGoogle Scholar
  102. Zahra SA, Neubaum DO (2006) Revising to be published: building trust to win the acceptance of journal editors and reviewers. In: Baruch Y, Sullivan SE, Schepmyer HN (eds) Winning reviews: a guide for evaluating scholarly writing. Palgrave Macmillan, Houndmills, pp 205–223Google Scholar

Copyright information

© Springer Fachmedien Wiesbaden GmbH, ein Teil von Springer Nature 2018

Authors and Affiliations

  1. 1.Organization and ManagementUniversity of HamburgHamburgGermany
  2. 2.CHESS Center for Higher Education and Science StudiesUniversity of ZurichZurichSwitzerland
  3. 3.Management, Strategy and LeadershipUniversity of KonstanzKonstanzGermany
  4. 4.Public ManagementUniversity of HamburgHamburgGermany

Personalised recommendations