(Why) Do you trust your reviewers? Influence behaviors, trustworthiness, and commitment to peer review

(Warum) Trauen Sie Ihren Gutachtern? Einflussnahme, Vertrauenswürdigkeit und Commitment gegenüber Peer Review

Abstract

Peer review in academic publishing relies on the voluntary engagement of scholars who are, at best, committed to that practice. Current debates on peer review suggest that this commitment is diminishing. Conceptualizing peer review as an instance of social exchange allows us to propose a conceptual model of commitment to peer review and test it by means of a structural equation analysis. Our empirical study is based on survey data from the social sciences (n = 359). Results show that authors are more committed to the practice of peer review if reviewers base their recommendations on rational arguments so that authors can trust them for their competence. By contrast, benevolent reviewers who try to collaborate with authors are not effective in fostering trust and commitment. Within the limitations of our data and with regard to reviewers’ behaviors and characteristics, we cannot support sweeping criticisms of the operational reliability of academic journals.

Zusammenfassung

Peer Review im akademischen Publizieren basiert auf dem freiwilligen Engagement von Wissenschaftlerinnen und Wissenschaftlern, die, im besten Fall, ein hohes Commitment zu dieser Praxis aufweisen. Aktuelle Debatten über den Peer-Review-Prozess deuten darauf hin, dass dieses Commitment abnimmt. Basierend auf der Social-Exchange-Theorie entwickeln wir ein konzeptionelles Modell, das wir mithilfe eines Strukturgleichungsmodells testen. Unsere empirische Studie basiert auf einer Umfrage in den Sozialwissenschaften (n = 359). Die Ergebnisse zeigen, dass Autoren ein höheres Commitment zur Praxis des Peer Review aufweisen, wenn Gutachter ihre Kommentare auf rationalen Argumenten gründen und die Autoren ihnen aufgrund dieser Kompetenz vertrauen können. Sind Gutachter allerdings zu wohlwollend oder versuchen sie mit den Autoren zu kollaborieren, erwecken sie kein Vertrauen und vermindern das Commitment der Autoren. Im Hinblick auf die Eigenschaften und Verhaltensweisen von Gutachtern und unter Berücksichtigung der Limitationen unserer empirischen Studie können wir die Kritik an der Funktionsweise des Peer Reviews in wissenschaftlichen Zeitschriften nicht bestätigen.

This is a preview of subscription content, log in to check access.

Fig. 1

References

  1. Aguinis H, Shapiro D, Antonacopoulou EP, Cummings TG (2014) Scholarly impact: a pluralist conceptualization. Acad Manage Learn Educ 13:1–17

    Google Scholar 

  2. Antonakis J, Bendahan S, Jacquart P, Lalive R (2010) On making causal claims: a review and recommendations. Leadersh Q 21:1086–1120

    Google Scholar 

  3. Aryee S, Budhwar, Chen ZX (2002) Trust as a mediator of the relationship between organizational justice and work outcomes: test of a social exchange model. J Organ Behav 23:267–286

    Google Scholar 

  4. Bandura A (1977) Self-efficacy: toward a unifying theory of behavioral change. Psychol Rev 84:191–215

    Google Scholar 

  5. Bedeian AG (2003) The manuscript review process: the proper roles of authors, referees, and editors. J Manag Inq 12:331–338

    Google Scholar 

  6. Bedeian AG (2004) Peer review and the social construction of knowledge in the management discipline. Acad Manag Learn Educ 3:198–216

    Google Scholar 

  7. Berkenkotter C (1995) The power and the perils of peer review. Rhetor Rev 13:245–248

    Google Scholar 

  8. Blau PM (1964) Exchange and power in social life. John Wiley, New York

    Google Scholar 

  9. Bollen K (1990) Overall fit in covariance structure models: two types of sample size effects. Psychol Bull 107:256–259

    Google Scholar 

  10. Bornmann L, Wolf M, Daniel H‑D (2011) Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics 91:843–856

    Google Scholar 

  11. Bornmann L, Herich H, Joos H, Daniel H‑D (2012) In public peer review of submitted manuscripts, how do reviewer comments differ from comments written by interested members of the scientific community? A content analysis of comments written for atmospheric chemistry and physics. Scientometrics 93:915–929

    Google Scholar 

  12. Bradley JV (1982) Editorial overkill. Bull Psychon Soc 19:271–274

    Google Scholar 

  13. Campanario JM (1998a) Peer review for journals as it stands today—part 1. Sci Commun 19:181–211

    Google Scholar 

  14. Campanario JM (1998b) Peer review for journals as it stands today—part 2. Sci Commun 19:277–306

    Google Scholar 

  15. Cardy RL, Dobbins GH (1986) Affect and appraisal accuracy: liking as an integral dimension in evaluating performance. J Appl Psychol 71:672–678

    Google Scholar 

  16. Chubin DE, Hackett EJ (1990) Peerless science: peer review and U.S. science policy. Sunny Press, New York

    Google Scholar 

  17. Clark T, Wright M (2007) Reviewing journal rankings and revisiting peer reviews: editorial perspectives. J Manag Stud 44:612–621

    Google Scholar 

  18. Colquitt JA, Scott BA, LePine JA (2007) Trust, trustworthiness, and trust propensity: a meta-analytic test of their unique relationships with risk taking and job performance. J Appl Psychol 92:909–927

    Google Scholar 

  19. Cropanzano R, Mitchell MS (2005) Social exchange theory: an interdisciplinary review. J Manage 31:874–900

    Google Scholar 

  20. Crossman A, Lee-Kelley L (2004) Trust, commitment and team working: the paradox of virtual organizations. Glob Netw (Oxf) 4:375–390

    Google Scholar 

  21. Cullen JB, Johnson JL, Sakano T (2000) Success through commitment and trust: the soft side of strategic alliance management. J World Bus 35:223–240

    Google Scholar 

  22. D’Andrea R, O’Dwyer JP (2017) Can editors save peer review from peer reviewers? PLoS ONE 12:18–20

    Google Scholar 

  23. Dulac T, Coyle-Shapiro JA, Henderson DJ, Wayne SJ (2008) Not all responses to breach are the same: the interconnection of social exchange and psychological contract processes in organizations. Acad Manage J 51:1079–1098

    Google Scholar 

  24. Eisenhart M (2002) The paradox of peer review: admitting too much or allowing too little? Res Sci Educ 32:241–255

    Google Scholar 

  25. Elsbach KD (2004) Managing images of trustworthiness in organizations. In: Kramer R, Cook K (eds) Trust and distrust in organizations. SAGE, New York, pp 275–292

    Google Scholar 

  26. Emerson RM (1976) Social exchange theory. Annu Rev Sociol 2:335–362

    Google Scholar 

  27. Frey BS (2003) Publishing as prostitution? Choosing between one’s own ideas and academic success. Public Choice 116:205–223

    Google Scholar 

  28. Fu P, Yukl G (2000) Perceived effectiveness of influence tactics in the United States and China. Leadersh Q 11:251–266

    Google Scholar 

  29. Giebe T, Gürtler O (2012) Optimal contracts for lenient supervisors. J Econ Behav Organ 81:403–420

    Google Scholar 

  30. Glogoff S (1988) Reviewing the gatekeepers: a survey of referees of library journals. J Am Soc Inf Sci 39:400–407

    Google Scholar 

  31. Gould-Williams J, Davies F (2005) Using social exchange theory to predict the effects of HRM practice on employee outcomes: an analysis of public sector workers. Public Manag Rev 7:1–24

    Google Scholar 

  32. Graf C, Wager E, Bowman A, Fiack S, Scott-Lichter D, Robinson A (2007) Best practice guide-lines on publication ethics: a publisher’s perspective. Int J Clin Pract 61:1–26

    Google Scholar 

  33. Hausman A, Johnston WJ (2010) The impact of coercive and non-coercive forms of influence on trust, commitment, and compliance in supply chains. Ind Mark Manag 39:519–526

    Google Scholar 

  34. Hemlin S, Rasmussen S (2006) The shift in academic quality control. Sci Technol Hum Values 31:173–198

    Google Scholar 

  35. Iacobucci D (2010) Structural equations modeling: fit indices, sample size, and advanced topics. J Consum Psychol 20:90–98

    Google Scholar 

  36. Kingshott RPJ, Pecotich A (2007) The impact of psychological contracts on trust and commitment in supplier-distributor relationships. Eur J Mark 41:1053–1072

    Google Scholar 

  37. Klein HJ, Molloy JC, Brinsfield CT (2012) Reconceptualizing workplace commitment to redress a stretched construct: revisiting assumptions and removing confounds. Acad Manage Rev 37:130–151

    Google Scholar 

  38. Klein HJ, Cooper JT, Molloy JC, Swanson JA (2014) The assessment of commitment: advantages of a unidimensional, target-free approach. J Appl Psychol 99:222–238

    Google Scholar 

  39. Korsgaard MA, Brower HH, Lester SW (2015) It isn’t always mutual: a critical review of dyadic trust. J Manage 41:47–70

    Google Scholar 

  40. Lance CE (2006) The sources of four commonly reported cutoff criteria: what did they really say? Organ Res Methods 9:202–220

    Google Scholar 

  41. Langfred CW (2004) Too much of a good thing? Negative effects of high trust and individual autonomy in self-managing teams. Acad Manage J 47:385–399

    Google Scholar 

  42. van Maanen J, Sørensen JB, Mitchell TR (2007) The interplay between theory and method. Acad Manage Rev 32:1145–1154

    Google Scholar 

  43. MacKinnon DP, Fairchild AJ, Fritz MS (2007) Mediation analysis. Annu Rev Psychol 58:593–614

    Google Scholar 

  44. Mahoney MJ (1977) Publication prejudices: an experimental study of confirmation bias in the peer review system. Cognit Ther Res 1:161–175

    Google Scholar 

  45. Malay DS (2009) Peer review and scientific misconduct: bad authors and trusting reviewers. J Foot Ankle Surg 48:283–284

    Google Scholar 

  46. Maloney P, Grawitch MJ, Barber LK (2011) Strategic item selection to reduce survey length: reduction in validity? Consult Psychol J Pract Res 63:162–175

    Google Scholar 

  47. Mayer R, Davis J (1999) The effect of the performance appraisal system on trust for management: a field quasi-experiment. J Appl Psychol 84:123–136

    Google Scholar 

  48. Mayer R, Davis J, Schoorman F (1995) An integrative model of organizational trust. Acad Manage Rev 20:709–734

    Google Scholar 

  49. Meade AW, Watson AM, Kroustalis CM (2007) Assessing common methods bias in organizational research. Annual Meeting of the Society for Industrial and Organizational Psychology. Society for Industrial and Organizational Psychology, New York

    Google Scholar 

  50. Merton R (1973) Recognition and excellence: instructive ambiguities. In: Merton R (ed) The sociology of science. Theoretical and empirical investigations. University of Chicago Press, Chicago, pp 419–437

    Google Scholar 

  51. Meyer JP, Allen NJ (1984) Testing the “side-bet theory” of organizational commitment: some methodological considerations. J Appl Psychol 69:372–378

    Google Scholar 

  52. Meyer JP, Stanley DJ, Herscovitch L, Topolnytsky L (2002) Affective, continuance, and normative commitment to the organization: a meta-analysis of antecedents, correlates, and consequences. J Vocat Behav 61:20–52

    Google Scholar 

  53. Miller CC (2006) Peer review in the organizational and management sciences: prevalence and effects of reviewer hostility, bias, and dissensus. Acad Manage J 49:425–431

    Google Scholar 

  54. Miller CC, Van de Ven AH (2015) Peer review, root canals, and other amazing life events. Acad Manag Discov 1:117–123

    Google Scholar 

  55. Miner JB (2003) Commentary on Arthur Bedeian’s “The manuscript review process: the proper roles of authors, referees, and editors.”. J Manag Inq 12:339–343

    Google Scholar 

  56. Molm LD, Takahashi N, Peterson G (2000) Risk and trust in social exchange: an experi-mental test of a classical proposition. Am J Sociol 105:1396–1427

    Google Scholar 

  57. Morgan R, Hunt S (1994) The commitment-trust theory of relationship marketing. J Mark 58:20–38

    Google Scholar 

  58. Mowday RT, Porter LW, Steers RM (1982) Employee-organization linkages: the psychology of commitment, absenteeism, and turnover. Academic Press, New York

    Google Scholar 

  59. Mulligan A, Hall L, Raphael E (2013) Peer review in a changing world: an international study measuring the attitudes of researchers. J Am Soc Inf Sci Technol 64:132–161

    Google Scholar 

  60. Murnighan JK (2002) The effects of contracts on interpersonal trust. Adm Sci Q 47:534–559

    Google Scholar 

  61. Newton DP (2010) Quality and peer review of research: an adjudicating role for editors. Account Res 17:130–145

    Google Scholar 

  62. Osterloh M (2010) Governance by numbers. Does it really work in research? Anal Krit 32:267–283

    Google Scholar 

  63. Osterloh M, Kieser A (2015) Double-blind peer review: how to slaughter a sacred cow. In: Welpe IM, Wollersheim J, Ringelhan S, Osterloh M (eds) Incentives and performance. Springer, Cham, pp 307–321

    Google Scholar 

  64. Petersen J, Hattke F, Vogel R (2017) Editorial governance and journal impact: a study of management and business journals. Scientometrics 112:1593–1614

    Google Scholar 

  65. Pillai R, Scandura TA, Williams EA (1999) Leadership and organizational justice: similarities and differences across cultures. J Int Bus Stud 30:763–779

    Google Scholar 

  66. Placier M (1995) But I have to have an A: probing the cultural meanings and ethical dilemmas of grades in teacher education. Teach Educ Q 22:45–62

    Google Scholar 

  67. Podsakoff PM, MacKenzie SB, Lee J‑Y, Podsakoff NP (2003) Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol 88:879–903

    Google Scholar 

  68. Prendergast C (1999) The provision of incentives in firms. J Econ Lit 37:7–63

    Google Scholar 

  69. Prendergast C, Topel RH (1996) Favoritism in organizations. J Polit Econ 104:958–978

    Google Scholar 

  70. Rabine JL (1999) Putting your trust in reviews: the ethics of book reviewing. Libr Collect Acquis Tech Serv 23:202–203

    Google Scholar 

  71. Raja U, Johns G, Ntalianis F (2004) The impact of personality on psychological contracts. Acad Manage J 47:350–367

    Google Scholar 

  72. Ramarajan L (2014) Past, present and future research on multiple identities: toward an intrapersonal network approach. Acad Manag Ann 8:589–659

    Google Scholar 

  73. Robinson SL (1996) Trust and breach of the psychological contract. Adm Sci Q 41:574–599

    Google Scholar 

  74. Roccas S, Brewer MB (2002) Social identity complexity. Pers Soc Psychol Rev 6:88–106

    Google Scholar 

  75. Rousseau DM (2001) Schema, promise and mutuality: the building blocks of the psychological contract. J Occup Organ Psychol 74:511–541

    Google Scholar 

  76. Rousseau DM, Tijoriwala SA (1998) Assessing psychological contracts: issues, alternatives and measures. J Organ Behav 19:679–695

    Google Scholar 

  77. Schoorman F, Mayer R, Davis J (2007) An integrative model of organizational trust: past, present, and future. Acad Manage Rev 32:344–354

    Google Scholar 

  78. Seibert S (2006) Anatomy of an R&R (Or, reviewers are an author’s best friends). Acad Manage J 49:203–207

    Google Scholar 

  79. Settoon RP, Bennett N, Liden RC (1996) Social exchange in organizations: perceived organizational support, leader-member exchange, and employee reciprocity. J Appl Psychol 81:219–227

    Google Scholar 

  80. Shapiro SP (1987) The social control of impersonal trust. Am J Sociol 93:623–658

    Google Scholar 

  81. Smith R (2006) Peer review: a flawed process at the heart of science and journals. J R Soc Med 99:178–182

    Google Scholar 

  82. Spector PE (2006) Method variance in organizational research. Truth or urban legend? Organ Res Methods 9:221–232

    Google Scholar 

  83. Spector PE, Brannick MT (2011) Methodological urban legends: the misuse of statistical control variables. Organ Res Methods 14:287–305

    Google Scholar 

  84. Squazzoni F, Bravo G, Takács K (2013) Does incentive provision increase the quality of peer review? An experimental study. Res Policy 42:287–294

    Google Scholar 

  85. Stanton JM, Sinar EF, Balzer WK, Smith PC (2002) Issues and strategies for reducing the length of self-report scales. Pers Psychol 55:167–194

    Google Scholar 

  86. Starbuck WH (2003) Turning lemons into lemonade: where is the value in peer reviews? J Manag Inq 12:344–351

    Google Scholar 

  87. Van der Stede WA, Young SM, Chen CX (2005) Assessing the quality of evidence in empirical management accounting research: the case of survey studies. Account Organ Soc 30:655–684

    Google Scholar 

  88. Teixeira da Silva JA, Dobránszki J (2015) Problems with traditional science publishing and finding a wider niche for post-publication peer review. Account Res 22:22–40

    Google Scholar 

  89. Tett RP, Meyer JP (1993) Job satisfaction, organizational commitment, turnover intention, and turnover: path analyses based on meta-analytic findings. Pers Psychol 46:259–293

    Google Scholar 

  90. Tsang E (2014) Ensuring manuscript quality and preserving authorial voice: the balancing act of editors. Manag Organ Rev 10:191–197

    Google Scholar 

  91. Tsang EWK, Frey BS (2007) The as-is journal review process: let authors own their ideas. Acad Manag Learn Educ 6:128–136

    Google Scholar 

  92. Vogel R, Hattke F (2018) How is the use of performance information related to performance of public sector professionals? Evidence from the field of academic research. Public Perfom Manag Rev 41(2):390–414

    Google Scholar 

  93. Vogel R, Hattke F, Petersen J (2017) Journal rankings in management and business studies: what rules do we play by? Res Policy 46:1707–1722

    Google Scholar 

  94. Wheeler B (2011) The ontology of the scholarly journal and the place of peer review. J Sch Publ 42:307–322

    Google Scholar 

  95. Whitley R (1984a) The intellectual and social organization of the sciences. Oxford University Press, Oxford

    Google Scholar 

  96. Whitley R (1984b) The fragmented state of management studies: Reasons and consequences. J Manag Stud 21:331–348

    Google Scholar 

  97. Whyte G, Saks AM, Hook S (1997) When success breeds failure: the role of self-efficacy in escalating commitment to a losing course of action. J Organ Behav 18:415–432

    Google Scholar 

  98. Yukl G, Tracey JB (1992) Consequences of influence tactics used with subordinates, peers, and the boss. J Appl Psychol 77:525–535

    Google Scholar 

  99. Yukl G, Lepsinger R, Lucia T (1992) Preliminary report on development and validation of the influence behavior questionnaire. In: Clark KE, Clark MB, Campbell DP (eds) Impact of leadership. Center for creative leadership, Greensboro, pp 417–427

    Google Scholar 

  100. Yukl G, Chavez C, Seifert CF (2005) Assessing the construct validity and utility of two new influence tactics. J Organ Behav 26:705–725

    Google Scholar 

  101. Yukl G, Seifert CF, Chavez C (2008) Validation of the extended influence behavior questionnaire. Leadersh Q 19:609–621

    Google Scholar 

  102. Zahra SA, Neubaum DO (2006) Revising to be published: building trust to win the acceptance of journal editors and reviewers. In: Baruch Y, Sullivan SE, Schepmyer HN (eds) Winning reviews: a guide for evaluating scholarly writing. Palgrave Macmillan, Houndmills, pp 205–223

    Google Scholar 

Download references

Acknowledgements

This research is supported by the German Federal Ministry of Education and Research (Grant No. 01PY13014).

Author information

Affiliations

Authors

Corresponding author

Correspondence to Fabian Hattke.

Appendix

Appendix

 

Table 6 Scales and items

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hattke, F., Bögner, I. & Vogel, R. (Why) Do you trust your reviewers? Influence behaviors, trustworthiness, and commitment to peer review. Managementforschung 28, 61–86 (2018). https://doi.org/10.1365/s41113-018-0022-2

Download citation

Keywords

  • Academic journals
  • Trust
  • Peer control
  • Reviewing
  • Science studies

Schlüsselwörter

  • Wissenschaftliche Zeitschriften
  • Vertrauen
  • Peer Control
  • Begutachtungsprozesse
  • Wissenschaftsforschung