Abstract
Peer reviews play a vital role in academic publishing. Authors have various feelings towards peer reviews. This study analyzes the experiences shared by authors in Scirev.org to understand these authors' psychological reactions to several aspects of peer reviews, including decisions, turnaround time, the number of reviews, and review quality. Text mining was used to extract different types of psychological reactions of authors, including affective processes and cognitive processes. Results show that authors' psychological responses to peer reviews are complex and cannot be summarized by a single numerical rating directly given by the authors. Rejection invokes anger, sadness, and disagreement, but not anxiety. Fast turnaround arouses positive emotions from authors, but slow peer review processes do not increase negative emotions as much. Low-quality reviews lead to a wide array of negative emotions, including anxiety, anger, and sadness.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Availability of data and materials
The original data is available in csv format.
Code availability
The source code used for analysis can be made available upon request.
References
Bail, C. A., Brown, T. W., & Mann, M. (2017). Channeling hearts and minds: Advocacy organizations, cognitive-emotional currents, and public conversation. American Sociological Review, 82(6), 1188–1213. https://doi.org/10.1177/0003122417733673
Bakanic, V., McPhail, C., & Simon, R. J. (1987). The manuscript review and decision-making process. American Sociological Review, 52(5), 631. https://doi.org/10.2307/2095599
Bayer, M., Sommer, W., & Schacht, A. (2010). Reading emotional words within sentences: The impact of arousal and valence on event-related potentials. International Journal of Psychophysiology, 78(3), 299–307. https://doi.org/10.1016/j.ijpsycho.2010.09.004
Belschak, F. D., & Den Hartog, D. N. (2009). Consequences of positive and negative feedback: The impact on emotions and extra-role behaviors. Applied Psychology, 58(2), 274–303. https://doi.org/10.1111/j.1464-0597.2008.00336.x
Bornmann, L., Wolf, M., & Daniel, H. D. (2012). Closed versus open reviewing of journal manuscripts: How far do comments differ in language use? Scientometrics, 91(3), 843–856. https://doi.org/10.1007/s11192-011-0569-5
Britton, R. (1994). Publication anxiety: Conflict between communication and affiliation. International Journal of Psycho-Analysis, 75(5), 1213–1224
Bunner, C., & Larson, E. L. (2012). Assessing the quality of the peer review process: Author and editorial board member perspectives. American Journal of Infection Control. https://doi.org/10.1016/j.ajic.2012.05.012
Casnici, N., Grimaldo, F., Gilbert, N., Dondio, P., & Squazzoni, F. (2017). Assessing peer review by gauging the fate of rejected manuscripts: the case of the Journal of Artificial Societies and Social Simulation. Scientometrics, 113(1), 533–546. https://doi.org/10.1007/s11192-017-2241-1
Dalton, R. (2001). Peers under pressure. Nature, 413, 102–104. https://doi.org/10.1038/35093252
Drentea, P., & Moren-Cross, J. L. (2005). Social capital and social support on the web: The case of an internet mother site. Sociology of Health and Illness, 27(7), 920–943. https://doi.org/10.1111/j.1467-9566.2005.00464.x
Drvenica, I., Bravo, G., Vejmelka, L., Dekanski, A., & Nedić, O. (2019). Peer review of reviewers: The author’s perspective. Publications, 7(1), 1–14. https://doi.org/10.3390/publications7010001
Fox, M. F. (1994). Scientific misconduct and editorial and peer review processes. The Journal of Higher Education. https://doi.org/10.2307/2943969
Gibson, M., Spong, C. Y., Simonsen, S. E., Martin, S., & Scott, J. R. (2008). Author perception of peer review. Obstetrics and Gynecology. https://doi.org/10.1097/AOG.0b013e31818425d4
Gollogly, L., & Momen, H. (2006). Ethical dilemmas in scientific publication: Pitfalls and solutions for editors. Revista De Saude Publica. https://doi.org/10.1590/s0034-89102006000400004
Hazra, R., Aryan, Aggarwal, H., Marsili, M., & Mukherjee, A. (2020). Characterising authors on the extent of their paper acceptance: A case study of the journal of high energy physics. In Proceedings of the ACM/IEEE joint conference on digital libraries, pp. 157–166. https://doi.org/https://doi.org/10.1145/3383583.3398527
Healey, M. L., & Grossman, M. (2018). Cognitive and affective perspective-taking: Evidence for shared and dissociable anatomical substrates. Frontiers in Neurology, 9, 491–499. https://doi.org/10.3389/fneur.2018.00491
Huisman, J., & Smits, J. (2017). Duration and quality of the peer review process: The author’s perspective. Scientometrics, 113(1), 633–650. https://doi.org/10.1007/s11192-017-2310-5
Humphreys, A., & Wang, R. J. H. (2018). Automated text analysis for consumer research. Journal of Consumer Research, 44(6), 1274–1306. https://doi.org/10.1093/jcr/ucx104
Jiang, S. (2020). Does use of health language improve social support outcome? Linguistic analysis of online health communities. In Proceedings of American conference on information systems, pp. 12–22.
Krch, D. (2018). Cognitive processing. In J. S., Kreutzer, B. Caplan, & J. DeLuca (Eds.), Encyclopedia of clinical neuropsychology (p. 859). Springer: Berlin. https://doi.org/https://doi.org/10.1007/978-3-319-57111-9_1443
Kübler, R. V., Colicev, A., & Pauwels, K. H. (2020). Social media’s impact on the consumer mindset: When to use which sentiment extraction tool? Journal of Interactive Marketing, 50, 136–155. https://doi.org/10.1016/j.intmar.2019.08.001
Lemerise, E., & Dodge, K. (2000). The development of anger and hostile interactions. In M. Lewis, J. M. H.- Jones, & L. F. Barrett (Eds.), Handbook of emotions (pp. 730–741).
Liu, X., Jiang, S., Sun, M., & Chi, X. (2020). Examining patterns of information exchange and social support in web-based health community: Exponential random graph models. Journal of Internet Medical Research, 29(22), e18062.
Lotriet, C. J. (2012). Reviewing the review process: Identifying sources of delay. Australasian Medical Journal, 5(1), 26. https://doi.org/10.4066/AMJ.2012.1165
Majumder, K. (2016). How do authors feel when they receive negative peer reviewer comments? An experience from Chinese biomedical researchers. European Science Editing, 42(2), 31–35
Nicholas, D., Watkinson, A., Jamali, H. R., Herman, E., Tenopir, C., Volentine, R., Allard, S., & Levine, K. (2015). Peer review: Still king in the digital age. Learned Publishing, 28(1), 15–21. https://doi.org/10.1087/20150104
Pennebaker, J. W. (2015). The development and psychometric properties of LIWC2015 James. University of Texas at Austin. https://doi.org/https://doi.org/10.2165/00044011-199815050-00006
Pöschl, U. (2012). Multi-stage open peer review: Scientific evaluation integrating the strengths of traditional peer review with the virtues of transparency and self-regulation. Frontiers in Computational Neuroscience, 6(1), 33–46. https://doi.org/10.3389/fncom.2012.00033
Pranić, S. M., Malički, M., Marušić, S. L., Mehmani, B., & Marušić, A. (2020). Is the quality of reviews reflected in editors’ and authors’ satisfaction with peer review? A cross-sectional study in 12 journals across four research fields. Learned Publishing. https://doi.org/10.1002/leap.1344
Pulverer, B. (2010). Transparency showcases strength of peer review. Nature, 468(7320), 29–31. https://doi.org/10.1038/468029a
Ralph, P. (2016). Practical suggestions for improving scholarly peer review quality and reducing cycle times. Communications of the Association for Information Systems. https://doi.org/https://doi.org/10.17705/1CAIS.03813
Resnik, D. B., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and Engineering Ethics, 14(3), 305–310. https://doi.org/10.1007/s11948-008-9059-4
Rigby, J., Cox, D., & Julian, K. (2018). Journal peer review: a bar or bridge? An analysis of a paper’s revision history and turnaround time, and the effect on citation. Scientometrics, 114(3), 1087–1105. https://doi.org/10.1007/s11192-017-2630-5
Smith, R. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 172–182. https://doi.org/10.1258/jrsm.99.4.178
Solomon, D. J., & Björk, B. C. (2012). Publication fees in open access publishing: Sources of funding and factors influencing choice of journal. Journal of the American Society for Information Science and Technology, 63(1), 98–107. https://doi.org/10.1002/asi.21660
Subasic, P., & Huettner, A. (2001). Affect analysis of text using fuzzy semantic typing. IEEE Transactions on Fuzzy Systems, 9(4), 483–496. https://doi.org/10.1109/91.940962
van den Besselaar, P., Sandström, U., & Schiffbaenker, H. (2018). Studying grant decision-making: a linguistic analysis of review reports. Scientometrics, 117(1), 313–329. https://doi.org/https://doi.org/10.1007/s11192-018-2848-x
Weber, E. J., Katz, P. P., Waeckerle, J. F., & Callaham, M. L. (2002). Author perception of peer review: Impact of review quality and acceptance on satisfaction. Journal of the American Medical Association. https://doi.org/10.1001/jama.287.21.2790
Funding
Joseph Healey Research Grant was used to support this research.
Author information
Authors and Affiliations
Contributions
The first author of this study executed the entire research.
Corresponding author
Ethics declarations
Conflicts of interest
This study has no conflicts of interest with other parties.
Ethical approval
Not applicable.
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Jiang, S. Understanding authors' psychological reactions to peer reviews: a text mining approach. Scientometrics 126, 6085–6103 (2021). https://doi.org/10.1007/s11192-021-04032-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-021-04032-8