Skip to main content
SpringerLink
Log in

What have we learned from OpenReview?

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Anonymous peer review is used by the great majority of computer science conferences. OpenReview is such a platform that aims to promote openness in peer review process. The paper, (meta) reviews, rebuttals, and final decisions are all released to public. We collect 11,915 submissions and their 41,276 reviews from the OpenReview platform. We also collect these submissions’ citation data from Google Scholar and their non-peer-reviewed versions from arXiv.org. By acquiring deep insights into these data, we have several interesting findings that could help understand the effectiveness of the public-accessible double-blind peer review process. Our results can potentially help writing a paper, reviewing it, and deciding on its acceptance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11

Similar content being viewed by others

Data availability

The data used in the paper is available at https://github.com/Seafoodair/Openreview/tree/master/data.

Code availability

The codes are available at https://github.com/Seafoodair/Openreview.

Notes

  1. https://openreview.net/

  2. International Conference on Learning Representations. https://iclr.cc/

  3. These datasets and the source code for the analysis experiment are available at https://github.com/Seafoodair/Openreview/

  4. All of the annotated data including manually annotated ones are publicly available at at https://github.com/Seafoodair/Openreview/.

  5. We compare paper creation date on arXiv with ICLR official notification date.

References

  1. Price, S., Flach, P.A.: Computational support for academic peer review: A perspective from artificial intelligence. Commun. ACM 60(3), 70–79 (2017)

    Article  Google Scholar 

  2. Kang, D., Ammar, W., Dalvi, B., van Zuylen, M., Kohlmeier, S., Hovy, E.H., Schwartz, R.: A dataset of peer reviews (peerread): Collection, insights and NLP applications. In: North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 1647–1661 (2018)

  3. Wang, K., Wan, X.: Sentiment analysis of peer review texts for scholarly papers. In: International ACM SIGIR Conference, pp 175–184 (2018)

  4. Li, S., Zhao, W.X., Yin, E.J., Wen, J.: A neural citation count prediction model based on peer review text. In: Natural Language Processing, pp 4913–4923 (2019)

  5. Ghosal, T., Verma, R., Ekbal, A., Bhattacharyya, P.: Deepsentipeer: Harnessing sentiment in review texts to recommend peer review decisions. In: Association for Computational Linguistics, pp 1120–1130 (2019)

  6. Stelmakh, I., Shah, N.B., Singh, A.: On testing for biases in peer review. In: NeurIPS, pp 5287–5297 (2019)

  7. He, H.: Some metadata for those curious about their #ICLR2020. https://twitter.com/cHHillee/status/1191823707100131329

  8. Maaten, L.v.d., Hinton, G.: Visualizing data using t-sne. J. Mach. Learn. Res. 9, 2579–2605 (2008)

    MATH  Google Scholar 

  9. Aslam, J.A., Pavlu, V.: Query hardness estimation using jensen-shannon divergence among multiple scoring functions. In: European Conference on Information Retrieval, pp 198–209 (2007)

  10. Clark, K., Luong, M.-T., Le, Q.V., Manning, C.D.: Electra: Pre-training text encoders as discriminators rather than generators. In: International Conference on Learning Representations (2020)

  11. Ito, T., Tsubouchi, K., Sakaji, H., Yamashita, T., Izumi, K.: Contextual sentiment neural network for document sentiment analysis. Data Sci. Eng. 5(2), 180–192 (2020)

    Article  MATH  Google Scholar 

  12. Gelman, A., Hill, J.: Causal Inference using Regression on the Treatment Variable. Analytical Methods for Social Research, pp 167–198. Cambridge University Press, Cambridge (2006)

    Google Scholar 

  13. Allison, P.D.: Multiple Regression: A Primer. Pine Forge Press, Pine Forge Press (1999)

  14. jiqizhixin: Artificial-Intelligence-Terminology. https://github.com/jiqizhixin/Artificial-Intelligence-Terminology

  15. Joe, H., Ward, J.: Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58(301), 236–244 (1963)

    Article  MathSciNet  Google Scholar 

  16. Trivedi: Causality or causal inference or conditions for causal inference. https://conceptshacked.com/causal-inference/ (2020)

  17. Alsaqr, A.M.: Remarks on the use of pearson’s and spearman’s correlation coefficients in assessing relationships in ophthalmic data. Afr. Vision Eye Health 80 (1), 10 (2021)

    Google Scholar 

  18. Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 4171–4186 (2019)

  19. Charlin, L., Zemel, R.: The toronto paper matching system: an automated paper-reviewer assignment system (2013)

  20. Mrowinski, M.J., Fronczak, P., Fronczak, A., Ausloos, M., Nedic, O.: Artificial intelligence in peer review: How can evolutionary computation support journal editors? PloS ONE 12(9), 0184711 (2017)

    Article  Google Scholar 

  21. Roos, M., Rothe, J., Scheuermann, B.: How to calibrate the scores of biased reviewers by quadratic programming. In: AAAI Conference on Artificial Intelligence, pp 255–260 (2011)

  22. Stelmakh, I., Shah, N., Singh, A.: On testing for biases in peer review. Adv. Neural Inf. Process. Syst. 32 (2019)

  23. Shah, N.B.: An overview of challenges, experiments, and computational solutions in peer review. Communications of the ACM (2021)

  24. Fallon, L., Grapin, S., Newman, D.S., Noltemeyer, A.: Promoting equity and social justice in the peer review process: Tips for reviewers. Sch. Psychol. Int. 43(1), 12–17 (2022)

    Article  Google Scholar 

  25. Manzoor, E., Shah, N.B.: Uncovering latent biases in text: Method and application to peer review. arXiv e-prints 2010 (2020)

  26. Mecca, J.T., Gibson, C., Giorgini, V., Medeiros, K.E., Mumford, M.D., Connelly, S.: Researcher perspectives on conflicts of interest: A qualitative analysis of views from academia. Sci. Eng. Ethics 21(4), 843–855 (2015)

    Article  Google Scholar 

  27. Wu, S., U, L.H., Bhowmick, S.S., Gatterbauer, W.: Pistis: A conflict of interest declaration and detection system for peer review management. In: Proceedings of the 2018 International Conference on Management of Data, pp 1713–1716 (2018)

  28. Aleman-Meza, B., Nagarajan, M., Ramakrishnan, C., Ding, L., Kolari, P., Sheth, A.P., Arpinar, I.B., Joshi, A., Finin, T.: Semantic analytics on social networks: experiences in addressing the problem of conflict of interest detection, pp 407–416 (2006)

  29. Wu, S., U, L.H., Bhowmick, S.S., Gatterbauer, W.: Conflict of interest declaration and detection system in heterogeneous networks, pp 2383–2386 (2017)

  30. CLOSET: ConfLict Of IntereSt DEtection & ManagemenT System. https://personal.ntu.edu.sg/assourav/research/DARE/closet.html

  31. Birukou, A., Wakeling, J.R., Bartolini, C., Casati, F., Marchese, M., Mirylenka, K., Osman, N., Ragone, A., Sierra, C., Wassef, A.: Alternatives to peer review: novel approaches for research evaluation. Front. Comput. Neurosci. 5, 56 (2011)

    Article  Google Scholar 

  32. Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., Miyao, Y.: Does my rebuttal matter? insights from a major NLP conference. In: North American Chapter of the Association for Computational Linguistics, pp 1274–1290 (2019)

  33. Cormode, G.: How not to review a paper: The tools and techniques of the adversarial reviewer. ACM SIGMOD Rec. 37(4), 100–104 (2009)

    Article  MathSciNet  Google Scholar 

  34. Stelmakh, I.: Experiments with the ICML 2020 Peer-Review Process. https://blog.ml.cmu.edu/2020/12/01/icml2020exp/

  35. Shah, N.B., Tabibian, B., Muandet, K., Guyon, I., Von Luxburg, U.: Design and analysis of the nips 2016 review process. J Mach. Learn. Res. 19, 1–34 (2018)

    MathSciNet  MATH  Google Scholar 

  36. Bharadhwaj, H., Turpin, D., Garg, A., Anderson, A.: De-anonymization of authors through arxiv submissions during double-blind review. arXiv:2007.00177 (2020)

  37. Tran, D., Valtchanov, A., Ganapathy, K., Feng, R., Slud, E., Goldblum, M., Goldstein, T.: An open review of openreview: A critical analysis of the machine learning conference review process. arXiv:2010.05137 (2020)

  38. Stelmakh, I., Rastogi, C., Liu, R., Chawla, S., Echenique, F., Shah, N.B.: Cite-seeing and reviewing: A study on citation bias in peer review. arXiv:2203.17239 (2022)

  39. Kohli, G.S., Kaur, P., Singh, M., Ghosal, T., Rana, P.S.: Arguably@ ai debater-nlpcc 2021 task 3: Argument pair extraction from peer review and rebuttals. In: Natural Language Processing and Chinese Computing, pp 590–602 (2021)

  40. Cheng, L., Bing, L., Yu, Q., Lu, W., Si, L.: Ape: argument pair extraction from peer review and rebuttal via multi-task learning. In: Empirical Methods in Natural Language Processing, pp 7000–7011 (2020)

Download references

Acknowledgements

We thank the anonymous reviewers of APWEB 2021 for their encouraging and constructive comments and suggestions.

Funding

This work was supported by the National Natural Science Foundation of China (62072082, U1811261, 62202088, U2241212), the Fundamental Research Funds for the Central Universities (N2216015, N2216012), and the Key R&D Program of Liaoning Province (2020JH2/10100037).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Northeastern University, Shenyang, 110169, China

    Gang Wang, Qi Peng, Yanfeng Zhang & Mingyang Zhang

  2. Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Shenyang, 110169, China

    Yanfeng Zhang

Authors
  1. Gang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qi Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanfeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mingyang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yanfeng Zhang and Gang Wang wrote the main manuscript text and Qi Peng prepared figures 2-4. Yanfeng Zhang contributed to the overall design of the study and some experimental designs. Gang Wang designed the model and analysed the data. Qi Peng crawled ICLR 2017-2020 data and analysed some data. Mingyan Zhang implements the production of some data sets. All authors reviewed the manuscript.

Corresponding author

Correspondence to Yanfeng Zhang.

Ethics declarations

Ethics approval

This article does not contain any studies involving human participants and/or animals by any of the authors.

Consent for publication

All authors have read and agreed to the published version of the manuscript.

Consent to participate

All authors have agreed to participate in the research described in this manuscript.

Conflict of interests

The authors declare no conflict of interest.

Human and animal ethics

Not applicable

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Qi Peng contributed equally to this work.

This article belongs to the Topical Collection: APWeb-WAIM 2021

Guest Editors: Yi Cai, Leong Hou U, Marc Spaniol, Yasushi Sakurai

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, G., Peng, Q., Zhang, Y. et al. What have we learned from OpenReview?. World Wide Web 26, 683–708 (2023). https://doi.org/10.1007/s11280-022-01109-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-022-01109-z

Keywords

Search

Navigation