Your Paper has been Accepted, Rejected, or Whatever: Automatic Generation of Scientific Paper Reviews

  • Alberto Bartoli
  • Andrea De Lorenzo
  • Eric Medvet
  • Fabiano Tarlao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9817)

Abstract

Peer review is widely viewed as an essential step for ensuring scientific quality of a work and is a cornerstone of scholarly publishing. On the other hand, the actors involved in the publishing process are often driven by incentives which may, and increasingly do, undermine the quality of published work, especially in the presence of unethical conduits. In this work we investigate the feasibility of a tool capable of generating fake reviews for a given scientific paper automatically. While a tool of this kind cannot possibly deceive any rigorous editorial procedure, it could nevertheless find a role in several questionable scenarios and magnify the scale of scholarly frauds.

A key feature of our tool is that it is built upon a small knowledge base, which is very important in our context due to the difficulty of finding large amounts of scientific reviews. We experimentally assessed our method 16 human subjects. We presented to these subjects a mix of genuine and machine generated reviews and we measured the ability of our proposal to actually deceive subjects judgment. The results highlight the ability of our method to produce reviews that often look credible and may subvert the decision.

References

  1. 1.
    Bartoli, A., Medvet, E.: Bibliometric evaluation of researchers in the internet age. Inf. Soc. 30(5), 349–354 (2014)CrossRefGoogle Scholar
  2. 2.
    Csiszar, A.: Peer review: troubled from the start. Nature 532(7599), 306–308 (2016)CrossRefGoogle Scholar
  3. 3.
    HEFC: Identification and dissemination of lessons learned by institutions participating in the research excellence framework (ref) bibliometrics pilot. Technical report, Higher Education Funding Council for England (2009)Google Scholar
  4. 4.
    Beall, J.: List of predatory publishers (2016). https://scholarlyoa.com/2016/01/05/bealls-list-of-predatory-publishers-2016. Accessed 29 April 2016
  5. 5.
    Bowman, J.D.: Predatory publishing, questionable peer review, and fraudulent conferences. Am. J. Pharm. Educ. 78(10), 6 pages (2014). http://www.ajpe.org/doi/abs/10.5688/ajpe7810176
  6. 6.
    Dadkhah, M., Alharbi, A.M., Al-Khresheh, M.H., Sutikno, T., Maliszewski, T., Jazi, M.D., Shamshirband, S.: Affiliation oriented journals: don’t worry about peer review if you have good affiliation. Int. J. Electr. Comput. Eng. 5(4), 621 (2015)Google Scholar
  7. 7.
    Butler, D., et al.: The dark side of publishing. Nature 495(7442), 433–435 (2013)CrossRefGoogle Scholar
  8. 8.
    Eldredge, N.: Mathgen paper accepted! Technical report, That’s Mathematics (2012)Google Scholar
  9. 9.
  10. 10.
    Qiu, J., Schrope, M., Jones, N., Borrell, B., Tollefson, J., Kaplan, M., Lovett, R.A., Dalton, R., Merali, Z.: News publish or Perish in China. Nature 463, 142–143 (2010)CrossRefGoogle Scholar
  11. 11.
    Reller, T.: Faking peer reviews. Technical report, Elsevier Connect (2012)Google Scholar
  12. 12.
    Fischman, J.: Fake peer reviews, the latest form of scientific fraud, fool journals. Technical report, The Chronicle of Higher Education (2012)Google Scholar
  13. 13.
    Ferguson, C., Marcus, A., Oransky, I.: Publishing: the peer-review scam. Nature 515(7528), 480–482 (2014)CrossRefGoogle Scholar
  14. 14.
    Callaway, E.: Faked peer reviews prompt 64 retractions. Nature, August 2015. http://www.nature.com/news/faked-peer-reviews-prompt-64-retractions-1.18202
  15. 15.
    Noorden, R.V.: Publishers withdraw more than 120 gibberish papers. Nature 24, February 2014. http://www.nature.com/news/publishers-withdraw-more-than-120-gibberish-papers-1.14763
  16. 16.
    Wen, T.H., Gasic, M., Mrkšić, N., Su, P.H., Vandyke, D., Young, S.: Semantically conditioned LSTM-based natural language generation for spoken dialogue systems, pp. 1711–1721, September 2015Google Scholar
  17. 17.
    Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)Google Scholar
  18. 18.
    Wright, A.: Algorithmic authors. Commun. ACM 58(11), 12–14 (2015)CrossRefGoogle Scholar
  19. 19.
    Labbé, C., Labbé, D.: Duplicate and fake publications in the scientific literature: how many scigen papers in computer science? Scientometrics 94(1), 379–396 (2013)CrossRefGoogle Scholar
  20. 20.
    Oh, A.H., Rudnicky, A.I.: Stochastic natural language generation for spoken dialog systems. Comput. Speech Lang. 16(3), 387–407 (2002)CrossRefGoogle Scholar
  21. 21.
    Belz, A.: Automatic generation of weather forecast texts using comprehensive probabilistic generation-space models. Nat. Lang. Eng. 14(4), 431–455 (2008)CrossRefGoogle Scholar
  22. 22.
    Rieser, V., Lemon, O.: Natural language generation as planning under uncertainty for spoken dialogue systems. In: Krahmer, E., Theune, M. (eds.) Empirical Methods in NLG. LNCS(LNAI), vol. 5790, pp. 105–120. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  23. 23.
    Kukich, K.: Where do phrases come from: Some preliminary experiments in connectionist phrase generation. In: Kempen, G. (ed.) Natural Language Generation. NATO ASI Series, vol. 135, pp. 405–421. Springer, Netherlands (1987)CrossRefGoogle Scholar
  24. 24.
    Mikolov, T., Karafiát, M., Burget, L., Cernockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: INTERSPEECH, vol. 2, p. 3 (2010)Google Scholar
  25. 25.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  26. 26.
    Graves, A.: Generating sequences with recurrent neural networks (2013). arXiv preprint: arXiv:1308.0850
  27. 27.
    Potash, P., Romanov, A., Rumshisky, A.: Ghostwriter: using an LSTM for automatic RAP lyric generation, pp. 1919–1924 (2015)Google Scholar
  28. 28.
    Zhang, X., Lapata, M.: Chinese poetry generation with recurrent neural networks. In: EMNLP, pp. 670–680 (2014)Google Scholar
  29. 29.
    Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128–3137 (2015)Google Scholar
  30. 30.
    Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., Yuille, A.: Deep captioning with multimodal recurrent neural networks (m-RNN) (2014). arXiv preprint: arXiv:1412.6632
  31. 31.
    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2015)Google Scholar
  32. 32.
    Venugopalan, S., Xu, H., Donahue, J., Rohrbach, M., Mooney, R., Saenko, K.: Translating videos to natural language using deep recurrent neural networks (2014). arXiv preprint: arXiv:1412.4729
  33. 33.
    Chatfield, K., Simonyan, K., Vedaldi, A., Zisserman, A.: Return of the devil in the details: Delving deep into convolutional nets (2014). arXiv preprint: arXiv:1405.3531
  34. 34.
    Finkel, J.R., Grenager, T., Manning, C.: Incorporating non-local information into information extraction systems by gibbs sampling. In: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pp. 363–370. Association for Computational Linguistics (2005)Google Scholar
  35. 35.
    Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, vol. 1, pp. 173–180. Association for Computational Linguistics (2003)Google Scholar
  36. 36.
    Narayanan, V., Arora, I., Bhatia, A.: Fast and accurate sentiment classification using an enhanced Naive Bayes model. In: Yin, H., Tang, K., Gao, Y., Klawonn, F., Lee, M., Weise, T., Li, B., Yao, X. (eds.) IDEAL 2013. LNCS, vol. 8206, pp. 194–201. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  37. 37.
    Manning, C., Klein, D.: Optimization, maxent models, and conditional estimation without magic. In: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: Tutorials, vol. 5, p. 8. Association for Computational Linguistics (2003)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2016

Authors and Affiliations

  • Alberto Bartoli
    • 1
  • Andrea De Lorenzo
    • 1
  • Eric Medvet
    • 1
  • Fabiano Tarlao
    • 1
  1. 1.Department of Engineering and ArchitectureUniversity of TriesteTriesteItaly

Personalised recommendations