Abstract
This study investigates whether negative citations in articles and comments posted on post-publication peer review platforms are both equally contributing to the correction of science. These 2 types of written evidence of disputes are compared by analyzing their occurrence in relation to articles that have already been retracted or corrected. We identified retracted or corrected articles in a corpus of 72,069 articles coming from the Engineering field, from 3 journals (Science, Tumor Biology, Cancer Research) and from 3 authors with many retractions to their credit (Sarkar, Schön, Voinnet). We used Scite to retrieve contradicting citations and PubPeer to retrieve the number of comments for each article, and then we considered them as traces left by scientists to contest published results. Our study shows that contradicting citations are very uncommon and that retracted or corrected articles are not more contradicted in scholarly articles than those that are neither retracted nor corrected but they do generate more comments on Pubpeer, presumably because of the possibility for contributors to remain anonymous. Moreover, post-publication peer review platforms, although external to the scientific publication process contribute more to the correction of science than negative citations. Consequently, post-publication peer review venues, and more specifically the comments found on it, although not contributing to the scientific literature, are a mechanism for correcting science. Lastly, we introduced the idea of strengthening the role of contradicting citations to rehabilitate the clear expression of judgment in scientific papers.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Formalized procedure of debate dating from the Middle Ages.
RetractionWatch is a blog launched in 2010 by two science journalists and is dedicated to the tracking of retracted papers. In 2018, they released an online database of all the cases they have recorded: of all the cases they have recorded http://retractiondatabase.org.
Committee on Publication Ethics, http://publicationethics.org/files/retraction%20guidelines.pdf.
"Crossmark gives readers quick and easy access to the current status of an item of content. With one click, you can see if content has been updated, corrected or retracted and access valuable additional metadata provided by the publisher" (source: http://www.crossref.org/services/crossmark).
All Science Journal Classification, which is used to classify journals and conferences in Scopus.
References
Barbour, B., & Stell, B. M. (2020). PubPeer: Scientific assessment without metrics. In M. Biagioli & A. Lippman (Eds.), Gaming the metrics: Misconduct and manipulation in academic research (pp. 149–155). Cambridge: The MIT Press.
Bar-Ilan, J., & Halevi, G. (2017). Post retraction citations in context: A case study. Scientometrics,113(1), 547–565. https://doi.org/10.1007/s11192-017-2242-0.
Bar-Ilan, J., & Halevi, G. (2018). Temporal characteristics of retracted articles. Scientometrics,116(3), 1771–1783. https://doi.org/10.1007/s11192-018-2802-y.
Baumgartner, H., & Pieters, R. (2003). The structural influence of marketing journals: A citation analysis of the discipline and its subareas over time. J Market,67(2), 123–139. https://doi.org/10.1509/jmkg.67.2.123.18610.
Bertin, M., & Atanassova, I. (2016). Weak links and strong meaning: The complex phenomenon of negational citations. In CEUR workshop proceedings (Vol. 1567, pp. 14–25).
Bordignon, F. (2020). Data for: “Self-correction of science: A comparative study of negative citations and post-publication peer review”. Mendeley Data. https://doi.org/10.17632/3gndhj3t6h.1.
Bornemann-Cimenti, H., Szilagyi, I., & Sandner-Kiesling, A. (2015). Perpetuation of retracted publications using the example of the Scott S. Reuben case: Incidences, reasons and possible improvements. Science and Engineering Ethics. https://doi.org/10.1007/s11948-015-9680-y.
Brainard, J. (2018). Rethinking retractions. Science,362(6413), 390–393. https://doi.org/10.1126/science.362.6413.390.
Brembs, B. (2018). Prestigious science journals struggle to reach even average reliability. Frontiers in Human Neuroscience. https://doi.org/10.3389/fnhum.2018.00037.
Brooks, T. A. (1985). Private acts and public objects: An investigation of citer motivations. Journal of the American Society for Information Science,36(4), 223–229. https://doi.org/10.1002/asi.4630360402.
Cano, V. (1989). Citation behavior: Classification, utility, and location. Journal of the American Society for Information Science,40(4), 284–290. https://doi.org/10.1002/(SICI)1097-4571(198907)40:4%3c284:AID-ASI10%3e3.0.CO;2-Z.
Catalini, C., Lacetera, N., & Oettl, A. (2015). The incidence and role of negative citations in science. Proceedings of the National Academy of Sciences,112(45), 13823–13826. https://doi.org/10.1073/pnas.1502280112.
Chubin, D. E., & Moitra, S. D. (1975). Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science, 5(4), 423–441.
De Solla Price, D. J. (1963). Little science, big science. Columbia: Columbia University Press.
Dougherty, M. V. (2019). Correcting the scholarly record for research integrity. Monash Bioethics Review,37(1–2), 76–78. https://doi.org/10.1007/s40592-019-00093-x.
Dubois, M., & Guaspare, C. (2019). “Is someone out to get me?” la biologie moléculaire à l’épreuve du Post-Publication Peer Review. Zilsel,6(2), 164. https://doi.org/10.3917/zil.006.0164.
Fanelli, D. (2009). How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE,4(5), e5738. https://doi.org/10.1371/journal.pone.0005738.
Fang, F. C., & Casadevall, A. (2011). Retracted science and the retraction index. Infection and Immunity,79(10), 3855–3859. https://doi.org/10.1128/IAI.05661-11.
Furman, J. L., Jensen, K., & Murray, F. (2012). Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine. Research Policy,41(2), 276–290. https://doi.org/10.1016/j.respol.2011.11.001.
Gabehart, M. E. (2005). An analysis of citations to retracted articles in the scientific literature. University of North Carolina, Chapel Hill, Master of Science thesis. https://ils.unc.edu/MSpapers/3050.pdf. Accessed 13 July 2019.
Garfield, E. (1964). Can citation indexing be automated? In Statistical association methods for mechanized documentation: Symposium proceedings, Washington 1964. (National Bureau of Standards Mliscellaneous Publication 269) (pp. 189–192). Washington, DC: U.S.G.P.O. http://www.garfield.library.upenn.edu/essays/V1p084y1962-73.pdf. Accessed June 26, 2019.
Garfield, E., & Welljams-Dorof, A. (1990). The impact of fraudulent research on the scientific literature. Journal of the American Medical Association,263(10), 1424–1426. https://doi.org/10.1001/jama.1990.03440100144021.
Garzone, M., & Mercer, R. E. (2000). Towards an automated citation classifier. In Canadian AI 2000. Lecture notes in computer science (Vol. 1822, pp. 337–346). Berlin, Heidelberg: Springer. https://doi.org/10.1007/3-540-45486-1_28.
Griesemer, J. R. (2020). Taking Goodhart’s law meta: Gaming, meta-gaming, and hacking academic performance metrics. In M. Biagioli & A. Lippman (Eds.), Gaming the metrics: Misconduct and manipulation in academic research (pp. 77–87). Cambridge: The MIT Press.
Gross, A. G., Harmon, J. E., & Reidy, M. S. (2002). Communicating science: The scientific article from the 17th century to the present. Oxford: Oxford University Press.
Hesselmann, F., Graf, V., Schmidt, M., & Reinhart, M. (2017). The visibility of scientific misconduct: A review of the literature on retracted journal articles. Current Sociology Review,65(6), 814–845. https://doi.org/10.1177/0011392116663807.
Hicks, D., Wouters, P., Waltman, L., De Rijcke, S., & Rafols, I. (2015, April 23). Bibliometrics: The Leiden Manifesto for research metrics. Nature. Nature Publishing Group. https://doi.org/10.1038/520429a.
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine,2(8), e124. https://doi.org/10.1371/journal.pmed.0020124.
Kaplan, N. (1965). The norms of citation behavior: Prolegomena to the footnote. American Documentation,16(3), 179–184. https://doi.org/10.1002/asi.5090160305.
Kochan, C. A., & Budd, J. M. (1992). The persistence of fraud in the literature: The Darsee case. Journal of the American Society for Information Science,43(7), 488–493. https://doi.org/10.1002/(SICI)1097-4571(199208)43:7%3c488:AID-ASI3%3e3.0.CO;2-7.
Korpela, K. M. (2010). How long does it take for the scientific literature to purge itself of fraudulent material? The Breuning case revisited. Current Medical Research and Opinion,26(4), 843–847. https://doi.org/10.1185/03007991003603804.
MacRoberts, M. H., & MacRoberts, B. R. (1984). The negational reference: Or the art of dissembling. Social Studies of Science,14(1), 91–94. https://doi.org/10.1177/030631284014001006.
Moravcsik, M. J., & Murugesan, P. (1975). Some results on the function and quality of citations. Social Studies of Science,5(1), 86–92. https://doi.org/10.1177/030631277500500106.
Mott, A., Fairhurst, C., & Torgerson, D. (2019). Assessing the impact of retraction on the citation of randomized controlled trial reports: An interrupted time-series analysis. Journal of Health Services Research & Policy,24(1), 44–51. https://doi.org/10.1177/1355819618797965.
Neale, A. V., Dailey, R. K., & Abrams, J. (2010). Analysis of citations to biomedical articles affected by scientific misconduct. Science and Engineering Ethics,16(2), 251–261. https://doi.org/10.1007/s11948-009-9151-4.
Oppenheim, C., & Renn, S. P. (1978). Highly cited old papers and the reasons why they continue to be cited. Journal of the American Society for Information Science,29(5), 225–231. https://doi.org/10.1002/asi.4630290504.
Peroni, S., & Shotton, D. (2012). FaBiO and CiTO: Ontologies for describing bibliographic resources and citations. Journal of Web Semantics,17, 33–43. https://doi.org/10.1016/j.websem.2012.08.001.
Popper, K. R. (1959). The logic of scientific discovery. London: Routledge.
Poworoznek, E. L. (2003). Linking of errata: Current practices in online physical sciences journals. Journal of the American Society for Information Science and Technology,54(12), 1153–1159. https://doi.org/10.1002/asi.10320.
Rousseau, R., Egghe, L., & Guns, R. (2018). Publication and citation analysis. Becoming Metric-Wise. https://doi.org/10.1016/b978-0-08-102474-4.00005-4.
Shadish, W. R., Tolliver, D., Gray, M., & Sen Gupta, S. K. (1995). Author judgements about works they cite: Three studies from psychology journals. Social Studies of Science,25(3), 477–498. https://doi.org/10.1177/030631295025003003.
Small, H. (1980). Co-citation context analysis and the structure of paradigms. Journal of Documentation,36(3), 183–196. https://doi.org/10.1108/eb026695.
Spiegel-Rosing, I. (1977). Science studies: Bibliometric and content analysis. Social Studies of Science,7(1), 97–113. https://doi.org/10.1177/030631277700700111.
Stremersch, S., Camacho, N., Vanneste, S., & Verniers, I. (2015). Unraveling scientific impact: Citation types in marketing journals. International Journal of Research in Marketing,32(1), 64–77. https://doi.org/10.1016/J.IJRESMAR.2014.09.004.
Teixeira da Silva, J. A., Al-Khatib, A., & Dobránszki, J. (2017). Fortifying the corrective nature of post-publication peer review: Identifying weaknesses, use of journal clubs, and rewarding conscientious behavior. Science and Engineering Ethics,23(4), 1213–1226. https://doi.org/10.1007/s11948-016-9854-2.
Teixeira da Silva, J. A., & Bornemann-Cimenti, H. (2017). Why do some retracted papers continue to be cited? Scientometrics,110(1), 365–370. https://doi.org/10.1007/s11192-016-2178-9.
Teufel, S., Siddharthan, A., & Tidhar, D. (2006). Automatic classification of citation function. In EMNLP’06 Proceedings of the 2006 conference on empirical methods in natural language processing (pp. 103–110). Sydney.
White, H. D. (2001). Authors as citers over time. Journal of the American Society for Information Science and Technology,52(2), 87–108. https://doi.org/10.1002/1097-4571(2000)9999:9999%3c:AID-ASI1542%3e3.0.CO;2-T.
Wouters, P. (1998). The signs of science. Scientometrics,41(1–2), 225–241. https://doi.org/10.1007/BF02457980.
Wouters, P., Costas, R., Franssen, T., Yegros-Yegros, A., Reyes, A., Suze Van Der Luijt-Jansen, E., et al. (2018). The Schön case: Analyzing in-text citations to papers before and after retraction. In STI 2018 conference proceedings. Leiden. https://hdl.handle.net/1887/64521.
Wu, Y., Ngai, E. W. T., Wu, P., & Wu, C. (2020). Fake online reviews: Literature review, synthesis, and directions for future research. Decision Support Systems,132, 113280. https://doi.org/10.1016/j.dss.2020.113280.
Acknowledgements
We are indebted to Joshua Nicholson from Scite for making citation polarity data available.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bordignon, F. Self-correction of science: a comparative study of negative citations and post-publication peer review. Scientometrics 124, 1225–1239 (2020). https://doi.org/10.1007/s11192-020-03536-z
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03536-z