Skip to main content
SpringerLink
Log in

How biomedical papers accumulated their clinical citations: a large-scale retrospective analysis based on PubMed

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

This paper explored the temporal characteristics of clinical citations of biomedical papers, including how long it takes to receive its first clinical citation (the initial stage) and how long it takes to receive two or more clinical citations after its first clinical citation (the build-up stage). Over 23 million biomedical papers in PubMed between 1940 and 2013 and their clinical citations are used as the research data. We divide these biomedical papers into three groups and four categories from clinical citation level and translational science perspectives. We compare the temporal characteristics of biomedical papers of different groups or categories. From the perspective of clinical citation level, the results show that highly clinically cited papers had obvious advantages of receiving clinical citations over medium and lowly clinically cited papers in both the initial and build-up stages. Meanwhile, as the number of clinical citations increased in the build-up stage, the difference in the length of time to receive the corresponding number of clinical citations among the three groups of biomedical papers significantly increased. From the perspective of translational science, the results reveal that biomedical papers closer to clinical science more easily receive clinical citations than papers closer to basic science in both the initial and build-up stages. Moreover, we found that highly clinically cited papers had the desperate advantage of receiving clinical citations over even the clinical guidelines or clinical trials. The robustness analysis of the two aspects demonstrates the reliability of our results. The indicators proposed in this paper could be useful for pharmaceutical companies and government policy-makers to monitor the translational progress of biomedical research. Besides, the findings in this study could be interesting for young scholars in biomedicine to get more attention from clinical science and to obtain success in their scientific careers, especially for those in basic science.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

This study’s data and code will be freely available upon request.

Code availability

This study’s data and code will be freely available upon request.

References

  • Burrel, Q. L. (2001). Stochastic modelling of the first-citation distribution. Scientometrics, 52(1), 3–12.

    Article  Google Scholar 

  • Caulley, L., Cheng, W., Catalá-López, F., Whelan, J., Khoury, M., Ferraro, J., Husereau, D., Altman, D. G., & Moher, D. (2020). Citation impact was highly variable for reporting guidelines of health research: A citation analysis. Journal of Clinical Epidemiology, 127, 96–104.

    Article  Google Scholar 

  • Du, J., & Wu, Y. (2018). A parameter-free index for identifying under-cited sleeping beauties in science. Scientometrics, 116(2), 959–971.

    Article  Google Scholar 

  • Egghe, L. (2000). A heuristic study of the first-citation distribution. Scientometrics, 48(3), 345–359.

    Article  Google Scholar 

  • Egghe, L., Bornmann, L., & Guns, R. (2011). A proposal for a First-Citation-Speed-Index. Journal of Informetrics, 5(1), 181–186.

    Article  Google Scholar 

  • Egghe, L., & Ravichandra Rao, I. K. (2001). Theory of first-citation distributions and applications. Mathematical and Computer Modelling, 34(1–2), 81–90.

    Article  Google Scholar 

  • Eriksson, M., Billhult, A., Billhult, T., Pallari, E., & Lewison, G. (2020). A new database of the references on international clinical practice guidelines: A facility for the evaluation of clinical research. Scientometrics, 122(2), 1221–1235.

    Article  Google Scholar 

  • Färber, M., & Ao, L. (2022). The Microsoft Academic Knowledge Graph enhanced: Author name disambiguation, publication classification, and embeddings. Quantitative Science Studies, 3(1), 51–98.

    Article  Google Scholar 

  • Goldberg, S. R., Anthony, H., & Evans, T. S. (2015). Modelling citation networks. Scientometrics, 105(3), 1577–1604.

    Article  Google Scholar 

  • Golosovsky, M. (2021). Universality of citation distributions: A new understanding. Quantitative Science Studies, 2(2), 527–543.

    Article  Google Scholar 

  • Grant, J., Cottrell, R., Cluzeau, F., & Fawcett, G. (2000). Evaluating “payback” on biomedical research from papers cited in clinical guidelines: Applied bibliometric study. BMJ, 320, 1107.

    Article  Google Scholar 

  • Huang, Y., Bu, Y., Ding, Y., & Lu, W. (2019). From zero to one: A perspective on citing. Journal of the Association for Information Science and Technology, 70(10), 1098–1107.

    Article  Google Scholar 

  • Huang, Y., Bu, Y., Ding, Y., & Lu, W. (2021). Partitioning highly, medium and lowly cited publications. Journal of Information Science, 47(5), 609–614.

    Article  Google Scholar 

  • Hutchins, B. I., Baker, K. L., Davis, M. T., Diwersy, M. A., Haque, E., Harriman, R. M., Hoppe, T. A., Leicht, S. A., Meyer, P., & Santangelo, G. M. (2019a). The NIH Open Citation Collection: A public access, broad coverage resource. PLOS Biology, 17(10), e3000385.

    Article  Google Scholar 

  • Hutchins, B. I., Davis, M. T., Meseroll, R. A., & Santangelo, G. M. (2019b). Predicting translational progress in biomedical research. PLOS Biology, 17(10), e3000416.

    Article  Google Scholar 

  • Kane, E. I., III., Daumit, G. L., Fain, K. M., Scherer, R. W., & McGinty, E. E. (2022). Evaluating the Revised National Institutes of Health clinical trial definition impact on recruitment progress. Research Evaluation, 31(2), 249–256.

    Article  Google Scholar 

  • Ke, Q. (2019). Identifying translational science through embeddings of controlled vocabularies. Journal of the American Medical Informatics Association, 26(6), 516–523.

    Article  Google Scholar 

  • Ke, Q. (2020). The citation disadvantage of clinical research. Journal of Informetrics, 14(1), 100998.

    Article  Google Scholar 

  • Ke, Q., Ferrara, E., Radicchi, F., & Flammini, A. (2015). Defining and identifying sleeping beauties in science. Proceedings of the National Academy of Sciences, 112(24), 7426–7431.

    Article  Google Scholar 

  • Kryl, D., Allen, L., Dolby, K., Sherbon, B., & Viney, I. (2012). Tracking the impact of research on policy and practice: Investigating the feasibility of using citations in clinical guidelines for research evaluation. British Medical Journal Open, 2(2), e000897.

    Google Scholar 

  • Lewison, G., & Paraje, G. (2004). The classification of biomedical journals by research level. Scientometrics, 60(2), 145–157.

    Article  Google Scholar 

  • Lewison, G., & Sullivan, R. (2008). The impact of cancer research: How publications influence UK cancer clinical guidelines. British Journal of Cancer, 98, 1944–1950.

    Article  Google Scholar 

  • Li, X., Rousseau, J. F., Ding, Y., Song, M., & Lu, W. (2020). Understanding drug repurposing from the perspective of biomedical entities and their evolution: Bibliographic research using aspirin. JMIR Medical Informatics, 8(6), e16739.

    Article  Google Scholar 

  • Li, X., & Tang, X. (2021). Characterizing interdisciplinarity in drug research: A translational science perspective. Journal of Informetrics, 15(4), 101216.

    Article  Google Scholar 

  • Li, X., Tang, X., & Cheng, Q. (2022). Predicting the clinical citation count of biomedical papers using multilayer perceptron neural network. Journal of Informetrics, 16(4), 101333.

    Article  Google Scholar 

  • Li, X., Tang, X., & Lu, W. (2023). Tracking biomedical articles along the translational continuum: A measure based on biomedical knowledge representation. Scientometrics, 128(2), 1295–1319.

    Article  Google Scholar 

  • Min, C., Sun, J., Pei, L., & Ding, Y. (2016). Measuring delayed recognition for papers: Uneven weighted summation and total citations. Journal of Informetrics, 10(4), 1153–1165.

    Article  Google Scholar 

  • Narin, F., Pinski, G., & Gee, H. H. (1976). Structure of the biomedical literature. Journal of the American Society for Information Science, 27(1), 25–45.

    Article  Google Scholar 

  • Nielsen, M. W., & Andersen, J. P. (2021). Global citation inequality is on the rise. Proceedings of the National Academy of Sciences, 118(7), e2012208118.

    Article  Google Scholar 

  • Price, D. D. S. (1976). A general theory of bibliometric and other cumulative advantage processes. Journal of the American Society for Information Science, 27(5), 292–306.

    Article  Google Scholar 

  • Radicchi, F., Fortunato, S., & Castellano, C. (2008). Universality of citation distributions: Toward an objective measure of scientific impact. Proceedings of the National Academy of Sciences, 105(45), 17268–17272.

    Article  Google Scholar 

  • Rousseau, R. (1994). Double exponential models for first-citation processes. Scientometrics, 30(1), 213–227.

    Article  Google Scholar 

  • Schubert, A., & Glänzel, W. (1986). Mean response time—A new indicator of journal citation speed with application to physics journals. Czechoslovak Journal of Physics, 36(1), 121–125.

    Article  Google Scholar 

  • Spencer, T. D. (2022). Clinical impact of research: Introduction to the forum. Perspectives of the ASHA Special Interest Groups, 7(3), 647–650.

    Article  Google Scholar 

  • Thelwall, M., & Kousha, K. (2016). Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov. Scientometrics, 109(2), 1341–1351.

    Article  Google Scholar 

  • Wallace, M. L., Larivière, V., & Gingras, Y. (2009). Modeling a century of citation distributions. Journal of Informetrics, 3(4), 296–303.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., & van Raan, A. F. J. (2012). Universality of citation distributions revisited. Journal of the American Society for Information Science and Technology, 63(1), 72–77.

    Article  Google Scholar 

  • Wang, K., Shen, Z., Huang, C., Wu, C.-H., Dong, Y., & Kanakia, A. (2020). Microsoft Academic Graph: When experts are not enough. Quantitative Science Studies, 1(1), 396–413.

    Article  Google Scholar 

  • Weber, G. M. (2013). Identifying translational science within the triangle of biomedicine. Journal of Translational Medicine, 11(1), 126.

    Article  Google Scholar 

  • Xu, J., Kim, S., Song, M., Jeong, M., Kim, D., Kang, J., Rousseau, J. F., Li, X., Xu, W., Torvik, V. I., Bu, Y., Chen, C., Ebeid, I. A., Li, D., & Ding, Y. (2020). Building a PubMed knowledge graph. Scientific Data, 7(1), 205.

    Article  Google Scholar 

  • Yang, S., & Han, R. (2015). Breadth and depth of citation distribution. Information Processing & Management, 51(2), 130–140.

    Article  Google Scholar 

  • Yue, J., Tabloski, P., Dowal, S. L., Puelle, M. R., Nandan, R., & Inouye, S. K. (2014). NICE to HELP: Operationalizing National Institute for Health and Clinical Excellence guidelines to improve clinical practice. Journal of the American Geriatrics Society, 62(4), 754–761.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant No. 72204090). This work was also supported by the Ministry of Education of Humanities and Social Science Project (Grant No. 22YJC870014). The computation is completed in the HPC platform of Huazhong University of Science and Technology. We also thank the two anonymous reviewers for improving the quality of this article.

Author information

Authors and Affiliations

  1. School of Medicine and Health Management, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, 430030, China

    Xin Li

  2. School of Information Management, Central China Normal University, Wuhan, 430079, China

    Xuli Tang

  3. School of Information Management, Wuhan University, Wuhan, 430074, China

    Wei Lu

Authors
  1. Xin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xuli Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuli Tang.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 1018 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Tang, X. & Lu, W. How biomedical papers accumulated their clinical citations: a large-scale retrospective analysis based on PubMed. Scientometrics (2024). https://doi.org/10.1007/s11192-024-05016-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11192-024-05016-0

Keywords

Search

Navigation