Skip to main content
SpringerLink
Log in

Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

An important way in which medical research can translate into improved health outcomes is by motivating or influencing clinical trials that eventually lead to changes in clinical practice. Citations from clinical trials records to academic research may therefore serve as an early warning of the likely future influence of the cited articles. This paper partially assesses this hypothesis by testing whether prior articles referenced in ClinicalTrials.gov records are more highly cited than average for the publishing journal. The results from four high profile general medical journals support the hypothesis, although there may not be a cause-and effect relationship. Nevertheless, it is reasonable for researchers to use citations to their work from clinical trials records as evidence of the possible long-term impact of their research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Akcan, D., Axelsson, S., Bergh, C., Davidson, T., & Rosén, M. (2013). Methodological quality in clinical trials and bibliometric indicators: No evidence of correlations. Scientometrics, 96(1), 297–303.

    Article  Google Scholar 

  • Boyack, K. W., & Jordan, P. (2011). Metrics associated with NIH funding: A high-level view. Journal of the American Medical Informatics Association, 18(4), 423–431.

    Article  Google Scholar 

  • Califf, R. M., Zarin, D. A., Kramer, J. M., Sherman, R. E., Aberle, L. H., & Tasneem, A. (2012). Characteristics of clinical trials registered in ClinicalTrials. gov, 2007–2010. JAMA, 307(17), 1838–1847.

    Article  Google Scholar 

  • Chiou, J. Y., Magazzini, L., Pammolli, F., & Riccaboni, M. (2016). Learning from successes and failures in pharmaceutical R&D. Journal of Evolutionary Economics, 26(2), 271–290.

    Article  Google Scholar 

  • Contopoulos-Ioannidis, D. G., Ntzani, E. E., & Ioannidis, J. P. (2003). Translation of highly promising basic science research into clinical applications. The American Journal of Medicine, 114(6), 477–484.

    Article  Google Scholar 

  • Drew, C. H., Pettibone, K. G., Finch Iii, F. O., Giles, D., & Jordan, P. (2016). Automated research impact assessment: A new bibliometrics approach. Scientometrics, 106(3), 987–1005.

    Article  Google Scholar 

  • Drolet, B. C., & Lorenzi, N. M. (2011). Translational research: Understanding the continuum from bench to bedside. Translational Research, 157(1), 1–5.

    Article  Google Scholar 

  • Fairclough, R., & Thelwall, M. (2015). More precise methods for national research citation impact comparisons. Journal of Informetrics, 9(4), 895–906. doi:10.1016/j.joi.2015.09.005.

    Article  Google Scholar 

  • Glänzel, W., & Schubert, A. (2001). Double effort = double impact? A critical view at international co-authorship in chemistry. Scientometrics, 50(2), 199–214.

    Article  Google Scholar 

  • Grant, J., Cottrell, R., Cluzeau, F., & Fawcett, G. (2000). Evaluating “payback” on biomedical research from papers cited in clinical guidelines: Applied bibliometric study. BMJ, 320(7242), 1107–1111.

    Article  Google Scholar 

  • Hirsch, B. R., Califf, R. M., Cheng, S. K., Tasneem, A., Horton, J., Chiswell, K., et al. (2013). Characteristics of oncology clinical trials: Insights from a systematic analysis of ClinicalTrials.gov. JAMA Internal Medicine, 173(11), 972–979.

    Article  Google Scholar 

  • Ioannidis, J. P. (2006). Evolution and translation of research findings: From bench to where. PLoS One, 1(7), e36.

    Google Scholar 

  • Jones, T. H., & Hanney, S. (2016). Tracing the indirect societal impacts of biomedical research: Development and piloting of a technique based on citations. Scientometrics, 107(3), 975–1003.

    Article  Google Scholar 

  • Kissin, I. (2010). Can a bibliometric indicator predict the success of an analgesic? Scientometrics, 86(3), 785–795.

    Article  Google Scholar 

  • Kousha, K., & Thelwall, M. (2015a). Patent citation analysis with Google. Journal of the Association for Information Science and Technology. doi:10.1002/asi.23608.

    Google Scholar 

  • Kousha, K., & Thelwall, M. (2015b). Web indicators for research evaluation, part 3: Books and non-standard outputs. El Profesional de la Información, 24(6), 724–736. doi:10.3145/epi.2015.nov.04.

    Article  Google Scholar 

  • Laine, C., Horton, R., DeAngelis, C. D., Drazen, J. M., Frizelle, F. A., Godlee, F., et al. (2007). Clinical trial registration—Looking back and moving ahead. New England Journal of Medicine, 356(26), 2734–2736.

    Article  Google Scholar 

  • Lancho-Barrantes, B. S., Bote, G., Vicente, P., Rodríguez, Z. C., & de Moya Anegón, F. (2012). Citation flows in the zones of influence of scientific collaborations. Journal of the American Society for Information Science and Technology, 63(3), 481–489.

    Article  Google Scholar 

  • Larivière, V., & Gingras, Y. (2010). The impact factor’s Matthew effect: A natural experiment in bibliometrics. Journal of the American Society for Information Science and Technology, 61(2), 424–427.

    Google Scholar 

  • Lewison, G., & Dawson, G. (1998). The effect of funding on the outputs of biomedical research. Scientometrics, 41(1–2), 17–27.

    Article  Google Scholar 

  • Mohammadi, E., & Thelwall, M. (2013). Assessing non-standard article impact using F1000 labels. Scientometrics, 97(2), 383–395.

    Article  Google Scholar 

  • Ogino, D., Takahashi, K., & Sato, H. (2014). Characteristics of clinical trial websites: Information distribution between ClinicalTrials.gov and 13 primary registries in the WHO registry network. Trials, 15(1), 428.

    Article  Google Scholar 

  • Palma, D. A., & Zietman, A. (2015). Clinical trial registration: A mandatory requirement for publication in the red journal. International Journal of Radiation Oncology, 91(4), 685–686.

    Article  Google Scholar 

  • Prayle, A. P., Hurley, M. N., & Smyth, A. R. (2012). Compliance with mandatory reporting of clinical trial results on ClinicalTrials.gov: Cross sectional study. BMJ, 344, d7373.

    Article  Google Scholar 

  • Riveros, C., Dechartres, A., Perrodeau, E., Haneef, R., Boutron, I., & Ravaud, P. (2013). Timing and completeness of trial results posted at ClinicalTrials.gov and published in journals. PLoS Med, 10(12), e1001566.

    Article  Google Scholar 

  • Romero, A., Cortés, J., Escudero, C., López, J., & Moreno, J. (2009). Measuring the influence of clinical trials citations on several bibliometric indicators. Scientometrics, 80(3), 747–760.

    Article  Google Scholar 

  • Ross, J. S., Mulvey, G. K., Hines, E. M., Nissen, S. E., & Krumholz, H. M. (2009). Trial publication after registration in ClinicalTrials.gov: A cross-sectional analysis. PLoS Med, 6(9), e1000144.

    Article  Google Scholar 

  • Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. BMJ: British Medical Journal, 314(7079), 498.

    Article  Google Scholar 

  • Smith, L. B., Mitchell, R. T., & McEwan, I. J. (2013). Testosterone: From basic research to clinical applications. Berlin: Springer.

    Book  Google Scholar 

  • Stockmann, C., Sherwin, C. M., Koren, G., Campbell, S. C., Constance, J. E., Linakis, M., et al. (2014). Characteristics and publication patterns of obstetric studies registered in ClinicalTrials.gov. The Journal of Clinical Pharmacology, 54(4), 432–437.

    Article  Google Scholar 

  • Thelwall, M., & Delgado, M. (2015). Arts and humanities research evaluation: No metrics please, just data. Journal of Documentation, 71(4), 817–833. doi:10.1108/JD-02-2015-0028.

    Article  Google Scholar 

  • Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. (2013). Do altmetrics work? Twitter and ten other candidates. PLoS One, 8(5), e64841. doi:10.1371/journal.pone.0064841.

    Article  Google Scholar 

  • Thelwall, M., Kousha, K., Dinsmore, A., & Dolby, K. (2016). Alternative metric indicators for funding scheme evaluations. Aslib Journal of Information Management, 68(1), 2–18. doi:10.1108/AJIM-09-2015-0146.

    Article  Google Scholar 

  • Thelwall, M., & Maflahi, N. (2016). Guideline references and academic citations as evidence of the clinical value of health research. Journal of the Association for Information Science and Technology, 67(4), 960–966. doi:10.1002/asi.23432.

    Article  Google Scholar 

  • Tse, T., Williams, R. J., & Zarin, D. A. (2009). Reporting “basic results” in ClinicalTrials.gov. CHEST Journal, 136(1), 295–303.

    Article  Google Scholar 

  • van Raan, A. F. (1998). In matters of quantitative studies of science the fault of theorists is offering too little and asking too much. Scientometrics, 43(1), 129–139.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. (2011a). Towards a new crown indicator: Some theoretical considerations. Journal of Informetrics, 5(1), 37–47.

    Article  Google Scholar 

  • Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. (2011b). Towards a new crown indicator: An empirical analysis. Scientometrics, 87(3), 467–481.

    Article  Google Scholar 

  • Wilsdon, J. et al. (2015). The metric tide: Report of the independent review of the role of metrics in research assessment and management. http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide/Title,104463,en.html.

  • Zarin, D. A., Tse, T., & Ide, N. C. (2005). Trial registration at ClinicalTrials.gov between May and October 2005. New England Journal of Medicine, 353(26), 2779–2787.

    Article  Google Scholar 

  • Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011). The ClinicalTrials.gov results database—Update and key issues. New England Journal of Medicine, 364(9), 852–860.

    Article  Google Scholar 

  • Zitt, M. (2012). The journal impact factor: Angel, devil, or scapegoat? A comment on JK Vanclay’s article 2011. Scientometrics, 92(2), 485–503.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Statistical Cybermetrics Research Group, University of Wolverhampton, Wulfruna Street, Wolverhampton, UK

    Mike Thelwall & Kayvan Kousha

Authors
  1. Mike Thelwall
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kayvan Kousha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mike Thelwall.

Appendix: Specific method details

Appendix: Specific method details

The free web crawler SocSciBot (http://socscibot.wlv.ac.uk) crawled the ClinicalTrials.gov index site at https://www.clinicaltrials.gov/ct2/crawl) and the link files in the SocSciBot link results folder imported into Excel and used to extract a complete list of URLs. From these URLs, a complete list of standard study URLs was generated.

A second SocSciBot crawl was then started, using a 1-s pause between requests (for politeness) and using a dummy fake startup URL and the above list of URLs as the start.txt initial list (this ensures that only URLs in the list are crawled).

When the crawl had finished, the free Webometric Analyst software (http://lexiurl.wlv.ac.uk) was used to extract the relevant information as follows.

  • The Services menu, ClinicalTrials.gov: Extract information from SocSciBot crawl (e.g., AgesEligible, MeshTerms, Publications, PubsLinkedViaNCT) menu item was used to extract relevant information from the crawled pages (pointing the program to the root folder for the second crawl). ≫ Record list file.

  • The Services menu, ClinicalTrials.gov: Summarise results of the above (e.g., study type, gender by year: pubs; keyword freqs.) menu item was used to generate summary information (pointing the program to the record list file generated above). ≫ Summary tables files.

  • The Citation menu, Clinicaltrials.gov (or similar): Extract citations from |-separated col, extract doi and year, match with record year menu item was used to generate lists of users-added publications (pointing the program to the record list file generated above, and selecting the user-added publications column). ≫ List of user added publications file.

  • Using Scopus, a list of all publications of type journal article was extracted for JAMA, BMJ, Lancet and NEJM. ≫ Scopus articles file.

  • The Scopus publication list was matched with each of the above two publications files by exact doi matching (case insensitive) or, for articles without DOI matches, title, journal name and year matching (case insensitive) using the Citation menu, Match DOIs or article title, year, journal from one file with same info from another menu item (selecting the Scopus articles file first, then the List of user added publications file). ≫ List of user added publications and matching Scopus records file.

  • Duplicate user added publication matches (i.e., the same publication cited by different trials records) were removed in Excel. Publications from the same or later year than the trial start year were also removed in Excel. ≫ List of unique user added prior publications and matching Scopus records file.

  • Geometric mean normalised citation counts for each file were created using the Citation menu, Calculate geometric mean normalised citation counts for a marked subset of articles, separately by year menu item (selecting the Scopus articles file first, then the List of user added publications file). ≫ geometric mean normalised user added results file.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Thelwall, M., Kousha, K. Are citations from clinical trials evidence of higher impact research? An analysis of ClinicalTrials.gov. Scientometrics 109, 1341–1351 (2016). https://doi.org/10.1007/s11192-016-2112-1

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-016-2112-1

Keywords

Search

Navigation