Skip to main content

A bibliometric measure of translational science

Scientometrics volume 125, pages 2349–2382 (2020)Cite this article

Abstract

Science funders are increasingly requiring evidence of the broader impacts of even basic research. Initiatives such as NIH’s CTSA program are designed to shift the research focus toward more translational research. However, tracking the effectiveness of such programs depends on developing indicators that can track the degree to which basic research is influencing clinical research. We propose a new bibliometric indicator, the TS score, that is relatively simple to calculate, can be implemented at scale, is easy to replicate, and has good reliability and validity properties. This indicator is broadly applicable in settings where the goal is to estimate the degree to which basic research is used in more applied downstream research, relative to use in basic research. The TS score should be of use for a variety of policy analysis and research evaluation purposes.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. One specific example of translational research is a study by Bhat et al. (1997) that found binding of fusion genes (BCR-ABL) and certain type of protein (c-CBL) only occurs when phosphate (PO43−) is added to acid (amino tyrosine) on a protein. Their finding was applied in treating genetic abnormality in chromosome 22 of leukemia cancer cell and led to the invention of Tasiga® (Sampat and Pincus 2015). Another example of translational research is a study by Garg and Hassid (1990). They found that proliferation of cell lines developed from disaggregated mouse embryos (BALB/c 3T3) are more active when muscle smoother is not present (CGMP-independent mechanism). Their finding was applied in solving respiratory failure problem and this led to the invention of INOmax® (Sampat and Pincus 2015).

  2. Examples include clinical study, clinical trial, phase 1 clinical trial, phase 2 clinical trial, phase 3 clinical trial, phase 4 clinical trial, controlled clinical trial, practice guideline, observational study and randomized controlled trial.

  3. More examples are available on request.

  4. In other words, if the paper was published in 2003, and had 3 cites in the first year (2004), it would appear in the 3 cite row, and the correlations are for the scores for the years 4 + in the future from 2004. If in the following year, it had reached 8 citations, then it would reappear in the 8 cite row, with a starting year of 2005, and the columns representing 4 + years from 2005. This means that the same paper can appear in multiple rows (but not generally every row), and also that a paper can drop out of the table if it acquires a total of 11 + cites. Hence, the Ns can fluctuate up and down across the rows.

  5. To the extent that this is not true, then our measure will include greater measurement error, biasing the correlations toward zero and hence giving us a more conservative test of the validity of our TS score.

  6. For simplicity, we ignore additional “co-first authors”. We do not expect there are a large share of such cases, and, furthermore, if the co-first authors are in the same department, or even in the same class of departments (clinical vs non-clinical), the results would be unchanged if we included them.

  7. There were some issues to consider when classifying publications into clinical papers and non-clinical papers, which led to doing the classification manually. First, the department information did not exist in some listings (e.g., Univ St Andrews, St Andrews KY 16 9ST, Fife, Scotland; RAND Corp, Santa Monica, CA 90401 USA). In these cases, these were coded as missing. Second, the order that university name, department or school name and city are listed were different between articles. Though the department information was listed second in most cases, it was listed first in some cases, later in other cases. Third, the details of the affiliation were different across publications. For instance, some publications provided very detailed information (e.g., Sch Med & Dent, Dept Biostat & Computat Bio) whereas some publications only provided information at the school level (e.g., Sch Life Sci & Technol). To be conservative on classifying a publication into clinical papers, we classified the publications into a clinical article only if there was clear and sufficient information on the discipline. For instance, we did not classify a publication as clinical if the most detailed information of the first author’s affiliation is “School of Medicine”. This is because, in many universities, the School of Medicine is composed of departments conducting basic science work (e.g., Department of Microbiology) as well as clinical work. Therefore, we classified a publication as a clinical paper only if the sub-division of the school is listed and it is closely related to fields of clinical research (e.g., Department of Pediatrics).

  8. LENS.ORG is an open public website managed by Cambia, an independent non-profit organization, that provides linkages between scholarly works, patents and biological sequences (LENS.ORG n.d.). Its patent database covers patent datasets from the USPTO, European Patent Office, WIPO and IP Australia and its scholarly dataset includes PubMed, Crossref and Microsoft Academics (LENS.ORG n.d.). With the collaboration with NIH Pubmed and Crossref teams, the Lens links publications’ Digital Object Identifier (DOI) with NPL in their patent database. Hence, using the DOI of publications, we could check if the publications were cited by patents or not.

  9. Using Pubmed ID instead of DOI for the search may give a higher retrieval rate. However, it would then be more difficult to track the forward citation links of publications to patents with only Pubmed ID but without the DOI.

  10. Both of these measures are available from the iCite website (https://icite.od.nih.gov/analysis). It is not clear what data window is used for calculating the APT scores published on the website. The data were downloaded June 18, 2020.

  11. The inferences from a linear probability model are the same (results available from contact author).

References

  • Bagnol, D., Mansour, A., Akil, H., & Watson, S. J. (1997). Cellular localization and distribution of the cloned mu and kappa opioid receptors in rat gastrointestinal tract. Neuroscience, 81(2), 579–591.

    Article  Google Scholar 

  • Bhat, A., Kolibaba, K., Oda, T., Ohno-Jones, S., Heaney, C., & Druker, B. J. (1997). Interactions of CBL with BCR-ABL and CRKL in BCR-ABL-transformed myeloid cells. Journal of Biological Chemistry, 272(26), 16170–16175.

    Article  Google Scholar 

  • Blümel, C. (2017). Translational research in the science policy debate: A comparative analysis of documents. Science and Public Policy, 45(1), 24–35.

    Article  Google Scholar 

  • Butler, D. (2008). Translational research: Crossing the valley of death. Nature, 453, 840–842. https://doi.org/10.1038/453840a.

    Article  Google Scholar 

  • Carpenter, C. R., Cone, D. C., & Sarli, C. C. (2014). Using publication metrics to highlight academic productivity and research impact. Academic Emergency Medicine, 21(10), 1160–1172.

    Article  Google Scholar 

  • Chen, C., & Hicks, D. (2004). Tracing knowledge diffusion. Scientometrics, 59(2), 199–211. https://doi.org/10.1023/B:SCIE.0000018528.59913.48.

    Article  Google Scholar 

  • Contopoulos-Ioannidis, D. G., Alexiou, G. A., Gouvias, T. C., & Ioannidis, J. P. (2008). Life cycle of translational research for medical interventions. Science, 321(5894), 1298–1299.

    Article  Google Scholar 

  • Cotropia, C. A., Lemley, M. A., & Sampat, B. (2013). Do applicant patent citations matter? Research Policy, 42(4), 844–854. https://doi.org/10.1016/j.respol.2013.01.003.

    Article  Google Scholar 

  • Drolet, B. C., & Lorenzi, N. M. (2011). Translational research: Understanding the continuum from bench to bedside. Translational Research, 157(1), 1–5.

    Article  Google Scholar 

  • EATRIS ERIC. (n.d.). About EATRIS European Infrastructure for translational medicine. https://eatris.eu/about/. Accessed 11 Aug 2020.

  • Fishburn, C. S. (2013). Translational research: The changing landscape of drug discovery. Drug Discovery Today, 18(9), 487–494.

    Article  Google Scholar 

  • Fontelo, P., & Liu, F. (2011). Finding translational science publications in MEDLINE/PubMed with translational science filters. Clinical and Translational Science, 4(6), 455–459.

    Article  Google Scholar 

  • Fukushima, M., & Kimura, Y. (n.d.). Translational Research Informatics Center (TRI) takes the initiative to reduce disability through Japanese academic research. https://s3-service-broker-live-19ea8b98-4d41-4cb4-be4c-d68f4963b7dd.s3.amazonaws.com/uploads/ckeditor/attachments/8326/TRi.pdf. Accessed 11 Aug 2020.

  • Garg, U. C., & Hassid, A. (1990). Nitric oxide-generating vasodilators inhibit mitogenesis and proliferation of BALBc 3T3 fibroblasts by a cyclic GMP-independent mechanism. Biochemical and Biophysical Research Communications, 171(1), 474–479.

    Article  Google Scholar 

  • Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The Qualitative Report, 8(4), 597–606.

    Google Scholar 

  • Grant, J., Cottrell, R., Cluzeau, F., & Fawcett, G. (2000). Evaluating “payback” on biomedical research from papers cited in clinical guidelines: Applied bibliometric study. BMJ, 320(7242), 1107–1111.

    Article  Google Scholar 

  • Han, X., Williams, S. R., & Zuckerman, B. L. (2018). A snapshot of translational research funded by the National Institutes of Health (NIH): A case study using behavioral and social science research awards and Clinical and Translational Science Awards funded publications. PLoS ONE, 13(5), e0196545. https://doi.org/10.1371/journal.pone.0196545.

    Article  Google Scholar 

  • Huang, M. H., Yang, H. W., & Chen, D. Z. (2015). Increasing science and technology linkage in fuel cells: A cross citation analysis of papers and patents. Journal of Informetrics, 9(2), 237–249.

    Article  Google Scholar 

  • Hutchins, B., Davis, M., Meseroll, R., & Santangelo, G. (2019). Predicting translational progress in biomedical research. PLoS Biology, 17(10), e3000416.

    Article  Google Scholar 

  • Ihli, M. I. R. F. (2016). Acknowledgement lag and impact: Domain differences in published research supported by the National Science Foundation (Master’s thesis). University of Tennessee-Knoxville, Knoxville, TN.

  • Kim, S. (2013). The concept of translational research and the requirements for the success (in Korean). STEPI Insight, 15(115), 1–25.

    Google Scholar 

  • Kim, Y. H. (2019). Bridging the valley of death in biomedicine with translational research: Assessing the impact of National Institutes of Health’s Clinical and Translational Science Award. Atlanta, GA: Georgia Institute of Technology.

    Google Scholar 

  • Knapke, J. M., Haynes, E. N., Kuhnell, P., & Tsevat, J. (2015). NIH grant awards as a metric of clinical and translational research training effectiveness. Clinical and Translational Science, 8(1), 52–56. https://doi.org/10.1111/cts.12232.

    Article  Google Scholar 

  • Lance, C. E., Butts, M. M., & Michels, L. C. (2006). The sources of four commonly reported cutoff criteria: What did they really say? Organizational Research Methods, 9(2), 202–220. https://doi.org/10.1177/1094428105284919.

    Article  Google Scholar 

  • Lander, B., & Atkinson-Grosjean, J. (2011). Translational science and the hidden research system in universities and academic hospitals: A case study. Social Science & Medicine, 72(4), 537–544.

    Article  Google Scholar 

  • Li, D., Azoulay, P., & Sampat, B. N. (2017). The applied value of public investments in biomedical research. Science, 356(63333), 78–81.

    Article  Google Scholar 

  • Liu, G., Chen, G., Sinoway, L. I., & Berg, A. (2013). Assessing the impact of the NIH CTSA program on institutionally sponsored clinical trials. Clinical and Translational Science, 6(3), 196–200. https://doi.org/10.1111/cts.12029.

    Article  Google Scholar 

  • Llewellyn, N., Carter, D. R., DiazGranados, D., Pelfrey, C., Rollins, L., & Nehl, E. J. (2019). Scope, influence, and interdisciplinary collaboration: The publication portfolio of the NIH clinical and translational science awards (CTSA) program from 2006 through 2017. Evaluation & the Health Professions. https://doi.org/10.1177/0163278719839435.

    Article  Google Scholar 

  • Llewellyn, N., Carter, D. R., Rollins, L., & Nehl, E. J. (2018). Charting the publication and citation impact of the NIH clinical and translational science awards (CTSA) program from 2006 through 2016. Academic Medicine: Journal of the Association of American Medical Colleges, 93(8), 1162–1170. https://doi.org/10.1097/ACM.0000000000002119.

    Article  Google Scholar 

  • Luke, D. A., Carothers, B. J., Dhand, A., Bell, R. A., Moreland-Russell, S., Sarli, C. C., et al. (2015). Breaking down silos: Mapping growth of cross-disciplinary collaboration in a translational science initiative. Clinical and Translational Science, 8(2), 143–149. https://doi.org/10.1111/cts.12248.

    Article  Google Scholar 

  • Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018). Google scholar, web of science, and scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160–1177. https://doi.org/10.1016/j.joi.2018.09.002.

    Article  Google Scholar 

  • McElroy, S. P., Jones, P. S., & Barrault, D. V. (2017). The SULSA Assay Development Fund: accelerating translation of new biology from academia to pharma. Drug Discovery Today, 22(2), 199–203.

    Article  Google Scholar 

  • Medical Research Councils UK. (2017). Translational research. https://www.mrc.ac.uk/funding/science-areas/translation/. Accessed 1 Dec 2017.

  • Meyer, M. (2000). Does science push technology? Patents citing scientific literature. Research Policy, 29(3), 409–434.

    Article  Google Scholar 

  • Morillo, F., Bordons, M., & Gómez, I. (2001). An approach to interdisciplinarity through bibliometric indicators. Scientometrics, 51(1), 203–222.

    Article  Google Scholar 

  • Morris, Z. S., Wooding, S., & Grant, J. (2011). The answer is 17 years, what is the question: Understanding time lags in translational research. Journal of the Royal Society of Medicine, 104(12), 510–520.

    Article  Google Scholar 

  • Myers, K. (forthcoming). The elasticity of science. American Economic Journal: Applied Economics. Available at https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3176991. Accessed 11 Aug 2020.

  • Narin, F., Hamilton, K., & Olivastro, D. (1997). The increasing linkage between U.S. technology and public science. Research Policy, 26(3), 317–330.

    Article  Google Scholar 

  • Narin, F., Pinski, G., & Gee, H. H. (1976). Structure of the biomedical literature. Journal of the American Society for Information Science, 27(1), 25–45.

    Article  Google Scholar 

  • National Library of Medicine. (2020). Translational Medical Research MeSH Descriptor Data 2020. https://meshb.nlm.nih.gov/record/ui?name=Translational%2520Medical%2520Research.

  • National Science Foundation. (2018). Classification of field of study. https://www.nsf.gov/statistics/nsf13327/pdf/tabb1.pdf. Accessed 11 Aug 2020.

  • Ortega, L., & Antell, K. (2006). Tracking cross-disciplinary information use by author affiliation: Demonstration of a method. College & Research Libraries, 67(5), 446–462.

    Article  Google Scholar 

  • Porter, A., & Chubin, D. (1985). An indicator of cross-disciplinary research. Scientometrics, 8(3–4), 161–176.

    Article  Google Scholar 

  • President’s Council of Advisors on Science and Technology. (2012). Executive summary report to the president on propelling innovation in drug discovery, development and evaluation. https://www.brookings.edu/wp-content/uploads/2012/11/PCAST-Executive-Summary.pdf. Accessed 11 Aug 2020.

  • Qin, J., Lancaster, F. W., & Allen, B. (1997). Types and levels of collaboration in interdisciplinary research in the sciences. Journal of the American Society for Information Science, 48(10), 893–916.

    Article  Google Scholar 

  • Roach, M., & Cohen, W. M. (2012). Lens or prism? Patent citations as a measure of knowledge flows from public research. Management Science, 59(2), 504–525. https://doi.org/10.1287/mnsc.1120.1644.

    Article  Google Scholar 

  • Rock, K. L., Gramm, C., Rothstein, L., Clark, K., Stein, R., Dick, L., et al. (1994). Inhibitors of the proteasome block the degradation of most cell proteins and the generation of peptides presented on MHC class I molecules. Cell, 78(5), 761–771. https://doi.org/10.1016/S0092-8674(94)90462-6.

    Article  Google Scholar 

  • Rubio, D. M., Schoenbaum, E. E., Lee, L. S., Schteingart, D. E., Marantz, P. R., Anderson, K. E., et al. (2010). Defining translational research: Implications for training. Academic Medicine: Journal of the Association of American Medical Colleges, 85(3), 470–475. https://doi.org/10.1097/ACM.0b013e3181ccd618.

    Article  Google Scholar 

  • Sampat, B. N., & Pincus, H. A. (2015). Citations in life science patents to publicly funded research at academic medical centers. Clinical and Translational Science, 8(6), 759–763.

    Article  Google Scholar 

  • Schneider, M., Kane, C. M., Rainwater, J., Guerrero, L., Tong, G., Desai, S. R., et al. (2017). Feasibility of common bibliometrics in evaluating translational science. Journal of Clinical and Translational Science, 1(1), 45–52. https://doi.org/10.1017/cts.2016.8.

    Article  Google Scholar 

  • Smith, C., Baveja, R., Grieb, T., & Mashour, G. A. (2017). Toward a science of translational science. Journal of Clinical and Translational Science, 1(4), 253–255. https://doi.org/10.1017/cts.2017.14.

    Article  Google Scholar 

  • Stevenson, D. K., Shaw, G. M., Wise, P. H., Norton, M. E., Druzin, M. L., Valantine, H. A., et al. (2013). Transdisciplinary translational science and the case of preterm birth. Journal of Perinatology, 33(4), 251.

    Article  Google Scholar 

  • Sung, H. Y., Wang, C. C., Huang, M. H., & Chen, D. Z. (2015). Measuring science-based science linkage and non-science-based linkage of patents through non-patent references. Journal of Informetrics, 9(3), 488–498.

    Article  Google Scholar 

  • Surkis, A., Hogle, J. A., DiazGranados, D., Hunt, J. D., Mazmanian, P. E., Connors, E., et al. (2016). Classifying publications from the clinical and translational science award program along the translational research spectrum: A machine learning approach. Journal of Translational Medicine, 14(1), 235–235. https://doi.org/10.1186/s12967-016-0992-8.

    Article  Google Scholar 

  • Translational Research Informatics Center. (2003). https://www.tri-kobe.org/english/index.html. Accessed 9 June 2016.

  • Trochim, W., Kane, C., Graham, M. J., & Pincus, H. A. (2011). Evaluating translational research: A process marker model. Clinical and Translational Science, 4(3), 153–162. https://doi.org/10.1111/j.1752-8062.2011.00291.x.

    Article  Google Scholar 

  • Weber, G. M. (2013). Identifying translational science within the triangle of biomedicine. Journal of Translational Medicine, 11(1), 126.

    Article  Google Scholar 

  • Williams, R. S., Lotia, S., Holloway, A. K., & Pico, A. R. (2015). From scientific discovery to cures: Bright stars within a galaxy. Cell, 163(1), 21–23. https://doi.org/10.1016/j.cell.2015.09.007.

    Article  Google Scholar 

  • Williams, S. (2016). Building a translational medicine powerhouse in China. Science, 352(6293), 1596.

    Google Scholar 

  • Woolf, S. H. (2008). The meaning of translational research and why it matters. JAMA, 299(2), 211–213. https://doi.org/10.1001/jama.2007.26.

    Article  Google Scholar 

  • Zerhouni, E. A. (2005). Translational and clinical science—Time for a new vision. New England Journal of Medicine, 353(15), 1621–1623. https://doi.org/10.1056/NEJMsb053723.

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Nicole Llewellyn for her help in providing detailed information on the CTSA program, as well as help in data collection and data coding. The authors also thank Bhaven Sampat, Alan Porter and Weihua An for helpful suggestions. The research was supported by NSF award numbers SMA-1646689; DRL-1150114 and EEC-1648035, as well as the National Center for Advancing Translational Sciences of the National Institutes of Health under award number UL1TR000454. The content is solely the responsibility of the authors and does not represent the official views of the U.S. National Institutes of Health, the U.S. National Science Foundation or the Ministry of Science and ICT, Republic of Korea. An earlier version of this work appeared in Kim (2019).

Author information

Authors and Affiliations

  1. Ministry of Science and ICT, Sejong, Republic of Korea

    Yeon Hak Kim

  2. School of Public Policy, Georgia Institute of Technology, Atlanta, GA, USA

    Aaron D. Levine & John P. Walsh

  3. Rollins School of Public Health, Emory University, Atlanta, GA, USA

    Eric J. Nehl

Authors
  1. Yeon Hak Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aaron D. Levine
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eric J. Nehl
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John P. Walsh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John P. Walsh.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kim, Y.H., Levine, A.D., Nehl, E.J. et al. A bibliometric measure of translational science. Scientometrics 125, 2349–2382 (2020). https://doi.org/10.1007/s11192-020-03668-2

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-020-03668-2

Keywords

  • Translational research
  • Indicators
  • Citation analysis
  • Research evaluation