Abstract
As federal programs are held more accountable for their research investments, The National Institute of Environmental Health Sciences (NIEHS) has developed a new method to quantify the impact of our funded research on the scientific and broader communities. In this article we review traditional bibliometric analyses, address challenges associated with them, and describe a new bibliometric analysis method, the Automated Research Impact Assessment (ARIA). ARIA taps into a resource that has only rarely been used for bibliometric analyses: references cited in “important” research artifacts, such as policies, regulations, clinical guidelines, and expert panel reports. The approach includes new statistics that science managers can use to benchmark contributions to research by funding source. This new method provides the ability to conduct automated impact analyses of federal research that can be incorporated in program evaluations. We apply this method to several case studies to examine the impact of NIEHS funded research.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
We use the word “grant” in this paper broadly, to include both projects that are conducted internally at NIH as well as “extramural” research that occurs beyond the walls of NIH.
Kostoff used the term research impact assessment to describe a broad range of methods to evaluate research, including peer review, retrospective methods, bibliometrics, co-occurrence, cost-benefit and economic analyses, and network analyses (Kostoff 1995). We coined the phrase automated research impact assessment without knowing that the term was already in use, but feel that there is a good fit between our new bibliometric method and Kostoff’s vision for broader research impact assessment activities.
See supplemental materials for an example of the raw data generated automatically in this report.
References
Ajiferuke, I., & Wolfram, D. (2010). Citer analysis as a measure of research impact: Library and information science as a case study. Scientometrics, 83, 623–638. doi:10.1007/s11192-009-0127-6.
Author Unknown. (1970). A trace of “traces”. Mosaic Magazine Science Articles Archive, 1(1), 14–19. Retrieved from http://www.mosaicsciencemagazine.org/index.php?mode=article&pk_magazine=109. Accessed 19 March 2015.
Bernstein, B. E., Stamatoyannopoulos, J. A., Costello, J. F., Ren, B., Milosavljevic, A., Meissner, A., et al. (2010). The NIH roadmap epigenomics mapping consortium. [opinion and comment]. Nature Biotechnology, 28(10), 1045–1048. doi:10.1038/nbt1010-1045. Retrieved from http://www.nature.com/nbt/journal/v28/n10/full/nbt1010-1045.html. Accessed 17 March 2015.
Borgman, C. L., & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology, 36, 3–72.
Boyack, K. W., & Jordan, P. (2011). Metrics associated with NIH funding: A high-level view. Journal of the American Medical Informatics Association, 18(4), 423–431. doi:10.1136/amiajnl-2011-000213.
Campbell, D., Picard-Aitken, M., Cote, G., Caruso, J., Valentim, R., Edmonds, S., et al. (2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation, 31(1), 66–83. doi:10.1177/1098214009354774.
Engel-Cox, J. A., Van Houten, B., Phelps, J., & Rose, S. W. (2008). Conceptual model of comprehensive research metrics for improved human health and environment. Environmental Health Perspectives, 116(5), 583–592. doi:10.1289/ehp.10925. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18470312.
Guthrie, K., Louie, J., David, T., & Foster, C. C. (2005). The challenge of assessing policy and advocacy activities: Strategies for a prospective evaluation approach. Los Angeles: The California Endowment.
Haak, L., Ferriss, W., Wright, K., Pollard, M., Barden, K., Probus, M., et al. (2012). The electronic Scientific Portfolio Assistant: Integrating scientific knowledge databases to support program impact assessment. Science and Public Policy, 39, 464–475. doi:10.1093/scipol/scs030.
Hanney, S., Home, P., Frame, I., Grant, J., Green, P., & Buxton, M. (2005). Indentifying the impact of diabetes research. Diabetic Medicine, 23, 176–184. doi:10.1111/j.1464-5491.2005.01753.x.
Hicks, D., Tomizawa, H., Saitoh, Y., & Kobayashi, S. (2004). Bibliometric techniques in the evaluation of federally funded research in the United States. Research Evaluation, 13(2), 76–86. doi:10.3152/147154404781776446. Retrieved from http://rev.oxfordjournals.org/content/13/2/7. Accessed 19 March 2015.
Howell, E. M., & Yemane, A. (2006). An assessment of evaluation designs: Case studies of 12 large federal evaluations. American Journal of Evaluation, 27(2), 219–236. doi:10.1177/1098214006287557. Retrieved from http://aje.sagepub.com/content/27/2/219.abstract.
Hughes, D., Docto, L., Peters, J., Lamb, A. L., & Brindis, C. (2013). Swimming upstream: The challenges and rewards of evaluating efforts to address inequities and reduce health disparities. Evaluation and Program Planning, 38, 1–12. Retrieved from http://www.sciencedirect.com/science/article/pii/S0149718913000050.
IIT Research Institute. (1968). Technology in retrospect and critical events in science: Prepared for the national science foundation (Vol. Volume 1 & 2). Chicago: Illinois Institute of Technology Research Institute under Contract NSF-C535.
Impact-EV. (2014). IMPACT-EV. http://impact-ev.eu/. Accessed 19 Aug 2015.
Jones, T., Donovan, C., & Hanney, S. (2012). Tracing the wider impact of biomedical research: A literature search to develop a novel citation categorisation technique. Scientometrics, 93, 125–134. doi:10.007/s11192-012-0642-8.
Jump, P. (2013). Australia prepares for (research) impact. Times Higher Education. June 22. Retrieved from http://www.timeshighereducation.co.uk/news/australia-prepares-for-research-impact/2005011.article.
Kan, M. (2010). ParsCit. https://www.comp.nus.edu.sg/entrepreneurship/ParsCit_kanmy.html. Accessed 19 Aug 2015.
Koplan, J. P., Milstein, R., & Wetterhall, S. (1999). Framework for program evaluation in public health. MMWR: Recommendations and Reports, 48, 1–40.
Kostoff, R. N. (1995). Federal research impact assessment: Axioms, approaches, applications. Scientometrics, 34(2), 163–206.
Kostoff, R. N. (1998). The use and misuse of citation analysis in research evaluation. Scientometrics, 43(1), 27–43.
Lane, J. (2010). Let’s make science metrics more scientific. Nature, 464(7288), 488–489. doi:10.1038/464488a.
Lane, J., & Bertuzzi, S. (2011). Measuring the results of science Investments. Science, 331(6018), 678–680. doi:10.1126/science.1201865. Retrieved from http://www.sciencemag.org/content/331/6018/678.short.
Lewison, G., Rippon, I., & Wooding, S. (2005). Tracking knowledge diffusion through citations. Research Evaluation, 14(1), 5–14. doi:10.3152/147154405781776319.
Leyedesdorff, L. (1998). Theories of citation? Scientometrics, 43(1), 5–25.
Liebow, E., Phelps, J., Van Houten, B., Rose, S., Orians, C., Cohen, J., et al. (2009). Toward the assessment of scientific and public health impacts of the National Institute of Environmental Health Sciences Extramural Asthma Research Program using available data. Environmental Health Perspectives, 117(7), 1147–1154. doi:10.1289/ehp.0800476. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/19654926.
Martin, B. R. (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36(3), 343–362. doi:10.1007/BF02129599. Retrieved from http://sci2s.ugr.es/hindex/pdf/Martin1996.pdf.
Milat, A. J., Bauman, A. E., & Redman, S. (2015). A narrative review of research impact assessment models and methods. Health Research Policy and Systems, 13(18). doi:10.1186/s12961-015-0003-1. Retrieved from http://www.health-policy-systems.com/content/pdf/s12961-015-0003-1.pdf.
Moed, H. F. (Ed.). (2005). Citation analysis in research evaluation (information science and knowledge management). Netherlands Springer: Dordrecht.
Narin, F. (1976). Evaluative bibliometrics. Cherry Hill, NJ: Computer Horizons.
National Academy of Sciences (NAS). (2001). Implementing the government performance and results act for research: A status report. Washington, DC: National Academy of Sciences, National Academy of Engineering, Institute of Medicine.
National Academy of Sciences (NAS). (2004). NIH Extramural center programs: Criteria for initiation and evaluation. Washington, DC: National Academy of Science.
National Institute of Environmental Health Sciences (NIEHS). (2012). Partnerships for environmental public health evaluation metrics manual (NIH publication no. 12-7825). Durham, NC. Retrieved from http://www.niehs.nih.gov/research/supported/assets/docs/a_c/complete_peph_evaluation_metrics_manual_508.pdf. Accessed 13 March 2015.
National Institutes of Health (NIH). (2014a). NIH RePORTER. http://projectreporter.nih.gov/reporter.cfm. Accessed 11 March 2015.
National Institutes of Health (NIH). (2014b). NIH public access policy website. http://publicaccess.nih.gov/policy.htm. Accessed 20 March 2015.
National Institutes of Health (NIH). (2015a). iEdison. https://public.era.nih.gov/iedison. Accessed 19 Aug 2015.
National Institutes of Health (NIH). (2015b). NIH clinical trials database. https://clinicaltrials.gov/. Accessed 19 Aug 2015.
National Library of Medicine (NLM). (2015). Health services/technology assessment texts http://www.ncbi.nlm.nih.gov/books/NBK16710/. Accessed 19 Aug 2015.
National Research Council (NRC). (1999). Arsenic in drinking water. Washington, D.C.: National Research Council (NRC). doi:10.17226/6444.
National Research Council (NRC). (2001). Arsenic in drinking water: 2001 update. Washington, D.C.: National Research Council (NRC). doi:10.17226/10194.
National Research Council (NRC). (1998). Assessing the value of research in the chemical sciences: Report of a workshop. In 6. Patents and publicly funded research. Washington (DC): National Academics Press. doi:10.17226/6200.
Orians, C., Abed, J., Drew, C., Rose, S. W., Cohen, J., & Phelps, J. (2009). Scientific and public health impacts of the NIEHS Extramural Asthma Research Program—Insights from primary data. Research Evaluation, 18(5), 375–385. doi:10.3152/095820209X480698. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/21921976.
ParaTools. (2004). ParaTools. http://paracite.eprints.org/developers/. Accessed 19 Aug 2015.
Pasterkamp, G., Rotmas, J., de Kleijn, D., & Borst, C. (2007). Citation frequency: A biased measure of research impact significantly influenced by the geographical origin of research articles. Scientometrics, 70(1), 153–165.
Phelan, T. J. (1999). A compendium of issues for citation analysis. Scientometrics, 45(1), 117–136. doi:10.1007/BF02458472.
Quinlan, K. M., Kane, M., & Trochim, W. M. K. (2008). Evaluation of large research initiatives: Outcomes, challenges, and methodological considerations. New Directions for Evaluation, 2008(118), 61–72. doi:10.1002/ev.261.
Research Excellence Framework. (2014). http://www.ref.ac.uk/. Accessed 19 Aug 2015.
Researchfish. (2014). https://www.researchfish.com/. Accessed 19 Aug 2015.
Reuters, T. (1994). The Thomson reuters impact factor. http://thomsonreuters.com/products_services/science/free/essays/impact_factor/. Accessed 19 March 2015.
Roessner, D., Carr, R., Feller, I., McGeary, M., & Newman, N. (1998). The role of NSF’s support of engineering in enabling technological innovation: Phase II, final report to National Science Foundation. Arlington, VA: SRI International.
Sayers, E. (2009). The E-utlities in-depth: Parameters, syntax and more. In Entrez programming utilities help. Bethesda, MD: National Center for Biotechnology Information, U.S. National Library of Medicine. Retrieved from ESearch: http://www.ncbi.nlm.nih.gov/books/NBK25499/#_chapter4_ESearch_. EFetch: http://www.ncbi.nlm.nih.gov/books/NBK25499/#_chapter4_EFetch_. ECitMatch: http://www.ncbi.nlm.nih.gov/books/NBK25499/#_chapter4_ECitMatch_.
Srivastava, C. V., Towery, N. D., & Zuckerman, B. (2007). Challenges and opportunities for research portfolio analysis, management, and evaluation. Research Evaluation, 16(3), 152–156. doi:10.3152/095820207x236385. Retrieved from http://rev.oxfordjournals.org/content/16/3/152.abstract.
Stuart, J. (2007). Necessity leads to innovative evaluation approach and practice. Evaluation Exchange, XII(1), 2.
Teles, S., & Schmitt, M. (2011). The elusive craft of evaluating advocacy. Stanford Social Innovation Review, 4. Retrieved from http://www.ssireview.org/images/digital_edition/2011SU_Feature_TelesSchmitt.pdf.
The Kellogg Foundation. (2004). Logic model development guide. Battle Creek, MI. Retrieved from http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx. Accessed 13 March 2015.
U.S. Environmental Protection Agency (EPA). (2009). Integrated science assessment for particulate Matter (EPA/600/R-08/139F). Research Triangle Park, NC. Retrieved from http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=216546. Accessed 13 March 2015.
U.S. Environmental Protection Agency (EPA). (2010). Integrated science assessment for carbon monoxide (EPA/600/R-09/019F). Research Triangle Park, NC. Retrieved from http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=218686. Accessed 13 March 2015.
U.S. Environmental Protection Agency (EPA). (2012). Integrated science assessment for lead (EPA/600/R-10/075B). Research Triangle Park, NC. Retrieved from http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=255721. Accessed 13 March 2015.
U.S. Government Accountability Office. (2012). Designing evaluations (Publication No. GAO-12-208G). Washington, DC. Retrieved from http://www.gao.gov/products/GAO-12-208G. Accessed 11 March 2015.
Wooding, S., Hanney, S., Buxton, M., & Grant, J. (2005). Payback arising from research funding: Evaluation of the Arthritis Research Campaign. Rheumatology, 44, 1145–1156. doi:10.1093/rheumatology/keh708.
World Health Organization (WHO). (2011). Arsenic in drinking water. Geneva: World Health Organization.
Acknowledgments
This work was supported by the National Institute of Environmental Health Sciences (NIEHS). The authors would like to acknowledge the work of Sheila Newton and Raymond Grissom, Jr., of the NIEHS Office of Planning and Policy Evaluation, who conducted an early manual review of an EPA Ozone Regulation as a test of this new bibliometric research method. Many thanks also to James Corrigan (National Cancer Institute) who provided comments on early versions of this paper; and to James Onken and Brian Haugen in the NIH Office of Extramural Research, Office of Data Analysis Tools and Systems, who currently manage the SPIRES database (and RePARS tool) for the National Institutes of Health.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Drew, C.H., Pettibone, K.G., Finch, F.O. et al. Automated Research Impact Assessment: a new bibliometrics approach. Scientometrics 106, 987–1005 (2016). https://doi.org/10.1007/s11192-015-1828-7
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-015-1828-7