Skip to main content
SpringerLink
Log in

Measuring the Impact of Scientific Research

  • Chapter
  • First Online:
Scientific Scholarly Communication

Part of the book series: Fascinating Life Sciences ((FLS))

Abstract

The body of scientific knowledge grows with incremental additions. Assessing the scientific quality and the impact of these contributions is necessary because future scientific research is based on previous knowledge. As the key literature consulted and influenced their work should be cited when researchers publish findings, measures based on citations metrics became the most widely accepted impact assessment tools, and citation analysis is considered an objective means to evaluate scholarly publications . Historical developments, strengths, and limitations in citation-based assessment tools, use of impact factor in measuring the scientific quality of scholarly journals, and use, misuse, and manipulation of the journal impact factor are examined in this chapter. The discussion also includes citation indexes and related issues, and other journal ranking systems. Assessing the performance of individual scientists using citation metrics , the Hirsch index , and many variations proposed to correct its deficiencies are discussed. Although citation metrics can be considered the best tools yet implemented to assess the quality and influence of scientific research, the importance of understanding their strengths, limitations, and implications when using them is stressed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 64.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Journal Immediacy Index is defined by JCR as “the measures how frequently the average article from a journal is cited within the same year as publication.”

  2. 2.

    Journal cited half-life is defined by JCR as “the age of cited articles by showing the number of years back from the current year that account for 50% of the total number of citations to a journal in the current year.”

  3. 3.

    Not to be confused with author self-citation that will be discussed later.

  4. 4.

    Available from the JCR provided as a part of the Web of Knowledge platform by Thomson Reuter.

  5. 5.

    Citation impact of a publication is calculated relative to the other publications in the same subject field.

  6. 6.

    Calculations are based on article citations in the JCR from the journal over the last five years, also taking into consideration which journals contributed the citations, giving more weight to the highly cited journals.

  7. 7.

    SCImago Journal & Country Rank http://www.scimagojr.com/aboutus.php.

  8. 8.

    Article Influence Score measures the average influence of each article in a journal. It normalizes the values by giving the mean article in the JCR an AIS of 1.00. So, a journal with an AIS of 2.00 means that its articles are on average two times more influential than the average article in the JCR. (Bergstrom et al. 2008).

References

  • Abbott, A., Cyranoski, D., Jones, N., Maher, B., Schiermeier, Q., & Van Noorden, R. (2010). Metrics: Do metrics matter? Nature News, 465(7300), 860–862.

    Article  CAS  Google Scholar 

  • Alonso, S., Cabrerizo, F. J., Herrera-Viedma, E., & Herrera, F. (2009). h-index: A review focused in its variants, computation and standardization for different scientific fields. Journal of Informetrics, 3(4), 273–289. doi:10.1016/j.joi.2009.04.001.

    Article  Google Scholar 

  • Banks, M. G. (2006). An extension of the Hirsch index: Indexing scientific topics and compounds. Scientometrics, 69(1), 161–168.

    Article  CAS  Google Scholar 

  • Basken, P. (2016). Better than impact factor? NIH team claims key advance in ranking journal articles. The Chronicle of Higher Education. Retrieved from http://www.chronicle.com.ezproxy.waterfield.murraystate.edu/article/Better-Than-Impact-Factor-NIH/237703

  • Bensman, S. J. (2007). Garfield and the impact factor. Annual Review of Information Science and Technology, 41(1), 93–155.

    Article  Google Scholar 

  • Bergstrom, C. T., West, J. D., & Wiseman, M. A. (2008). The Eigenfactor™ metrics. The Journal of Neuroscience, 28(45), 11433–11434.

    Article  CAS  PubMed  Google Scholar 

  • Bollen, J., Rodriquez, M. A., & Van de Sompel, H. (2006). Journal status. Scientometrics, 69(3), 669–687.

    Article  CAS  Google Scholar 

  • Bornmann, L., & Marx, W. (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics, 98(1), 487–509.

    Article  Google Scholar 

  • Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011). A multilevel meta-analysis of studies reporting correlations between the h index and 37 different h index variants. Journal of Informetrics, 5(3), 346–359.

    Article  Google Scholar 

  • Bucur, O., Almasan, A., Zubarev, R., Friedman, M., Nicolson, G. L., Sumazin, P., … Kunej, T. (2015). An updated h-index measures both the primary and total scientific output of a researcher. Discoveries, 3(3).

    Google Scholar 

  • Burrell, Q. L. (2007). Hirsch’s h-index: A stochastic model. Journal of Informetrics, 1(1), 16–25.

    Article  Google Scholar 

  • Cantín, M., Muñoz, M., Roa, I., CantÍN, M., MuÑOz, M., & Roa, I. (2015). Comparison between impact factor, Eigenfactor score, and SCImago journal rank indicator in anatomy and morphology journals. International Journal of Morphology, 33(3), 1183–1188.

    Article  Google Scholar 

  • Chew, M., Villanueva, E. V., & Van Der Weyden, M. B. (2007). Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994–2005) and their Editors’ views. Journal of the Royal Society of Medicine, 100(3), 142–150. doi:10.1258/jrsm.100.3.142.

    Article  PubMed  PubMed Central  Google Scholar 

  • Durieux, V., & Gevenois, P. A. (2010). Bibliometric indicators: Quality measurements of scientific publication 1. Radiology, 255(2), 342–351.

    Article  PubMed  Google Scholar 

  • Egghe, L. (2006). Theory and practise of the g-index. Scientometrics, 69(1), 131–152.

    Article  Google Scholar 

  • Egghe, L. (2008). Mathematical theory of the h-and g-index in case of fractional counting of authorship. Journal of the American Society for Information Science and Technology, 59(10), 1608–1616.

    Article  Google Scholar 

  • Egghe, L., & Rao, I. K. (2008). Study of different h-indices for groups of authors. Journal of the American Society for Information Science and Technology, 59(8), 1276–1281.

    Article  Google Scholar 

  • Falagas, M. E., Kouranos, V. D., Arencibia-Jorge, R., & Karageorgopoulos, D. E. (2008). Comparison of SCImago journal rank indicator with journal impact factor. The FASEB Journal, 22(8), 2623–2628.

    Article  CAS  PubMed  Google Scholar 

  • Fassoulaki, A., Paraskeva, A., Papilas, K., & Karabinis, G. (2000). Self-citations in six anaesthesia journals and their significance in determining the impact factor. British Journal of Anaesthesia, 84(2), 266–269.

    Article  CAS  PubMed  Google Scholar 

  • Garfield, E. (1955). Citation indexes for science—new dimension in documentation through association of ideas. Science, 122(3159), 108–111. doi:10.1126/science.122.3159.108.

    Article  CAS  PubMed  Google Scholar 

  • Garfield, E. (1972). Citation analysis as a tool in journal evaluation. Science (New York, NY), 178(4060), 471.

    Article  CAS  Google Scholar 

  • Garfield, E. (1976). Significant journals of science. Nature, 264, 609–615.

    Article  CAS  PubMed  Google Scholar 

  • Garfield, E. (1999). Journal impact factor: A brief review. Canadian Medical Association Journal, 161(8), 979–980.

    CAS  PubMed  PubMed Central  Google Scholar 

  • Garfield, E., & Merton, R. K. (1979). Citation indexing: Its theory and application in science, technology, and humanities (Vol. 8). New York: Wiley.

    Google Scholar 

  • Garfield, E., & Sher, I. H. (1963). New factors in the evaluation of scientific literature through citation indexing. American documentation, 14(3), 195–201.

    Article  CAS  Google Scholar 

  • Glänzel, W., & Moed, H. F. (2002). Journal impact measures in bibliometric research. Scientometrics, 53(2), 171–193.

    Article  Google Scholar 

  • González-Pereira, B., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of informetrics, 4(3), 379–391.

    Article  Google Scholar 

  • Hakkalamani, S., Rawal, A., Hennessy, M. S., & Parkinson, R. W. (2006). The impact factor of seven orthopaedic journals factors influencing it. Journal of Bone & Joint Surgery, British, 88(2), 159–162.

    Article  CAS  Google Scholar 

  • Hirsch, J. E. (2005). An index to quantify an individual’s scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102(46), 16569–16572. doi:10.1073/pnas.0507655102.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Hutchins, B. I., Yuan, X., Anderson, J. M., & Santangelo, G. M. (2016). Relative Citation Ratio (RCR): A new metric that uses citation rates to measure influence at the article level. bioRxiv, 029629.

    Google Scholar 

  • Imperial, J., & Rodríguez-Navarro, A. (2007). Usefulness of Hirsch’s h-index to evaluate scientific research in Spain. Scientometrics, 71(2), 271–282.

    Article  Google Scholar 

  • Jin, B., Liang, L., Rousseau, R., & Egghe, L. (2007). The R-and AR-indices: Complementing the h-index. Chinese Science Bulletin, 52(6), 855–863.

    Article  Google Scholar 

  • King, J. (1987). A review of bibliometric and other science indicators and their role in research evaluation. Journal of information science, 13(5), 261–276.

    Article  Google Scholar 

  • Kirchhof, B., Bornfeld, N., & Grehn, F. (2006). The delicate topic of the impact factor. Graefe’s Archive for Clinical and Experimental Ophthalmology, 245(7), 925–927.

    Article  Google Scholar 

  • Kostoff, R. N. (2002). Citation analysis of research performer quality. Scientometrics, 53(1), 49–71.

    Article  Google Scholar 

  • Krell, F.-T. (2010). Should editors influence journal impact factors? Learned Publishing, 23(1), 59–62.

    Article  Google Scholar 

  • Kurmis, A. P. (2003). Current concepts review—understanding the limitations of the journal impact factor. Journal of Bone and Joint Surgery-American, 85A(12), 2449–2454.

    Article  Google Scholar 

  • Leydesdorff, L., & Opthof, T. (2010). Normalization at the field level: Fractional counting of citations. Journal of Informetrics, 4(4), 644–646.

    Article  Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. F. (1989). Problems of citation analysis: A critical review. Journal of the American Society for Information Science, 40(5), 342–349.

    Article  Google Scholar 

  • MacRoberts, M. H., & MacRoberts, B. R. (2010). Problems of citation analysis: A study of uncited and seldom-cited influences. Journal of the American Society for Information Science and Technology, 61(1), 1–12.

    Article  Google Scholar 

  • Mavrogenis, A. F., Ruggieri, P., & Papagelopoulos, P. J. (2010). Self-citation in publishing (editorial). Clinical Orthopaedics and Related Research, 468(10), 2803–2807.

    Article  PubMed  PubMed Central  Google Scholar 

  • McVeigh, M. E. (2004). Journal self-citation in the Journal Citation Reports®-Science Edition (2002): A Citation Study from The Thomson Corporation. Retrieved from http://wokinfo.com/essays/journal-self-citation-jcr/

  • Miguel, A., & Martí-Bonmatí, L. (2002). Self-citation: Comparison between Radiología, European radiology and radiology for 1997–1998. European Radiology, 12(1), 248–252.

    Article  PubMed  Google Scholar 

  • Moed, H. F. (2010). Measuring contextual citation impact of scientific journals. Journal of Informetrics, 4(3), 265–277.

    Article  Google Scholar 

  • Molinari, A., & Molinari, J.-F. (2008). Mathematical aspects of a new criterion for ranking scientific institutions based on the h-index. Scientometrics, 75(2), 339–356.

    Article  Google Scholar 

  • Pendlebury, D. A. (2009). The use and misuse of journal metrics and other citation indicators. Archivum immunologiae et therapiae experimentalis, 57(1), 1–11.

    Article  PubMed  Google Scholar 

  • PLoS Medicine E. (2006). The impact factor game. PLoS Med, 3(6), e291.

    Article  Google Scholar 

  • Ramin, S., & Shirazi, A. S. (2012). Comparison between Impact factor, SCImago journal rank indicator and Eigenfactor score of nuclear medicine journals. Nuclear Medicine Review, 15(2), 132–136.

    PubMed  Google Scholar 

  • Romanovsky, A. A. (2012). Revised h index for biomedical research. Cell Cycle, 11(22), 4118–4121.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Saha, S., Saint, S., & Christakis, D. A. (2003). Impact factor: A valid measure of journal quality? Journal of the Medical Library Association, 91(1), 42.

    PubMed  PubMed Central  Google Scholar 

  • Sahel, J.-A. (2011). Quality versus quantity: assessing individual research performance. Science translational medicine, 3(84), 84cm13–84cm13.

    Google Scholar 

  • Schiermeie, Q. (2008). Self-publishing editor set to retire. Nature, 456(432).

    Google Scholar 

  • Schreiber, M. (2007). Self-citation corrections for the Hirsch index. EPL (Europhysics Letters), 78(3), 30002.

    Article  Google Scholar 

  • Schreiber, M. (2008). A modification of the h-index: The hm-index accounts for multi-authored manuscripts. Journal of Informetrics, 2(3), 211–216.

    Article  Google Scholar 

  • Van Eck, N. J., Waltman, L., van Raan, A. F. J., Klautz, R. J. M., & Peul, W. C. (2013). Citation analysis may severely underestimate the impact of clinical research as compared to basic research. PloS One, 8(4), e62395. 

    Google Scholar 

  • van Leeuwen, T. (2012). Discussing some basic critique on journal impact factors: Revision of earlier comments. Scientometrics, 92(2), 443–455.

    Article  PubMed  PubMed Central  Google Scholar 

  • Van Leeuwen, T. N., Moed, H. F., & Reedijk, J. (1999). Critical comments on institute for scientific information impact factors: A sample of inorganic molecular chemistry journals. Journal of Information Science, 25(6), 489–498.

    Article  Google Scholar 

  • Vanclay, J. K. (2009). Bias in the journal impact factor. Scientometrics, 78(1), 3–12.

    Article  CAS  Google Scholar 

  • Vanclay, J. K. (2012). Impact factor: Outdated artefact or stepping-stone to journal certification? Scientometrics, 92(2), 211–238. doi:10.1007/s11192-011-0561-0.

    Article  Google Scholar 

  • Vinkler, P. (2007). Eminence of scientists in the light of the h-index and other scientometric indicators. Journal of Information Science, 33(4), 481–491.

    Article  Google Scholar 

  • Waltman, L., & van Eck, N. J. (2013). Source normalized indicators of citation impact: An overview of different approaches and an empirical comparison. Scientometrics, 96(3), 699–716.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Murray State University, Murray, Kentucky, USA

    Pali U. K. De Silva & Candace K. Vance

Authors
  1. Pali U. K. De Silva
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Candace K. Vance
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pali U. K. De Silva .

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

De Silva, P.U.K., K. Vance, C. (2017). Measuring the Impact of Scientific Research. In: Scientific Scholarly Communication. Fascinating Life Sciences. Springer, Cham. https://doi.org/10.1007/978-3-319-50627-2_7

Download citation

Publish with us

Policies and ethics

Search

Navigation