, Volume 1, Issue 4, pp 359–375 | Cite as

Is citation analysis a legitimate evaluation tool?

  • E. Garfield


A comprehensive discussion on the use of citation analysis to rate scientific performance and the controversy surrounding it. The general adverse criticism that citation counts include an excessive number of negative citations (citations to incorrect results worthy of attack), self-citations (citations to the works of the citing authors), and citations to methodological papers is analyzed. Included are a discussion of measurement problems such as counting citations for multiauthored papers, distinguishing between more than one person with the same last name (homographs), and what it is that citation analysis actually measures. It is concluded that as the scientific enterprise becomes larger and more complex, and its role in society more critical, it will become more difficult, expensive and necessary to evaluate and identify the largest contributors. When properly used, citation analysis can introduce a useful measure of objectivity into the evaluation process at relatively low financial cost.


Evaluation Process Financial Cost Evaluation Tool Citation Analysis Citation Count 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    A. A. JOHNSON, R. B. DAVIS. The Research Productivity of Academic Materials Scientists,Journal of Metals 27 (1975) No. 6, 28–29.Google Scholar
  2. 2.
    D. SHAPLEY. NSF: A ‘Populist’ Pattern in Metallurgy, Materials Research?Science, 189 (1975) No. 4203, 622–624.Google Scholar
  3. 3.
    C. A. WERT. The Citation Index Revisited,Journal of Metals 27 (1975) No. 12, 20–22.Google Scholar
  4. 4.
    T. GUSTAFSON, The Controversy Over Peer Review,Science, 190 (1975) No. 4219, 1060–1066.Google Scholar
  5. 5.
    D. SHAPLEY, Materials Research: Scientists Show Scant Taste for Breaking Ranks,Science, 191 (1976) No. 4222, 53.Google Scholar
  6. 6.
    R. ROY, Comments on Citation Study of Materials Science Departments,Journal of Metals, 28 (1976) No. 6, 29–30.Google Scholar
  7. 7.
    G. C. CROSBIE, R. W. HECKEL, Citation Criteria for Ranking Academic Departments,Journal of Metals, 28 (1976) No. 9, 27–28.Google Scholar
  8. 8.
    N. ARBITER, Letter to the Editor,Journal of Metals, 28 (1976) No. 12, 33.Google Scholar
  9. 9.
    J. C. AGARWAL, et al., Letter to the Editor,Journal of Metals, 28 (1976) No. 12, 33.Google Scholar
  10. 10.
    C. J. ALTSTETTER, Letter to the Editor,Journal of Metals, 28 (1976) No. 12, 33–35.Google Scholar
  11. 11a.
    N. WADE, Citation Analysis: A New Tool for Science Administrators,Science, 188 (1975) No. 4187, 429–432.Google Scholar
  12. 11b.
    Citation Analysis. Letters in response to Wade by M. Klerer, H. J. M. Hanley, J. Arditti, R. E. Machol,Science, 188 (1975) No. 4193, 1064. [N. WADE, Citation Analysis: A New Tool for Science Administrators,Science, 188 (1975) No. 4187, 429–432.]Google Scholar
  13. 12.
    G. P. KOSHY, The Citeability of a Scientific Paper.Proc. of Northeast Regional Conference of American Institute for Decision Sciences, Philadelphia, Pa., April/May 1976, 224–227.Google Scholar
  14. 13.
    E. GARFIELD, Is the Ratio Between Number of Citations and Publications Cited a True Constant?Essays of an Information Scientist, Vol. 2, Philadelphia: ISI Press, 1977, 419–421.Google Scholar
  15. 14.
    E. GARFIELD, Will ISI'sArts & Humanities Citation Index Revolutionize Scholarship?Current Contents, No. 32 (August 8, 1977) 5–9.Google Scholar
  16. 15.
    I. H. SHER, E. GARFIELD, New Tools for Improving and Evaluating the Effectiveness of Research, inResearch Program Effectiveness, M. C. YOUITS, D. M. GILFORD, R. H. WILCOX, E. STAVELY, H. D. LEMER, (Eds). New York; Gordon and Breach, 1966, p. 135–146.Google Scholar
  17. 16.
    J. P. MARTINO, Citation Indexing for Research and Development Management,IEEE Transactions on Engineering Management, EM-18 (1971) No. 4, 146–151.Google Scholar
  18. 17.
    A. E. BAYER, J. FOLGER, Some Correlates of a Citation Measure of Productivity in Science,Sociology of Education, 39 (1966) 381–390.Google Scholar
  19. 18.
    J. A. VIRGO, A Statistical Procedure for Evaluating the Importance of Scientific Papers,Library Quarterly 47 (1977) No. 4, 415–430.Google Scholar
  20. 19.
    E. GARFIELD, Citation Indexing for Studying Science,Nature, 227 (1970) No. 5259, 669–671.CrossRefGoogle Scholar
  21. 20.
    C. L. BERNIER, W. N. GILL, R. G. HUNT, Measures of Excellence of Engineering and Science Departments: A Chemical Engineering Example,Chemical Engineering Education, 9 (1975) 94–97.Google Scholar
  22. 21.
    G. M. CARTER, Peer Review, Citations, and Biomedical Research Policy: NIH Grants to Medical School Faculty,Rand Report, R-1583-HEW, Santa Monica, California; Rand Corporation, 1974.Google Scholar
  23. 22.
    A. J. MEADOWS,Communication in Science, London Butterworths, 1974, p. 45.Google Scholar
  24. 23.
    H. G. SMALL,Characteristics of Frequently Cited Papers in Chemistry, Final Report on NSF Contract #C795, 1974. See also E. GARFIELD, A list of 100 Most Cited (Chemical) Articles,Current Contents, No. 10 (March 9, 1977) 5–12.Google Scholar
  25. 24.
    O. H. LOWRY, N. J. ROSEBROUGH, A. L. FARR, R. J. RANDAL, Protein measurement with the Folin phenol reagent,Journal of Biological Chemistry, 193 (1951) 265–275.PubMedGoogle Scholar
  26. 25.
    O. H. LOWRY, Personal communication to D. de S. PRICE, November 11, 1969.Google Scholar
  27. 26a.
    H. G. SMALL, B. C. GRIFFITH, The Structure of Scientific Literatures, I: Identifying and Graphing Specialties,Science Studies, 4 (1974) 17–40.Google Scholar
  28. 26b.
    B. C. GRIFFITH, H. G. SMALL, J. A. STONEHILL, S. DEY, The Structure of Scientific Literatures, II: Towards a Macro- and Microstructure for Science,Science Studies 4 (1974) 339–365.Google Scholar
  29. 27.
    R. K. MERTON,On the Shoulders of Giants-A Shandean Postcript, New York, Harcourt Brace & World, 1965, p. 218–219.Google Scholar
  30. 28.
    J. LEDERBERG, A View of Genetics,Science, 131 (1960) 269–276.Google Scholar
  31. 29.
    E. GARFIELD, I. H. SHER, Citation Indexes in Sociological and Historical Research,American Documentation, 14 (1963) 289–291.Google Scholar
  32. 30.
    F. NARIN,Evaluative Bibliometrics: The USE of Publication and Citation Analysis in the Evaluation of Scientific Activity, Cherry Hill, N.J.; Computer Horizons, Inc., 1976, p. 500, NTIS-PB252339/AS.Google Scholar
  33. 31.
    J. R. COLE, S. COLE, The Ortega Hypothesis,Science, 178 (1975) 368–375.Google Scholar
  34. 32a.
    E. GARFIELD, Citation Analysis and the Anti-Vivisection Controversy,Current Contents, No. 17 (April 25, 1977) 5–10.Google Scholar
  35. 32b.
    E. GARFIELD, Citation Analysis and the Anti-Vivisection Controversy. Part II. An Assessment of Lester R. Aronson's Citation Record,Current Contents, No. 48 (November 28, 1977) 5–14.Google Scholar
  36. 33.
    E. GARFIELD, The 250 Most-Cited Primary Authors, 1961–1975. Part II. The Correlation Between Citedness, Nobel Prizes, and Academy Memberships,Current Contents, No. 50 (December 12, 1977) 5–15.Google Scholar
  37. 34.
    E. GARFIELD, The 250 Most-Cited Primary Authors, 1961–1975. Part III. Each Author's Most Cited Publication,Current Contents, No. 51 (December 19, 1977) 5–20.Google Scholar
  38. 35.
    E. GARFIELD, The 300 Most-Cited Authors, 1961–1976, Including Co-Authors at Last. Part I. How the Names Were Selected,Current Contents, No. 28 (July 10, 1978) 5–18.Google Scholar
  39. 36.
    J. R. COLE, S. COLE,Social Stratification in Science, Chicago; University of Chicago Press, 1973, p. 32–33.Google Scholar
  40. 37.
    D. LINDSEY, G. W. BROWN, Problems of Measurement in the Sociology of Science: Taking Account of Collaboration, (unpublished, 1977).Google Scholar
  41. 38.
    R. ROY, Approximating Total Citation Counts From First-Author Counts and Total Papers, Working Paper, April 1977.Google Scholar
  42. 39.
    D. de S. PRICE,Little Science, Big Science, New York, Columbia University Press, 1963.Google Scholar
  43. 40.
    D. de S. PRICE, D. B. BEAVER, Collaboration in an Invisible College,American Psychologist, 21 (1966) 1011–1018.Google Scholar
  44. 41a.
    N. L. GELLER, J. S. CANI, R. E. DAVIES, Lifetime-Citation Rates as a Basis for Comparisons Within a Scientific Field,Proc. of the American Statistical Association. Social Statistics Section, Washington, D. C., 1975, p. 429–433.Google Scholar
  45. 41b.
    N. L. GELLER, J. S. de CANI, R. E. DAVIES, Lifetime-Citation Rates to Compare Scientists' Work,Social Science Research, (in press).Google Scholar
  46. 42.
    E. GARFIELD, Caution Urged in the Use of Citation Analysis,Trends in Biochemical Sciences, 2 (1977) No. 4, N84.CrossRefGoogle Scholar
  47. 43.
    D. LINDSEY, G. W. BROWN, The Measurement of Quality in Social Studies of Science: Critical Reflections on the Use of Citation Counts, (unpublished, 1977).Google Scholar

Copyright information

© Akadémiai Kiadó 1979

Authors and Affiliations

  • E. Garfield
    • 1
  1. 1.Institute for Scientific Information® (ISI®)Philadelphia(USA)

Personalised recommendations