Scientometrics

, Volume 15, Issue 3–4, pp 189–203

Using citation counts as a measure of quality in science measuring what's measurable rather than what's valid

  • D. Lindsey
Article

Abstract

Empirical work in the social studies of science has progressed rapidly with the availability and development of the citation indexes. Citation counts have become a widely accepted measure of the quality of a scientific contribution. However, there are several problems involved in the use of citation counts as a measure of quality in science. First, citation counts are sensitive to popular trends in science. In this sense, they approximate a Nielsen rating for science. Second, the distribution of citations restricts their utility to separating the extremes. Third, citation counts are not sensitive to the ethical and moral dimensions of the quality of a scientific contribution. Fourth, citation counts underestimate the contribution of applied scientists. This paper examines these limitations.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    J. G. MANIS, Some academic influences upon publication productivity,Social Forces, 29 (1951) 267.Google Scholar
  2. 2.
    D. LINDSEY,The Scientific Publication System in Social Science, San Francisco, Jossey-Bass, 1978.Google Scholar
  3. 3.
    B. T. SHAW,The Use of Quality and Quantity of Publications as, Criteria for Evaluating Scientists, Agriculture Research Service, USDA Miscellaneous Publication No. 1041, Washington, D. C., U.S. Department of Agriculture, 1967.Google Scholar
  4. 4.
    N. D. GLENN, W. VILLEMEZ, The productivity of sociologists in 45 American universities.American Sociologist, 5 (1970) 244.Google Scholar
  5. 5.
    L. L. HARGENS,Patterns of Scientific Research: A Comparative Analysis of Research in Three Scientific Fields, Washington, D. C., The American Sociological Association, 1975, p. 87.Google Scholar
  6. 6.
    H. W. MENARD,Science: Growth and Change, New York, Harper and Row, 1971.Google Scholar
  7. 7.
    D. CRANE,Invisible Colleges: Diffusion of Knowledge in Scientific Communities, Chicago, University of Chicago Press, 1972.Google Scholar
  8. 8.
    N. C. MULLINS, The development of specialties in social science: the case of ethnomethodology,Science Studies, 3 (1973) 245.Google Scholar
  9. 9.
    N. C. MULLINS, New causal theory: An elite specialty in social sciences,History of Political Economy, 7 (1975) 499.Google Scholar
  10. 10.
    S. COLE, The growth of scientific knowledge: theories of deviance as a case study, In:The Idea of Social Structure, L. COSER (Ed.), New York, Harcourt, Brace, Jovanovich, 1976.Google Scholar
  11. 11.
    L. FREESE, Cumulative sociological knowledge,American Sociological Review, 37 (1972) 472.Google Scholar
  12. 12.
    T. PARSONS,Essays in Sociological Theory, (re. ed.) New York, Free Press, 1954.Google Scholar
  13. 13.
    G. J. STIGLER, The pattern of citation practices in economics. inThe Economist as Preacher and Other Essays, 173–191, Chicago, The University of Chicago Press, 1982.Google Scholar
  14. 14.
    D. LINDSEY, Assessing precision in the manuscript review process: a little better than a dice roll,Scientometrics, 14 (1988) 61.Google Scholar
  15. 15.
    H. F. MOED, W. J. M. BURGER, J. G. FRANKFORT, A. F. J. Van RAAN, The use of bibliometric data for measurement of university research performanceResearch Policy, 14 (1985) 131.Google Scholar
  16. 16.
    R. E. QUANDT, Some quantitative aspects of the economics journal literature.Journal of Political Economy, 84 (1976) 741.Google Scholar
  17. 17.
    H. A. ZUCKERMAN,The Scientific Elite, New York, Free Press, 1977.Google Scholar
  18. 18.
    E. GARFIELD,Citation Indexing-its Theory and Applications in Science, Technology, and Humanities, New York, Wiley, 1979.Google Scholar
  19. 19.
    A. J. DIAMOND, JR. What is a citation worth?The Journal of Human Resources, 21 (1986) 200.Google Scholar
  20. 20.
    J. COLE, S. COLE,Social Stratification in Science, Chicago, University of Chicago Press, 1973, p. 21.Google Scholar
  21. 21.
    M. KOCHEN, B. PERKEL, Improving referee-selection and manuscript evaluation, in:Proceedings of the First International Conference of Scientific Editors, J. McCARTNEY (Ed.), April 24–29, 1977, Jerusalem, Dordrecht: Reidel, 1977, p. 1.Google Scholar
  22. 22.
    E. GARFIELD, The most-cited life science articles highlight AIDS research,Current Contents, 49 (1986) 3.Google Scholar
  23. 23.
    M. J. MAHONEY, Open exchange and epistemic progress,American Psychologist, 40 (1985) 29.Google Scholar
  24. 14.
    E. GARFIELD, The most-cited 1983 chemistry articles focus on NMR but the increasing impact of materials science and computer methods is evident,Current Contents, 51-52 (1986) 3.Google Scholar
  25. 25.
    T. KUHN,The Structure of Scientific Revolutions, Chicago, University of Chicago Press, 1962.Google Scholar
  26. 26.
    G. HOLTON, Can science be measured? In:Toward a Metric of Science, Y. ELKANA and others (Eds), New York, Wiley-Interscience, 1978.Google Scholar
  27. 27.
    B. GRIFFITH, H. G. SMALL,A Philadelphia study of structure of science: The structure of social and behavioral science literature, Proceedings of the 1st International Conference on Social Studies of Science, Ithaca, New York November 4–6, 1976.Google Scholar
  28. 28.
    E. GARFIELD, The 250 most-cited primary authors 1961–1975. Part 3: Each authors's most-cited publication,Current Contents, 51 (1977) 5.Google Scholar
  29. 29.
    J. WIENER, The footnote fetish,Telos, 31 (1977) 172, p. 174.Google Scholar
  30. 30.
    H. M. BEVILLE, Jr.Audience Ratings: Radio, Television and Cable, Hillsdale, N. J., Lawrence Erlbaum Associates, Publisher, 1985.Google Scholar
  31. 31.
    A. C. NIELSEN,What Do the Ratings Really Say? Chicago, A. C. Nielsen, 1964.Google Scholar
  32. 32.
    L. MEYERS, On the reliability of the ratings,Television Quarterly, 1 (1962) 50.Google Scholar
  33. 33.
    M. H. MACROBERTS, B. R. MACROBERTS, Quantitative measures of communication in science: A study of the formal level,Social Studies of Science, 16 (1986) 151.Google Scholar
  34. 34.
    D. DIEKS, H. CHANG, Differences in impact of scientific publications: Some indices derived from a citation analysis.Social Studies of Science, 6 (1976) 247.Google Scholar
  35. 35.
    M. J. MORAVCSIK, P. MURUGESAN, Some results on the function and quality of citations,Social Studies of Science, 5 (1975) 86.Google Scholar
  36. 36.
    M. J. MULKAY, Norms and ideology in science,Social Science Information, 15 (1976) 637.Google Scholar
  37. 37.
    R. A. ROTHMAN, A dissenting view on the scientific ethos,British Journal of Sociology, 23 (1972) 102.Google Scholar
  38. 38.
    J. LAW, Theories and methods in the sociology of science: an interpretive approach,Social Science Information 13 (1974) 163.Google Scholar
  39. 39.
    M. JAY,The Dialectical Imagination: A History of the Frankfurt School and the Institute of Social Research, 1923–50, Boston, Little, Brown, 1973.Google Scholar
  40. 40.
    F. CUNNINGHAM,Objectivity in Social Science, Toronto, University of Toronto Press, 1973.Google Scholar
  41. 41.
    R. POOLE,Toward Deep Subjectivity, Middlesex, England, Penguin Books, 1972., p. 12.Google Scholar
  42. 43.
    R. S. LECKER, Whose side are we on?Social Problems, 14 (1967) 239.Google Scholar
  43. 43.
    R. S. LYNN,Knowledge for What? Boston, Grove Press, 1938.Google Scholar
  44. 44.
    M. POLANYI,Personal Knowledge, Chicago, University of Chicago Press, 1958.Google Scholar
  45. 45.
    E. GARFIELD, The 250 most-cited autors in the Arts and Humanities Citation Index, 1976–1983,Current Contents 48 (1986) 3.Google Scholar
  46. 46.
    M. J. MAHONEY, Publication prejudices: An experimental study of confirmatory bias in the peer review system,Cognitive Therapy and Research, 1 (1977) 161.Google Scholar
  47. 47.
    T. S. KUHN,The Structure of Scientific Revolutions, Chicago, University of Chicago Press, 1962.Google Scholar
  48. 48.
    B. F. RESKIN, Sex differences in status attainment in science: the case of the postdoctoral fellowship,American Sociological Review, 42 (1977) 491.Google Scholar
  49. 49.
    M. H. MACROBERTS, B. R. MACROBERTS, Testing the Ortega Hypothesis: Facts and artifacts,Scientometrics, 12 (1987) 293. Also see the comments and discussions of others regarding this paper in this same issue.Google Scholar
  50. 50.
    A. V. CICOUREL,Method and Measurement in Sociology, New York, Free Press, 1964.Google Scholar

Copyright information

© Akadémiai Kiadó 1989

Authors and Affiliations

  • D. Lindsey
    • 1
  1. 1.College of Human Development and PerformanceUniversity of Oregon(USA)

Personalised recommendations