, Volume 71, Issue 2, pp 191–202 | Cite as

On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications

  • Günter KrampenEmail author
  • Ralf Becker
  • Ute Wahner
  • Leo Montada


In reference to the increasing significance of citation counting in evaluations of scientists and science institutes as well as in science historiography, it is analyzed empirically what is cited in which frequency and what types of citations in scientific texts are used. Content analyses refer to numbers of references, self-references, publication language of references cited, publication types of references cited, and type of citation within the texts. Validity of citation counting is empirically analyzed with reference to random samples of English and German journal articles as well as German textbooks, encyclopedias, and test-manuals from psychology. Results show that 25% of all citations are perfunctory, more than 50% of references are journal articles and up to 40% are books and book-chapters, 10% are self-references. Differences between publications from various psychological sub-disciplines, publication languages, and types of publication are weak. Thus, validity of evaluative citation counting is limited because at least one quarter refers to perfunctory citations exhibiting a very low information utility level and by the fact that existing citation-databases refer to journal articles only.


Journal Article Citation Analysis Citation Counting Publication Language Publication Type 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Aristotle (undated/1959), Über die Seele (De anima) [= E. Gruber (Hrsg.), Werke Aristoteles, Bd. 13], Wissenschaftliche Buchgesellschaft, Darmstadt.Google Scholar
  2. Bagby, R. M., Parker, J. D., Bury, A. S. (1990), A comparative citation analysis of attribution theory and the theory of cognitive dissonance, Personality and Social Psychology Bulletin, 16: 274–283.CrossRefGoogle Scholar
  3. Bonzi, S., Snyder, H. W. (1991), Motivations for citation: A comparison of self citation and citation of others, Scientometrics, 21: 245–254.CrossRefGoogle Scholar
  4. Brooks, T. A. (1986), Evidence of complex citer motivations, Journal of the American Society for Information Science, 37: 34–36.CrossRefGoogle Scholar
  5. Brooks, T. A. (1988), Citer motivations. In: A. Kent (Ed.), Encyclopedia of Library and Information Science, Volume 43. Marcel Dekker, New York, NY, pp. 48–59.Google Scholar
  6. Brožek, J. (1980), The echoes of Wundt’s work in the United States, 1887–1977: A quantitative citation analysis, Psychological Research, 42: 103–107.CrossRefGoogle Scholar
  7. Chubin, D. E., Moitra, S. D. (1975), Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science, 5: 423–441.CrossRefGoogle Scholar
  8. Cole, J., Cole, S. (1971), Measuring the quality of sociological research: Problems in the use of the Science Citation Index, American Sociologist, 6: 23–29.Google Scholar
  9. Cronin, B. (1984), The Citation Process: The Role and Significance of Citations in Scientific Communication, Taylor Graham, London, Great Britain.Google Scholar
  10. Endler, N. S., Rushton, J. P., Roediger, H. L. (1978), Productivity and scholary impact (citations) of British, Canadian, and U.S. Departments of Psychology (1975), American Psychologist, 33: 1064–1082.CrossRefGoogle Scholar
  11. Garfield, E. (1979), Citation Indexing: Its Theory and Application in Science, Technology, and Humanities, Wiley, New York, NY.Google Scholar
  12. Gray, P. H. (1983), Using science citation analysis to evaluate administrative accountability for salary variance, American Psychologist, 38: 116–117.CrossRefGoogle Scholar
  13. Liu, M. (1993), Progress in documentation — The complexities of citation practice: A review of citation studies, Journal of Documentation, 49: 370–408.Google Scholar
  14. May, R. M. (1997), The scientific wealth of nations, Science, 275: 793–796.CrossRefGoogle Scholar
  15. McPherson, A., Bonem, M., Green, G., Osborne, J. G. (1984), A citation analysis of the influence on research of Skinner’s Verbal Behavior, Behavior Analyst, 7: 157–167.Google Scholar
  16. Moravsik, M. J., Murugesan, P. (1975), Some results on the function and quality of citations, Social Studies of Science, 5: 86–92.Google Scholar
  17. Peritz, B. (1983), A classification of citation roles for the social sciences and related fields, Scientometrics, 5: 303–312.CrossRefGoogle Scholar
  18. Perlman, D. (1984), Recent developments in personality and social psychology: A citation analysis, Personality and Social Psychology Bulletin, 10: 493–501.CrossRefGoogle Scholar
  19. Schui, G., Krampen, G. (2006), Möglichkeiten und Grenzen der Anwendung bibliometrischer Indikatoren im Rahmen von Evaluationsvorhaben in der deutschsprachigen Psychologie (Possibilities and limits of applications of bibliometrical indicators within evaluations of psychology). In: G. Krampen, H. Zayer (Eds), Didaktik und Evaluation in der Psychologie (Teaching methods and evaluation in psychology). Hogrefe, Göttingen (Germany), pp. 11–26.Google Scholar

Text material of authored and edited textbooks and encyclopaedias

  1. Amelang, M., Bartussek, D. (2001), Differentielle Psychologie und Persönlichkeitsforschung (Differential psychology and personality research) (5th ed.), Kohlhammer, Stuttgart (Germany).Google Scholar
  2. Baumann, U., Perrez, M. (Eds) (1998), Lehrbuch Klinische Psychologie — Psychotherapie (Textbook clinical psychology — psychotherapy) (2nd ed.), Huber, Bern (Switzerland).Google Scholar
  3. Birbaumer, N., Schmidt, R. (2003), Biologische Psychologie (Biological psychology) (5th ed.), Springer, Berlin (Germany).Google Scholar
  4. Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1994–1997). Enzyklopädie der Psychologie: Pädagogische Psychologie (Encyclopedia of psychology: Educational psychology) (Serie D/I, Bd. 1–4), Hogrefe, Göttingen (Germany).Google Scholar
  5. Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2000), Enzyklopädie der Psychologie: Differentielle Psychologie und Persönlichkeitsforschung (Encyclopedia of Psychology: Differential psychology and personality research) (Serie C/VIII, Bd. 1–4), Hogrefe, Göttingen (Germany).Google Scholar
  6. Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2001), Enzyklopädie der Psychologie: Klinische Psychologie (Encyclopedia of psychology: Clinical psychology) (Serie D/II, Bd. 1–4), Hogrefe, Göttingen (Germany).Google Scholar
  7. Birbaumer, N., Frey, D., Kuhl, J., Schneider, W., Schwarzer, R. (Eds) (1996–2002), Enzyklopäde der Psychologie: Biologische Psychologie (Encyclopedia of psychology: Biological psychology) (Serie C/I, Bd. 1–7), Hogrefe, Göttingen (Germany).Google Scholar
  8. Davison, G., Neale, J., Hautzinger, M. (2002), Klinische Psychologie (Clinical psychology) (6th ed.), Beltz, Weinheim (Germany).Google Scholar
  9. Ehlert, U. (Ed.) (2003), Verhaltensmedizin (Behavioral medicine), Springer, Berlin (Germany).Google Scholar
  10. Frey, D., Irle, M. (Eds) (1998/2002), Theorien der Sozialpsychologie (Theories of social psychology) (3 Vol.), Huber, Bern (Switzerland).Google Scholar
  11. Graumann, C. G. (Ed.) (1972/1975), Sozialpsychologie (Social psychology) [= Gottschaldt, K., Lersch, P., Sander, F., Thomae, H. (Eds) Handbuch der Psychologie in 12 Bänden, Bd. 7(1) und Bd. 7(2)], Hogrefe, Göttingen (Germany).Google Scholar
  12. Herkner, W. (2001), Lehrbuch Sozialpsychologie (Textbook social psychology) (2nd ed.), Huber, Bern (Switzerland).Google Scholar
  13. Herrmann, T., Lantermann, E.-D. (Eds) (1985), Persönlichkeitspsychologie (Psychology of personality), Urban & Schwarzenberg), München (Germany).Google Scholar
  14. Krapp, A., Weidenmann, B. (Eds) (2001), Pädagogische Psychologie (Educational psychology) (4th ed.), Beltz, Weinheim (Germany).Google Scholar
  15. Oerter, R., Montada, L. (Eds) (2002), Entwicklungspsychologie (Developmental psychology) (5th ed.), Beltz, Weinheim (Germany).Google Scholar
  16. Tausch, R., Tausch, A. (1998), Erziehungspsychologie (Psychology of education) (11th ed.), Hogrefe, Göttingen (Germany).Google Scholar
  17. Thomae, H. (Ed.) (1959), Entwicklungspsychologie (Developmental psychology) [= Gottschaldt, K., Lersch, P., Sander, F., Thomae, H. (eds) Handbuch der Psychologie in 12 Bänden, Bd. 3], Hogrefe, Göttingen (Germany).Google Scholar
  18. Trautner, H. M. (1992), Lehrbuch der Entwicklungspsychologie (Textbook developmental psychology) (2 Vol., 2nd ed.), Hogrefe, Göttingen (Germany).Google Scholar

Text material of test-manuals

  1. Borkenau, P., Ostendorf, F. (1993), Neo-Fünf-Faktoren-Inventar (Neo-Five-Factors-Inventory) NEO-FFI), Hogrefe, Göttingen (Germany).Google Scholar
  2. Fahrenberg, J., Hampel, R., Selg, H. (2001), Freiburger Persönlichkeitsinventar (Freiburg Personality-Inventory) (FPI-R; 7th ed.), Hogrefe, Göttingen (Germany).Google Scholar
  3. Franke, G. H. (2002), Die Symptom-Checkliste von L.R. Derogatis (Symptom-Checklist) (SCL-90-R; 2nd ed.), Beltz, Weinheim (Germany).Google Scholar
  4. Grimm, H., Schöler, H. (1991), Heidelberger Sprachentwicklungstest (Heidelberg language development test) (HSET; 2nd ed.), Hogrefe, Göttingen (Germany).Google Scholar
  5. Kastner-Koller, V., Deimann, P. (1998), Der Wiener Entwicklungstest (Vienna development test) (WET), Hogrefe, Göttingen (Germany).Google Scholar
  6. Krohne, H.-W., Pulsack, A. (1995), Erziehungsstilinventar (Education-style inventory) (ESI; 2nd ed.), Beltz, Weinheim (Germany).Google Scholar
  7. Müller, R. (1980), Diagnostisches Soziogramm (Diagnostic sociogram), Westermann, Braunschweig (Germany).Google Scholar
  8. Rosenzweig, S., Hörmann, H., Moog, W. (1957), Rosenzweig P-F Test, Form für Erwachsene (Rosenzweig P-F Test, Adults) (PFT), Hogrefe, Göttingen (Germany).Google Scholar
  9. Tewes, U., Rossmann, P., Schallberger, U. (2000), Hamburg-Wechsler-Intelligenztest für Kinder (Hamburg-Wechsler-Intelligence-Test for Children) (HAWIK-III; 3rd ed.), Huber, Bern (Switzerland).Google Scholar
  10. Wittchen, H.-U., Zaudig, M., Fydrich, T. (1997), Strukturiertes Klinisches Interview für DSM IV — Achsen I und II (Structured clinical interview for DSM-IV, Axis I and II) (SKID), Hogrefe, Göttingen (Germany).Google Scholar

Copyright information

© Akadémiai Kiadó 2007

Authors and Affiliations

  • Günter Krampen
    • 1
    Email author
  • Ralf Becker
    • 1
  • Ute Wahner
    • 1
  • Leo Montada
    • 1
  1. 1.Department of Psychology and Institute for Psychology Information (ZPID)University of TrierTrierGermany

Personalised recommendations