Skip to main content

Journal-to-journal citation data: Issues of validity and reliability

Abstract

Citation analysis is a useful method for studying a wide range of topics in bibliometrics and the sociology of science. However, many challenges have been made to the validity and reliability of the underlying assumptions, the data, and the methods used in citation studies. This article addresses these issues in three parts. First is a brief review of validity and reliability issues in citation research. Next we explore measurement error in a principal source of journal-to-journal citation data, the Institute for Scientific Information'sJournal Citation Reports. Possible sources of measurement error include discrepancies between citing and cited data, changed or deleted journal titles, aberrant abbreviations, and listing algorithms. The last section is a detailed description of ways to overcome some of the measurement errors. The data and examples are drawn from a journal-to-journal citation study in the fields of Communication, Information Science, and Library Science.

This is a preview of subscription content, access via your institution.

References

  1. J. BENIGER (1988). Information and communication: The new convergence.Communication Research, 15 (2), 198–218.

    Google Scholar 

  2. R. W. BOHANNON (1986). Letter to the editor.Physical Therapy, 66(9), 1431–1432.

    Google Scholar 

  3. C. L. BORGMAN & J. SCHEMENT (1988). Information science and communication science: An essay on convergence. In: PRENTICE, A. (Ed.),Information science as a discipline, (in press) NY: Schuman.

    Google Scholar 

  4. B. BOYCE & C. BANNING (1979). Data accuracy in citation studies. RQ, 18, 349–150.

    Google Scholar 

  5. R. BROADUS (1987). Early approaches to bibliometrics.Journal of the American Society for Information Science, 38(2), 127–129.

    Google Scholar 

  6. L. D. BROWN & J. C. GARDNER (1985). Using citation analysis to assess the impact of journals and articles on contemporary accounting research.Journal of Accounting Research, 23(1), 84–109.

    Google Scholar 

  7. R. S. BURT (1978). Stratification and prestige among elite experts in methodological and mathematical sociology circa 1975.Social Networks, 1, 105–158.

    Google Scholar 

  8. R. S. BURT & P. DOREIAN (1982). Testing a structural model of perception: Conformity and deviance with respect to journal norms in elite sociological methodology.Quality and Quantity, 16, 109–150.

    Google Scholar 

  9. A. R. BUSS & J. R. McDERMOTT (1976). Ratings of psychology journals compared to objective measures of journal impact.American Psychologist, 31(9), 675–678.

    Google Scholar 

  10. M. CARPENTER & F. NARIN (1973). Clustering of scientific journals.Journal of the American Society for Information Science, 23, 425–436.

    Google Scholar 

  11. H. CASON & M. LUBOTSKY (1936). The influence and dependence of psychological journals on each other.Psychological Bulletin, 33(2), 95–103.

    Google Scholar 

  12. A. CAWKELL (1978). Evaluating scientific journals with Journal Citation Reports: A case study in acoustics.Journal of American Society for Information Science, 29, 41–46.

    Google Scholar 

  13. D. CHUBIN (1983).Sociology of sciences: An annotated bibliography on invisible colleges, 1972–1981, NY: Garland Publishing.

    Google Scholar 

  14. J. COLE & S. COLE (1971). Measuring the quality of sociological research: Problems in the use of the Science citation index.American Sociologist, 6, 23–29.

    Google Scholar 

  15. D. CRANE (1972).Invisible colleges: Diffusion of knowledge in scientific communities. Chicago: University of Chicago Press.

    Google Scholar 

  16. D. CRANE (1974). The gatekeepers of science: Some factors affecting the selection of articles for scientific journals.The American Sociologist, 2, 195–201.

    Google Scholar 

  17. L. CUMMINGS & P. FROST (Eds), (1985).Publishing in the organizational sciences. Homewood, ILL: Irwin.

    Google Scholar 

  18. J. DICKMAN (1984). How do secondary services select journals?Cleveland Clinic Quarterly, 51, 575–578.

    Google Scholar 

  19. J. DICKMAN & G. PLATEAU (1983). Foreign literature challenges to Chemical Abstracts Service.Journal of Chemical Information and Computer Science, 23(2), 70–73.

    Google Scholar 

  20. P. DOREIAN (1985). Structural equivalence in a psychology journal network.Journal of the American Society for Information Science, November, 411–417.

  21. P. DOREIAN (1985). A measure of standing of journals in stratified networks.Journal of the American Society for Information Science, 8(5/6), 341–363.

    Google Scholar 

  22. P. DOREIAN & T. FARARO (1985). Structural equivalence in a journal network.Journal of the American Society for Information Science, 36(1), 28–37.

    Google Scholar 

  23. D. EDGE (1979). Quantitative measures of communication in science: A critical review.History of Science, 17, 102–133.

    Google Scholar 

  24. E. GARFIELD (1979).Citation indexing: Its theory and application in science, technology and humanities. NY: Wiley.

    Google Scholar 

  25. E. GARFIELD (1984). The 100 most-cited papers ever and how we select citation classics.Current Contents, 23, 3–9.

    Google Scholar 

  26. W. GARVEY (1979).Communication: The essence of science. Facilitating information exchange among librarians, scientists, engineers, and students. NY: Pergamon Press.

    Google Scholar 

  27. J. GOODRICH & C. ROLAND (1977). Accuracy of published medical reference citations.Journal of Technical Writing and Communication, 7, 15–19.

    Google Scholar 

  28. M. GORDON (1982). Citation ranking versus subjective evaluation in the determination of journal hierarchies in the social sciences.Journal of the American Society for Information Science, 33, 55–57.

    Google Scholar 

  29. W. HAGSTROM (1965).The scientific community. NY: Basic Books.

    Google Scholar 

  30. P. J. HART (1987).The use and role of commercial online databases in four organizations. Unpublished PhD dissertation. Los Angeles: University of Southern California, Annenberg School of Communications.

  31. G. HIRST (1978). Discipline impact factors: A method for determining core journal lists.Journal of the American Society for Information Science, 29, 171–172.

    Google Scholar 

  32. R. HJERPPE (1980).A bibliography of bibliometrics and citationindexing and analysis. (TRITA-LIB-2013). Stockholm: Royal Institute of Technology Library.

    Google Scholar 

  33. Institute for Scientific Information. (1985).Journal Citation Reports. Philadelphia: Institute for Scientific Information.

    Google Scholar 

  34. D. LINDSEY (1978).The scientific publication system in social science. San Francisco: Josey-Bass.

    Google Scholar 

  35. M. LINE (1985). Changes in rank lists of serials over time: Interluding versus citation data.College & Research Libraries, 46(1), 77–79.

    Google Scholar 

  36. S. LOCK (1986).A difficult balance: Editorial peer review in medicine. Philadelphia, PA: Institute for Scientific Information.

    Google Scholar 

  37. T. LUCE & D. JOHNSON (1978). Rating of education and psychological journals.Educational Researcher, 7, 8–10.

    Google Scholar 

  38. K. MACE & H. WARNER (1973). Ratings of psychology journals.American Psychologist, 28(2), 184–186.

    Google Scholar 

  39. P. McALLISTER; R. ANDERSON & F. NARIN (1980). Comparison of peer and citation assessment of the influence of scientific journals.Journal of the American Society for Information Science, 31, 147–152.

    Google Scholar 

  40. N. MIDORIKAWA (1983). Citation analysis of physics journals: Comparison of subfield of physics.Scientometrics, 5(6), 361–374.

    Google Scholar 

  41. F. NARIN; M. CARPENTER & N. BERLT (1972). Inter-relationships of scientific journals.Journal of the American Society for Information Science, 23, 323–331.

    Google Scholar 

  42. F. NARIN & D. GARSIDE (1972). Journal relationships in special education.Exceptional Children, 38(9), 695–703.

    Google Scholar 

  43. W. NORD (1985). Looking at ourselves as we look at others: An exploration of the publication system for organizational research. In: L. CUMMINGS & P. FROST (Eds),Publishing in the organizational sciences. Homewood, ILL: Irwin.

    Google Scholar 

  44. W. PAISLEY (1984). Communication in the communication sciences. In: B. DERVIN & M. VOIGT (Eds),Progress in the communication sciences, Vol. 5 (pp. 1–43.) Norwood, NJ: Ablex.

    Google Scholar 

  45. D. PETERS & S. CECI (1982). Peer review practices of psychological journals: The fate of published articles, submitted again.Behavioral and Brain Science, 2(5), 187–195. (Cited inLock, 1985).

    Google Scholar 

  46. A. PORTER (1977). Citation analysis: Queries and caveats.Social Studies of Science, 7, 257–267.

    Google Scholar 

  47. W. POWELL (1985).Getting into print: The decision-making process in scholarly publishing. Chicago: University of Chicago Press.

    Google Scholar 

  48. B. REEVES & C. L. BORGMAN (1983). A bibliometric evaluation of core journals in communication research.Human Communication Research, 10(1), 119–136.

    Google Scholar 

  49. R. E. RICE; C. L. BORGMAN & B. REEVES (1988). Citation networks of communication journals, 1977–1985.Human Communicatiin Research, 15(2), 256–283.

    Google Scholar 

  50. J. RUSHTON (1984). The validity of citation counts.Bulletin of the British Psychological Society, 37, 33–36.

    Google Scholar 

  51. W. SATARIANO (1978). Journal use in sociology: Citation analysis versus readership patterns.Library Quarterly, 48(3), 293–300.

    Google Scholar 

  52. A. SINGLETON (1976). Journal ranking and selection: A review in physics.Journal of Documentation, 32(4), 258–289.

    Google Scholar 

  53. W. SCOTT (1974). Interreferee agreement on some characteristics of manuscripts submitted to theJournal of Personality and Social Psychology.American Psychologist, 29, 698–702. (Cited in Bohannon, 1986).

    Google Scholar 

  54. H. SMALL (1981). The relationship of information science to the social sciences: A co-citation analysis.Information Processing and Management, 17, 39–50.

    Google Scholar 

  55. J. SMART (1983). Perceived quality and citation rates of education journals.Research in Higher Education, 19(2), 175–182.

    Google Scholar 

  56. L. SMITH (1981). Citation analysis.Library Trends, Summer, 83–106.

  57. N. SMITH & D. CAULLEY (1981). The evaluation of education journals through the study of citations.Educational Researcher, 10, 11–12.

    Google Scholar 

  58. C. SO (1988). Citation networks of communication journals.Human Communication Research, 15(2), 236–255.

    Google Scholar 

  59. SSCI Journal citation reports; a bibliometric analysis of social sciences journals in the ISI data base. Social Sciences Citation Index 1981 annual, vol. 7. (1981, 1984, 1985, 1986). Philadelphia: Institute for Scientific Information.

  60. F. THORNE (1977). The Citation Index: Another case of spurious validity.Journal of Clinical Psychology, October, 33, 1157–1161.

    Google Scholar 

  61. Ulrich's International Periodicals Guide; A classified Guide to a Selected List of Current Periodicals, Foreign and Domestic. 25th ed, 1986–87. (1986). NY: R. R. Bowker.

    Google Scholar 

  62. H. WHITE & B. GRIFFITH (1982). Authors as markers of intellectual space: Co-citation on studies of science, technology and society.Journal of Documentation, 38(4), 255–272.

    Google Scholar 

  63. M. WHITE & K. WHITE (1977). Citation analysis of psychology journals.American Psychologist, 32, 301–305.

    Google Scholar 

  64. N. WHITMAN & B. EYRE (1985). The pattern of publishing previously rejected articles in selected journals.Family Medicine, 17(1), 26–28.

    Google Scholar 

  65. S. Jr. WIBERLEY (1982). Journal rankings from citation studies: A comparison of national and local data from social work.Library Quarterly, 52(4), 348–359.

    Google Scholar 

  66. M. E. WILLIAMS (Ed.), (1985).Computer-Readable Databases: A Directory and Sourcebook, Vols 1–2. Chicago: American Library Association.

    Google Scholar 

  67. L. ZHIGNESSE & C. OSGOOD (1967). Bibliographic citation characteristics of the psychological journal network in 1950 and 1960.American Psychologist, 22, 778–791.

    Google Scholar 

  68. J. ZIMAN (1968).Public knowledge. Cambridge, England: Cambridge University Press.

    Google Scholar 

  69. H. ZUCKERMAN & R. MERTON (1971). Patterns of evaluation in science: Institutionalization, structure, and functions of the referee system.Minerva, 9, 66–100. (Cited in BOHANNON 1986.)

    Google Scholar 

Download references

Author information

Affiliations

Authors

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Rice, R.E., Borgman, C.L., Bednarski, D. et al. Journal-to-journal citation data: Issues of validity and reliability. Scientometrics 15, 257–282 (1989). https://doi.org/10.1007/BF02017202

Download citation

Keywords

  • Measurement Error
  • Scientific Information
  • Citation Analysis
  • Principal Source
  • Citation Data