Limitations of the databases that are used in citation analyses hinder and complicate evaluations of the citation success of different scientific publication types (e.g., books, chapters, journal articles, test manuals, and therapy manuals) and publication genres (e.g., theoretical, methodological, or empirical studies, literature reviews/overview, textbooks, monographs, etc.). This can result in serious unfairness in scientometric evaluations of individual scientists, research teams, departments, institutions, and nations as well, because some of their scientific and scholarly work is underrepresented or—even worse—neglected.
This problem applies—for example—to textbooks and teaching materials published in book chapters, because these publications appertain to the scholarly performance of scientists, which report and integrate primary research results and theory building in (more or less) creative and—more important—didactically clear, structured form, which is motivating for students in education and further training. At least in medicine and psychology, this applies also to test manuals, therapy and prevention manuals as well as diagnostic and therapeutic guidelines, because of their outstanding significance not only for education and training, but also for applied diagnostic and therapeutic work in medical and psychotherapeutic treatments. Thus, these scientific publication types and publication genres must be considered in evaluations of scientists and science as well as primary research reports. Besides other criteria and indicator variables of scientific productivity and performance (e.g., research funding, invited talks at international congresses, innovations, and letter patents), this belongs to the criteria of scientific and scholarly publication output which constitute multiple bibliometric indicators and—if cited—scientometric indicators of productivity and its impact in the sciences (see, e.g., Glänzel and Moed 2013; Moed and Halevi 2014).
The problem is that the selectivity of international and interdisciplinary databases (e.g., Web of Science and Scopus) that are used in citation analyses hinders evaluations of the citation success of different scientific publication types (e.g., books, chapters, journal articles, test manuals, and therapy manuals) and publication genres (e.g., theoretical, methodological, or empirical studies, literature reviews/overview, textbooks, monographs, etc.), and this is a situation which can result in serious unfairness in scientometric evaluations. For example, the Web of Science (WoS) and Scopus, surely the most frequently used citation databases, pushes journal articles (mainly English-language papers from the Anglo-American research community; see, e.g., Albarrán et al. 2010; González-Alcaide et al. 2012; Krampen 2009) and ignore widely—with very few and really only English-language exceptions—books, book chapters, test manuals, and intervention (therapy) manuals in its documentation of publications and of reference lists (see, e.g., Chi 2014; Ossenblok et al. 2013; van Leeuwen 2013). More than that, comparative scientometric results document significant differences in the coverage, e.g., of WoS and Scopus (Alves et al. 2014) and others as well (see, e.g., Cavacini 2015, for computer sciences).
At the other extreme, Google Scholar checks the “complete” Internet for citations of publication titles (or author names) including all and perhaps nothing, because the references identified refer not only to citations in scientific publications, but to these at homepages of individuals, institutions, and organizations, in excerpts, homework of high school and college students, examination schedules and exam papers, self-presentations, advertising and advertisements, etc. without any type of quality control. Therefore, citation success according to Google Scholar can be manipulated very easily and, therefore, can be considered poor and not very useful for scientometrics (see, e.g., Aguillo 2012; Beel and Gipp 2010; Mayr and Walter 2007; Mingers and Lipitakis 2010; for a recent discussion of the weaknesses of Google Scholar and its comparison with the WoS see, e.g., de Winter et al. 2014). In addition, there are huge problems with homonyms and identical or similar titles of publications. Somewhat in between of the high selective, restrictive WoS and the not selective, but complete open Google Scholar without any professional specialist quality control and scientific quality management are databases like Ovid, which are dominated by publishers and/or database providers because they tend to selectively include primarily their “own” products (see, e.g., Larsen and von Ins 2010; Mingers and Lipitakis 2010). Thus, the argument is that these are selective, or bad quality or publisher controlled databases but yet no complete, open, quality databases.
Therefore, for more profound and serious citation analyses we need databases that cover all types and genres of scientific and scholarly publications including their complete reference lists. Because of a better manageable population of scientists and their publications it is more feasible to work first of all with local and disciplinary databases, which may be merged or combined to international multidimensional databases subsequently. Citation analyses, which refer to such exhaustive databases, can show what proportions of publication output are neglected in citation analyses, which refer to the multidimensional, international databases (as the WoS, Scopus, etc.; see, e.g., Hicks 2005, on publication types; Chi 2014, on publication languages). Furthermore, it can be expected that the results of such citation analyses will have better acceptance not only by the evaluators but also and especially by the evaluated scientists, that is, by the scientific community.
Since the publication year 2009, PSYNDEX, the database of psychology publications from Austria, Germany, Liechtenstein, and the German-speaking regions of Luxemburg and Switzerland published either in German or in English language, includes the complete reference lists of publications documented. Thus, PSYNDEX is a local disciplinary database for psychology and its neighbouring disciplines. Access to PSYNDEX is provided, e.g., by the platforms Ovid and Ebsco as well as www.zpid.de, open access is possible for individual users in PubPsych (http://www.zpid.de/index.php?wahl=products&uwahl=pubpsych).
PSYNDEX is a domain-specific literature database which covers the scientific literature of a discipline better than citation databases such as the multidisciplinary WoS (coverage ≤20 % of the PSYNDEX-documents in the WoS; see, e.g., Krampen 2009), for which Larsen and von Ins (2010) report in addition a general decline in coverage of scientific publications. A special feature of PSYNDEX is its completeness for the psychology publication output of its defined area, that is, the German-speaking countries of the world, which have sometimes been called the “D–A–CH–L–L” countries (D = Germany, A = Austria, CH = Switzerland, L = Luxembourg, L = Liechtenstein). Thus, the following scientometric analyses refer to all the references that are cited in the psychology publications documented in PSYNDEX for the publication years 2009, 2010, and 2011. Analyses are conducted separately for the references cited in the publications documented in PSYNDEX with the publication year (1) 2009, (2) 2010, and (3) 2011. This has the advantage that cross-sectional time comparisons are possible thus allowing conclusions to be made on the stability of the results.
Main objective of our scientometric study is to analyse the frequencies by which different publication types and publication genres are cited. Because this question can not be answered up to now by the use of multidisciplinary international databases (focus at journals, neglect or even ignorance of book chapters, books, test manuals, etc.), we conduct a case study with the local disciplinary databases PSYNDEX. The first research question refers to (1) the frequencies of different publication types (i.e., journal articles, books, book chapters) to which reference is given by citations in the publications of the years 2009, 2010, and 2011. The second research question refers to (2) the replication of the first analysis; however, data refer only to the cited references in publications from the German-speaking countries that are also documented in PSYNDEX (11 % of all cited references) and whose PSYNDEX documentation contains meta-data about the publication genre and on the psychological sub-discipline to which they belong. The third research question refers to (3) the characteristics of the most frequently cited references in the psychology publications from the German speaking countries published in 2009, 2010, and 2011. These top-cited references are analysed with the help of PSYNDEX meta-data for the relative frequencies of (3a) different document types, i.e., journal articles, books, chapters, (3b) different scientific publication genres, i.e., empirical and non-empirical publications, and (3c) different psychological sub-disciplines following the classification codes (CC) of the Thesaurus of Psychological Index Terms of the American Psychological Association (APA; Gallagher Tuleya 2007), i.e., methodology, basic psychology, and applied psychology, etc. In addition, the citation success over time of the top-cited publications (see, e.g., Johnston et al. 2013) is briefly described.