PART A: questionable and Web-of-Science-listed journals
A.1. Share of cited papers from questionable journals
Between 2012 and 2018, 65 analysed journals published 25,146 papers, of which 3234 (13%) were cited by WoS journals.
Figure 3 shows the highest share, 19%, in 2012 and 2013. The mean number of papers published by a single journal in the analysed period was 53.5 (min = 43, max = 2176). On average, 11% of papers published by a journal were cited by WoS journals. The highest shares were found for journals that published 1748 and 259 papers. The shares are 36% (635 papers) and 35% (91 papers), respectively.
Table 1 shows the number and shares of cited papers according to lists of questionable journals. In the analysed sample, 8327 papers were published in journals listed in Beall’s list (10.3% of them were cited) and 13,910 papers from journals indexed in Cabell’s list (14.7% were cited). 5587 papers were published by journals indexed in both Beall’s and Cabell’s lists. The shares presented in the table may suggest that there is a significant relationship between the number of articles published by a journal and the share of cited papers. However, the Spearman correlation coefficient revealed that there was no significant relationship between the size of the journal and its share (rs = 0.019).
We checked whether 65 analysed journals were covered by Scopus, as this could potentially increase the share of citations. We found that five of 65 were or have been covered: one has been covered by the whole analysed period (24% of papers were cited whereas in the whole sample 13% of papers were cited), one was removed from Scopus before the analysed period (23%), one was covered and removed in the analysed period (36% papers), one was covered in the last year of the analysed period (19%) and one was covered before the analysed period and removed during the period (8%).
A.2. Web of science journals citing questionable journals
We found that 2338 unique WoS journals cited 3234 questionable papers 6750 times. The number of citations per questionable journal from WoS journals is 2.88 (median = 1, minimum = 1, maximum = 218). The mean number of papers from WoS journals that cited analysed questionable papers was 2.88. Half of the citations were from 261 journals. Eighty-nine of 2338 journals cited at least 10 times papers from questionable journals, and four WoS journals cited over 100 questionable papers. In the analysed period, one WoS journal published 183 papers, which cited questionable papers from our sample 218 times (all except one published in one questionable journal). One of these questionable papers was cited 36 times by this WoS journal.
We analysed in which WoS product (JCR, ESCI, A&HCI) a journal was indexed when a citing paper was published. We considered the publication year and whether a journal was included in the WoS product in the year in question. We found that 1152 of 2338 journals were indexed in ESCI, 35 in A&HCI, and 1047 in JCR. 104 journals that published papers that produces 366 citations were neither in ESCI, A&HCI nor in JCR, which means they were either in SCIE or SSCI indexes but not yet JCR (e.g., waiting for calculation of their impact factor) or dropped from the indexes because of quality issues or manipulations such as citation stacking or excessive self-citation rates. Figure 4 shows how questionable journals were cited by WoS journals. Of the 6750 citations, 2502 (37.1%) were from JCR journals, 3821 (56.6%) were from ESCI journals and 61 (0.9%) were from A&HCI journals. 366 (5.4%) citations were from journals indexed in SCIE or SSCI indexes but not yet JCR.
The questionable journals were selected only from social sciences; however, citations from all fields in WoS were considered for the evaluation of citations. The distribution of subjects to WoS subjects and broader OECD classifications (Clarivate Analytics, 2012) is shown in Fig. 5. The results are important because they prove the existence of citations from different subjects (such as medical sciences and agriculture) to social-science papers appearing in questionable journals.
Figure 5 shows that 25% of WoS citations were from one field, Education and Educational Research. This is followed by Management (8.9%) and Business (8.7%). In total, 63% of journals were classified in Social Sciences. According to the WoS classification of journals, a journal may be classified into two or more different subject fields. Six-hundred and sixty-five papers in 256 journals in our dataset were classified into two or more different categories. The Kruskal–Wallis test results show that there are significant differences between the journals’ OECD category and the number of citations to questionable journals (χ2 = 75.641, df = 6, p < 0.001). When each subject category was compared to each other using the Mann Whitney U Test, the sources of differences were determined for Engineering and Social Sciences (U = 41,585.500, Z= − 2.924, p = 0.003), Humanities and Medical Sciences (U = 15,038.500, Z = − 3.276, p = 0.001), Humanities and Social Sciences (U = 92,069.500, Z = − 2.192, p = 0.028), Medical Sciences and Natural Sciences (U = 17,338.500, Z = − 2.627, p = 0.009), Medical Sciences and Social Sciences (U = 128,110.000, Z = − 7.648, p < 0.001) and Natural Sciences and Social Sciences (U = 96,527.000, Z = − 3.310, p = 0.001).
A.3. Impact-factor journals citing questionable journals
The fact that a journal has a valid impact factor or is included in citation indexes is used by policymakers and managers to determine the level of that journal. We analysed the relationship between the impact-factor journals and their citations to questionable journals. To be able to make accurate statistical analyses, the impact factor of all journals cited in questionable journals were gathered with yearly changes. For example, if two articles in the same journal cited the questionable journals in 2018 and 2019, JCR 2017 and JCR 2018 were used. As a result, 1600 impact factors for 1047 IF journals were obtained.
Before presenting the comparisons between impact factors and citations to questionable journals, it is worth mentioning those journals dropped from citation indexes. Twenty impact-factor journals that cited questionable journals 125 times were dropped from JCR or WoS for different reasons. Fifteen of them were dropped from the index without listing any unethical concerns. This means that coverage of the journals did not meet the WoS selection criteria (Clarivate, 2018). Scientific World Journal was suppressed from JCR based on citation stacking and four journals (Business Ethics: A European Review, Environmental Engineering and Management Journal, Eurasia Journal of Mathematics Science and Technology Education and Industria Textila) were dropped for their excessive self-citation rates. These five journals cited questionable journals 39 times. Furthermore, although they were not indexed in JCR and did not have an impact factor, 15 journals were excluded from ESCI after being indexed for a couple of years in ESCI. All these findings can be commented as questionable journals in WoS cited questionable journals. However, statistical tests did not confirm this comment: the Spearman’s Rho correlation coefficient shows that the correlation between journal impact factors and the number of citations to questionable journals is very low, at a 99% confidence level (rs = 0.090, p < 0.001). Also, according to the Kruskal–Wallis test results, the differences between journal impact-factor quartiles and the number of citations to questionable journals were not significant (χ2 = 7.785, df = 3, p = 0.051). However, the Mann Whitney U test revealed that the only differences were found between Q1 and Q4 journals’ number of citations to questionable journals (U = 72,661.500, Z = − 2.648, p = 0.008).
The impact-factor range of questionable journal citers is from 0 to 27.604 (mean = 1.689, median = 1.378, SD = 1.471, 25% = 0.745, 75% = 2.252), while the minimum impact factor of the whole JCR between 2011 and 2018 is 0 and the maximum is 244.585 (mean = 2.072, median = 1.373, SD = 3.310, 25% = 0.704, 75% = 2.462).
Eighty percent of the journals in JCR cited questionable journals only one time, and there is a significant difference between the impact factors of one-time citers and the others (U = 174,977.000, Z = − 3.668, p < 0.001). However, the surprising result is that the average impact factor of journals that cited questionable journals more than once is 1.896 (median = 1.634), and this is higher than that of one time citers (mean = 1.639, median = 1.318).
Table 2 shows the main features of 30 impact-factor journals that cited questionable journals more than 10 times.
Having open access mega journals such as SageOpen, IEEE Access and Plos One on the list of the most cited IF journals (see Table 2) may suggest that open access mega journals cite questionable journals more frequently. However, the Mann Whitney U test result reveals that there is no significant relationship between open access feature of the journal (gold open or not) and the number of citations to the questionable journals (U = 75,233, p = 0.988). Similarly, the Spearman’s Rho correlation coefficient shows that there is no correlation between journals’ open access rates and the number of citations to questionable journals (rs = − 0.065).
All the test results on impact-factor journals prove that it is impossible to evaluate the questionable journals by looking at the impact factors or impact-factor percentiles of the journals because no pattern is identified. The impact factor is neither a descriptor of the quality of a paper nor the quality of citation. For example, the Journal of Business Ethics, which is one of the journals with the highest number of citations to questionable journals, is listed among the top 50 financial journals by the Financial Times (Ormans, 2016). On the other hand, Journal of Cleaner Production, which is also on the list, has been shown among the problematic journals by Clarivate Analytics due to its self-citation practices (Clarivate Analytics, 2021). Therefore, it reveals the importance of content-based analysis in understanding the purpose of citations to questionable journals.
Part B: self-citation analyses
We have analysed the countries of the corresponding authors of cited and citing papers. Table 3 presents the top 10 countries of cited and citing papers. Corresponding authors of papers published in questionable journals were most often from Turkey (335 papers), whereas citing papers were most often from the USA (555 papers).
We have calculated pairs of the most-often citing countries using the affiliation of corresponding authors. Figure 6 presents the top 50 pairs.
Italian researchers published the highest number of papers in WoS journals, which cited papers authored by Italian researchers in questionable journals. This result is not expected because previous studies have shown that Italian researchers did not publish intensively in questionable journals. Thus, we also conducted an author-level analysis of self-citations, which revealed that Italian researchers cited multiple questionable papers, of which they were co-authors.
In the analysis of self-citation on the author level, we have considered all the authors (not only the corresponding ones). By the analysis of bibliographical data and PDF files of the paper, we have found that 641 (9.5%) of 6750 are self-citations from 369 WoS journals to 53 questionable journals. The highest number of author self-citations from one WoS journal is 65 (all citations to one questionable journal; the corresponding authors of 55 from 65 of those WoS journal papers are affiliated in Italy). The highest number of author self-citations of questionable papers from one journal is 147.
Table 4 presents the top 10 self-citing authors from our sample. In total, we found 641 authors who self-cited their questionable papers.