Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland

Several countries have established performance-based research funding systems (PRFSs). A PRFS can be based on either a peer review or on an indicator-based model (Hicks 2012). The Research Excellence Framework in England is the most well-known example of a peer review model (Wilsdon et al. 2015). Indicator-based models have been implemented in Flanders (Belgium), Italy, Nordic countries, and Poland, among others. These indicator-based models provide classifications for publication channels in terms of the publications’ quality. In the Nordic system and in Flanders, full coverage national databases for recording and validating academic publications have been developed (Verleysen et al. 2014). In Italy, publication quality is assessed through a combination of citations and journal metrics (Abramo and D’Angelo 2016). In Finland, the characteristics of all relevant publication channels are provided by the JuFo database (Saarela et al. 2016). The Polish database, Polish Scholarly Bibliography, records all publications by Polish scholars since 2009.

Much has been written about the journal rankings and journal classifications (Hodge and Lacasse 2011; Serenko and Dohan 2011). On the one hand, journal rankings are useful tools for researchers, stakeholders, and policy makers. Scholars publishing in high-ranked journals receive higher salaries (Gomez-Mejia and Balkin 1992). Various higher-education institutions have implemented policies for rewarding their faculty for publishing in top-tier journals (Manning and Barrette 2005). Publishing in the high-ranked journals is perceived as one of the most important aims of science. On the other hand, there are many controversies surrounding the rating of journals. Most of them focus, among others things, on the ranking impact on academic life (Brembs et al. 2013; Reuter 2011; Wheeler 2011), the marginalisation of non-English countries (Wolters 2013), the relation of the citation metrics to the expert judgment (Haddawy et al. 2016; Sangster 2015; Serenko and Dohan 2011), and building a ranking for the social sciences and humanities (SSH) journals (Ferrara and Bonaccorsi 2016).

The two most often used methods of constructing journal rankings have relied on expert-based and citation-based evaluation. Serenko and Dohan (2011) compared various rankings based on both the expert surveys and the citation-based measures. They conclude that these two methods cannot be used as theirs substitutes. An expert-based evaluation and an evaluation based on the journal-citation measures should instead be used as complementary approaches. Saarela et al. show that based on the Finnish JuFo system, most expert-based rankings can be predicted and explained using automatically constructed reference models. Thomas and Watkins (1998) claim that the expert surveys and the citation-based measures strongly correlate, whereas Maier (2006) points out no significant positive correlation between the Impact Factors and the peer judgments. Schloegl and Stock (2004) state that the ranking lists, based on these two approaches, are very inconsistent. Nonetheless, in light of the present paper aim, it is worth highlighting that most rankings using these two approaches relied on the journal citation measures based on the Web of Science (WoS), Scopus, and Google Scholar (Ahlgren and Waltman 2014; Bontis and Serenko 2009; Haddow and Genoni 2010; Pajić 2015; Vanclay 2008). However, these rankings are often limited to some narrow fields or disciplines, such as forestry journals (Vanclay 2008) or Taiwanese journals in the SSH (Kao et al. 2008).

In addition to national databases, in which publications are recorded and validated, various journal rankings have been carried out at the national level. In France, Agence d’Evaluation de la Recherche et de l’Enseignement Superieur published a list of 6305 journals organised into three categories. In Australia within the Excellence in Research for Australia, an expert-based classification of more than 20,000 journals was published in 2010. Currently, the Australian list (i.e., the ERA 2015 Submission Journal List) consists of more than 16,000 journals. In Serbia, national journals are ranked and categorized annually in the Journal Bibliometric Report (Šipka 2013). Other national solutions have also been implemented in Norway, Taiwan, Brazil, Colombia, and the Netherlands. It is noteworthy to mention the international experience on the classification journals (i.e., the European Reference Index for the Humanities [ERIH]), which was built by the European Science Foundation in 2001 and evolved over various cycles (Wolters 2013). Finally, the ERIH was transferred to the Norwegian Centre for Research Data and has become a part of the ERIH PLUS.

In Poland since 2009, the research output has been assessed using a new system—the Comprehensive Evaluation of Scientific Units—developed by the Ministry of Science and Higher Education in Poland (Koczkodaj et al. 2014; Kulczycki 2017). An important component of this system is the Polish Journal Ranking (PJR) that allocates journals into three lists—A, B, and C—which translates the ‘quality’ of articles published in these journals into ‘the points’. National and non-national journals across all disciplines are represented in the ranking, similarly to, for example, the Australian, French, and Spanish systems (Ferrara and Bonaccorsi 2016; Haddow and Genoni 2010). A PJR, annually prepared by the Specialist Team for the Evaluation of Scientific Journals (STESJ), has a major impact on how the higher education institutions and researchers are financially and reputationally rewarded.

PJRs are built for and used in scientific unit funding decisions. Publication in a journal is the most important parameter of evaluation for which a scientific unit could possibly obtain the highest number of points. In the last cycle of evaluation in 2013, the percentage of funding depending on the results of evaluation ranged from 4 to 22% of the statutory funding for higher education institutions. PJRs are also used in promotion procedures. A Polish law required that only publications in journals indexed on the PJRs might be considered in decisions on a habilitation (the highest scientific degree). According to Moya et al. (2015), such a situation resembles the Spanish and Romanian solutions. In Poland, a candidate for a degree can list other publications, however, those that are not included in the government-controlled list ‘do not count’. Moreover, some universities require that a candidate collects a certain number of points (e.g. 250 points) and their h-index should be, for example, at least 3, if they want to obtain a habilitation. This is not a government criterion but a university recommendation or a prerequisite for those who want to start the procedure. However, these recommendations are explicitly expressed in terms of the points that can be obtained by publishing in journals indexed in the PJRs (Kulczycki 2017).

Nearly 3000 scientific journals have been published in Poland of which 2477 are indexed in the 2015 PJR. The remainder of the journals that do not fulfil the PJR criteria—for instance, they are published non-periodically—are indexed in the ARIANTA, which is the largest continuously updated Polish journal reference list. Currently in the WoS, however, there are only 231 Polish journals indexed. These include the Journal Citation Reports (JCR): 137, Arts and Humanities Citation Index: 6, and Emerging Sources Citation Index: 88. In SCOPUS, there are 350 journals indexed, and there are 139 Polish journals included in the ERIH. Because the PJRs are used for evaluating all Polish higher-education institutions, the local publications that are not indexed in the international databases must be taken into account. Thus, a national journal ranking in Poland, which should represent all disciplines, could not be based only on the Impact Factor or other bibliometrics indicators, which are not suitable for assessing, for example, publications in the humanities. The degree of coverage in the SSH in the Web of Science, which is the only government-acknowledged database in Poland, is low (Sivertsen and Larsen 2012), particularly for non-English publications.

The regulations that determine the PJRs have initiated many discussions and controversies. In Poland, there are much fewer publications in English than in other non-English countries such as Norway. According to Sivertsen (2016), in Norway, humanists published 61.1% of their publications in international languages and in the social sciences, the value of this indicator was even higher—71.7%. Sivertsen concludes that researchers are normally bilingual in their publication practices in the SSH. We cannot directly compare Norwegian and Polish practices because of the different levels of data collection. Sivertsen presents data at the level of individual researchers, whereas in Poland, data are collected at the level of all higher education institutions and at the level of Polish journals. In the 2009–2012 period, Polish scholars in all fields of sciences published and submitted 19,764 monographs to the Comprehensive Evaluation of Scientific Units, of which 86.7% were published in Polish; in the SSH, this value was slightly higher—87.8%. There were 144,873 book chapters published, of which 77.4% were published in Polish; in the SSH—80.2%. According to the Polish journals not indexed in the JCR or in the ERIH (N = 2035), the mean percentage of the articles published in international languages is 26% for the social sciences, 26% for the humanities, and 46% for the so-called hard sciences. These numbers show that Polish scholars publish mostly in Polish.

Until 2014, the PJRs were based on the two dimensions of evaluation: the formal and the bibliometric. In 2015, the STESJ, taking the criticism into account, decided to add the other dimension (i.e., the expert-based dimension, but only for the evaluation of national journals that are not indexed in the JCR or the ERIH).

Vieira and Gomes (2015) show that a bibliometric evaluation relates to an expert-based evaluation in three different ways: (1) bibliometrics are used for analysing the expert-based evaluation, (2) the expert-based evaluation uses bibliometrics as an auxiliary tool, and (3) the expert-based evaluation is a way of correcting the results of the bibliometric assessment. These methods have been implemented in various national models. For example, in Norway (as well as Denmark and Finland, which follow the Norwegian model), citation-based indicators are first collected and then presented to experts who can make their judgment based on them as well as other available journal information. In contrast, in the Polish system, the experts are provided neither with citation-based indicators nor the formal evaluation of the journal. According to this classification, the third relation characterises the case of the 2015 PJR.

The aim of the present paper is to determine how the expert-based evaluation influenced the results of final evaluation (2015 PJR). In our analysis, we tested three models designed separately for three fields of science: the humanities (H), the social sciences (SS), and the engineering, natural and medical sciences (ENM).

Like every scientometric tool, a PJR has some drawbacks and advantages (the most important are characterised in the present paper). It is noteworthy that building such a national ranking in a non-English country is the most important value for scholars and stakeholders. Thus, a PJR should and could be developed. We have conducted our research to show how to improve the PJRs and reduce the drawbacks resulting from overregulation and using incorrect bibliometric indicators.

The paper is structured as follows: in “The 2015 PJR” section, we present the framework of the 2015 PJR and explain the evaluation process. In “Material and Methods” section, the Materials and methods are described, and the next section presents the results, focusing on the theoretical models designed for the three fields of science (H, SS, and ENM). In the final section, we discuss the main findings and conclusion.

The 2015 PJR allocates 17,437 journals into three lists—A, B, and C. The set of A, B, C lists makes up a national (Polish) journal ranking. An isolated list (i.e., the B list) constitutes a ranking of Polish journals, the structure and regulations of which will be examined in the present paper. The B list plays a major role in the science policy in Poland because Polish scientists have published more frequently in journals indexed in the B list than in journals indexed in the A or C lists.

The STESJ built the 2015 PJR according to Komunikat Ministra Nauki i Szkolnictwa Wyższego z dnia 2 czerwca 2015 r. w sprawie kryteriów i trybu oceny czasopism naukowych (the Regulation of the Ministry of Science and Higher Education of 2 June, 2015 on criteria and procedure of the evaluation of scientific journals). There were two technical operators that supported the STESJ work: the National Information Processing Institute and the Index Copernicus International.

All Polish journals had to submit the Ankieta czasopisma naukowego (The Questionnaire of Scientific Journal) through the Polska Bibliografia Naukowa (the Polish Scholarly Bibliography) to be indexed on the B list of the 2015 PJR. A Polish journal is defined as a journal for which the editorial office or the publishing house is located in Poland. Moreover, a journal had to be published at least for two years: data from the last two years were submitted through the questionnaire. Both the questionnaire and the final evaluation of each journal in all dimensions have been published and available on the Polska Bibliografia Naukowa.

The B list was built in two steps in which: (1) a fulfilment of the entrance criteria was verified, and (2) fulfilment of the required minimal values of parameters was verified.

In Step 1, each journal had to fulfil at least five of six entrance criteria to go to Step 2. As Table 1 shows, the entrance criteria were formal and related to the editorial policies and standards.

In Step 2, each journal was assessed within three dimensions of the evaluation: formal, bibliometric, and expert-based. As Table 2 shows, various parameters were defined and required minimal values for the fulfilment of these parameters. There is a single exception in the expert-based dimension in which the value of the only parameter, that is, the expert-based evaluation, is equal to the points obtained for it. This means that fulfilment of this parameter allows the journal to get 1–5 points.

During a submission of the above-mentioned questionnaire, a journal editor must choose one of the three fields of science in which a journal should be evaluated (i.e., the H, the SS, or the ENM). Moreover, a journal editor must indicate at least one and at most three disciplines that characterise a scientific profile of the published articles. For all fields of science, the same sets of parameters were designed. According to the field, however, different required minimal values were assigned for the fulfilling parameter, and a number of points were assigned for it. Points obtained by fulfilling the parameters are totalled. This sum of the points provides the result of the multidimensional evaluation of a given journal.

To illustrate how required minimal values could differ as a result of fulfilling the parameters, let us examine two cases. The first case concerns points for fulfilling the same parameter in different fields of the sciences. For instance, a journal from the H could obtain 1 point if at least 5% of articles in the evaluated period were published by the authors from foreign countries. Conversely, a journal from the SS or the ENM could obtain only a half point and only when at least 10% of the articles were published by the authors from foreign countries. The other case concerns journals from the same field of the sciences that have similar parametric values but have obtained different numbers of points because of the required minimal values. For instance, 9% of authors from the H in the first journal are from foreign countries and 10% of authors from the H in the second journal are from foreign countries. The second journal obtains 1 point for the ‘Internationalisation of authors’ parameter and the first journal receives no points for this parameter because of the required minimal values (10% in the H).

Formal dimension was based on the data submitted through the questionnaire. Moreover, journal editors could provide the bibliographic data to the POL-index. This database has been designed for collecting the references from all articles published in the Polish journals. The Ministry of Science and Higher Education has announced that, on the basis of the POL-index, the Polish Impact Factor will be calculated (orig. Polski Współczynnik Wpływu) in the next cycles of journal evaluation, probably in 2017. The STESJ verified the data quality submitted by various journals, which were pointed out by the technical operators and the STESJ members. The editors were not obligated to send electronic or paper versions of the evaluated issues during the questionnaire submission. Therefore, preparing the final formal ranking was founded on trust in the provided information by the journal representatives.

Bibliometric dimension was based on two bibliometric indicators: Predicted Impact Factor (PIF) and Scimago Impact Factor (SIF). On the B list, only journals without an Impact Factor could be indexed. In addition, SCOPUS is not an acknowledged database in the Polish science policy. However, to implement some bibliometric assessment within a ranking of national journals, the STESJ has proposed the two indicators in question.

During the evaluation, the Library of the University of Silesia in Katowice collected the data, and then the technical operators calculated PIF and SIF for the submitted journals. If a journal had calculated both SIF and PIF, a higher value of indicators was taken into account for a given journal in the final bibliometric ranking prepared in the bibliometric dimension.

We have found that using SIF and PIF will allow us to evaluate 90% of journals from the ENM, but in the two other fields of science, the percentage is significantly lower (i.e., 48% for the SS and 48%) for the ENM. Thus, the multidimensional evaluation should not be reduced only to the bibliometric evaluation if it has to serve for assessing journals from all fields of science: too many journals (52%) from the SS and the H could be omitted in the evaluation because of a lack of citations in the WoS or SCOPUS. Our findings confirm that journals in the SSH are underrepresented in the major citation databases (Sivertsen 2016).

Expert-based dimension involved the Committees of the Polish Academy of Sciences. Committees represent the Polish scientific community according to a discipline (e.g., the Committee on Psychological Sciences and the Committee on Geodesy). A technical operator of evaluation sent committees a list of journals that best fit the profile of a given committee. The process of assigning journals to committees was conducted on the basis of indicated discipline(s) in the submitted questionnaire by a journal and on the basis of the committee’s scientific profile. However, during a process of the evaluation, committees may have to add other journals if they decide that they are qualified to assess the journals in question. In this way, two committees received more than 500 journals for evaluation, 41 committees received more than 100 journals, 46 committees received from 10 to 100 journals, and only 5 committees received less than 10 journals. Finally, 94 committees prepared the expert recommendations.

Each committee could decide how many experts to invite to the evaluation process. Moreover, experts could be external (i.e., not members of a committee). The regulations regarding how to proceed this evaluation were designed by each committee. The STESJ specified only one guideline: each committee was asked to recommend, on the basis of the set of assigned journals, at most 10% of the journals that the experts (appointed by a given committee) acknowledged as ‘outstanding’ and to recommend at most 10% as ‘significant’. Finally, 1367 journals were acknowledged as ‘outstanding’ or ‘significant’ by at least one committee. On the basis of the expert-based evaluation prepared by all committees, the STESJ assigned the points to journals. Taken into consideration were the types and the number of recommendations for a given journal (an ‘outstanding’ recommendation was more important than a ‘significant’ one). Some factors that could disturb the evaluation and the results comparison between the committees include access to the previous evaluation results (if some journal was in the previous PJRs, it might mean that is ‘good enough’ to be recognised as a good one), an alphabetic order of the journals indicated for the evaluation, a different number of experts in the various committees (from one to over a dozen), and the committees’ diligence (some committees did not conduct their duties and recommended no journals).

In conclusion, our findings show that the multidimensional ranking of local journals should not be constructed only on the basis of the bibliometric evaluation. In the Polish case, reducing the 2015 PJR dimensions to the bibliometric dimension could keep 52% of journals from the H and the SS from being included in the ranking. According to our analysis, this dimension is significant only to the ENM in the multidimensional evaluation of national journals.

As presented, the expert-based dimension plays an important role in multidimensional evaluation. Theoretical models that include this dimension explain the majority of the variability of results evaluation in all fields of science.

However, when we look at the interrelationships between the indicators in the analysed models, we can see that including the expert-based dimension has increased the model fitting in the H and the SS models, but this effect is not observed with respect to the ENM.

The expert-based evaluation has been introduced in the current journal evaluation system, as requested by the majority of Polish scholars. However, the procedure of the evaluation itself was not properly designed and should be improved in many aspects. The experts might be affected by the previous evaluation of journals because they could check if a journal was previously assessed and how many points were assigned. The experts did not know how journals were evaluated in formal and bibliometric evaluation in the current cycle of assessment. Moreover, the experts, who received too many journals for assessment may have experienced the halo effect. According to Serenko and Dohan (2011), the extremely high or low quality of a preceding journal may affect quality perceptions of the subsequent journals. There are many other limitations of the expert-based evaluation (e.g. experts’ research interests or the time and costs of implementation). Thus, the procedures should be continuously improved so these valuable resources of academic community will not be wasted.

Development of the PJRs is possible but should be implemented gradually to make it possible to assess and redesign a new version of this journal-evaluation system. According to our findings, we suggest four main steps that could improve the current solution and may be relevant for the journal evaluation systems in other non-English countries.

The first two suggestions are relevant for the journal evaluation from all fields of science: (1) the formal evaluation should be reduced only to such parameters that could be verified by the STESJ members: the current solution is based mostly on the trust in the data provided by the journal editors and is perceived by many scholars and stakeholders as insufficient; (2) the expert-based evaluation should be conducted on the basis of some common guidelines for the experts and, moreover, the committees should provide the internal regulations for their experts that might be adjusted for various fields of science.

The next two suggestion are especially important for the social sciences and the humanities: (3) Non-Polish journals that are indexed neither in the WoS nor the ERIH should also be indexed in the B list: the current solution does not allow for keeping the balance between the humanities and other fields of science; (4) in the next cycles of PJRs, improved bibliometric indicators should be used that fit to publication languages, different disciplines, and publication patterns: one of the possible solutions is to re-define the announced Polish Impact Factor and to collect the bibliographic data of books that are not only Polish in the POL-index. Currently, only references from articles have been gathered.

In our analysis, we have focused on showing how adding the expert-based dimension affected the results in the three science groups. Nonetheless, the data and the relations between the variables presented in the present paper might be an important component in the discussion concerning the multidimensional journal evaluation in general. For instance, in Table 5, one can see that there is no correlation or only a weak correlation between the electronic version and the variables constituting journal internationalisation. However, international databases (e.g., Web of Science Core Collection and SCOPUS), which index journals with a high level of internationalisation, expect the electronic versions. This may be evidence that using indicators specified for international journals and relying on an electronic version in the evaluation of local journals may not be well grounded.

In some areas of science policy in Poland, the collection of points and use of PJRs in the academic-promotion procedures have become targets themselves. Analysis of such unintended uses could provide some new arguments regarding how to change the research-evaluation system in Poland. Thus, the aim of the PJRs’ improvement must be not only a good quality evaluation but for scientometricians and policy makers to take into account that each change in this system affects some new, unintended uses that should be reduced.