Assessing the effects of publication requirements for professorship on research performance and publishing behaviour of Ukrainian academics

Abstract

This article aims to explore the effects of Ukrainian policy reform, introducing Scopus and WoS publication requirements for professorship, on the publication behaviour and research performance of professors. Our analysis reveals a better scientific profile, at the time of promotion, of those who obtained professorship after the reform as compared to those who obtained it before. Also, we observe a bandwagon effect since the research performance gap between the two observed cohorts decreased after the introduction of the publication requirements. The statistical difference in differences tests revealed that in general, the incentive to produce more indexed publications worked. Nevertheless, it did not always led to higher research performance. Evidently, in several cases, the increase in research output was obtained at the expense of research impact. The effects of the reform could be far greater if combined with initiatives aimed at assessing Ukrainian professor performance regularly and extending the requirements and assessment to the impact of research.

Introduction

Professors are expected, among others, to have a larger number of high-quality research works as compared to lower-rank academics. Nowadays most academic stakeholders tend to associate high-quality research with works published in journals indexed in Web of Science (WoS) or Scopus (Hicks, 2012). Indexed publications are those used to make world-university performance rankings. Shanghai Ranking uses WoS as a data source to measure research performance, while Times Higher Education and QS use Scopus as their source. Consequently, several state authorities have introduced Scopus- and WoS- (S&W) indexed publication thresholds for doctoral degree, professorship, research funding allocation (Grancay et al., 2017; Hladchenko & Moed, 2021a). However, publishing in journals indexed in S&W does not necessarily imply high-quality research (Abramo et al., 2019a, 2019b; Bornmann & Daniel, 2008). That is why the states that aim to increase research output without a decline in impact, give extra weight to impactful publication channels e.g. Norway (Bloch & Schneider, 2016), Spain (Jiménez-Contreras et al., 2003). Prior studies highlight that the incentive schemes linked to only the number of publications can result in academics increasing the number of publications by publishing in lower-impact journals, e.g. Australia (Butler, 2003a, 2003b), Denmark (Ingwersen & Larsen, 2014). Negative consequences of policies focused only on increasing the number of S&W publications are stronger in (semi-) peripheral scientific countries that have recently introduced the requirements of publishing in S&W journals e.g. Ukraine (Nazarovets, 2020), Kazakhstan (Kuzhabekova & Ruby, 2018), Uzbekistan (Eshchanov et al., 2021), Vietnam (Pham-Duc et al., 2022), Turkey (Demir, 2018; Önder & Erdil, 2017).

This study aims to contribute to the body of knowledge on the impact of research policy reforms on the performance and publication behaviour of academics. Specifically, it explores whether the introduction of S&W publication requirements for professorship in Ukraine resulted in increased research productivity,¹ and changes in the publication behaviour of professors along several dimensions, namely intensity of publication, average impact, research collaboration, publication document type, and language. The following research questions can express the purpose of our work:

• RQ1: Had the academics who obtained professorship after the introduction of the S&W publication requirements a better scientific profile than those who obtained it before?

• RQ2: Did the research performance gap between the two cohorts decrease after the introduction of the publication requirements (i.e. was there a bandwagon effect?)

• RQ3: Did the publication requirements determine an increase in the research performance of academics who obtained professorship after their introduction?

We focus our bibliometric analysis on professors in the STEM sciences.

The study is structured as follows. In the next section, we review the literature on the effects of research evaluation policies.  In "Ukrainian higher education system and publication trends" section, we present the main traits of the higher education system in Ukraine. In "Data and methods" section, we describe the indicators and methodologies adopted for the investigation, while the results are reported in "Results" section. We discuss them and draw our conclusions in "Discussion and conclusions" section.

Effects of research assessment policies

There is a rich literature on the effects of research evaluation policies on the individual behaviour of scientists both in Western (Abramo et al., 2019a, 2019b; Moher et al., 2018; Schneider et al., 2016; Butler, 2003a) and non-Western societies (Grancay et al., 2017; Good et al., 2015; Hladchenko, 2022).

A stream of research focuses on the unintended effects often associated with the opportunistic behaviour within academies (De Rijcke et al., 2016; Pajic, 2015). Opportunistic behaviour may take different forms: (i) publishing more articles but of lower-quality by slicing the results of research to least publishable units (Weingart, 2005); (ii) signing papers without contributing to the research, also referred to as gift authorship (Kovacs, 2013); (iii) scientific misconduct, i.e. altering research results to facilitate acceptance for publication; plagiarism; or inappropriate self-citation (Hazelkorn, 2010; Edwards & Roy, 2017; Seeber et al., 2019; Abramo et al., 2021).

Another stream of research investigates the effectiveness of research policy reforms and the relevant incentive systems put into place. Research performance evaluation schemes based on output quantity only, generally resulted in an increase in the number of publications on average in lower-impact journals e.g. Australia (Butler, 2003a, 2003b), Indonesia (Rochmyaningsih, 2019), Kazakhstan (Kuzhabekova, 2019), Slovakia (Pisár & Šipikal, 2017), Turkey (Demir, 2018); of lower impact (Butler, 2003a, 2003b; Vanecek, 2014); and encoded in document types that are more easily published and less often cited than articles, e.g. proceedings papers (Vanecek & Pecha, 2020). In Uzbekistan, research assessment policies led the country to have the largest share of publications in journals discontinued from Scopus (59.67%) (Eshchanov et al., 2021). It has been also observed a growing trend to publishing in journals which generally use an open access mode to obtain financial gains without providing the expected publishing services and peer-review quality (Beall, 2015; Butler, 2013). Though this practice is more spread in scientific (semi-)peripheral countries e.g. Turkey (Demir, 2018), Kazakhstan (Macháček & Srholec, 2021), Uzbekistan (Eshchanov et al., 2021), it is not infrequent in western scientific countries either (Bagues et al., 2019; Moher et al., 2018; NDR, 2018). Demir (2018) revealed that 15.85% of the publications in 832 supposedly “predatory” journals explored by him were written by academics from developed countries.

Another effect is for academics to submit manuscripts to national journals or journals of neighbouring countries. This tendency is reported in Belgium (Ossenblok et al., 2012), Kazakhstan (Kuzhabekova, 2019), Central and Eastern European countries (Nazarovetz, 2020; Machacek & Srholec, 2017; Grancay et al., 2017; Pajic, 2015; Pajic & Jevremov, 2014). The majority of S&W journals originate from western states but since 2000 Scopus and WoS have extended their coverage by noticeably increasing the number of the journals from non-western countries e.g. Latin America and the Caribbean (Collazo-Reyes, 2014; Chinchilla‐Rodríguez et al., 2015), Central and Eastern European countries (Grancay et al., 2017). Prior studies indicate that national S&W journals have a comparatively low impact (Collazo-Reyes, 2014; Grancay et al., 2017).

Other side-effects observed were discouraging research diversification, interdisciplinary and innovative research (Abramo et al., 2018; Hicks, 2012; Rafols et al., 2012; Wilsdon, 2015); and tilting time and energies from teaching to research activities (Enders et al., 2015; De Philippis, 2021).

Ukrainian higher education system and publication trends

After the 1991 independence, the higher education system in Ukraine has gradually undergone marketisation, massification and internalisation (Oleksiyenko & Shchepetylnykova, 2021) but a significant part of the Soviet model has been preserved (Gomilko et al., 2016; Oleksiyenko, 2016; Shevchenko, 2019.). Despite Ukraine joined the Bologna process in 2005, whereby the doctor of philosophy or PhD is the highest academic degree, the two-level system of doctoral degrees remained unchanged. In Ukraine, PhD is the first-level doctoral degree, obtained after four years of study by submitting and publicly defending a thesis and passing the required examinations. The second-level doctoral degree, Doctor of Sciences is comparable to “habilitation” in some western countries. It is awarded to candidates already holding a PhD, on the successful presentation and defence of the dissertation.²

In addition to the two-level system of doctoral degrees, a two-level system of academic titles—docent (associate professor) and professor inherited from the Soviet model has also been preserved. The first-level doctoral degree allows promotion to associate professor (docent), while the second-level doctoral degree allows promotion to professor. Doctoral degrees and academic titles involve extra payment to the salary of academics: 15% and 35% of a salary increase respectively for the PhD and Doctor of Sciences, and 25% and 33% for the scientific titles of associate professor and professor (Ministry of Education and Science of Ukraine, 2005).³ Salaries of professors and other academic staff are regulated at the central level. As a rule, the salary of academics, including professors, does not depend on merit.

After 1991, publications in Ukrainian journals were required to obtain doctoral degrees and academic titles of associate professor and professor, as well as for the allocation of research funding and the state attestation of Ukrainian higher education institutions. Specifically, publication requirements for the title of professor included textbooks and ten publications produced after the second-level doctoral thesis and published in Ukrainian journals from the list approved by the Ministry of Education and Science. An academic was required to have already the academic title of associate professor.⁴

In 2013, publications in international journals and in Ukrainian journals indexed in S&W became a requirement to obtain doctoral degrees. In 2015, changes were made in publication requirements for academic titles, and it was announced that they would come into force in 2016. Specifically, articles in S&W journals were also introduced into publication requirements for the academic titles of associate professor and professor. According to these requirements for the scientific title of associate professor (professor), academics with a PhD (second-level doctoral degree) are required to have a five-year (ten-year) work experience, an article (two articles) in an S&W journal, a B2 level English certificate, and a study visit or participation in a conference or symposium in OECD countries or European Union countries.⁵ The B2 level English certificate can be substituted in both cases with 10 articles in S&W journals. Researchers affiliated with the research institutes of the academies of sciences to be promoted to professor are required to have three articles in S&W journals and three successfully defended doctoral candidates.

More recently, the scope of application of publication requirements was extended. In 2018, articles in S&W journals became one of the requirements for the state attestation of Ukrainian higher education institutions.⁶ In 2019, publications in S&W journals were included among the criteria for assessing research projects applying for state funding.⁷

A recent study on the research output of Ukrainian academics in Scopus highlights that the total number of publications has risen dramatically since 2011. However, firstly, the share of Q3 + Q4 by SNIP⁸ exceeded Q1 + Q2. Secondly, the share of publications in Ukrainian Scopus-indexed journals reached the peak of 47.3% in 2015. In the following years, it fell up to 31.8% in 2019 (Hladchenko, 2022). In 2016, Scopus indexed 47 Ukrainian journals, and in 2014–2016 publications by Ukrainian authors constituted 68.2% of overall publications in these journals (Hladchenko & Moed, 2021b).These findings resonate with Nazarovets (2020), who revealed that in 2015–2019, the most popular journals among Ukrainian authors, according to the number of publications, were journals of Ukrainian publishers and English-language translations of Ukrainian journals. CiteScore⁹ values indicate that these are generally low impact journals. Furthermore, in 2011–2020 in the field of Economics, Econometrics, and Finance, 47% of papers authored by Ukrainian academics were published in journals discontinued from Scopus (Nazarovets, 2022).

Data and methods

To set up the dataset for the analysis, we performed the following steps. Initially, we extracted from the website of the Ministry of Education and Science of Ukraine¹⁰ data on all Ukrainian academics that were awarded the title of (full) professor in 2015–2018. The metadata included name, discipline based on the department of affiliation, and year of awarding. We then classified professors by discipline. From the compiled list of Ukrainian professors, we selected those in STEM disciplines (medicine, engineering, computer science, physics, agricultural and biological sciences, psychology, material science, chemistry, mathematics, pharmacology, environmental science, health profession, biochemistry, genetics & molecular biology, veterinary, earth and planetary sciences). We excluded the social sciences and arts & humanities because the coverage of bibliographic repertories is still insufficient for a reliable representation of research output in these fields (Aksnes & Sivertsen, 2019; Hicks, 1999; Larivière et al., 2006). The final dataset is made of 805 professors.

Then, we manually retrieved the Scopus_ID of each professor of them. In doing so, we faced the formidable task of matching the Cyrillic name of each professor in the national register with the Latin name with which they are indexed in Scopus, e.g. Гoтькo Євгeн (Hotko Yevhen), Кaтюжинcькa Cвiтлaнa (Katyuzhynska Svitlana), Клiтинcькa Oкcaнa (Klitynska Oksana). While retrieving the Scopus_ID of each professor, we matched Ukrainian disciplines with the subject areas of Scopus. Based on Scopus_ID, we downloaded publications for each Ukrainian professor from our list. Table 1 represents data on the number of academics in STEM who were awarded the title of professor in 2015 (458 in all) and 2016–2018 (347 in all) and their breakdown by discipline. For 116 professors (14.4% of the total 805) we were not able to identify any Scopus profiles. Among 2016–2018 professors, 19 show no publications in Scopus. The reason for that may be that the journals in which they published were indexed only in WoS but not in Scopus. As expected, the 2016–2018 cohort is more represented in Scopus (94.5% of awarded professors have a Scopus profile) than the 2015 one (78.8%).

Table 1 Professor dataset of analysis by year of awarding and discipline

At the discipline level, Medicine, Engineering, and Computer science account for over 50% of total observations. Note that in “Biochemistry, genetics and molecular biology” all 17 awarded academics have been identified in Scopus, while only 20 (out of 39) professors in Psychology have a Scopus profile.

We intend to assess each professor’s research performance over a period of time and professors’ publication behaviour. To the first aim, we recur to the Fractional Scientific Strength (FSS) indicator of research productivity,¹¹ defined as

$${\rm{FSS}} = \sum\limits_{i = 1}^N {\frac{{{c_i}}}{{\bar c}}} {f_i},$$

(1)

where ${c}_{i}$ = citations received by publication i,$\overline{c }$ = average of the distribution of citations received by all publications in the same year and Scopus subject area of publication i, ${{f}}_{{i}}$ = fractional contribution of the professor to publication i, given by the inverse of the number of co-authors in the byline, N = number of publications¹² of the professor in the period under observation. To investigate each professor’s publication behaviour, we also measure single components of the FSS, that may be of some use in understanding which “dimension” mainly drives the performance. In particular, we will consider the following:

$$\mathrm{Output}= N$$

(2)

$${\rm{Fractional\,output}} = \sum\limits_{i = 1}^N {{f_i}},$$

(3)

$$\mathrm{Average\,impact }=\frac{1}{N} \sum_{i=1}^{N}\frac{{c}_{i}}{\overline{c}}.$$

(4)

Finally, we investigate the scientific output by document type (article, review article, conference paper, book chapter), publication language, and professors’ collaboration behaviour.

For each professor, we observe production in fixed publication windows of three years, before and after professorship, as shown in Table 2. Publication and citation count date 30 April 2022.

Table 2 Time windows considered for the before/after analysis

To assess the effect of the 2016 publication requirement initiative in stimulating higher intensity of publication and productivity, we apply a difference in differences statistical technique, whereby we contrast the after/before reform variations of 2015 awardees (Cohort A), not subject to the incentive, vs 2016–2018 awardees (Cohort B), subject to the incentive. For this analysis, the length of the observation periods varies according to the year the researchers obtained the professorship.

Results

The 689 professors in the dataset authored in total, during their career to date, 14,679 unique publications indexed in Scopus, for a total of 15,186 authorships, obtained by multiple counting co-authorships of the same publication. Limiting the observation to the time windows shown in Table 2, we count 5434 unique publications and 5735 authorships, 2481 (43.3%) of which relative to the period before the awarding and 3254 (56.7%) in the period after.

In the following, we report first the descriptive analysis of research performance and publication behaviour at the aggregate and the individual levels, and then the statistical analysis to assess the cause-effect link between the publication requirement initiative and research performance.

The research performance and publication behaviour of the two cohorts at the aggregate level

Table 3 reports a descriptive analysis at the aggregate level of the research performance of the two cohorts before and after promotion. It reveals that in the period before obtaining a professorship, Cohort B outperforms Cohort A in all disciplines. This means that at the time of promotion, the academics who became professors after the reform had a better scientific profile than those promoted before (RQ1). This is evident in all disciplines, with a peak in Medicine.

Table 3 Average Fractional Scientific Strength (FSS) of the two cohorts of professors by discipline, before and after the award

Both Cohort A and B professors increased their average FSS in the three years after the professorship award in all disciplines but Physics (Cohort A) and Mathematics (Cohort B). The latter is the only discipline where Cohort A outperforms B after professorship. Among Cohort A, professors in Mathematics had the highest average FSS after. Finding the reasons for the above exceptions would require delving into the single disciplines through ad-hoc interviews.

Focusing attention on the variations of the performance gap between the two cohorts, we observe a bandwagon effect in most disciplines (RQ2). The performance gap between the two cohorts tends to decrease after professorship in all disciplines but Agricultural & biological sciences, Pharmacology, and Physics. As mentioned above, the interpretation of exceptions is not straightforward.

We now analyse the differences in publication behaviour of the two cohorts along the three relevant FSS components (e.g. output, fractional output, average impact). In the next "The effect of professorship on publication patterns" section, we move on to the other dimensions of the language of publication and collaboration behaviour.

Table 4 shows that in the period before obtaining a professorship, Cohort B on average published more than Cohort A in all disciplines, even accounting for co-authors’ contributions. Cohort B’s supremacy also occurs by the average impact of publications, in all disciplines but Physics. After promotion, the same holds true per number of publications and fractional output, in all disciplines and per average impact in all disciplines but in Mathematics. The latter explains why in Mathematics, Cohorts A overcomes B per productivity (FSS) after promotion. The gap after-before between the two cohorts tends to decrease per average number of publications and fractional output in all disciplines but Pharmacology; and per average impact in all disciplines but Agriculture and biology, and Physics. Analysing the variations within the individual cohorts, it can be seen that for professors in Cohort A, output and fractional output increased in all disciplines but Pharmacology after promotion. The same applies to average impact with the exception, in this case, of Physics. As for Cohort B, after promotion the output increased on average in all disciplines but Chemistry. Fractional output decreased in Chemistry, Engineering, Medicine, and Physics. Finally, the average impact increased in all disciplines but Pharmacology.

Table 4 Average output, fractional output and impact of the two cohorts of professors by discipline, before and after the award

The research performance and publication behaviour of the two cohorts at the individual level

In this subsection, we delve into the research performance and publication behaviour at the individual level. We analyse after-before variations on scores of research performance, output, fractional output, and average impact for each professor in the dataset.

We start describing (Table 5) the share of professors in each “status” (productive and unproductive, by FSS)¹³ three years before and three after promotion. Only 19% of Cohort A professors result as productive both before and after the award, as against 55% of Cohort B professors. On the other side, the share of professors unproductive in both periods is 58.3% for Cohort A and 24.8% for Cohort B. Interestingly, 18.3% of professors in Cohort A change their status, passing from unproductive to productive after promotion. Only 4.4% of the same Cohort shows the opposite change. On the contrary, 7.8% of professors in Cohort B become productive after promotion, against 12.4% of those showing the opposite shift.

Table 5 Share of productive (FSS > 0) and unproductive (FSS = 0) professors, by cohort, before and after promotion

Table 6 shows that referring to Cohort A professors the share of those experiencing an improvement in FSS after the promotion is higher than the share of those experiencing a worsening: 30.8% vs 10.9%, at the overall level. It must be noted that the remaining 58.3% are unproductive (FSS = 0), both before and after promotion. A different situation emerges for Cohort B professors: at overall level 34.9% of them register an increase in FSS, against 40.3% registering a decrease (and 24.8% remaining unproductive). Looking at the single components of research performance, results are very similar: the percentage of professors in Cohort A experiencing an increase in the scores of the indicators (output, fractional output or average impact) is almost three times the percentage of those registering a decrease. At the same time, the two subsets are numerically equivalent for Cohort B. This shows a bandwagon effect of Cohort B on A.

Table 6 Share of professors registering an increased/decreased score of each indicator, after becoming professor, by cohort

At the discipline level, and limiting the focus to FSS, Fig. 1 shows that for Cohort A, the share of professors experiencing an improvement in research performance is higher than the share of those experiencing a worsening in all disciplines but Physics. In Pharmacology, the percentage of professors improving their research performance is the same as those worsening it. Data in Fig. 2 show that for Cohort B, in five disciplines, the share of professors experiencing an improvement in research performance is higher than the share of those experiencing a worsening. The opposite holds in the other three (Chemistry, Pharmacology and Physics).

Fig. 1

Share of professors awarded in 2015 registering an increase/decrease in FSS score after becoming professor, by discipline

Fig. 2

Share of professors awarded in 2016–2018 registering an increase/decrease in FSS score after becoming professor, by discipline

The effect of professorship on publication patterns

We now move on to analyse the distribution of metadata of the publications in the dataset to highlight possible changes in publication patterns by professors. Table 7 reports the breakdown of scientific output by document type. It highlights that the increase in research output after academics were awarded the title of professor involved a significant growth in book chapters (+ 170.4%) and conference papers (+ 93.7%) and only a slight increase in articles and reviews (+ 12.7%). This occurs because publication requirements for promotion to professor include only S&W articles and academics are not motivated to publish any other publication types. Nevertheless, after being promoted to professor they increase their output through publications with laxer admission criteria.

Table 7 Document types of publications authored by professors before and after promotion

Table 8 reveals that the average number of co-authors has increased after promotion (+ 21.1%) from 6.49 to 7.86, mainly due to the increase recorded for cohort B (+ 31.3%), indicating an evident propensity for these professors to participate in larger research teams after promotion.

Table 8 Average number of co-authors in publications authored by the awarded professors

The share of publications resulting from international collaboration remained almost unchanged before and after the title of professor was awarded (Table 9), with a slight increase for Cohort A (from 27.4% before, to 30.2% after).

Table 9 Share of publications resulting from international collaboration

Regarding the language of publication, after being awarded the title of professor, academics increased the share of publications in English and decreased the percentage of publications in Russian and Ukrainian (Table 10).

Table 10 Language of publications authored by awarded professors

Impact of publications

After obtaining the professorship, the average SNIP percentile, average CiteScore percentile, and average SJR percentile of academics’ publications slightly decreased by 2.5%, 0.7%, and 1% respectively (Table 11).¹⁴ Conversely, the average Field-Weighted Citation Impact (FWCI) of their publications increased by 86.9%. In other words, the papers of academics published after they were awarded the title of professor received more citations than the papers that they published before, but their editorial placement slightly worsened. In particular, the share of Q1 publications decreased by 1.9% per SNIP, 3.2% per CiteScore, and 1.7% per SJR (Table 12).

Table 11 Impact indicators of publications authored by awarded professors

Table 12 Share of publications by awarded professors in the first quartile by journal impact indicators

Table 13 shows that: i) the average FWCI increased for both cohorts after the introduction of the publication requirements; ii) it increased slightly more for Cohort A; and iii) the FWCI score recorded for Cohort B is always higher (both before and after) than that recorded for Cohort A. As for the impact of journals, for SNIP there is an average percentile increase for both cohorts, slightly higher for Cohort A (+ 3.9) than for Cohort B (+ 1.3). Differently, for the other two indicators (Citescore and SJR), the average percentile changes are negative for Cohort A and practically nihil for Cohort B.

Table 13 Average impact indicators variations, by Cohort

The difference in differences statistical analysis

To assess the impact of the S&W publication requirements, announced in 2015 and become effective in 2016, on the intensity of publication and productivity of professors under analysis (RQ3), we adopted a difference in differences statistical technique. In particular, we conducted an ANOVA test to verify if researchers of Cohort B (subject to the incentive of increasing S&W publications to obtain professorship), increased the number of S&W publications more than professors of Cohort A. Depending on the year of promotion, Cohort B professors can be divided into three subsets, those awarded in 2016 (B|2016), those awarded in 2017 (B|2017), and finally those awarded in (B|2018). For Cohort B|2018, we tested differences vs Cohort A in publication variations between the periods 2016–2018 and 2013–2015; for Cohort B|2017, between the periods 2016–2017 and 2014–2015. For robustness reasons, we run no test for Cohort B|2016, because for them the periods of analysis should have been limited to one year only, 2016 vs 2015.

Table 14 shows the results of such two ANOVA tests and reveals that:

• All cohorts under analysis increased their S&W publications after the introduction of the publication requirements (F-values of the “period” variable, are high and statistically significant);

• The increases of Cohorts B|2017 and B|2018 were statistically higher than that of Cohort A.

Table 14 Two-way ANOVA tests for differences in the output of professors before and after the introduction of publication requirements, by cohort

Increasing research output does not entail necessarily increasing the scholarly outcome of research activities. We, therefore, repeated the same analysis by productivity. Results reported in Table 15 confirm an increase in productivity for all cohorts. The comparison between cohorts shows that the increases in productivity of Cohort B|2017 and B|2018 are higher than that of Cohort A, but statistical significance occurs only for Cohort B|2017 vs Cohort A.

Table 15 Two-way ANOVA tests for differences in productivity of professors before and after the introduction of publication requirements, by cohort

Discussion and conclusions

This study aimed to explore the effects of the introduction of S&W publication requirements for professorship in Ukraine on the research performance and publication behaviour of two cohorts of Ukrainian scholars.

The results of the study highlighted that the academics who became professors after the reform (Cohort B), at the time of promotion had a better scientific profile than the colleagues who were promoted before it (Cohort A). In fact, before obtaining the professorship, the FSS of Cohort B professors exceeded the FSS of Cohort A, overall and in every single discipline. Both Cohorts A and B, increased their average FSS in the three years after the award of professorship, in all disciplines but Physics (Cohort A), and Mathematics (Cohort B). Cohort B outperforms Cohort A also in the years after promotion, with the only exception of professors in Mathematics. Cohort B professors outperform Cohort A also along each component of FSS, i.e. output, fractional output, and average impact. The only exceptions for this last indicator occur for professors in Physics (before promotion), and Mathematics (after promotion). As such publication requirements had negative effects on academics from Physics, whose FSS before changes in publication requirements was significantly higher in comparison with academics from other disciplines, maybe indicating that in this discipline, Ukrainian academic community is already highly developed and internationalised.

The bandwagon effect of Cohort B performance on that of Cohort A shows in the reduction of the performance gap after promotion in all disciplines but Agricultural & biological sciences, Pharmacology, and Physics. We ascribe it to a natural tendency to emulate better performers operating in the same institutional environment.

The analysis at the individual level reveals that the overall performance improvement is due to a small share of professors, 34.9% of Cohort B (40.3% registered a decrease) and 30.8% of Cohort A (10.9% registered a decrease). The high share of Cohort B professors experiencing a decrease in performance suggests that to avoid professors resting on their laurels and stimulate continuous improvement, periodic monitoring of performance and incentive systems based on it, should be considered by policymakers and university managers. This findings resonate with Sasvari et al. (2022) regarding the performance of professors in Hungary. Moreover, 24.8% of Cohort B professors, although producing enough publications to pass the quantitative requirements for promotion, produced no scholarly impact with those publications (zero citations).

The statistical difference in differences tests revealed that in general the incentive to produce more S&W publications worked, but it did not always translate into higher research productivity which should be the ultimate aim of incentive schemes in research systems. Evidently, in several cases, the increase in S&W publications was obtained at the expense of research impact. It must be said that at the overall level, the average impact of publications authored by professors of the two cohorts increased following the awarding. Conversely, the editorial placement of the same publications slightly worsened.

We also found evidence that, before and after the introduction of S&W publication requirements, after being awarded the title of professor the production of book chapters and conference papers increased much more than the number of articles. This phenomenon too calls for the consideration of research evaluation policies involving the impact of research as the primary criterion for performance assessment.

These findings support the earlier studies on research-evaluation policies which indicate that in order to prevent academics from increasing the number of publications at the expense of research impact, evaluation policies should give weight to impactful journals (Bloch & Schneider, 2016; Korytkowski & Kulczycki, 2019).

Finally, after introducing the S&W publication requirements, the percentage of publications resulting from international collaborations registered a slight increase. Still, a significant shift to publishing mainly in truly international English journals occurred.

To conclude, we warn the reader that all assumptions and limits of the bibliometric analysis apply when interpreting the results. First, the new knowledge produced is not only embedded in publications, and the bibliographic repertories (such as Scopus, used here) do not register all publications. Furthermore, the measurement of the value of publications using citation-based indicators is a prediction, not definitive. Also, citations can also be negative or inappropriate; in any case they certify only scholarly impact, forgoing other types of impact.