Introduction

Like many research funders, the Academy of Finland (AKA) as a major Finnish public research funding agency is interested in the effects of its funding. As a partial measure for its goal of promoting high quality, renewal and impact of research, AKA uses a bibliometric indicator to compare the citation impact of AKA-funded publications to the citation impact of the entire Finland. This indicator has been in use since 2016 and the main elements of the method have been described by Auranen and Leino (2019). The indicator is called the Top10 index, which describes the unit of analysis’ relative share of the 10% most cited publications in the world, world average being 1 (Academy of Finland, 2020a). Web of Science (WoS) is used as a data source for calculating the Top10 index. Publications from two AKA’s funding instruments are included in the calculation of the indicator: Academy Projects and Academy Research Fellows. They represent the bottom-up funding instruments in AKA’s funding portfolio (Academy of Finland, 2020b, 2020c). Results show that AKA-funded publications have a higher Top10 index than publications from Finland on average (1.29 vs 1.09 in 2020). However, Top10 index for Academy Research Fellows is usually higher than for Academy Projects (1.38 vs 1.26 in 2020).

Web of Science allows international bench-marking. However, its main disadvantage is the narrow focus on peer-reviewed international journal articles. Comparisons with the comprehensive institutional CRIS data, which in some countries—including Finland—has been integrated at the national level, have shown that WoS and Scopus coverage is seriously lacking in the social sciences and the humanities (SSH) (Aksnes & Sivertsen, 2019; Pölönen et al., 2020b; Sivertsen, 2019; van Leeuwen et al., 2016). This is because in the SSH fields, local publication languages and books play an important role in the scholarly communication (Engels et al., 2018; Kulczycki et al., 2018, 2020). Due to lack of WoS coverage, AKA’s bibliometric indicator misses a large share of scientific publishing particularly in the SSH, as 41% of the peer-reviewed articles reported from the social sciences and 79% from the humanities are not included in the indicator (Auranen & Leino, 2019). In addition, especially in the SSH but also in other fields, dissemination of research knowledge within and beyond academia involves a broad range of publications that are not peer-reviewed (Hicks, 2004). Also these types of publications, which are highly relevant for the societal impact of research, are excluded from the WoS-based analyses but they are covered in the CRISs. In Finland, peer-reviewed and not-peer-reviewed publications are comprehensively covered in the national VIRTA publication information service, which integrates the publication metadata from the local CRISs of 13 Finnish universities (Pölönen, 2018).

One challenge in using the national publication data for assessing research performance, as opposed to WoS or Scopus, is the lack of citation data. In several countries, including Norway, Denmark and Finland, performance-based research funding systems (PRFS) use a comprehensive list of peer-reviewed publication channels as a quality-index (Hicks, 2012; Pölönen et al., 2020a; Sivertsen, 2016). In Finland, Publication Forum classification (in short, JUFO) has been developed by the Federation of Finnish Learned Societies since 2010 to support the Ministry of Education and Culture’s PRFS for allocating part of core-funding annually to universities. The main rationale is to reward universities not only based on quantity but also quality of output, namely publishing in channels that are valued by the scientific community, are demanding in terms of peer reviews, and reach the widest critical expert audience. Several Finnish universities have also used local CRIS data and JUFO levels, in addition to bibliometric citation analyses based on WoS or Scopus data, to inform expert-panels conducting institutional research assessments (Pölönen et al, 2021; Wang et al., 2014). So far, the national VIRTA data or JUFO classifications have not been used to assess the performance or broader scholarly communication practices of the AKA funded research.

In JUFO classification (Publication Forum, 2021), domestic and foreign peer-reviewed publication channels (journals and book publishers) are classified according to four levels (1 = basic, 2 = leading, 3 = top, 0 = other) by Finnish experts in the field (Auranen & Pölönen, 2012; Pölönen, 2018). The evaluation of channels is entrusted to 250 experts in 23 field-specific panels, who represent the Finnish research community. The experts’ main tasks are (1) to identify reliable peer-reviewed channels, and (2) indicate the leading channels of their field in terms of average quality, impact and prestige. JUFO evaluation is informed but not constrained by citation-based journal metrics, such as the Journal Impact Factor (JIF), and Source Normalized Impact per Paper (SNIP). In the SSH fields, also some of the national language journals and publishers without impact factors are included among the leading channels (Pölönen et al., 2021).

According to the international initiatives for responsible metrics, evaluation should take into consideration the disciplinary diversity and plurality of research outputs (https://sfdora.org; Hicks et al., 2015; Wilsdon et al., 2015). Also the Responsible Research and Innovation (RRI) and Open Science policies have called for a broader-based evaluation of research taking into account societal interaction and impact. In this study, we compare the research performance and scholarly communication profiles of the Finnish universities and the AKA-funded research based on comprehensive national publication data and the JUFO classification. We pose the following research questions:

  1. 1.

    How does AKA-funded research at Finnish universities perform in publishing compared with the entire research activity at Finnish universities? We analyse the research performance by means of the following analyses:

    1. 1.

      JUFO-levels: we use the share of peer-reviewed outputs published in journals, conferences and book publishers in JUFO levels 2 (“leading”) and 3 (“top”) as the indicator of publication performance.

    2. 2.

      Channel-based metrics: we compare results based on JUFO levels with those based on other indicators of channel quality or impact, namely the national level ratings produced by expert-panels in Norway and Denmark (Pölönen et al., 2020a; Sivertsen, 2016), as well as the Scopus-based 2019 journal metrics SNIP (Moed, 2010; Waltman et al., 2012) and CiteScore (Van Noorden, 2016).

    3. 3.

      Top10 index: we also compare the results based on JUFO levels with those based on the WoS-based Top10 index (the AKA bibliometric indicator).

  2. 2.

    What is the scholarly communication profile of AKA-funded research at Finnish universities in comparison with the entire publication output of Finnish universities? We compare the scholarly communication profiles according to the following indicators derived from VIRTA data across the main fields of science:

    1. 1.

      Science communication: share of not-peer-reviewed publications aimed at academic, professional and general audiences.

    2. 2.

      Bibliodiversity: share of peer-reviewed book publications (chapters, monographs and edited volumes) and conference articles.

    3. 3.

      Multilingualism: share of peer-reviewed publications in languages other than English (Finnish, Swedish and other languages).

    4. 4.

      Country of publisher: share of peer-reviewed publications in journals and books published in Finland.

    5. 5.

      Open access: share of peer-reviewed open access publications, including gold, hybrid and green OA.

    6. 6.

      Collaboration: share of peer-reviewed publications with co-authors, co-authors from more than one Finnish university, and co-authors affiliated with foreign institutions.

Previous research

There is an ample number of studies into the effects of research funding, both with regard to the effects of competitive research funding allocated by the various funding agencies and the effects of wider funding environments and systems, for example PRFS. On the latter, see for example Bloch and Schneider (2016), Sandström and Van den Besselaar (2018) and Schneider et al. (2015). However, of special interest for us is the research on the effects of competitive research funding which is typically granted by research councils or similar funding agencies. This research has concentrated on different types of effects of funding: publication output and citation impact, societal impact and benefits of research, and academic careers and practices of conducting research. Here we concentrate on the branches of research which are closest to our topic: effects of competitive research funding on publication output and citation impact, and effects on wider dissemination of research results.

Langfeldt et al. (2015) compared the recipients of Danish and Norwegian research grants to rejected applicants of the same grants. They found that the publication productivity of grant recipients increased more than that of rejected applicants when comparing productivity before and after the grant period. Grantees also published more highly cited papers. However, the increases in mean normalized citation scores were not significantly higher for grantees than for rejected applicants. Neufeld (2016) used a similar approach to compare the research performance of funded and non-funded applicants for a German DFG research grant before and after the funding decisions in biology and medicine. Small to moderate effects of funding on performance were found in biology, but not in medicine. Seus and Bührer (2017) studied the effects of the Austrian Science Fund’ START programme on grantees’ research performance and career development. Publication productivity of the grantees increased during and after the grant period and was higher than that of the control group. Grantees also increased their collaboration and collaborated more than the control group. Effect on citation rate was smaller. The career development of the START grantees was rather similar in comparison with other groups of researchers.

Not surprisingly, funding agencies themselves are interested in the effects of their funding and sometimes commission studies to investigate these effects. An example is an analysis by Robitaille et al. (2015) on the applicants of the European Research Council (ERC) grants. Analysis showed an increase in publication productivity of the grantees after the funding decision, as well as slight increases in citation impact. Interestingly, authors observe a decrease in citation impact among junior ERC-funded researchers after the funding decision. Among borderline applicants (those with lowest scores above the funding threshold and those with highest scores below the funding threshold), non-funded applicants typically had higher publication productivity and citation impact after the funding decision than funded applicants.

Heyard and Hottenrott (2021) expanded the scope of analysis from scientific effects of research funding to cover also the wider dissemination of research results by including altmetric indicators in their study of the applicants of the Swiss National Science Foundation funding. They found that SNSF funding increases the publication productivity and citation impact of the grantees compared to the researchers who did not receive the funding. Grantees also have higher altmetric scores than the others, indicating wider popular attention for their research. Larivière et al. (2015) report higher numbers of altmetric events (mentions in social media, blogs and news media) for funded applicants of ERC funding than for non-funded, although differences with regard to blogs and media were smaller than with regard to social media.

This literature review indicates that previous studies on the research performance of the researchers funded by competitive funding typically focus on publication and citation analyses based on Web of Science or Scopus data, although alternative data and methods, such as altmetrics, are also utilized. Citation databases have well-known limitations regarding different publication types and languages, and coverage of scholarly publishing. These limitations make a comprehensive analysis of research performance difficult. In addition, analyses based on citation databases usually don’t look at scholarly communication and science communication profiles. Comprehensive national publication data employed in our study can provide new insights into the effects of competitive research funding.

Data and methods

To investigate the research questions, we created a dataset based on three sources:

  1. 1.

    VIRTA publication data, consisting of 158,029 publications (publication year 2015–2018), validated by the 13 Finnish universities and reported annually to the Ministry of Education and Culture. The Ministry used this data to allocate 13% of core funding annually to universities in the performance-based research funding system. Publications with authors affiliated with more than one Finnish university figure in the data as duplicates.

  2. 2.

    AKA publication data, consisting of 7971 publications (publication year 2015–2018), which the PIs of the Academy Projects and Academy Research Fellows from the call years 2011–2013 have reported as outputs. Publication reported by more than one project or fellow figure in the data as duplicates. Since 2017, AKA-funded research outputs have been reported using VIRTA as one of the information sources.

  3. 3.

    JUFO publication channel classification, consisting of 31,597 journals and series, conferences and book publishers evaluated and rated according to quality, impact and prestige by national expert-panels. Information about the channel identified by unique JUFOID and the JUFO level is included in the VIRTA data for all peer-reviewed publications. In addition to JUFO-levels, the JUFO-register of publication channels (JUFO-portal: https://jfp.csc.fi/en/web/haku/julkaisukanavahaku) includes information on the Norwegian and Danish level ratings as well as Scopus-based SNIP and CiteScore indicators for journals and series with ISSN.

We matched AKA and VIRTA publications, and indicated in the VIRTA data which publications by the Finnish universities have been produced with funding for AKA Projects and Fellows (Table 1). We also deduplicated the VIRTA data to arrive at full-counts at national level. Our final dataset consists of 142,742 Finnish universities’ publications, of which 6143 (4%) are an AKA-funded subset. Dataset comprises only publications where at least one of the authors is affiliated at a university. It should be noted that while the Academy of Finland grants funding to researchers with various affiliations, appr. 80% of the funding is granted to researchers at universities (see Academy of Finland, 2020d).

Table 1 Finnish universities’ publications and Academy of Finland (AKA) funded publications 2015–2018 by scholarly status
Full size table

The identification of peer-review status, target audience, publication type, language, open access status, number of authors and international co-authorship of publications in VIRTA is based on researchers’ self-reports and/or validation by the data-collection personnel at the universities. Over two-thirds (69%) of all Finnish Universities’ publications reported in VIRTA are peer-reviewed scientific publications, including articles in journals, conferences and books, as well as monographs and edited volumes (Table 1). Almost one-third of the outputs are not peer-reviewed publications for academic, professional and general audiences. Of 6143 publications reported as outputs of AKA-funded research, the vast majority (89%) are peer-reviewed.

Of the AKA-funded publications, 4359 are reported as Projects’ outputs and 1896 as Fellows’ outputs (Table 2). This includes 112 publications that have been reported as outputs of both AKA-funded Projects and Fellows. All publication outputs are assigned to OECD FOS main fields based on the field-classification in the VIRTA data based on researchers’ self-reports. The share of Natural sciences and Engineering publications is larger, and that of SSH publications smaller, in the AKA-funded output compared to the universities’ entire output.

Table 2 Finnish universities’ publications and Academy of Finland (AKA) funded Projects’ and Fellows’ publications 2015–2018 by main field of science
Full size table

Results

Publication performance of AKA-funded research compared to Finnish universities’ entire peer-reviewed publication output in 2015–2018

In this part, we limit our analysis to 98,472 peer-reviewed publications, of which 5478 (5.6%) are AKA-funded research outputs, including 3892 (4%) outputs related to AKA Projects and 1681 (1.7%) to AKA Fellows.

1. JUFO-levels. Overall, both the universities’ peer-reviewed output and the AKA-funded research is published in channels in all JUFO level categories from 0 to 3 (Table 3). The AKA-funded research is, however, more strongly concentrated on channels the national expert-panels in Finland have rated as JUFO level 2 "leading" and 3 "top" channels. The share of JUFO level 2 and 3 publications is 45% for the AKA-funded research, compared to 32% for the Finnish universities in general (Fig. 1). Also, a larger share of publications by AKA-funded Fellows (48%) is on JUFO levels 2 and 3 than publications by AKA-funded Projects (43%). Our analysis also shows that AKA-funded research outperformed Finnish universities in all fields, and in case of each university’s peer-reviewed output (Fig. 2). These results are consonant with the WoS-based bibliometric analysis, as the AKA-funded research has a higher Top10 index than the Finnish research in general, and also the research by AKA Fellows has a higher Top10 index than AKA projects.

Table 3 Research performance: share of peer-reviewed publications of Finnish universities and Academy of Finland (AKA) funded research according to JUFO level of the publication channel
Full size table
Fig. 1
figure 1

Share of JUFO-level 2 and 3 channels of all peer-reviewed publications of Finnish universities and AKA-funded projects and fellows by the main field of science. (Agriculture is excluded from comparison of AKA-projects and AKA-fellows because the number of fellows’ outputs is less than 50). (Color figure online)

Full size image
Fig. 2
figure 2

Share of JUFO-level 2 and 3 channels of all peer-reviewed publications of 9 Finnish universities and AKA-funded projects and fellows (Lapland, Hanken, UNIArts and Vaasa are excluded because the number of AKA-funded outputs is less than 50). (Color figure online)

Full size image

2. Other channel-based metrics. In addition to JUFO-levels, we compared the research-performance of universities and AKA-funded research based on alternative channel-based quality and impact metrics: the Norwegian and Danish level ratings and Scopus-based impact metrics SNIP and CiteScore. Because all these metrics are available only for the journals and series with ISSN, we limited this analysis to 70,050 peer-reviewed journal articles. We calculated for the Finnish universities and AKA-funded research the share of articles in channels rated to level 2 in Norway or level 2 and 3 in Denmark, as well as the average SNIP and CiteScore of journals in which the articles were published in. It is important to notice that whereas JUFO-levels cover all 70,050 publications, the Norwegian level rating cover 89%, the Danish ratings 73%, SNIP 79% and CiteScore 83% of the Finnish universities’ journal articles. The coverage of these alternative channel-based indicators is weakest in the humanities (Norwegian ratings: 74%, Danish ratings: 55%, SNIP: 39% and CiteScore: 48%). Our analysis shows that all the channel-based indicators produce consonant results in the sense that AKA-funded research outperforms the universities’ total output, and AKA fellows outperform the AKA projects (Fig. 3).

Fig. 3
figure 3

Comparison of research performance of peer-reviewed journal articles of Finnish universities and Academy of Finland (AKA) funded research according to JUFO level, Norwegian level and Danish level, as well as the SNIP and CiteScore of the publication channel. Absolute values have been scaled based on universities’ research performance (universities = 1). (Color figure online)

Full size image

3. Top10 index. We also compared the results based on JUFO levels calculated for all 98,472 peer-reviewed publications, and those of the WoS based Top10 index calculated based on Web of Science data for all research with Finnish affiliations. Our analysis shows that the results are highly consonant, as the AKA-funded research outperform the universities’ total output, and AKA fellows the AKA projects (Fig. 4).

Fig. 4
figure 4

Comparison of Top10 index for Finnish research and share of JUFO levels 2 and 3 of Finnish universities peer-reviewed output as research performance indicators. Both indicators have been calculated including all peer-reviewed publication types: journal, conference and book articles, as well as monographs. (Color figure online)

Full size image

Publication profiles of AKA-funded research compared to Finnish universities’ entire publication output in 2015–2018

In this part, the first analysis of science communication includes all 142,742 outputs, including 44,270 not peer-reviewed publications. The rest of the analyses of bibliodiversity, multilingualism, country of publisher, open access and collaboration is limited to 98,472 peer-reviewed publications.

1. Science communication. According to our analysis, both Finnish universities and AKA-funded research (Fellows and Projects) use a broad range of publications, mostly peer-reviewed publications but also non-scholarly publications aimed at academic, professional and general audiences (Table 4). The AKA-funded research is, however, much more focused on peer-reviewed scholarly communication within academia. The share of not-peer-reviewed publications of the total output is 11% for the AKA-funded research compared to 31% in the case of Finnish universities. The share of not-peer-reviewed publications is much larger in the SSH fields in case of both AKA-funded research and Finnish universities’ output (Fig. 5).

Table 4 Scholarly communication profiles of Finnish universities and the AKA-funded research
Full size table
Fig. 5
figure 5

Scholarly communication profiles of Finnish universities and AKA-funded Projects and Fellows by main fields of science. (Color figure online)

Full size image

2. Bibliodiversity. Both Finnish universities and AKA-funded research use peer-reviewed journal, conference and book publications for scholarly communication (Table 4). Nevertheless, the AKA-funded research is more focused on using peer-reviewed journal articles. The share of articles in conferences and books, as well as monographs, of the total peer-reviewed output is 22% for the AKA-funded research compared to 29% in the case of Finnish universities. Both AKA-funded research and Finnish universities’ output shows traditional disciplinary differences in use of different publication types, especially in engineering (conferences) and the SSH (book publications). Interestingly, in the case of humanities the share of journal articles is slightly smaller for AKA-funded research than Finnish universities in general (Fig. 5).

3. Multilingualism. Both Finnish universities and AKA-funded research use multiple languages in peer-reviewed scholarly communication, including English, Finnish and Swedish (Finland’s two national languages), as well as other languages (Table 4). The AKA-funded research is, however, much more focused on English language publications. The share of peer-reviewed publications in languages other than English is 4,5% for the AKA-funded research compared to 11% in the case of Finnish universities’ entire peer-reviewed output. Despite the strong focus on English language publishing, the share of publications in languages other than English is much larger in the SSH fields in case of both AKA-funded and Finnish universities’ output (Fig. 5).

4. Country of publisher. Finnish universities and AKA-funded research is published with both Finnish and foreign publishers (Table 4). A much smaller share, however, of AKA-funded research (5.7%) than of the universities’ total output (11.9%) is published with the Finnish publishers (Fig. 5). As expected, in the case of both universities and AKA-funded research the share of outputs published with the Finnish publishers is larger in the SSH fields, in which a considerable share of original research is published in the national languages. We also looked at differences in shares of the 25 most frequently used publishers (Table 5). A larger share of AKA-funded research is published with the largest international publishers, such as Elsevier, Springer-Nature and Wiley. Taylor & Francis is a clear exception, perhaps because of the strong SSH portfolio. The one Finnish publisher of medical Finnish language journal Duodecim is used more frequently by universities in general. Of the new open access publishers PLOS and Frontiers are used more frequently, and MDPI less frequently, by AKA-funder research compared to the universities’ total output.

Table 5 25 most frequently used publishers and their share of peer-reviewed outputs of 14 Finnish universities and AKA-funded research (* = Finnish publishers)
Full size table

5. Open Access. Both Finnish universities and AKA-funded research use different routes, including gold, hybrid and green OA, to enable open access to peer-reviewed publication outputs (Table 4). Overall, the share of Open Access publications is larger in case of the AKA-funded research. While the share of outputs in gold OA channels is almost the same, AKA-funded research has a larger share of hybrid and green (self-archived) OA outputs. Overall, the share of Open Access peer-reviewed output is 53% for the AKA-funded research compared to 48% in the case of Finnish universities. There are some differences between fields in the overall Open Access share, however the advantage of AKA-funded research in share of OA outputs is stronger in medicine, agriculture and social sciences, and non-existent or modest in case of natural sciences, engineering and humanities (Fig. 5).

6. Collaboration. According to our analysis, both Finnish universities and AKA-funded research produce the vast majority of peer-reviewed publications in collaboration between two or more authors, who are often affiliated with other Finnish or foreign universities (Table 4). The AKA-funded research is, however, more focused on research collaboration. The share of co-authored publications of the total peer-reviewed output is 91% for the AKA-funded research compared to 82% in the case of Finnish universities. Also the share outputs produced in Finnish inter-university collaboration and international collaboration is larger for the AKA-funded research (23% and 50%, respectively) compared to Finnish universities in general (12% and 46%, respectively). Single-authorship is most common in the SSH fields, so the increased collaboration related to AKA-funded research is visible especially in these fields (Fig. 5). Perhaps surprisingly, the share of internationally co-authored publications is smaller for AKA-funded research in all fields except medicine.

Discussion and conclusions

Our literature review shows that earlier studies of research performance of competitive research funding typically focus on publication and citation analysis based on Web of Science and/or Scopus data. These data sources have well-known limitations regarding the different publication types and languages as well as the coverage of scholarly publishing especially in the social sciences and humanities. Some studies have also used altmetrics to study broader dissemination of results. Prior studies have not used comprehensive national publication data to investigate research performance or scholarly communication and science communication profiles.

In this study we first compared the research performance of AKA-funded research and research conducted in the Finnish universities in general. Instead of bibliometric citation analysis based on WoS data, we measured research performance by using comprehensive national publication data including all publication types and languages, and the Finnish national expert-based JUFO classification of journals and book publishers as a quality-index. A considerably larger share of AKA-funded research than universities total peer-reviewed output was published in JUFO-level 2 and 3 publication channels, and also AKA-fellows outperformed AKA projects. The same was observed across all fields and in the case of individual universities. In the case of peer-reviewed journal articles, we found that the JUFO-based results were highly consonant with those obtained by using as an indicator of research performance the share of outputs in the Norwegian level 2 or the Danish level 2 and 3 journals. Similar differences between output of universities and AKA-funded research in general, and AKA fellows and AKA projects in particular, appeared when using as measure of research performance the average SNIP or CiteScore of journals.

We also compared the JUFO-based results with the bibliometric indicator (Top-10 index) calculated for AKA based on WOS citation data. Indicator based on JUFO levels and the WoS based Top10 index produced again consonant results, as the AKA-funded research outperforms the baseline (Finland or Finnish universities) both in the citation impact as well as in the share of peer-reviewed outputs published in journals, conferences and book publishers the JUFO expert-panels have nominated as “leading” publication channels (levels 2 and 3). Similarly, AKA Fellows outperformed AKA Projects based on both indicators. JUFO-based indicator has the advantage of taking into account the disciplinary diversity and plurality of peer-reviewed research outputs.

The national publication data also provides added value by enabling assessment of the scholarly communication profile. We looked at publication profiles in the second part of this study, and discovered that both the Finnish universities and AKA-funded research take care of the responsibilities of science communication by disseminating research knowledge in not-peer-reviewed publications aimed at the academic, professional and general audiences. Both AKA-funded research and universities also show considerable bibliodiversity and multilingualism in peer-reviewed scholarly communication, and both show the same traditional field specific differences in target audiences, publication types, languages, country of publishing and collaboration. Nevertheless, the AKA-funded research is also more focused than the universities in general on using peer-reviewed publications, articles published in journals, English as publication language, and the largest international publishers. Also, a larger share of AKA funded research is Open Access, and published with new OA publishers, notably PLOS and Frontiers.

The Academy of Finland employs international peer-review to select most promising research for funding, with a goal of promoting high quality, renewal and impact of research. Furthermore, the AKA funding instruments included in our analysis are bottom-up funding opportunities with an emphasis on high quality of (basic) research, international collaboration and established position in the international scientific community. Funding for AKA Projects and Fellows is also rather selective, with success rates hovering between 12 and 15% in recent years (see Table 1 at Academy of Finland, 2020e). It is expected that AKA-funded research would have a strong scientific impact internationally, as indeed shown by Top10 index.

Strong emphasis on international impact and application success rates may explain our findings that AKA-funded research is published in leading journals and book publishers, and with relatively strong preference for English as a publication language and foreign publishers. AKA is interested in the societal impact of the research conducted by Projects and Fellows; information about this is requested both in the application and reporting phase of the funding process. However, our results indicate that in publishing, science communication is not a top priority for the Projects and Fellows. AKA funding criteria and policies do not privilege journal publishing as such over conference and book publishing, and indeed our findings suggest that also AKA-funded research follows traditional disciplinary patterns in scholarly communication and collaboration. As a cOAlition S member, AKA is strongly committed to Open Access to research published with its funding (see Academy of Finland, 2020f), and this policy readily explains the larger OA share of AKA-funded research, as well as increased publishing with some of the new OA publishers.

One of the limitations of our study is that we were not able to analytically examine the influence of disciplinary variations in comparison of AKA Top10 index and share of JUFO levels 2 and 3 outputs. The main obstacle is the limited number of outputs covered by WoS in some SSH fields, as well as the small number of AKA-funded outputs in VIRTA when differentiated according to the main field. Another challenge for this comparison is due to specific disciplinary grouping used in AKA’s WoS-based bibliometric analyses that is difficult to match to disciplinary classification used in VIRTA publication data.

Another possible limitation relates to reporting of outputs by researchers to institutional CRISs, from which VIRTA data is integrated, as well as reporting of outputs of AKA-funded research by the principal investigators (PI). In both cases, reporting may be less comprehensive in case of the other than peer-reviewed outputs, and there may be field-specific differences. In general, universities have a financial incentive to report outputs comprehensively, including publications for professional and general audiences as these are included among the Finnish Ministry of Education’s funding criteria since 2015. It is not clear, however, if PIs of the AKA-funded projects comprehensively report these types of publications, or if they tend to overreport the output of their projects. In addition, PIs may report only outputs published by the time they submit the final report. Reporting usually takes place several months after the termination of the project, but due to publication delays we have not been able to identify all AKA-funded outputs.

Future analyses could extend comparison to other funding instruments, of which funding programmes of the Strategic Research Council (STN)—hosted by AKA—would be of particular interest. This is because STN funding programmes emphasise the societal impact and interaction, which might be expected to result in somewhat different scholarly communication profiles compared to AKA funding instruments.

In all, we conclude that national VIRTA data and JUFO levels complement WoS based bibliometric analyses of research performance with a comprehensive coverage of publication output including all fields, publication types and languages. As an indicator of research performance, JUFO-levels do not directly measure the impact of outputs but produced similar results compared to other channel-based quality indicators (the Norwegian and Danish level ratings) and impact metrics (SNIP and CiteScore). It is important to notice that both citations and expert-based classifications provide a relatively narrow measure of research quality (Aksnes et al., 2019; Pölönen et al., 2020a). Especially from the perspective of the Finnish performance-based research funding system, in which also the quantity of universities’ entire publication output is considered, it is important that JUFO—unlike the other indicators considered—covers the entire publication output from all fields (Pölönen et al., 2021). In addition to PRFS, JUFO-levels and comprehensive national publication data offer a relevant information source for a responsible macro level assessment and monitoring of publication activity in several contexts in Finland. These include, for example, research assessment of organizations and research fields, or analyses of research funded via competitive mechanisms, such as the Academy of Finland funding. We want to emphasize, however, in accordance with the DORA declaration, that channel-based indicators, such as impact factors or level ratings for journals or book publishers, should not be used for assessing or comparing individual researchers.

Our study focused on the Finnish context, however our results suggest that CRIS data can be a useful resource for responsible assessment also in other countries. Hundreds of research performing and funding organisations have institutional CRISs, and also in several other countries institutional CRIS data has been integrated at national level (Sīle et al., 2017, 2018). Completeness, correctness, consistency, and timeliness are required of any publication data used for evaluative metrics, however CRIS data requires relatively strong effort to ensure sufficiently rich and structured metadata on outputs including all types and languages (Azeroual & Schöpfel, 2019; Sīle et al., 2019). Self-reporting of outputs by researchers and/or other personnel may result in less than optimal coverage, findability, traceability, comparability and reproducibility of CRIS data (Bosman & Kramer, 2019; van Leeuwen et al., 2016). Institutional incentives and requirements for reporting, as well as data-collection guidelines and communities for dedicated data-collection personnel, are important means to promote the quality and comprehensiveness of CRIS data (Pölönen et al., 2020b; Sivertsen, 2019).

As mentioned above, integration of CRIS data at national level is not something that is unique to Finland; in fact some countries, like Norway for example, predated Finland in this (Sīle et al., 2017; Sivertsen, 2019). Possibilities for producing uniform national level data on publications and corresponding CRIS depend on at least a couple of factors. There needs to be a motive for the government and research performing organizations (RPO) to create such systems and data. Often this motive stems from national higher education policy. In the case of Finland, a strong motive around 10 years ago was the goal to give publications a wider role as an indicator in the government’s core funding model for universities (Auranen & Pölönen, 2012; Pölönen et al., 2021). This required reliable and uniform publication metadata from universities that didn’t exist before the 2010s (Pölönen, 2018; Poropudas et al., 2007). Technical implementation for the integration of CRIS data must be provided by one or more organizations with capabilities to do so. All this tends to require sustained collaboration between RPOs and government agencies (for example the ministry responsible for higher education and science). Historically formed relations between RPOs and the government, such as the level of autonomy of the RPOs from the government or governance models used in science and higher education policy, as well as the sheer size of a country and its science and higher education system influence the possibilities to create national level integrated CRIS and CRIS data.

In this study, nationally integrated CRIS data enabled us to take the field variation in publication practices and locally relevant research into account, as well as to present a multidimensional assessment of research output including aspects of science communication, multilingualism, collaboration and open access. It can be considered a very important advancement toward more diverse and inclusive metrics for responsible assessment (as envisioned e.g., by European Commission, 2021). Yet, we were not able to include in our analysis non-traditional output types, such as preprints, datasets, software, teaching materials or artistic work. Therefore, we were not able to assess, for example, the integrity and transparency of the research process. When it comes to societal impact, we were able to count some impact-enhancing interactions, such as the number of publications targeted at professional and general audiences, but we could not trace impacts or measure citizen science. In VIRTA, outputs are at national level linked to organisations but not to individual researchers, so we were not able to investigate, for example, differences in performance or scholarly communication profiles based on gender or career-stage (in local CRISs the link between outputs and researchers exists). CRISs have a great potential for supporting responsible assessment with comprehensive, reliable, comparable, and transparent information on research activities, however they still need to be fully adapted for the documentation of open science practices (Mustajoki et al., 2021). To enhance international comparisons, we also suggest international integration of institutional and national CRIS data (Puuska et al, 2020; Sivertsen, 2019).