Introduction

The Open Science movement has gained special relevance in recent years, becoming a scientific practice in which diverse actors “can collaborate and contribute, where research data, lab notes and other research processes are freely available, under terms that enable reuse, redistribution and reproduction of the research and its underlying data and methods” (FOSTER, 2016). This paradigm shift in the way scientific knowledge is produced and released covers a wide scope. In response to the wide-reaching change, a variety of organisations has instituted numerous policies and strategies to promote and consolidate open science in recent years. The European Union has been one of the leading global promoters, and, as Anglada and Abadal (2018) have noted, its motivations for encouraging open science have been primarily of two kinds: political/social and scientific. Europe’s political and social motivations are based on its need to maintain and increase its levels of well-being, and to do that it needs a strong economy with research-led innovation at its core. As for the scientific motivations, the open science movement seeks to adjust scientific communication so that it can take advantage of all of today’s technological possibilities and disseminate scientific findings more effectively.

The European Commission’s adoption of open access in 2012 was one of its first steps in pursuit of open science. That beginning has since been consolidated in measures such as the requirement of open publication of the findings of publicly funded research (European Commission, 2016a). Initiatives to promote open science have also spread to the field of infrastructure, with initiatives like OpenAIRE, the creation of the European Open Science Cloud (EOSC) (European Commission, 2016b) and the development of the Open Science Monitor (European Commission, 2017).

Under the European Commission’s recommendation on access to and preservation of scientific information, numerous countries have rolled out policies along these same lines. Finland has its Open Science and Research Initiative (Finland, 2014; Forsström & Haataja, 2016); the Netherlands, the National Plan Open Science (van Wezenbeek et al., 2017). Portugal adopted and applied an open scientific policy (Portugal, 2016), France launched the French Plan for Open Science in 2018 (Plan, 2021), and Greece kicked off the National Open Science Plan for Greece in 2020 (Athanasiou et al., 2020). Spain has its 2017–2020 State Plan on Scientific and Technical Research and Innovation, which makes it mandatory for results and research data obtained under public funding to be open (Spain, 2017). Moreover, Spain’s new Act 17/2022 of 5 September on science (Boletín Oficial del Estado, 2022) lays particular stress on initiatives “to facilitate free access and management of data produced through research (open data), in accordance with international FAIR principles (Findability, Accessibility, Interoperability and Reusability), to develop open infrastructure and platforms, to foster the publication of scientific results in open access and civil society’s open participation in scientific processes”.

A great many non-European countries, like Canada and South Africa (OECD, 2015), are also making noteworthy efforts. In Latin America public policies to develop and promote open science have begun to arise in the last decade in countries like Argentina, Brazil, Chile, Colombia, Mexico, Peru and Uruguay (De Filippo & D’Onofrio, 2019).

While the most well-developed open science policies in most countries are those having to do with open access, the need to pursue fresh strategies in fields like research evaluation (development of new metrics, open evaluation, etc.) is becoming increasingly evident. The European Commission, among other organisations, has taken up the challenge in reports like Mutual Learning Exercise Open Science: Altmetrics and Rewards (European Commission, 2018), Evaluation of Research Careers Fully Acknowledging Open Science Practices (European Commission, 2017), Next-generation Metrics: Responsible Metrics and Evaluation for Open Science (Wilsdon et al., 2017), Indicator Frameworks for Fostering Open Knowledge Practices in Science and Scholarship (Wouters et al., 2019) and Open Science and Intellectual Property Rights (de la Cueva & Méndez, 2022).

Numerous factors have contributed to this paradigm shift associated with open science. Among them, the irruption of Web 2.0 and the resulting multiplication of informal communication channels, which has posed a challenge to analysts of scientific activity (Mohammadi & Thelwall, 2013). Also, bibliometrics’ hegemony over the analysis and evaluation of scientific and technological activity has come in for debate -in documents such as the Declaration on Research Assessment, or DORA (2012), Science in Transition (2013) and the Leiden Manifesto (Hicks et al., 2015)-, making it clear that traditional evaluation systems are proving obsolete in the face of the diverse scientific practices and the variety of communication channels that now exist. Critical voices have also said that traditional metrics focus on the evaluation of research results, instead of research processes. In this sense, as mentioned by Eva Méndez (2021), several authors have argued that we should focus more on data and code sharing, open access articles, compliance with methodological standards and dataset FAIRness. Several reports point in this direction, stating that, more than opening research results, open science implies that data are shared, findable, accessible, interoperable and reusable (Wilkinson et al., 2016). Another of the new challenges in relation to research results is the so-called “future of scientific communication”, which has to do with how the forms of scientific communication and the business models for funding science are changing. This shift has been addressed as a central issue and discussed by various national and international organisations (Guédon et al., 2019).

In this sense, bibliometrics’ traditional toolbox has been expanded for a little over a decade by the addition of altmetric indicators (Priem & Hemminger, 2010) to analyse scientific production’s impact on non-scientific audiences by studying information sources like social networks and the media. There has been a great deal of argument over these indicators’ possibilities for measuring the real impact of documents and authors (Martín-Martín et al., 2018; Neylon & Wu, 2009; Orduña-Malea et al., 2016) and the characteristics and scope of various platforms (Robinson-García et al., 2014; Torres-Salinas et al., 2013). The advantages and limitations of altmetric indicators have also been fully described in the scientific literature (Gumpenberger et al., 2016; Martín-Martín et al., 2018; Moed, 2017), as have the characteristics that may affect a document’s social impact (De Filippo & Serrano-López, 2018; Haustein et al., 2015; Robinson-Garcia et al., 2017) and the relationship between bibliometric and altmetric impact (Cabezas-Clavijo & Torres-Salinas, 2010; Costas et al., 2014; Eysenbach, 2011; Schloegl & Gorraiz, 2010; Serrano-López et al., 2017). Thus, the challenge has to do with the need to generate new metrics to analyse and evaluate scientific activity in the framework of open science.

Some countries’ practices for evaluating their own institutions reflect concern over and interest in evaluating R + D + i, as in the case of the United Kingdom’s well-known Research Assessment Exercise (RAE) and its current Research Excellence Framework (RAF) implemented in 2014, which shows how the importance of research’s impact (and not just its academic impact) has gradually changed the focus of evaluation.

Spanish R + D + i institutions have a range of evaluation mechanisms weighing factors like teaching, research and transfer. These mechanisms are embodied in the various nationwide, regional and institutional calls sent out for research proposals. While there are numerous calls targeting university teachers and researchers, they contain few if any criteria related with open science. Traditional publication and project participation remain the leading concerns, after which minor consideration is accorded to other activities, such as scientific dissemination (España, 2019) and the social influence of research.

Despite the importance of recognising, measuring and analysing activities related with open science, detection and evaluation are not simple matters. As Bornmann (2013) says, research’s social impact often takes years to become apparent, and in many cases, it is hard to identify the chain of cause and effect between research and its influence. Furthermore, sundry authors claim that expected social impact differs widely by research area: an engineer’s scientific work may be anticipated to have a different impact than the work of a sociologist or historian (Martin, 2011; Molas-Gallart et al., 2002). The same thing happens in fields like health and economics, whose expected social impacts may differ very much indeed. A single evaluation mechanism may therefore be untenable (Bornmann, 2013).

This makes evaluation difficult, especially in view of the variety of impacts that may arise, some of which are less than desirable or positive (e.g., from the short-term economic standpoint).

Furthermore, it is important to differentiate between research’s influence on social networks (measured in terms of mentions in sources and media as obtained from altmetric indicators) and more long-term social impact, which affects diverse actors. As other authors have mentioned (Bornmann, 2014), social impact measurements and altmetric measurements appear to be very different types of indicators, because the time scale used in altmetrics works with the time scales usually employed by current research evaluations, but it is no good for measuring research’s effects on society; that requires a more long-term perspective.

So, to bolster how social impact is evaluated (beyond simply counting up mentions on social networks), it might be interesting to look at new information sources that collect the documents with potential impact on society. One example studied in recent years is policy documents. Studies propose that the relationship between scientific publications and government publications (measured in terms of citation presence) may be a good indicator of how science is influencing the institutions that have decision-making power or an influence on society. Several areas of research have endeavoured to analyse how scientific knowledge flows from academia to policy documents. Bornmann et al. (2016) show how the influence of scientific knowledge about climate change on policy decisions has been researched within the environmental sciences. Newson et al. (2018) relate how numerous medical studies have analysed the interaction between scientific publications and clinical guidelines. Pinheiro et al. (2021) analyse the influence of interdisciplinary scientific studies on policy documents and the likelihood that these policy documents will have a greater influence on policy decisions. Lastly, in a recent study, Yin et al. (2022) found a high degree of alignment between publicly funded scientific research and public interest in certain problems addressed by science, showing how certain areas influence policy documents more than other areas do.

Analyses like these have traditionally been done by applying text-mining techniques to policy documents. The creation of the Altmetric.com tool was the first step toward the construction of databases of policy documents. In 2019 Overton (overton.io) arose, a database that in 2021 had over 30,000 policy documents from governments, think tanks, governmental organisations and ONGDs (Szomszor & Adie, 2022). As Szomszor and Adie say, Overton offers a set of documents that can be highly informative for reviewing scientific policies and assessing research and its impact in disciplines related with health, economics and the environment.

This possibility of analysing the relationship between the scientific world and the policy world offers a framework that can be usefully applied to the study of open science. Since the topic of open science has a great academic, social and political impact, we are interested in learning how scientific products related with open science feed back into the public policy sphere. Accordingly, we have identified the scientific activity of Spanish institutions in matters of open science as our object of study. In addition to identifying and analysing its characteristics, we are focusing especially on the influence of this scientific activity as a conceptual, methodological or empirical reference for the generation of national and international policy documents. We use knowledge flows as our framework of analysis. “Knowledge flows” is a concept developed in previous research (De Filippo & Serrano-López, 2018) that enables us to track scientific research from its origin in a research project (knowledge production phase) to the project’s output in the form of scientific publications (research output phase), and then to analyse the scientific publications’ influence in non-academic environments. In this sense, we see scientific projects and publications as an indication of the scientific community’s sensitivity toward or engagement with a specific set of problems, while public policies reflect political and social interest in a given topic.

In this paper we use the concept of public policies proposed by Oscar Oszlak and Guillermo O'Donnell (1981), who define public policies as a set of strategies and actions that express decisions taken simultaneously or successively over time by one or more nationwide organisations, constituting the state’s reaction to affairs that elicit the attention, interest or mobilisation of other civil society actors. Public policies on open science, therefore, refer to state strategies and actions (specifically, by the nationwide organisations in charge of creating and coordinating scientific and technological policies, nationwide funding agencies and national research councils) aimed at promoting the principles and practices of open science.

This analysis seeks to reach two complementary goals. First, we wish to analyse the main characteristics of scientific activity about open science. Second, we want to study how scientific activity about open science influences non-academic audiences in the particular realm of public policy.

To achieve these goals, a series of questions concerning both methodological and conceptual issues has been established:

  1. Q1.

    How can scientific activities about open science be identified?

  2. Q2.

    What are the characteristics of projects and publications about open science?

  3. Q3.

    What entities fund research into open science, and what extent does their funding cover?

  4. Q4.

    What are the characteristics of the papers about open science that are cited in policy documents?

  5. Q5.

    Who are the main non-academic users of research into open science?

These research questions form the central thread of this study. We shall return to them in the discussion section.

Sources

The following information sources were used for this study:

  • The European Commission’s CORDIS database, which provides information about the projects funded by the European Union’s research and innovation programmes (https://cordis.europa.eu/).

  • The Spanish Ministry of Science and Innovation’s website, which provides access to all calls for projects under the National Research Plan since 2000 and the related decisions (https://www.ciencia.gob.es/Convocatorias.html).

  • Clarivate Analytics’ Web of Science, for the detection and analysis of scientific publications. The Web of Science was accessed under the licence issued to the Spanish Foundation for Science and Technology (Fundación Española para la Ciencia y la Tecnología, FECYT).

  • The Overton database, to trace the relationships between policy documents and scientific documents through their citations. Since 2019 Overton has been storing information drawn from organisations like governments, IGOs (including intergovernmental organisations and international organisations), think tanks and ONGDs. Overton was accessed using an institutional account (for access to the information offered in the free version).

Methodology

The methodology was divided into phases. First, a search strategy was defined to identify projects and scientific publications about open science in which at least one Spanish institution participated. This keyword-based strategy was designed on the basis of the review of scientific literature and was tested in a number of previous studies and validated by experts in the field. Second, the Spanish and European projects about open science greenlighted between 2010 and 2020 were identified, and their identification codes were collected. Next, the scientific publications about open science were located using two methods: (a) through the previously defined search strategy and (b) through the publications stemming from projects about open science (by searching for target project’s code in the “Funding Acknowledgement” field of the Web of Science Core Collection).

Once all the projects and publications had been retrieved, the bibliometric indicators of scientific activity, specialisation, collaboration, funding, impact and visibility were obtained. The methodology followed in these stages can be seen in greater detail in De Filippo and Lascurain (2023). Lastly, the DOIs of the publications were extracted and entered in the Overton search engine. Thus, the policy documents citing the retrieved scientific publications were found (Fig. 1).

Fig. 1
figure 1

Steps taken to analyse activities concerning open science

Full size image

Results

The study of research projects about open science shows that there were 37 projects under national calls and 105 projects under European framework programmes. Publication analysis enabled 1491 publications about open science signed by Spanish institutions from 2011 to 2020 to be identified. Of these, 1124 publications were the result of national and international projects about open science (142 projects), and another 1128 were retrieved via the keyword-based search strategy. Ninety-five percent of all the publications (1,418 publications) had a DOI, and of these 15% (215 publications) were mentioned in policy documents. Figure 2 shows the volume of projects and publications about open science that were used in the study and the total number of times these publications were mentioned in policy documents.

Fig. 2
figure 2

Source Authors, based on De Filippo and Lascurain (2023)

Projects, publications and mentions in policy documents.

Full size image

The information gathered from each of the sources is presented below.

Publications and projects

Table 1 shows a summary of the results. It includes the results of the analysis of projects (project results) and the analysis of publications (publication results). As can be seen, there are major differences between domestic projects and European projects in terms of the amount of funding received and the quantity of participants (the European projects have much more funding and many more participants). These differences also influence project orientation; in domestic projects the number of participating institutions is lower, the participants are mainly universities, and the topics related with open science revolve around open data, open access, open education and citizen participation, primarily from a qualitative perspective. In the European projects, however, there is a larger technological component or large field studies, with participants from different institutional sectors and a larger number of countries. Projects about open innovation, open data and citizen science from a combined quantitative and qualitative perspective tend to be frequent among the European projects, as do large-scale studies. Accessibility is another of the interesting points, since information and results from European projects can be obtained not only from the projects’ own websites, but also from the CORDIS database.

Table 1 Main characteristics of projects and publications about open science
Full size table

Seventy-four percent of the scientific publications are led by an author belonging to a Spanish institution. Sixty-six percent of the publications about open science stem from projects funded by calls for domestic or international projects. The most frequent type of collaboration is collaboration with European or US institutions. Universities are the main author institutions, and the most frequent topics concern open data, open innovation, open access and citizen science. Interestingly, nearly 60% of the publications are in open access, primarily the green route, which enhances result visibility. Table 1 shows the results disaggregated into two groups: project results (both national and international projects) and publication results (scientific papers).

Mentions in policy documents

Out of all the documents with a DOI (1418 publications), the publications cited in policy documents were identified. The result was 215 publications, or 15% of all scientific publications on open science.

The publications mentioned in policy documents were published in 150 journals. Some of these journals (related with ecology and the environment) had high percentages of citations in policy documents; over 75% of the published documents were picked up in policy sources (Table 2).

Table 2 Journals with the highest number of documents cited in policy sources (> 2 docs cited)
Full size table

It was observed that 66% of the documents cited in policy documents were written by authors from Spanish centres in collaboration with foreign institutions. In addition, over half of the documents cited were published in journals in the first quartile (Q1) (Fig. 3).

Fig. 3
figure 3

Document distribution by quartile and by number of mentions

Full size image

These 215 publications received 544 mentions in policy documents, with a heterogeneous distribution. One publication with 20 mentions was found, although the vast majority of documents were mentioned only once. Figure 3 shows the distribution of publications with mentions, classed by publishing journal quartile and by number of mentions per scientific document.

Nearly 60% of the citing organisations are government organisations. The rest are primarily either IGOs (37%) or think tanks (31%). Another 7% are other kinds of institutions. Figure 4 shows the distribution in absolute values by type of citing institution. Since the same document may be cited more than once, the sum of the absolute values is greater than the total.

Fig. 4
figure 4

Types of organisations citing Spanish publications on open science

Full size image

The Publications Office of the European Union is the organisation with the largest number of cited documents (Table 3). Other major citing organisations include the Organisation for Economic Co-operation and Development (OECD), the APO (Analysis & Policy Observatory) open-access repository and the IPBES (Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services), an intergovernmental institution associated with the United Nations for the conservation of biodiversity.

Table 3 Citing organisations (> 4 docs)
Full size table

Relationships between variables

A series of variables was analysed to pinpoint the differences and similarities between publications cited in policy documents and publications not cited in policy documents. Table 4 shows that over half the documents mentioned were the product of funded research (in other words, 44% of the cited documents were not the outcome of funded projects). Moreover, the publications cited in policy documents were more generally published in open access than the documents not cited in policy documents. International collaboration and the percentage of first-quartile (Q1) publications were also higher in the cited documents (with statistically significant differences between the two groups).

Table 4 Comparison of cited and uncited documents
Full size table

When classed by publication topic area (according to the 250 WoS categories), the documents on open science fall mainly into four disciplines: Information Science and Library Science, Computer Science Information Systems, Management and Environmental Sciences. Documents on open science in disciplines related with physics, chemistry, the humanities and medical science, on the other hand, are few. The four disciplines with the largest volume of documents also have a considerable volume of mentions in policy documents. To reduce the volume effect, the activity index (AI) is calculated, which indicates the intensity of mentions in a given scientific area and therefore detects disciplines that have received an unexpectedly high number of mentions. The AI is defined as a discipline’s percentage of documents (cited in policy texts) versus that same discipline’s percentage of documents (on open science). When the AI is > 1, the mentions are understood to be higher than expected in the light of the initial volume of documents. We find that publications on Economics and Environmental publications are mentioned most frequently in policy documents. Figure 5 shows the activity index calculated for each discipline (only disciplines with more than five cited documents are shown). The category of Economics can be seen to have been cited nearly five times more than expected (1.17% of Spain’s publications on open science belong to the Economics category, vs 5.6% of mentions in policy documents). The same situation is observed in Ecology and other disciplines related with the environment, which have high AIs. This indicator is interesting, because it shows normalised values, not just absolute figures.

Fig. 5
figure 5

Topics of the journals most cited in policy documents (> 5 docs cited)

Full size image

Among the organisations and countries funding scientific publications, the European Union plays a decisive role in the funding of research related with open science (it is responsible for 14% of the publications on the topic) and its subsequent use in policy documents created by the EU’s various science and innovation committees. Thirty percent of the mentions come from the Office of Publications of the EU.

The Spanish government funded 36% of the publications on open science through nationwide and regional competitive calls for projects, but it used a very small number of articles (5%) as input for its policy documents. Governmental (multinational) organisations were responsible for quite a small percentage of funding; they funded under 1% of the publications on open science. However, they were the organisations that cited the most documents in their reports (38% of the mentions). Countries like the United Kingdom, the USA, Germany, Australia, Canada, Belgium and France numbered among the foremost citers, although their proportions of funding for open science research signed by Spanish centres was very low. Figure 6 shows the main open science publication funding and citing countries.

Fig. 6
figure 6

Countries funding and citing publications on open science

Full size image

A flow diagram (Fig. 7) has been prepared to illustrate the relationship between funder organisations and users of publications on open science. It shows how the number of documents from funded research is nearly double the number of documents produced without funding. The important presence of government funding in the target publications is also quite apparent. Over 55% of the publications mentioned in policy documents come from funded research (mostly funded with government money). Interestingly, the figure helps visualise how not only national and regional governments themselves but also IGOs draw upon scientific publications, illustrating the dynamics of the flow of knowledge from its origin to its final “use” in policy documents.

Fig. 7
figure 7

Relationship between funding and citing organisations of publications on open science

Full size image

Discussion

In the light of the study, a series of aspects has been identified that will help increase our knowledge of activities related with open science and their impact on the sphere of public policies. These findings are outlined below, and their scope and limitations are discussed in the attempt to find answers to the research questions steering this paper.

In connection with Q1, “How can scientific activities about open science be identified”, a methodology is proposed in this study that finds the relationships among several aspects enabling the activities linked with open science to be “measured”. Although the range of actions related with open science is extremely wide, we have stressed the analysis of elements that we regard as key in the process of producing and disseminating scientific knowledge: research projects, publications and the social influence of research. In this sense, analysis of research projects has proved to be a valuable focus for finding information about lines of study proposed by researchers and accepted by funding organisations and to learn about significant aspects of research in progress (Plaza, 2001). Previous studies have been found along these same lines analysing the results of the Third Framework Programme and the Fourth Framework Programme (Arnold et al., 2005), as well as more recent studies focusing on the Seventh Framework Programme to report on the progress made in certain scientific fields or the role played by funding (De Filippo et al.,; 2020; Edler & James, 2015; Haanstra et al., 2016; Pohoryles, 2014). Furthermore, scientific communication has traditionally been the main means whereby scientists and academics have shared new knowledge, especially through scientific publications. Analysis of scientific publications has enabled us to study scientific productivity (of researchers, groups, institutions, disciplines and countries) and its impact on the scientific community itself. However, other, more innovative applications have arisen as well, such as the detection of new research fronts and emerging fields, the analysis of topics of interest in the various disciplines, the study of the make-up of collaborating networks of institutional actors and the identification of research niches. In recent years funding agencies have shown growing interest in ascertaining the impact of the projects they fund, and bibliometrics has proved to be a rather good tool for that, despite the difficulties involved in a bibliometric approach (Costas & Van Leeuwen, 2012; Wang & Shapira, 2011). In this sense, analysis of the publications stemming from projects (examination of the funding information appearing in papers) is an interesting means of helping us learn about the impact of the activity taking place in a given scientific field (Paul-Hus et al., 2016), and it may prove useful for measuring and evaluating the impact of funding organisations (Rigby, 2013).

The last component analysed is social influence, understood as the impact of research outside the purely academic realm. A great deal of research on this topic has explored different types of “impacts”; in this paper we chose to focus on the policy sphere. Our choice is directly linked to the “research flow” approach we decided to take, in which we look at the different points or stages of scientific research. Thus, we set out to learn about the origins of research, its results and dissemination and its “use” by other social actors. In short, our goal was to explore a continuous process, since policy documents (which cite papers) lay down the rules for the implementation of future guidelines that will steer agencies and indicate the paths of future research.

Although we consider that the methodology (in both its conceptual and its technical aspects) may be of interest for other researchers, we are aware of the limitations we are facing. One of the study’s methodological limitations is the fact that we are examining scientific activity through its formal written results. Realms like open science, however, contain many invisible practices that cannot be grasped through quantitative analysis, like citizen science actions, the development of open-code software and free hardware projects, and incipient experiments in the open evaluation of research work. These and many other practices are being conducted in Spain and highlight a valuable aspect of the dynamics of open, collaborative science (De Filippo & D’Onofrio, 2019; Fressoli & De Filippo, 2021). Lastly, the limitations and biases of the sources we used influence the results as well, so we must assume that this research, like all research, is a slice of reality that furnishes us with approximate information about a given situation.

With respect to the question “What are the characteristics of projects and publications about open science?” (Q2), our results show that it is important for Spain to participate in European projects, both in the role of partner and in the role of coordinator, and that a growing amount of funding is being provided through calls for projects. Spain participated in over 100 projects related with open science in the last two calls under the Framework Programme, and it led 55% of the projects in the Seventh Framework Programme and 44% of the projects in the Horizon 2020 programme. Topics related with open data and open access predominate, although considerable differences have been found between domestic and European projects. Universities are the majority beneficiary institutions of domestic projects, which are mainly qualitative. In the case of European projects, the range of institutions widens, and this contributes to the development of technological projects and projects of greater scope. The institutions that participate most often are polytechnic universities and R + D + i centres.

In the case of scientific publications, according to data from Clarivate Analytics (2022), over 18,000 documents about open science were published in the Web of Science database between 2010 and 2020. Six percent of that production was by authors belonging to Spanish centres. This figure rose from 3% in 2011 to 7% in 2020, underscoring the growing interest in research into open science. Previous studies of Spanish scientific output on open science in SCOPUS have also made clear the growth of production on the topic during the last decade, coinciding with the launch of numerous European and Iberian initiatives and policies focusing on promoting and consolidating the Open Science movement (De Filippo et al., 2019).

Spain collaborates primarily with other European and U.S. centres, mainly universities and research centres. Publication availability is an interesting point; it has been shown that 58% of the 1491 documents analysed here were available in open access, especially through the green route. Computer Science and Social Science were identified as the topics most strongly related with open science. This is logical, since many of the publications address the technical concerns involved in open access to information, the interconnection of different data sources, and so on. Furthermore, publications on issues affecting society, such as citizen participation in new knowledge production processes, information transparency and changes in communication, are also important. Documents on open science are thus found to specialise in certain topics, and this finding agrees with previous studies performed in different contexts (Bautista-Puig et al., 2019; De Filippo et al., 2019).

It was also verified that universities are the institutions that produce the most documents about open science, especially universities that have implemented strategies to promote scientific openness, primarily through the creation of institutional repositories, projects about Open Science, participation in networks for the development of OA, etc.

In connection with Q3, “What entities fund research into open science, and what extent does their funding cover”, it was found that public funding plays a central role in research project development. This agrees with the observations of other studies, such as the work by Beaudry and Allaoui (2012), which demonstrates that factors like public funding, patenting capacity and collaboration among researchers significantly increased scientific production in the nanotechnology area in Canada, an especially sensitive, funding-dependent area due to the high costs of its infrastructure. Furthermore, Payne and Siow (2003) looked at the impact of public funding on 73 universities in the United States and found that an investment of an additional one million dollars results in the production of 11 to 18 additional scientific publications. Other researchers, among them Costas and van Leeuwen (2012), carried out a study of all the publications on the Web of Science that contain funder acknowledgements and discovered that these studies have much more impact than unfunded publications, and that their visibility varies a great deal depending on knowledge area and country. Similarly, Álvarez-Bornstein and Bordons (2021) observed that, in the case of Spain, funded publications are published in journals having higher prestige and receive a higher number of citations.

Given the methodology followed in our work, all the projects we analysed had funding from national or European governments, since that was the initial selection requirement. Sixty-six percent of the publications were the result of funded projects, mainly funded by government organisations (87% of the funded publications). It was observed that, among the publications stemming from projects funded by government organisations, 61% of the underlying projects were funded by the government of Spain through calls for projects such as Spain’s National Research Plans (nationwide annual calls) or calls issued by autonomous communities (regional governments). Another 25% were calls under the European Union’s Framework Programme, and 4% were calls issued by the USA. European calls for projects in recent years have contained specific lines devoted to open science and citizen science, such as the SWAFT (Science With And For Society) calls of the Horizon 2020 programme. This has resulted in the launching of numerous research projects in this field and a rise in publications on the topic. However, it is important to bear in mind that the strategic lines of each funding programme set conditions that the scientific output of the funded research must meet. The European Union, for instance, especially stresses that the funded research must have social impact, so the projects it funds are expected to produce not only scientific publications (Wang et al., 2020).

This majority prevalence of public and government funding seems to be the norm, especially in research fields like the social sciences. In open science the relationship between funding organisations and the government entities in charge of creating public policy tends to be very close, since they are often the same institutions (those that lay down the lines of policy also fund what is considered priority research). So, we can see how, in various international experiences, public policies on open science were initially oriented toward the need to disseminate—through open access—the results of research, on the basis of recognition that the findings of publicly funded research ought to be unconstrainedly available to the public. The scope of open science policies has now broadened, so that we can find national policies that encourage a diversity of open science practices, as well as specific provisions in laws, regulations and directives (European Commission, 2017). Funding organisations have thus taken on a dominant role in motivating research oriented toward the challenges of society (Braun, 1998; Yin et al, 2022) and are some of the main supporters of the open science model.

With respect to the question “What are the characteristics of the papers about open science that are cited in policy documents?” (Q4), we have observed that 15% of scientific articles are cited in policy documents.

These documents were published in a total of 150 journals, some of which contained high percentages of articles mentioned in policy documents. More than half of the publications cited in policy documents were published in journals in the first quartile of their area. This finding agrees with those of other researchers (De-Filippo & Serrano-López, 2018) who say that one of the most significant factors for research results to have an impact on social networks, the media and policy documents is the prestige and reputation of the journal that publishes them. It could be interesting to look into the role of scientific journals themselves as agents that leverage and promote the research they publish, since the most influential journals usually have press offices that help them manage their media communications, public relations and visibility (Elías-Pérez, 2008; Franzen, 2012). It may also prove interesting to examine the role researchers themselves play in their papers’ presence on social networks and in the media, since the vast majority of researchers see Twitter as a useful instrument for improving scientific communication and reaching audiences outside their own academic sphere (Alonso-Flores et al., 2020). This fact may explain how documents published in Q1 journals have more visibility and therefore more possibilities of being cited in policy documents, and again it is related with the Overton database’s own characteristics and the source coverage it provides (Szomszor & Adie, 2022).

It was also observed that 61% of the publications cited in policy documents were published in open access. This is slightly higher than the figure for the publications that were not mentioned in policy documents (58%). The comparison may be considered in close relationship with the considerations mentioned above, since accessibility is another means of increasing visibility that may contribute positively to research’s impact in non-academic realms, as shown by earlier studies (Bruns & Stieglitz, 2012; De Filippo et al., 2019). Cited documents also have higher proportions of international collaboration (66% vs 46%), and that too may be a factor that facilitates visibility and impact (Adams et al., 2005; Bordons et al., 1993; Katz & Martin, 1997).

Furthermore, the topics garnering the most mentions in policy documents were Environmental Sciences, Computer Science and Library and Information Sciences. These data are in line with the topics covered by the publications. First of all, Environmental Sciences contains numerous papers touching on citizen science, where scientific knowledge is gained in a collaborative arrangement between scientists and citizens who have ties to the environment being analysed (Pelacho et al., 2021). In Computer Science great interest is observed in topics related with data processing (FAIR data) and the design of open digital infrastructure to facilitate information sharing. These aspects of open science are a legacy from open source movements pushing for free software, and they are still very important in this community today (Mombach et al., 2018). Lastly, the area of Library and Information Sciences is especially sensitised to open science, because open science is a cross-cutting issue there. Examples include open peer review, the importance of citizen participation in processes producing new knowledge (citizen science), open access to information, information transparency and scientific communication outside academia.

However, looking beyond the absolute values, it has been observed that, relatively speaking (considering the disciplines’ activity index), the Economics area has been cited five times more than expected, far above the rate shown by the rest of the areas. At the same time, the areas related with Environmental Sciences also have activity indices of above 1. This agrees with the observations shared by Szomszor and Adie (2022) about the preponderance of citations in areas of social, economic and environmental sciences in the Overton database. As a result, we must consider the possibility of running future studies to find out if the influence of the Economics area is due to overrepresentation in the database or effective influence on policy documents.

Lastly, in reply to question Q5, “Who are the main non-academic users of research into open science”, the results show that the main countries that cite Spanish scientific publications are the global leaders in scientific knowledge production (the USA, the UK, Germany, Australia, Canada, Belgium and France, for example), not counting China. In addition, multinational organisations like the European Union and the OECD are responsible for a large quantity of policy documents that cite scientific knowledge related with open science. Strikingly, Spain is ranked eighth (behind France) by volume of mentions in policy documents, while it is the main funder of our target publications and the other countries contribute very little to their funding. This difference in ranks between funding volume and document use may be related with the biases built into the database we used and the national sources Overton includes. As stated by its developers, the country most strongly represented in Overton is the USA, with two million documents from 359 institutions, followed by the United Kingdom and Canada (257,641 documents from 253 institutions and 497,242 documents from 52 institutions, respectively). International organisations as a whole (which includes institutions like the World Bank and UNESCO) have 495,559 documents and 78 institutions. Other countries and institutions, on the other hand, have no more than 200,000 documents. For Spain the database holds 49,547 documents from 9 institutions. It is true, however, that the European Union has good coverage in the database (Szomszor & Adie, 2022).

Governments themselves are the main consumers of scientific results, since half of all the mentions in policy documents come from this institutional sector. IGOs are responsible for 37% of the documents citing scientific publications, and think tanks, 19%.

Conclusions

The study has enabled us to verify the importance of open science activities for the Spanish scientific community, as attested by the number of projects (mainly European projects) led by or participated in by Spanish institutions and the surge in international publications about open science in recent years. We observe that research results related to open science tend to be closely related to disciplines such as Computer Sciences and Social Sciences. About 15% of the papers on open science have been cited in policy documents and share a set of specific characteristics: they have high proportions of open access, international collaboration and inclusion in high-prestige journals. Within open science production, publications in disciplines such as Economics and environmental disciplines are found to have had a particular impact on policy documents.

Among the most outstanding findings, public funding agencies have been identified as playing a central role in the scientific production process. The results show, firstly, that there is a knowledge flow feedback such that the public institutions funding research related with open science are making use of that same research in their institutional reports. Furthermore, these scientific publications may be influencing the process whereby the institutions and governments using them forge policies. Future research might seek to discover if the presence of scientific citations in institutional reports is really in line with the implementation of public policies or whether, on the contrary, scientific findings are mentioned just for show and are never actually implemented through policy. Moreover, think tanks exert an undeniable influence on the policymaking process (Fraussen & Halpin, 2017), and their presence is evidence that the open science movement has permeated other actors of the political ecosystem that strongly sway political agendas yet are independent of public institutions proper. The open science movement has therefore crossed the bounds of academia and impacted audiences that bear responsibilities in the policy-making process, and this achievement stresses the structural, transformative nature of open science.

To wind up, we would like to underline the fact that public policies on open science, which in Europe have been rolled out primarily in the last decade, have had a positive impact on the research done at Spanish centres, and there has been an increase in the number of projects and publications. This has been possible largely thanks to economic backing from the European Union, but mainly due to support from the government of Spain, which has recognised the need to support the development of open science. In order to implement open science right across the board, it is definitely fundamental to keep running continuous diagnostics of the activities done and to keep public policies adjusted accordingly. Analysis of how scientific knowledge is fed back into funding organisations enables the academic and social impacts of research to be assessed, and these impacts are fundamental as an indicator of interest in and development of a topic by scientists and society as a whole. This line has opened up new research outlooks for us, and we are working on them to develop new metrics for responsible, inclusive evaluation in the context of open science.