Introduction

Academic research grants account for billions of pounds in many countries and so the funders may naturally want to assess their value for money in the sense of financing desirable outcomes at a reasonable cost (Raftery et al., 2016). Since many of the benefits of research are long term and difficult to identify or quantify financially, it is common to benchmark against previous results or other funders to judge progress and efficiency. This is a complex task because academic funding has many small and large variations and is influenced by, and may influence, many aspects of the work and environment of the funded academics (e.g., Reale et al., 2017). The goal of this article is to support future analyses of the effectiveness or influence of grant funding by providing a typology of the important dimensions to be considered in evaluations (or otherwise acknowledged as limitations). The focus is on grant funding rather than block funding.

The ideal way to assess the value of a funding scheme would be a counterfactual analyses that showed its contribution by identifying what would have happened without the funding. Unfortunately, counterfactual analyses are usually impossible because of the large number of alternative funding sources. Similarly, comparisons between successful and unsuccessful bidders are faced with major confounding factors that include groups not winning one grant winning another (Neufeld, 2016), and complex research projects attracting funding of different kinds from multiple sources (Langfeldt et al., 2015; Rigby, 2011). Even analyses with effective control groups, such as a study of funded vs. unfunded postdocs (Schneider & van Leeuwen, 2014), cannot separate the effect of the funding from the success of the grant selection process: were better projects funded or did the funding or reviewer feedback improve the projects? Although qualitative analyses of individual projects help to explain what happened to the money and what it achieved, large scale analyses are sometimes needed to inform management decision making. For example: would a funder get more value for money from larger or smaller, longer or shorter, more specific or more general grants? For such analyses, many simplifying assumptions need to be made. The same is true for checks of the peer review process of research funders. For example, a funder might compute the average citation impact of publications produced by their grants and compare it to a reference set. This reference set might be as outputs from the rejected set or outputs from a comparable funder. The selection of the reference set is crucial for any attempt to identify the added value of any funding, however defined. For example, comparing the work of grant winners with that of high-quality unsuccessful applicants (e.g., those that just failed to be funded) would be useful to detect the added value of the money rather than the success of the procedure to select winners, assuming that there is little difference in potential between winners and narrow losers (Van den Besselaar & Leydesdorff, 2009). Because of the need to make comparisons between groups of outputs based on the nature of their funding, it is important to know the major variations in academic research funding types.

The dimensions of funding analysed in previous evaluations can point to how the above issues have been tackled. Unfortunately, most evaluations of the effectiveness, influence, or products of research funding (however defined) have probably been private reports for or by research funders, but some are in the public domain. Two non-funder studies have analysed whether funding improves research in specific contexts: peer review scores for Scoliosis conference submissions (Roach et al., 2008), and the methods of randomised controlled trials in urogynecology (Kim et al., 2018). Another compared research funded by China with that funded by the EU (Wang et al., 2020). An interesting view on the effect of funding on research output suggests that a grant does not necessarily always result in increased research output compared to participation in a grant competition (Ayoubi et al., 2019; Jonkers et al., 2017). Finally, a science-wide study of funding for journal articles from the UK suggested that it associated with higher quality research in at least some and possibly all fields (the last figure in: Thelwall et al., 2023).

From a different perspective, at least two studies have investigated whether academic funding has commercial value. The UK Medical Research Council (MRC) has analysed whether medical spinouts fared better if they were from teams that received MRC funding rather than from unsuccessful applicants, suggesting that funding helped spin-outs to realise commercial value from their health innovations (Annex A2.7 of: MRC, 2019). Also in the UK, firms participating in UK research council funded projects tended to grow faster afterwards compared to comparator firms (ERC, 2017).

Discussing the main variations in academic research funding types to inform analyses of the value of research funding is the purpose of the current article. Few prior studies seem to have introduced any systematic attempt to characterise the key dimensions of research funding, although some have listed several different types (e.g., four in: Garrett-Jones, 2000; three in: Paulson et al., 2011; nine in: Versleijen et al., 2007). The focus of the current paper is on grant-funded research conducted at least partly by people employed by an academic institution rather than by people researching as part of their job in a business, government, or other non-academic organisation. The latter are presumably funded usually by their employer, although they may sometimes conduct collaborative projects with academics or win academic research funding. The focus is also on research outputs, such as journal articles, books, patents, performances, or inventions, rather than research impacts or knowledge generation. Nevertheless, many of the options apply to the more general case. The list of dimensions relevant to evaluating the value of research funding has been constructed from a literature review of academic research about funding and insights from discussions with funders and analyses of funding records. The influence of funding on individual research projects is analysed, rather than systematic effects of funding, such as at the national level (e.g., for this, see: Sandström & Van den Besselaar, 2018; Van den Besselaar & Sandström, 2015). The next sections discuss dimensions in difference in the funding awarded, the influence of the funding on the research, and the way in which the funding is recorded.

Funding sources

There are many types of funders of academic research (Hu, 2009). An effort to distinguish between types of funding schemes based on a detailed analysis of the Dutch government budget and the annual reports of the main research funders in the Netherlands found the following nine types of funding instruments (Versleijen et al., 2007), but the remainder of this section gives finer-grained breakdown of types. The current paper is primarily concerned with all these except for the basic funding category, which includes the block grants that many universities receive for general research support. Block grants were originally uncompetitive but now may also be fully competitive, as in the UK where they depend on Research Excellence Framework scores, or partly competitive as in the Netherlands, where they partly depend on performance-based parameters like PhD completions (see also: Jonkers & Zacharewicz, 2016).

  • Contract research (project—targeted—small scale)

  • Open competition (project—free—small scale)

  • Thematic competition (project—targeted—small scale)

  • Competition between consortia (project—targeted—large scale)

  • Mission oriented basic funding (basic—targeted—large scale)

  • Funding of infrastructure and equipment (basic—targeted—diverse)

  • Basic funding for universities and public research institutes (basic—free—large scale)

  • International funding of programs and institutes (basic, both, mainly large scale)

  • EU funding (which can be subdivided in the previous eight types)

Many studies of the influence of research funding have focused on individual funders (Thelwall et al, 2016) and funding agencies’ (frequently unpublished) internal analyses presumably often compare between their own funding schemes, compare overall against a world benchmark, or check whether a funding scheme performance has changed over time (BHF, 2022). Public evaluations sometimes analyse individual funding schemes, particularly for large funders (e.g., Defazio et al., 2009). The source of funding for a project could be the employing academic institution, academic research funders, or other organisations that sometimes fund research. There are slightly different sets of possibilities for equipment and time funding.

Who funded the research project (type of funder)?

A researcher may be funded by their employer, a specialist research funding organisation (e.g., government-sponsored or non-profit) or an organisation that needs the research. Commercial funding seems likely to have different requirements and goals from academic funding (Kang & Motohashi, 2020), such as a closer focus on product or service development, different accounting rules, and confidentiality agreements. The source of funding is an important factor in funding analysis because funders have different selection criteria and methods to allocate and monitor funding. This is a non-exhaustive list.

  1. 1.

    Self-funded or completely unfunded (individual). Although the focus of this paper is on grant funding, this (and the item below) may be useful to record because it may partly underpin projects with other sources and may form parts of comparator sets (e.g., for the research of unfunded highly qualified applicants) in other contexts.

  2. 2.

    University employer. This includes funding reallocated from national competitive (e.g., performance-based research funding: Hicks, 2012) or non-competitive block research grants, from teaching income, investments and other sources that are allocated for research in general rather than equipment, time, or specific projects.

  3. 3.

    Other university (e.g., as a visiting researcher on a collaborative project).

  4. 4.

    National academic research funder (e.g., the UK’s Economic and Social Research Council: ESRC).

  5. 5.

    International academic research funder (e.g., European Union grants).

  6. 6.

    Government (contract, generally based on a tender and not from a pot of academic research funding)

  7. 7.

    Commercial (contract or research funding), sometimes called industry funding.

  8. 8.

    NGO (contract or research funding, e.g., Cancer Research charity). Philanthropic organisations not responsible to donors may have different motivations to charities, so it may be useful to separate the two sometimes.

Who funded the time needed for the research?

Research typically needs both people and equipment, and these two are sometimes supported separately. The funding for a researcher, if any, might be generic and implicit (it is part of their job to do research) or explicit in terms of a specified project that needs to be completed. Clinicians can have protected research time too: days that are reserved for research activities as part of their employment, including during advanced training (e.g., Elkbuli et al., 2020; Voss et al., 2021). For academics, research time is sometimes “borrowed” from teaching time (Bernardin, 1996; Olive, 2017). Time for a project may well be funded differently between members, such as the lead researcher being institutionally supported but using a grant to hire a team of academic and support staff. Inter-institutional research may also have a source for each team. The following list covers a range of different common arrangements.

  1. 1.

    Independent researcher, own time (e.g., not employed by but emeritus or affiliated with a university).

  2. 2.

    University researcher, own time (e.g., holidays, evenings, weekends).

  3. 3.

    University, percentage of the working time of academic staff devoted for research. In some countries this is large related to the amount of block finding versus project funding (Sandström & Van den Besselaar, 2018).

  4. 4.

    University, time borrowed from other activities (e.g., teaching, clinical duties, law practice).

  5. 5.

    Funder, generic research time funding (e.g., Gates chair of neuropsychology, long term career development funding for a general research programme).

  6. 6.

    University/Funder, specific time allocated for research programme (e.g., five years to develop cybersecurity research expertise).

  7. 7.

    University/Funder, employed for specific project (e.g., PhD student, postdoc supervised by member of staff).

  8. 8.

    University/Funder, specific time allocated for specific study (e.g., sabbatical to write a book).

Who funded the equipment or other non-human resources used in the research?

The resources needed for a research project might be funded as part of the project by the main funder, it may be already available to the researcher (e.g., National Health Service equipment that an NHS researcher could expect to access), or it may be separately funded and made available during the project (e.g., Richards, 2019). Here, “equipment” includes data or samples that are access-controlled as well as other resources unrelated to pay, such as travel. These types can be broken down as follows.

  1. 1.

    Researcher’s own equipment (e.g., a musician’s violin for performance-based research or composition; an archaeologist’s Land Rover to transport equipment to a dig).

  2. 2.

    University equipment, borrowed/repurposed (e.g., PC for teaching, unused library laptop).

  3. 3.

    University equipment, dual purpose (e.g., PC for teaching and research, violin for music teaching and research).

  4. 4.

    University/funder equipment for generic research (e.g., research group’s shared microbiology lab).

  5. 5.

    University/funder equipment research programme (e.g., GPU cluster to investigate deep learning).

  6. 6.

    University/funder equipment for specific project (e.g., PCs for researchers recruited for project; travel time).

  7. 7.

    University/funder equipment for single study (e.g., travel for interviews).

Of course, a funder may only support the loan or purchase of equipment on the understanding that the team will find other funding for research projects using it (e.g., “Funding was provided by the Water Research Commission [WRC]. The Covidence software was purchased by the Water Research fund”: Deglon et al., 2023). Getting large equipment working for subsequent research (e.g., a space telescope, a particle accelerator, a digitisation project) might also be the primary goal of a project.

How many funders contributed?

Although many projects are funded by a single source, some have multiple funders sharing the costs by agreement or by chance (Davies, 2016), and the following seem to be the logical possibilities for cost sharing.

  1. 1.

    Unfunded.

  2. 2.

    Partially funded from one source, partly unfunded.

  3. 3.

    Partially funded from multiple sources, partly unfunded.

  4. 4.

    Fully funded from multiple sources.

  5. 5.

    Fully funded from a single source.

As an example of unplanned cost sharing, a researcher might have their post funded by one source and then subsequently bid for funding for equipment and support workers to run a large project. This project would then be part funded by the two sources, but not in a coordinated way. It seems likely that a project with a single adequate source of funding might be more efficient than a project with multiple sources that need to be coordinated. Conversely, a project with multiple funders may have passed through many different quality control steps or shown relevance to a range of different audiences. Those funded by multiple sources may also be less dependent on individual funders and therefore more able to autonomously follow their own research agenda, potentially leading to more innovative research.

How competitive was the funding allocation process?

Whilst government and charitable funding is often awarded on a competitive basis, the degree of competition (e.g., success rate) clearly varies between countries and funding calls and changes over time. In contrast, commercial funding may be gained without transparent competition (Kang & Motohashi, 2020), perhaps as part of ongoing work in an established collaboration or even due to a chance encounter. In between these, block research grants and prizes may be awarded for past achievements, so they are competitive, but the recipients are relatively free to spend on any type of research and do not need to write proposals (Franssen et al., 2018). Similarly, research centre grants may be won competitively but give the freedom to conduct a wide variety of studies over a long period. This gives the following three basic dimensions.

  1. 1.

    The success rate from the funding call (i.e., the percentage of initial applicants that were funded) OR

  2. 2.

    The success rate based on funding awarded for past performance (e.g., prize or competitive block grant, although this may be difficult to estimate) OR

  3. 3.

    The contract or other funding was allocated non-competitively (e.g., non-competitive block funding).

How was the funding decision made?

Who decides on which researchers receive funding and through which processes is also relevant (Van den Besselaar & Horlings, 2011). This is perhaps one of the most important considerations for funders.

  • The procedure for grant awarding: who decided and how?

There is a lot of research into the relative merits of different selection criteria for grants, such as a recent project to assess whether randomisation could be helpful (Fang & Casadevall, 2016; researchonresearch.org/experimental-funder). Peer review, triage, and deliberative committees are common, but not universal, components (Meadmore et al., 2020) and sources of variation include whether non-academic stakeholders are included within peer review teams (Luo et al., 2021), whether one or two stage submissions are required (Gross & Bergstrom, 2019) and whether sandpits are used (Meadmore et al., 2020). Although each procedure may be unique in personnel and fine details, broad information about it would be particularly helpful in comparisons between funders or schemes.

What were the characteristics of the research team?

The characteristics of successful proposals or applicants are relevant to analyses of competitive calls (Grimpe, 2012), although there are too many to list individually. Some deserve some attention here.

  • What are the characteristics of the research team behind the project or output (e.g., gender, age, career status, institution)?

  • What is the track record of the research team (e.g., citations, publications, awards, previous grants, service work).

Gender bias is an important consideration and whether it plays a role is highly disputed in the literature. Recent findings suggest that there is gender bias in reviews, but not success rates (Bol et al., 2022; Van den Besselaar & Mom, 2021). Some funding schemes have team requirements (e.g., established vs. early career researcher grants) and many evaluate applicants’ track records. Applicants’ previous achievements may be critical to success for some calls, such as those for established researchers or funding for leadership, play a minor role in others, or be completely ignored (e.g., for double blind grant reviewing). In any case, research team characteristics may be important for evaluating the influence of the funding or the fairness of the selection procedure.

What were the funder’s goals?

Funding streams or sources often have goals that influence what type of research can be funded. Moreover, researchers can be expected to modify their aspirations to align with the funding stream. The funder may have different types of goal, from supporting aspects of the research process to supporting relevant projects or completing a specific task (e.g., Woodward & Clifton, 1994), to generating societal benefits (Fernández-del-Castillo et al., 2015).

A common distinction is between basic and applied research, and the category “strategic research” has also been used to capture basic research aiming at long term societal benefits (Sandström, 2009). The Frascati Manual uses Basic Research, Applied Research and Experimental Development instead (OECD, 2015), but this is more relevant for analyses that incorporate industrial research and development.

Research funding does not necessarily have the goal to fund research because some streams support network formation in the expectation that the network will access other resources to support studies (Aagaard et al., 2021). European Union COST (European Cooperation in Science and Technology) Actions are an example (cost.eu). Others may have indirect goals, such as capacity building or creating a strong national research base that helps industry or attracts to international business research investment (Cooksey, 2006), or promoting a topic (e.g., educational research: El-Sawi et al., 2009). As a corollary to the last point, some topics may be of little interest to most funders, for example because they would mainly benefit marginalised communities (Woodson & Williams, 2020).

Since the early 2000s, many countries have also issued so-called career grants which have become prestigious. At the European level career grants started in 2009: the European Research Council (ERC) grants. These grants have a career effect (Bloch et al., 2014; Danell & Hjerm, 2013; Schroder et al., 2021; Van den Besselaar & Sandström, 2015) but this dimension, and the longer-term effects of funding other than on specific outputs, is not considered here. A funding scheme may also have multiple of the following goals.

  • Basic research (e.g., the Malaysia Toray Science Foundation supports basic research by young scientists to boost national capacity: www.mtsf.org).

  • Strategic research (e.g., the UK Natural Environment Research Council’s strategic research funding targets areas of important environmental concern, targeting long term solutions: www.ukri.org/councils/nerc/).

  • Applied research (e.g., the Dutch NWO [Dutch Research Council] applied research fund to develop innovations supporting food security: www.nwo.nl/en/researchprogrammes/food-business-research).

  • Technology transfer (i.e., applying research knowledge or skills to a non-research problem) or translational research.

  • Researcher development and training (including career grants).

  • Capacity building (e.g., to support research in resource-poor settings).

  • Collaboration formation (e.g., industry-academia, international, inter-university).

  • Research within a particular field.

  • Research with a particular application area (e.g., any research helping Alzheimer’s patients, including a ring-fenced proportion of funding within a broader call).

  • Tangible academic outputs (e.g., articles, books).

  • Tangible non-academic outputs (e.g., policy changes, medicine accreditation, patents, inventions).

Extent of the funding

The extent of funding of a project can vary substantially from a small percentage, such as for a single site visit, to 100%. A project might even make a surplus if it is allowed to keep any money left over, its equipment survives the project, or it generates successful intellectual property. The financial value of funding is clearly an important consideration because a cheaper project delivering similar outcomes to a more expensive one would have performed better. Nevertheless, grant size is often ignored in academic studies of the value of funding (e.g., Thelwall et al., 2023) because it is difficult to identify the amount and to divide it amongst grant outputs. This section covers four dimensions of the extent of a grant.

What proportion of the research was funded?

A research project might be fully funded, funded for the extras needed above what is already available, or deliberately partly funded (Comins, 2015). This last approach is sometimes called “cost sharing”. A grant applied on the Full Economic Cost (FEC) model would pay for the time and resources used by the researchers as well as the administrative support and accommodation provided by their institution. The following seem to be the main possibilities.

  1. 1.

    Unfunded.

  2. 2.

    Partly funded.

  3. 3.

    Fully funded but on a partial FEC or sub-FEC model cost sharing model.

  4. 4.

    FEC.

  5. 5.

    FEC plus surplus.

The Frascatti Manual about collecting research and development statistics distinguishes between funding internally within a unit of analysis or externally (OECD, 2015) but here the distinction is between explicit and implicit funding, with the latter being classed as “Unfunded”.

How was the funding delivered?

Whilst a research grant would normally be financial, a project might be supported in kind by the loan or gift of equipment or time. For instance, agricultural research might be supported with access to relevant land or livestock (Tricarico et al., 2022). Here are three common approaches for delivering funding.

  1. 1.

    In kind—lending time or loaning/giving equipment or other resources.

  2. 2.

    Fixed amount of money.

  3. 3.

    A maximum amount of money, with actual spending justified by receipts.

How much funding did the project receive?

Project funding can be tiny, such as a few pounds for a trip or travel expenses, or enormous, such as for a particle accelerator. Grants of a few thousand pounds can also be common in some fields and for some funders (e.g., Gallo et al., 2014; Lyndon, 2018). In competitive processes, the funder normally indicates the grant size range that it is prepared to fund. The amount of funding for research has increased over time (Bloch & Sørensen, 2015).

  • The money awarded and/or claimed by the project.

How long was the funding for?

Funded projects can be short term, such as for a one-day event, or very long term, such as a 50-year nuclear fusion reactor programme. There seems to be a trend for longer term and larger amounts of funding, such as for centres of excellence that can manage multiple different lines of research (Hellström, 2018; OECD, 2014).

  • The intended or actual (e.g., due to costed or non-costed extensions) duration of the project.

Influence of the funding on the research project

A variety of aspects of the funding system were discussed in the previous sections, and this section and the next switch to the effects of funding on what research is conducted and how. Whist some grant schemes explicitly try to direct research (e.g., funding calls to build national artificial intelligence research capacity), even open calls may have indirect influences on team formation, goals, and broader research directions. This section discusses three different ways in which funding can influence a research project.

Influence on what the applicant did

Whilst funding presumably has a decisive influence on whether a study occurs most of the time because of the expense of the equipment or effort (e.g., to secure ethical approval for medical studies: Jonker et al., 2011), there may be exceptions. For example, an analysis of unfunded medical research found that it was often hospital-based (Álvarez-Bornstein et al., 2019), suggesting that it was supported by employers. Presumably the researcher applying for funding would usually have done something else research-related if they did not win the award, such as conducting different studies or applying for other funding. The following seem to be the main dimensions of variation here.

  1. 1.

    No influence (the study would have gone ahead without the funding).

  2. 2.

    Improved existing study (e.g., more time to finish, more/better equipment, more collaborators, constructive ideas from the peer review process). An extreme example of the latter is the Medical Research Council’s Developmental Pathway Funding Scheme (DPFS), which has expert input and decision making throughout a project.

  3. 3.

    Made the study possible, replacing other research-related activities (e.g., a different type of investigation, supporting another project, PhD mentoring).

  4. 4.

    Made the study possible, replacing non-research activities (e.g., teaching, clinical practice).

Researchers may conduct unfunded studies if financing is not essential and they would like to choose their own goals (Edwards, 2022; Kayrooz et al., 2007), or if their research time can be subsidised by teaching revenue (Olive, 2017). Some types of research are also inherently cheaper than others, such as secondary data analysis (Vaduganathan et al., 2018) and reviews in medical fields, so may not need funding. At the other extreme, large funding sources may redirect the long-term goals of an entire research group (Jeon, 2019). In between these two, funding may improve the quality of a study that would have gone ahead anyway, such as by improving its methods, including the sample size or the range of analyses used (Froud et al., 2015). Alternatively, it may have changed a study without necessarily improving it, such as by incorporating funder-relevant goals, methods, or target groups. Scholars with topics that do not match the major funding sources may struggle to be able to do research (Laudel, 2005).

Influence on research goals or methods

In addition to supporting the research, the nature of the influence of the source of funding can be minor or major, from the perspective of the funded researcher. It seems likely most funding requires some changes to what a self-funded researcher might otherwise do, if only to give reassurance that the proposed research will deliver tangible outputs (Serrano Velarde, 2018), or to fit specific funder requirements (Luukkonen & Thomas, 2016). Funding influence can perhaps be split into the following broad types, although they are necessarily imprecise, with considerable overlaps.

  1. 1.

    No influence (the applicant did not modify their research goals for the funder, or ‘relabelled’ their research goals to match the funding scheme).

  2. 2.

    Partial influence (the applicant modified their research goals for the funder)

  3. 3.

    Strong influence (the applicant developed new research goals for the funder, such as a recent call for non-AI researchers to retrain to adopt AI).

  4. 4.

    Full determination (the funder specified the project, such as a pharmaceutical industry contract to test a new vaccine).

Focusing on more substantial changes only, the funding has no influence if the academic did not need to consider funder-related factors when proposing their study, or could select a funder that fully aligned with their goals. On the other hand, the influence is substantial if the researcher changed their goals to fit the funder requirements (Currie-Alder, 2015; Tellmann, 2022). In between, a project goals may be tailored to a funder or funding requirements (Woodward & Clifton, 1994). An indirect way in which health-related funders often influence research is by requiring Patient and Public Involvement (PPI) at all levels of a project, including strategy development (e.g., Brett et al., 2014). Funding initiatives may aim to change researchers’ goals, such as to encourage the growth of a promising new field (Gläser et al., 2016). The wider funding environment may also effectively block some research types or topics if it is not in scope for most grants (Laudel & Gläser, 2014).

It seems likely that funding sources have the greatest influence on researchers’ goals in resource intensive areas, presumably including most science and health research, and especially those that routinely issue topic-focused calls (e.g., Laudel, 2006; Woelert et al., 2021). The perceived likelihood of receiving future funding may also influence research methods, such as by encouraging researchers to hoard resources (e.g., perform fewer laboratory experiments for a funded paper) when future access may be at risk (Laudel, 2023).

Influence on research team composition

The funder call may list eligibility requirements of various types. For example, the UK national funders specify that applicants must be predominantly UK academics. One common type of specification seems to be team size and composition since many funders (e.g., EU) specify or encourage collaborative projects. Funding may also encourage commercial participants or end user partnerships, which may affect team composition (e.g., Gaughan & Bozeman, 2002). Four different approaches may be delineated as follows.

  1. 1.

    No influence (the funder allows any team size).

  2. 2.

    Partial influence (the applicant chooses a team size to enhance their perceived success rate).

  3. 3.

    Funder parameters (the funder specifies parameters, such as a requirement for collaboration or partners from at least three EU countries, disciplinary composition or interdisciplinarity mandate).

  4. 4.

    Full determination (the funder specifies the team size, such as individual applicants only for career-related grants).

The influence of funders on research team composition is unlikely to be strict even if they fully determine grant applicant team sizes because the funded researchers may choose to collaborate with others using their own grants or unfunded.

Influence of the funding on the research outputs

The above categories cover how research funding helps or influences research studies. This section focuses on what may change in the outputs of researchers or projects due to the receipt of funding. This is important to consider because research outputs are the most visible and countable outcomes of research projects, but they are not always necessary (e.g., funding for training or equipment) and different types can be encouraged. Four relevant dimensions of influence are discussed below.

Influence of funding on the applicant’s productivity

Funding can normally be expected to support the production of new outputs by an academic or team (Bloch et al., 2014; Danell & Hjerm, 2013), but this may be field dependent. Studying the factors affecting productivity, DFG grants had a positive effect on the productivity for German political scientists (Habicht et al., 2021). However, in some cases funding may produce fewer tangible outputs because of the need to collaborate with end users or conduct activities of value to them (Hottenrott & Thorwarth, 2011), or if the funding is for long-term high-risk investigations. In areas where funding is inessential or where or core/block funding provides some baseline capability, academics who choose not to apply for it can devote all their research time to research rather than grant writing, which may increase their productivity (Thyer, 2011). Although simplistic, the situation may therefore be characterised into three situations.

  1. 1.

    Reduction in the number or size of outputs of relevant types by the applicant(s) during and/or after the project.

  2. 2.

    No change in the number or size of outputs of relevant types by the applicant(s) during and/or after the project.

  3. 3.

    Increase in the number or size of outputs of relevant types by the applicant(s) during and/or after the project.

Funding can also have the long-term indirect effect of improving productivity, though career benefits for those funded, such as making them more likely to attract collaborators and future funding (Defazio et al., 2009; Heyard & Hottenrott, 2021; Hussinger & Carvalho, 2022; Saygitov, 2018; Shimada et al., 2017). Writing grant applications may also provide an intensive learning process, which may help careers (Ayoubi et al., 2019; Jonkers et al., 2017).

Influence of funding on the applicant’s research output types

Funding may change what a researcher or research team produces. For example, a commercial component of grants may reduce the number of journal articles produced (Hottenrott & Lawson, 2017). Funder policies may have other influences on what a researcher does, such as conditions to disseminate the results in a certain way. This may include open access, providing accessible research data, or writing briefings for policy makers or the public. Whilst this may be considered good practice, some may be an additional overhead for the researcher. This may be summarised as follows, although the distinctions are qualitative.

  1. 1.

    No change in the nature of the outputs produced.

  2. 2.

    Partial change in the nature of the outputs produced.

  3. 3.

    Complete change in the nature of the outputs produced (e.g., patents instead of articles).

Influence of funding on the impact or quality of the research

Although cause-and-effect may be difficult to prove (e.g., Aagaard & Schneider, 2017), funding seems likely to change the citation, scholarly, societal, or other impacts of what a researcher or research team produces. For example, a reduction in citation impact may occur if the research becomes more application-focused and an increase may occur if the funding improves the quality of the research.

Most studies have focused on citation impact, finding that funded research, or research funded by a particular funder, tends to be more cited than other research (Álvarez-Bornstein et al., 2019; Gush et al., 2018; Heyard & Hottenrott, 2021; Rigby, 2011; Roshani et al., 2021; Thelwall et al., 2016; Yan et al., 2018), albeit with a few exceptions (Alkhawtani et al., 2020; Jowkar et al., 2011; Muscio et al., 2017). Moreover, unfunded work, or work that does not explicitly declare funding sources, in some fields can occasionally be highly cited (Sinha et al., 2016; Zhao, 2010). Logically, however, there are three broad types of influence on the overall impacts of the outputs produced, in addition to changes in the nature of the impacts.

  1. 1.

    Reduction in the citation/scholarly/societal/other impact of the outputs produced.

  2. 2.

    No change in the citation/scholarly/societal/other impact of the outputs produced.

  3. 3.

    Increase in the citation/scholarly/societal/other impact of the outputs produced.

The quality of the research produced is also important and could be assessed by a similar list to the one above. Research quality is normally thought to encompass three aspects: methodological rigour, innovativeness, and societal/scientific impact (Langfeldt et al., 2020). Considering quality overall therefore entails attempting to also assess the rigour and innovativeness of research. These seem likely to correlate positively with research impact and are difficult to assess on a large scale. Whilst rigour might be equated with passing journal peer review in some cases, innovation has no simple proxy indictor and is a particular concern for funding decisions (Franssen, et al., 2018; Whitley et al., 2018).

The number and types of outcomes supported by a grant

When evaluating funding, it is important to consider the nature and number of the outputs and other outcomes produced specifically from it. Research projects often deliver multiple products, such as journal articles, scholarly talks, public-facing talks, and informational websites. There may also be more applied outputs, such as health policy changes, spin-out companies, and new drugs (Ismail et al., 2012). Since studies evaluating research funding often analyse only the citation impact of the journal articles produced (because of the ease of benchmarking), it is important to at least acknowledge that other outputs are also produced by researchers, even if it is difficult to take them into account in quantitative analyses.

  • The number and type of outcomes or outputs associated with a grant.

Of course, the non-citation impacts of research, such as policy changes or drug development, are notoriously difficult to track down even for individual projects (Boulding et al., 2020; Raftery et al., 2016), although there have been systematic attempts to identify policy citations (Szomszor & Adie, 2022). Thus, most types of impacts could not be analysed on a large scale and individual qualitative analyses are the only option for detailed impact analyses (Guthrie et al., 2015). In parallel with this, studies that compare articles funded by different sources should really consider the number of outputs per grant, since a grant producing more outputs would tend to be more successful. This approach does not seem to be used when average citation impact is compared, which is a limitation.

A pragmatic issue for studies of grants: funding records

Finally, from a pragmatic data collection perspective, the funding for a research output can be recorded in different places, not all of which are public. A logical place to look for this information is within the output, although it may be recorded within databases maintained by the funder or employer. Related to this, it is not always clear how much of an output can be attributed to an acknowledged funding source. Whilst the location of a funding record presumably has no influence on the effectiveness of the funding, so is not relevant to the goals of this article, it is included here an important practical consideration that all studies of grant funding must cope with. Three relevant dimensions of this ostensibly simple issue are discussed below.

Where the funding is recorded inside the output

Funding can be acknowledged explicitly in journal articles (Aagaard et al., 2021) and other research outputs, whether to thank the funder or to record possible conflicts of interest. This information may be omitted because the authors forget or do not want to acknowledge some or all funders. Here is a list of common locations.

  1. 1.

    A Funding section.

  2. 2.

    An Acknowledgements section.

  3. 3.

    A Notes section.

  4. 4.

    A Declaration of Interests section.

  5. 5.

    The first footnote.

  6. 6.

    The last footnote.

  7. 7.

    The last paragraph of the conclusions.

  8. 8.

    Elsewhere in the output.

  9. 9.

    Not recorded in the output.

The compulsory funding declaration sections of an increasing minority of journals are the ideal place for funder information. These force corresponding authors to declare funding, although they may not be able to track down all sources for large, multiply-funded teams. This section also is probably the main place where a clear statement that a study was unfunded could be found. A Declaration of Interests section may also announce an absence of funding, although this cannot be inferred from the more usual statement that the authors have no competing interests. Funding statements in other places are unsystematic in the sense that it seems easy for an author to forget them. Nevertheless, field norms may dictate a specific location for funding information (e.g., always a first page footnote), and this seems likely to reduce the chance that this step is overlooked.

Where the funding is recorded outside the output

Large funders are likely to keep track of the outputs from their funded research, and research institutions may also keep systematic records (Clements et al., 2017). These may be completed by researchers or administrators and may be mandatory or optional. Funders usually also record descriptive qualitative information about funded projects that is not essential for typical large-scale analyses of funded research but is important to keep track of individual projects. It may also be used large scale descriptive analyses of grant portfolio changes over time. For example, the UKRI Gateway to Research information includes project title, abstract (lay and technical), value (amount awarded by UKRI—so usually 80% FEC), funded period (start and end), project status (whether still active), category (broad research grant type—e.g., Fellowship), grant reference, Principle Investigator (PI) (and all co-Investigators), research classifications (e.g. Health Research Classification System [HRCS] for MRC grants), research organisations involved (whether as proposed collaborators or funding recipients/partners), and, as the project progresses, any outputs reported via Researchfish.

Academic employers may also track the outputs and funding of their staff in a current research information system or within locally designed databases or spreadsheets. Dimensions for Funders (Dimensions, 2022), for example, compiles funding information from a wide range of sources. Other public datasets include the UKRI Gateway to Research (extensive linkage to outputs), the Europe PMC grant lookup tool (good linkage to publications) or the UKCDR covid funding tracker (some linkage to publications via Europe PMC), or the occasional UK Health Research Analysis (.net), and the European commission CORDIS dataset. There are also some initiatives to comprehensively catalogue who funds what in particular domains, such as for UK non-commercial health research (UKCRC, 2020). Of course, there are ad-hoc funding statements too, such as in narrative claims of research impact in university websites or as part of evaluations (Grant & Hinrichs, 2015), but these may be difficult to harvest systematically. The following list includes a range of common locations.

  1. 1.

    In a university/employer public/private funding record.

  2. 2.

    In the academic’s public/private CV.

  3. 3.

    In the funder’s public/private record.

  4. 4.

    In a shared public/private research funding system used by the funder (e.g., Researchfish).

  5. 5.

    In publicity for the grant award (if output mentioned specifically enough).

  6. 6.

    In publicity for the output (e.g., a theatre programme for a performance output).

  7. 7.

    Elsewhere outside the output.

  8. 8.

    Not recorded outside the output.

From the perspective of third parties obtaining information about funding for outputs, if the employer and/or funder databases are private or public but difficult to search then online publicity about the outputs or funding may give an alternative record.

What is the connection between outputs and their declared funders?

Some outputs have a clear identifiable funder or set of funders. For example, a grant may be awarded to write a book and the book would therefore clearly be the primary output of the project. Similarly, a grant to conduct a specified randomised controlled trial seems likely to produce an article reporting the results; this, after passing review, would presumably be the primary research output even though an unpublished statistical summary of the results might suffice in some cases, especially when time is a factor. More loosely, a grant may specify a programme of research and promise several unspecified or vaguely specified outputs. In this case there may be outputs related to the project but not essential to it that might be classed as being part of it. It is also possible that outputs with little connection to a project are recorded as part of it for strategic reasons, such as to satisfy a project quota or gain a higher end-of-project grade. For example, Researchfish (Reddick et al., 2022) allows grant holders to select which publications on their CVs associate with each grant. There are also genuine mistakes in declaring funding (e.g., Elmunim et al., 2022). The situation may be summarised with the following logical categories.

  1. 1.

    Direct, clear connection (e.g., the study is a named primary output of a project).

  2. 2.

    Indirect, clear connection (e.g., the study is a writeup of a named project outcome).

  3. 3.

    Indirect, likely connection (e.g., the study is an output of someone working on the project and the output is on the project topic).

  4. 4.

    Tenuous connection (e.g., the study was completed before the project started, by personnel not associated with the project, or by project personnel on an unrelated topic).

  5. 5.

    No connection at all (such as due to a recording error; presumably rare).

Conclusions

This paper has described dimensions along which research funding differs between projects, with a focus on grant funding. This includes dimensions that are important to consider when analysing the value of research funding quantitatively. This list is incomplete, and not all aspects will be relevant to all future analyses of funding. Most qualitative and rarer dimensions of difference associated with funding are omitted, including the exact nature of any societal impact, support for researcher development, and support for wider social, ethical or scientific issues (e.g., promoting open science).

Organisations that compile funding datasets or otherwise record funding information may also consult the lists above when considering the records that are desirable to collect. Of course, the providers of large datasets, such as the Dimensions for Funders system, may often not be able to find this information for inclusion (not provided by funders) or not be able to adequately process it (e.g., simply too many variations in funding types, and no straightforward way to present this data to users).

When comparing funding sources or evaluating the impact of funding, it is important to consider as many dimensions as practically possible to ensure that comparisons are fair as achievable, whilst acknowledging the remaining sources of variation as limitations. Even at the level of funding schemes, all have unique features but since comparisons must be made for management purposes, it is important to consider differences or to at least be aware of them when making comparisons.