Background

The COVID-19 pandemic has made the importance of high-quality evidence abundantly clear to policy-makers. Hence, the pressure for policy-makers to gather and assess all available evidence when making decisions is increasing [1]. However, the process of including scientific evidence in public health decision-making has been—so far—not fully systematic and is complicated by barriers such as specific contexts and traditions, political priorities, individual beliefs and preferences, social values, and available resources [2,3,4].

Research also shows a significant gap between the scientific evidence that is produced by researchers and the actual scientific evidence demanded by policy-makers and other stakeholders [5,6,7,8]. Identifying what scientific evidence different groups of stakeholders prioritise may help to bridge this gap [7,8,9]. A promising yet underused approach is to involve stakeholders systematically in a structured priority setting process to ensure that their needs are accommodated by the produced evidence [10,11,12,13,14].

Funders of public health research have to decide which research projects to support while facing competing demands and scarce resources [15,16,17,18]. However, decision-making about which research should be conducted first is often not evidence-based [19]. Without rigorous research priority setting (RPS), funders risk that research topics will be chosen arbitrarily or are determined based on subjective goals [20,21,22,23]. Hence, RPS should be conducted in a structured matter to allow for better-informed decisions regarding the direction of future research investments.

Research priority setting

Although several frameworks for priority setting in health research are suggested [14, 16, 17, 24,25,26,27,28,29,30], no established, single framework exists that fits all RPS purposes, not least due to varying aims and target groups in RPS [25, 31, 32]. Nevertheless, several studies on health-related RPS agree that a best-practice RPS is a multi-stage process combining multiple methodological approaches [15, 29, 31].

Most notably, many studies incorporated Delphi-like techniques in RPS [33]. The Delphi technique often consists of two or more rounds, in which a panel of experts can give their opinions about an issue. In the following Delphi rounds, they are encouraged to give anonymous controlled feedback to the results of previous stages, which allows them to reflect, reassess and revise their opinions and judgements if needed [34,35,36]. The Delphi technique in health research is especially appropriate when seeking shared preferences from multiple stakeholders and when available evidence is incomplete [34, 35, 37, 38]. The implementation of this technique assumes that a result yielded from a larger panel of experts with various views will be more well-grounded than a conclusion reached by only one stakeholder [34]. Furthermore, the Delphi technique offers additional advantages, such as uncomplicated incorporating various metrics-based techniques and methods for consensus building [35, 38]. The Delphi technique also does not require face-to-face interactions, which reduces time and cost investments. More importantly, this reduces potential drawbacks of the in-person approach that lead to inaccurate estimations because of social pressure and/or dominance of individuals within groups [31, 39, 40].

Moreover, rating of research topics should preferably be done using more than a single assessment criterion to measure different dimensions of why specific topics are prioritised [23, 29, 31].

RPS studies focussing on the field of public health are rare: Selected public health topics are considered occasionally in RPS studies that focus on health in general or on a (sub)field that overlaps with public health (e.g. health services [41], nursing [42], obesity [43] or mental health [44]). We identified three RPS studies [40, 45, 46] that focused specifically on all public health topics, but for the purpose of prioritising topics for conducting systematic literature reviews for the Cochrane Collaboration only.

In Germany, where public institutions fund more than 3 billion EUR per year on health research [47], no RPS study specifically encompassing all public health research topics has been published so far.

Objectives

Our main objective was to investigate which research topics in public health should be prioritised in Germany according to a wide range of stakeholders by conducting a structured priority setting study. We also wanted to identify potential similarities and differences between stakeholder groups.

Methods

Our RPS study (see Fig. 1) was conceptually split into a scoping stage and a Delphi stage and consisted of four rounds:

  • Scoping round I. Gathering expert insights during two initial workshops

  • Scoping round II. Establishing the framework for the study

  • Delphi round I. Collecting research topics and assessment criteria with stakeholders

  • Delphi round II. Rating the collected research topics with the collected assessment criteria

Fig. 1
figure 1

The prioritisation process

Full size image

In the scoping stage, we conducted two initial workshops, established an advisory board, and set the framework for the remainder of the study. For the Delphi stage, we administered two rounds of online questionnaires.

Scoping round I—initial workshops

We conducted our study in collaboration with the German Public Health Association (DGPH). Participants of two workshops, organised by the DGPH in 2015 and 2016, discussed and proposed potential research topics that should be covered in an RPS study. The invitations for the workshops were sent to all members of the DGPH. Additionally, individuals with a clear public health expertise, mainly consisting of established researchers in Germany and representatives of German research funding agencies, were invited. In total, the number of participants during the workshops varied between 40 and 50 individuals.

During the first workshop called “Priority topics for public health research”, members of the DGPH were invited during a 5-h-long exploratory roundtable discussion to discuss which research topics for public health research need most attention. The workshop was publicly announced and open to all members of the DGPH. The participants could propose topics or broader research areas themselves for different domains of public health and they could critically reflect on the topics that were proposed by other participants. The exploratory results were recorded in the minutes and distributed to the members of the DGPH by email.

In order to frame the discussions in this stage regarding potential research topics, we defined domains, i.e., broader public health research areas, to group similar research topics together to ensure comparability. We also used the domains as guidance for the respondents in the first Delphi round, allowing the respondents to consider the different broader public health research areas. The domains were

  • Research on current/contemporary issues

  • Effectiveness research

  • Policy research

  • Implementation and/or participatory research

  • Theories and theoretical concepts

  • Methodological research

  • Research on indicators

The second 2-h-long workshop took place during the annual conference of the German Society for Social Medicine and Prevention (DGSMP). In this plenary discussion the results of the first workshop and the planning for our RPS study were presented and discussed with representatives of German research funding agencies.

Scoping round II—establishing framework for the study

The author team approached established researchers and practitioners in the field of public health in Germany to form an advisory board. The advisory board consisted of five researchers and practitioners in total (see also the acknowledgements below), with both subject matter and methodological expertise [25, 31, 48].The advisory board members were not allowed to participate in the following Delphi stage.

The advisory board members reviewed the design and the proposed analyses of the study, particularly with regards to the scoping stage of the study and the questionnaire design. The advisory board was in particular helpful for framing if a research topic can be reasonably considered within the realm of public health research. In our definition for this study, public health research encompasses population level and health systems research, which excludes research topics that are predominantly clinical or biomedical research.

Moreover, we identified the stakeholder groups that are active in German public health and who are therefore relevant for inclusion in our RPS study [25, 44, 49]:

  • Public health research and education

  • Public health administration and policy-making

  • Non-governmental organisations (NGO) and representatives of the public

  • Representatives of health professionals and health care institutions

  • Self-governing associations of health providers and statutory health insurance

The stakeholder groups stand for different professional fields who are either producers, facilitators, or consumers of public health research in Germany. Hereafter, we identified specific organisations that fall within each of the stakeholder groups. We used this list of organisations for the recruitment of individual respondents for the Delphi stage (see Additional file 1 for a full list).

Delphi round I—collecting research topics and assessment criteria

In the first Delphi round, we distributed an online questionnaire to individual respondents. We presented the initial list of proposed research topics from the workshops and proposed assessment criteria that were defined during the scoping stage in the online questionnaire. The respondents could vote from this list which research topics should be included and which assessment criteria should be used for the rating of the research topics in the second Delphi round. Moreover, the respondents could propose further research topics and assessment criteria.

Thereafter, we coded and aggregated all proposed research topics using content analysis [50,51,52] to minimize redundancies and overlaps. The aggregated research topics should be interpreted as broader research themes that can be used for, e.g., setting the focus of future calls for proposals.

Delphi round II—in-depth rating of research topics with assessment criteria

In the second Delphi round, we presented the final list of research topics based on the first Delphi round. For each research topic, we also reported how many respondents voted for a particular research topic during the previous stage [15, 29, 53]. Subsequently, the respondents were asked to rate each research topic with the assessment criteria on a 4-point Likert scale (also with the alternative choice “I cannot assess this"). In order to limit the workload for the respondents, we presented each respondent a random sample of approximately 50% of the research topics only.

Recruitment

We based our recruitment strategy of respondents for the Delphi stage on a previous priority setting study in the field of health services research [41]: We invited public health relevant organisations, i.e. stakeholders, to nominate individuals as respondents from within their own organisation for participation in the study. We asked the organisations to nominate individuals who they believed are most suitable for assessing and prioritising research topics in public health. We pre-defined this as somebody who had multiple years of experience working in the field of public health and who is either a researcher or a user of research.

In total, we invited 140 organisations to nominate up to three respondents each. Table 1 shows the number of organisations by stakeholder group (see Additional file 1 for a complete list of the invited organisations).

Table 1 Respondents during the Delphi stage by stakeholder group
Full size table

Of the 140 invited organisations, 94 (67%) participated and nominated in total 230 individuals. The participation rate in the first Delphi round was 87% (201 respondents) and in the second Delphi round 88% (203 respondents).

Table 1 also shows the professional area the respondents assigned themselves: approx. 50% of the respondents are working in research and/or higher education. The other respondents assigned themselves to the following stakeholder groups: Administration and/or politics, representatives of the general public, self-governing associations of health providers and statutory health insurance, and healthcare professionals. The respondents had an average of 16 years of experience in the field of public health.

Results

We will report the results of each Delphi round in turn.

First Delphi round: collecting research topics and assessment criteria with stakeholders

The respondents voted for research topics and assessment criteria from the initial expert list and could also propose further research topics and assessment criteria for inclusion in the second Delphi round. In total, the respondents proposed 529 research topics and 50 assessment criteria which we aggregated through a content analysis into 76 sufficiently distinct research topics and 6 assessment criteria, respectively.

The content analysis of the assessment criteria revealed, however, two types of research topics that could not be assessed meaningfully with the same set of criteria: Substantive research topics and methodological-theoretical research topics. Substantive research topics focused on specific thematic contents, for example “Climate change and health” or “Health literacy”. Methodological-theoretical research topics focused on the application or development of specific research methods and paradigms as well as on the development of theories and concepts for public health research, for example “Participation in health research” or “Interdisciplinary research”. This split resulted in 46 substantive research topics and 30 methodological-theoretical research topics, each with three associated assessment criteria. The operational definitions of the 6 assessment criteria are shown in Box 1 (see Additional file 2a and b for the final list of research topics based on the content analysis).

Second Delphi round: rating of research topics

In the second Delphi round the respondents rated the research topics using the assessment criteria on a 4-point Likert scale, ranging from 1 “totally agree” to 4 “totally disagree”. Alternatively, the respondents could also select the option “I cannot assess this”.

Table 2 shows the average rating by assessment criterion. Methodological-theoretical research topics received on average lower ratings than substantive research topics. Furthermore, for the two criteria that required broader insight into existing public health research (criterion S3 “Insufficient research” and criterion M3 “Potential for innovative insights”), the share of respondents who answered “I cannot assess this” is highest.

Table 2 Assessment results per criterion of both substantive and methodological-theoretical research topics
Full size table

This table shows the results of the assessments of all the respondents per criterion. The respondents had to rate the research topics according to the assessment criteria on a 4-point Likert scale, ranging from 1 “totally agree” to 4 “totally disagree”. Alternatively, the respondents could also select the option “I cannot assess this”.

We asked each respondent to assess approximately 50% of all the research topics in order to reduce the workload. The selection and order of the research topics that each respondent had to rate was randomised. Therefore, the total number of respondents that assessed a certain topic differs slightly.

Substantive research topics

As shown in Table 3, the average overall rating score for all substantive research topics was 1.96, ranging from 1.42 for the research topic “Interventions in everyday life” (ranked highest) to 2.48 for the research topic “Accidents, violence, self-harm” (ranked lowest).

Table 3 Results of the rating of substantive research topics
Full size table

In addition to the rating score, we also provided the rank of each research topic. For some research topics, the ranking per criterion showed slight differences, whereas for other research topics we found larger differences. For example, the research topic “Social inequality and injustice” was not ranked very high according the assessment criterion “Insufficient research” (ranked 20th out of 46); however, according to the assessment criteria “Improving health” and “Health justice”, the same research topic was ranked very high (respectively, 2nd and 1st).

Methodological-theoretical research topics

Table 4 shows the rating and ranking of methodological-theoretical research topics for the three assessment criteria separately (criteria M1 to M3). The average overall rating score for all methodological-theoretical research topics was 2.07, ranging from 1.64 for the research topic “Interdisciplinary research” (ranked highest) to 2.46 for the research topic “Mobility concepts” (ranked lowest). For all the three criteria, the highest-ranked research topics were virtually identical. For other (not highly-ranked) research topics we found larger ranking differences between the criteria. For example, the research topic “Comparative effectiveness research (CER)” was ranked 9th with the assessment criterion “Potential for innovative insights”. With the assessment criteria “Impact on public health research” and “Impact on public health practice”, the same research topic was ranked much lower (17th and 25th, respectively).

Table 4 Results of the rating of methodological-theoretical research topics
Full size table

Comparison by stakeholder groups

As shown in Table 1, approximately 50 percent of the respondents belonged to the "Public health research and/or higher education" group. The other stakeholder groups, "Representatives of the general public", "Administration and/or politics", "Self-governing associations of health providers and statutory health insurance" and "Healthcare professionals" were grouped together and defined as "Public health practitioners". This allowed for a comparison between these two major stakeholder groups.

Additional file 3a and b show the ranking of substantive and methodological-theoretical research topics, respectively, by stakeholder group. For the majority of the substantive research topics, we did not see a major difference between the two stakeholder groups in the ranking. For nine research topics, however, there was a very large difference of ten or more positions in the ranking: The research topics "Digitalisation and health", "Nutrition and health (cultural, physiological, social)", and "Infectious diseases and vaccination" showed the largest differences (all ranked 14 places higher by public health practitioners as compared to public health researchers). Among the methodological-theoretical research topics (see Additional file 3b), we saw a very large difference (ten or more positions) in the ranking of seven methodological-theoretical research topics, with the research topics "Process evaluation", "Causal analyses / Experiments", and "Modelling studies / Decision analysis" showing the largest difference (up to 15 ranking positions difference).

As the judging difference in ranking as "large" or "very large" may seem somewhat arbitrary, we calculated the interrater reliability by stakeholder group, i.e., public health researchers vs. public health practitioners. The interrater reliability can be quantified by calculating an agreement coefficient (AC). We calculated Gwet’s AC because it allows for multiple rates and multiple topics [54, 55]. The interpretation is straightforward: Values close to 1 indicate almost total agreement among the respondents, values close to 0 indicate agreement is mostly due to chance. The analysis was conducted using the Stata software version 17 [56, 57].

In Table 5 we report the degree of agreement among respondents by the two major stakeholder groups (public health researchers vs. public health practitioners). For all three substantive assessment criteria the agreement of raters within both stakeholder groups could be considered as moderate (Gwets AC between 0.41 and 0.60). The agreement was highest for the criterion S1 (“Improving health”). For the criterion S3 (“Insufficient research”), the agreement among public health researchers and public health practitioners was the lowest among the substantive assessment criteria and the difference between both stakeholder groups was not statistically significant anymore.

Table 5 Agreement coefficient† by stakeholder groups (public health researchers vs. public health practitioners) for each criterion
Full size table

For all three methodological rating criteria the agreement among public health researchers could be considered moderate (Gwets AC between 0.41 and 0.60). However, among public health practitioners there was only fair agreement (Gwets AC between 0.21 and 0.40). Moreover, the coefficients for the methodological-theoretical assessment criteria were slightly lower than for the substantive assessment criteria. Concerning criteria M2 (“Impact on public health practice”) there was no statistically significant difference between the two groups of stakeholders.

Discussion

Funding decisions in public health research are complex and multiple criteria play a role. Consequently, priority setting becomes a challenge that policy-makers cannot solve easily [23, 58]. Therefore, policy-makers need structured and transparent RPSs that take all relevant criteria into account [25, 29, 59].

We conducted an RPS study with the aim to investigate which research topics in public health should be prioritised in Germany according to different stakeholders. To the best of our knowledge, we conducted the first structured and transparent RPS study for public health research topics in Germany.

Improved transparency in RPS can strengthen the acceptability of the prioritised research topics, not least because research efforts and funding can be directed towards research that is relevant to all stakeholders [14]. Involving public health researchers and policy-makers simultaneously in RPS studies is not rare, however, Cortier and colleagues [60] demonstrated that other stakeholder groups such as public and advocacy organisations are not frequently included. By using a modified Delphi technique, we involved a wide range of stakeholders in order to investigate which research topics should be prioritised in public health in Germany.

We used a multi-stage approach to balance two conflicting aims, i.e., (i) eliciting proposals from a wide range of stakeholders and (ii) yet ensure that the proposals are within the realm of public health research. In order to ensure the latter, we defined domains that each stand for a broader public health research area and created—by conducting expert workshops in cooperation with the DGPH—an initial list of relevant research topics.

In the first Delphi round, we collected additional proposals for research topics from the participating stakeholders. The findings of the first Delphi round demonstrated clearly the need to split research topics into two groups (46 substantive vs. 30 methodological-theoretical research topics). We applied a different set of assessment criteria to these two types of research topics in the second Delphi round, as methodological-theoretical research topic cannot be assessed by the same criteria as substantive research topics.

The overall rating score for a particular research topic is the average of the rating scores for the three corresponding assessment criteria. No research topic received an overall score lower than 2.5. That is, on average no research topic was considered unimportant.

We found large differences in the rating and ranking of the research topics when differentiating the results along the three assessment criteria. These results corresponded to our expectation that the assessment of a particular research topic depends on the criterion applied (e.g. improving population health vs. insufficient research). It shows that the assessment criteria are measuring distinct dimensions of a research topics and can give an indication on why a particular research topic is prioritised high or low. Although many studies highlighted the importance of selecting multiple assessment criteria that fit to the specific context and that can sufficiently discriminate between the assessment of different research topics [25, 29, 31, 61], most RPS studies do not involve stakeholders in the selection of relevant assessment criteria [60]. The use of multiple assessment criteria also makes it easier for the respondents to rate the research topics as it provides clarity of what aspect of the research topic they were rating exactly.

The descriptive comparison of the priority ranking of the research topics by stakeholder groups, showed that both public health practitioners and public health researchers ranked predominantly similar research topics as top priorities. However, it should also be noted that clear differences exist in the priority ranking of many (non-top priority) research topics between the two stakeholder groups. Moreover, the degree of agreement among the respondents on the importance of research topics differed by stakeholder group: Public health practitioners had on all criteria a lower degree of agreement than public health researchers. This overall lower degree of agreement among public health practitioners might be a result from the existing variation within the stakeholder group, as several more narrowly defined stakeholder groups were aggregated together and labelled ‘public health practitioners´. Further research is needed to investigate how different and/or more narrowly defined stakeholder groups might produce differing results in an RPS. However, the group sizes of the more narrowly defined stakeholder groups in our RPS study were too small, which prevented a comparison between them.

Public health is a complex and multifaceted field that is mainly conducted in a real-life setting, which makes it a challenge for researchers and other stakeholders in public health research to develop a common understanding of what research is most relevant or important [62].The top priorities derived from our study (see Tables 3 and 4) also clearly highlight the importance of inter- or transdisciplinary research in a complex and real-life setting.

Funders in Germany can use the results of this RPS study to discuss future calls in a transparent and structured manner. The overall results show which research topics are prioritised highest by a respective stakeholder groups, while the assessment criteria also help to explain why particular topics are rated lower or higher. A worthwhile next research step should be to investigate if and how funders and other policy-makers in Germany make use of the findings of this RPS study.

Strengths and limitations

This is, to our knowledge, the first RPS study for public health research in Germany. So far no widely accepted standards exist on how to conduct such an RPS. Hence, our works was in many ways novel, but limitations have to be acknowledged.

We recruited respondents for the Delphi stage by identifying relevant organisations in the field of public health and ask them for the nomination of respondents. The approach may seem onerous, but it has been successfully applied already before [41] and had several advantages: The workload and cost for recruiting was relatively low and the approached organisations provided the contact details of the respondents. Furthermore, we had a very high participation rate (close to 90%) in both Delphi rounds as compared to other RPS studies that did not use this specific recruitment approach for their Delphi stage [13, 63, 64]. Also, the approached organisations nominated respondents they considered to have sufficient expertise in public health to participate meaningfully in an RPS study.

The validity of the sample is limited by the specific public health organisations we identified in our search (see Additional file 1) that nominated respondents. As organisations could only nominate up to three respondents, no single organisation could influence the rating unduly. We explicitly refrained from asking the organisations to form and present an official position; we merely asked their nominees to propose and prioritise research topics based on their expertise. In absence of a readily available list of all relevant public health experts and/or stakeholders in Germany, the only alternative would have been an open call to participate in the Delphi stage with all its disadvantages (e.g., unclear participation rates, unclear biases, and uncertain expertise of respondents).

Furthermore, we did not collect data on the context of the respondents within their organisations, such as position, decision-making power and organizational culture, although these factors could have an influence on the actual responses. However, we believe that this was necessary in order to ensure the anonymity of the respondents and to limit the time respondents had to spend for participating in our modified Delphi, therefore ensuring a sufficiently high participation rate. Future research is needed to investigate how contextual factors influence the RPS.

Our RPS approach incorporated a modified Delphi technique, i.e., the results of the rating in the second Delphi round were not discussed by the respondents in an additional follow-up round. We modified the Delphi process to reduce the workload for the respondents and to be able to include more stakeholders. Therefore, the results of our Delphi study should be interpreted as an indication of preferences rather than a consensus. On the other hand, this modified process increased the involvement of stakeholders who otherwise might be marginalised in the discussion of a full Delphi study, or who would not be invited at all as one of the top experts for a Delphi study [35, 65]. The open nature of the first Delphi round, i.e., the possibility to propose additional research topics, required us to conduct a content analysis to aggregate the large number of redundant or overlapping proposals (529 research topics) into a practicable number of sufficiently distinct research topics. However, an aggregation of data often means a loss of information. Furthermore, we are also aware that the formulation of the research topics through the content analysis is based on partly subjective assessments by the author team; a procedure that is however inherent to content analysis and a well-established method for condensing vast amounts of qualitative data [51, 52, 66,67,68].

Despite wide agreement that distinct criteria should be applied to rate the different dimensions of a research topic, no standard exists. We used the input of the respondents and were able to develop and apply 6 distinctive assessment criteria. However, further research is needed to investigate if additional or perhaps different assessment criteria are needed. Moreover, it is not clear if generic assessment criteria exist that can be used in any RPS study or if they must be specific to the research field, e.g., public health vs. health service research. Nevertheless, our study demonstrated that for public health research topics, depending on whether a research topic is substantive or methodological-theoretical, different assessment criteria should be used.

As this is the first RPS study of its kind in Germany that considers all public health research domains, it is unclear how long the results of the study remain sufficiently valid to inform policy-makers. This study was conducted before the outbreak of the Covid-19 epidemic in 2020 and an update may already yield different results. However, we do believe that many proposed research topics, such as “Health in all policies” or “Health literacy promotion”, will be considered relevant for the foreseeable future. A replication of the study after a few years would help to distinguish which research topics are rather long-term priorities and which research topics are rather short-term.

Funders and other research teams could easily use our study as a template for their own RPS studies in a different field of study. Although conducting our structured RPS study is a time intensive exercise initially, we believe that a follow-up study would not need much additional work.

Conclusions

We conducted a structured RPS study involving a wide range of stakeholders in the field of public health in Germany. Our study demonstrates how a multi-stage RPS study, using an anonymous and online modified Delphi technique in which stakeholders rate and rank a comprehensive list of research topics, can be implemented with limited resources. Being the first structured RPS study for public health that can be replicated easily, it may lay important groundwork for future RPS studies in public health and related fields of health research.