Images of the future shape the expectations of what tomorrow’s tomorrows will bring. They do so by influencing mind-sets, decisions, actions, and the allocation of funds [1, 2]. Progress in science and technology is a major driver for shaping the possibility space of futures, and simultaneously effecting the expectations of various actors. Those future-oriented actions and decisions impact the lives of many people, yet traditionally very few actor groups set the agenda, or have the opportunity to implement their expectations, for instance via research and innovation policy programming.

In order to harness expectations and provide policy advice for actively shaping futures, forward-looking activities (FLA) often elicit experts’ opinions. Additionally, stakeholder engagement has become a norm over the course of the past two decades [3]. This shift towards democratizing knowledge production has had a long development, described for instance as the participatory turn in science [4] or post-normal science [5]. Here, the co-production of socially robust knowledge by linking different types of knowledge is a main aim of transdisciplinary research (e.g., [6,7,8]. And also in futures studies, the shift towards participation which began in the 1970s [9, 10] gained momentum in the 1990s [11] and led to the development of a number of participatory frameworks for futures research [12,13,14].

Societies increasingly face complex, transnational, highly interrelated problems involving high degrees of uncertainty while being potentially open-ended issues—wicked problems in planning theory terminology [15, 16]—such as climate change, resource scarcity, or economic crises. Despite the age of the discourse and referenced problems, policy making is struggling to find new ways of addressing these complex challenges [17, 18].

However, transdisciplinary knowledge co-production is recognized as being effective for addressing societal challenges with regard to sustainability transitions [19, 20]. Public engagement literature states producing better decisions and raising legitimacy are among the most prominent reasons for commissioning and conducting participatory processes [21,22,23,24]. Additionally, on EU-level, public participation is pursued to strengthen EU-citizenship and build trust in EU institutions [25, 26], and can be traced back to projects like 2006’s “Meeting of the Minds,” a transnational citizens’ deliberation on Brain Sciences [27].

The European Commission reframed such wicked problems as societal challenges and built a large part of its framework program for funding research and innovation (Horizon 2020) on the expectation that science and technology will support solutions for those challenges [28]. Here, the governance concept of Responsible Research and Innovation (RRI) was implemented to strengthen this role in research and development actors with the guiding principles of inclusiveness, anticipation, reflexivity and responsiveness [29,30,31]. All principles embrace public engagement at their center—responsiveness, for example, aims at orienting research and innovation (R&I) processes towards societal needs and demands. However, Zwart et al. state that the newness of RRI mainly resides in an emphasis on socio-economic impacts, such as valorisation, employment, and competitiveness [32].

Participatory agenda setting for increasing responsiveness to societal needs and values

On the one hand, participatory methods for public engagement in research and innovation are abundant [33]. A recent analysis shows that with regard to research program development—agenda setting—the involvement of civil society organizations or other stakeholders is common practice, while citizens are only involved seldom [34]. On the other hand, expert- and stakeholder-centered forward-looking activities (FLA) have proven to be effective in supporting RI programming [35,36,37,38]. However, lay-people’s involvement in FLA with regard to advice-giving mechanisms for science and innovation programming is relatively new and rare [39,40,41,42].

Yet, program development offers a timely and unique space for the entry of public needs, values, and demands into the innovation process. Here, participatory foresight and co-creation can offer tools to support the shaping of responsive research and innovation agendas. Lay-people engagement by means of visioning evades some of the challenges attached to early engagement, such as opaqueness of the issue to be discussed. Visioning allows for moving away from immediate concerns or feasibility thinking and towards shared explorations of alternative, more sustainable futures, especially when applying longer time horizons [43, 44]. In this way, tacit knowledge, needs, and values can be collected and integrated with stakeholder and expert knowledge, and used in the co-creation of socially robust knowledge for responsible agenda setting. The next section will give an overview of such a process, applied during CIMULACT—Citizen and Multi-Actor Consultation on Horizon 2020. A more detailed positioning and theoretical background on participatory science, technology and innovation agenda setting, especially with regard to the project discussed in this contribution is described in [45].

Aim of this paper

Participatory agenda setting entails planning and conducting complex participatory processes, which include challenges in data analysis and knowledge management, in motivating and selecting participants or in ensuring the aimed for impact on political decision making. Several theoretical and practical challenges for public engagement in science and technology have been put forward in literature, for instance the lack of measurable increases in rationality considering the respective topic in deliberation, the failure to open up the debate, or the lack of impact on decision making [46,47,48,49,50,51,52,53,54].

Against this background, we compare results of a large-scale Europe-wide participatory process for research and innovation agenda setting (CIMULACT) with the findings of 16 expert-based foresight studies with a similar aim. Authors have had central roles of work package and task leaders in designing and implementing the discussed project. We are aware that this may hamper objective judgement of the process; however, we believe that this close proximity to the CIMULACT project, and our familiarity with both its results and the comparative method it employed, allows for in-depth extension of research work initiated therein. In doing so, we aim to shed light on the following research question: Considering the elaborate process, invested resources and intertwined challenges, is lay engagement in STI agenda setting worth the trouble? And, to be more specific, are results sufficiently distinct from expert-based studies to make the case for the continued or expanded use of citizen-based, large-scale participatory foresight processes?

CIMULACT—citizen and multi-actor consultation on Horizon 2020

CIMULACT was a coordination and support action (CSA) funded under Horizon 2020, H2020-ISSI-2014-2015: Call for integrating Society in Science and Innovation, Topic ISSI-2-2014: Citizens and multi-actor engagement for scenario building implemented between 2015 and 2018, and coordinated by the Danish Board of Technology. The project’s main aim was to engage citizens, stakeholders, and experts in co-creating research topics based on validated and shared visions, needs, and demands. Results of the project informed decision makers, while drafting the European research agenda Horizon 2020, national research agendas as well as the upcoming European framework program for research and innovation (FP9), therefore making these research and innovation more responsive to societal needs, one of RRI’s main objectives.

The project comprised an extensive participatory process consisting of several interconnected steps, applying several well-established participatory methods as well as developing new methods. The project’s overall methodology was based on previous large-scale experiences with participatory STI agenda setting, such as delivered by EU projects CIVISTI, or VOICES was an EU-wide consultation process, in scale comparable to the first phase of CIMULACT, engaging citizens in focus groups to produce advice for Horizon 2020 on the specific topic of urban waste as resource. CIVISTI is a long-term participatory foresight method developed and applied to find to identify new, emerging topics for Horizon 2020 [39, 40]. Within the method, citizens developed visions of a desirable future, while experts and stakeholders used these to derive recommendations for research and development policy. Finally, citizens prioritized these recommendations. The method was later adapted and applied to local and national scale in several case-studies [41].

The demand-oriented CIVISTI-logic, starting with collecting citizens’ visions to then derive advice for R&I policy, is still present in CIMULACT. Nevertheless, the methodology applied in CIMULACT is considerably different, for instance with regard to mixing actor groups (co-creation) or the final product—research topics instead of recommendations. It comprises six main steps: (1) Applying a standardized visioning method in thirty European countries, more than thousand citizens produced 179 visions of a desirable future. CIVISTI—Citizens Visions on Science, Technology and Innovation (2008–2011), (2) From these visions, experts extracted 26 underlying, crosscutting, implicitly, and explicitly mentioned societal needs. (3) Selected citizens from the visioning workshops, stakeholders and experts—overall more than a hundred participants—co-created 48 suggestions for research topics on the basis of the visions and societal needs in a facilitated workshop. VOICES—Views, Opinions & Ideas of Citizens in EU on Science (2013–2014), (4) In a second national face-to-face consultation phase, the consortium engaged multiple actor groups in all 30 countries to review and enrich the results and shape them towards becoming tangible research topics. Deliverable 3.2 Programmes and concepts for all citizen and multi-actor consultations, In parallel, an online consultation engaged more than 3400 participants in an argumentative Delphi to prioritize and enrich the drafted topics. (5) Integration of results of the online and face-to-face consultations to synthesize 48 one-paged summaries of the research topics with the sections challenge, scope, and expected impact. (6) With the aim of increasing applicability to EU research programming, a workshop was held to engage EC policy officers and experts, who merged highly similar topics, resulting in 46, and selected and refined 23 final topics, Deliverable 2.2 Final citizen’s based research topics,

The selection processes regarding citizen participants had two distinct methodological breaks within CIMULACT. To select citizen participants for the visioning workshops, broad principles were defined by the CIMULACT consortium to find common ground concerning the “ideal” participant pool. “In a highly deliberative collective visioning process, more than 1088 citizens in 30 European countries were engaged and developed 179 visions of desirable futures. Here, the method is not aiming at inviting a statistically representative sample of participants, but building a sample of maximum heterogeneity to include a high number of different perspectives. Thus, each national consultation invited around 36 participants, ensuring a high diversity within the following criteria: age, gender, education, occupation, and place of living. For example, six age groups were defined, ranging from 16 to 60 and older, to then have one member of each age group in all smaller working groups (table level). Four educational levels—pre-primary to higher education—and several occupation groups were defined to distribute participants at table level aiming at a high heterogeneity of backgrounds. The criterion of place of living ensured engaging city as well as country dwellers.” [45]. To accommodate alternative national cultures and institutional processes, each of the 30 national partners was allowed to implement their own selection method within a framework of standardized methods and approaches. In Germany and Austria, for example, participants were paid a small daily stipend; in order to incentivize participants who might not normally engage such an activity, in Denmark, paying such an incentive would often be considered an affront.

Citizens participating in follow-up workshops were often selected from these visioning participant pool based on availability and the participants level of engagement within prior workshops. It is also worth noting that examination of the national visions allowed the CIMULACT project to identify trends, issues, and prioritizations that were particular to specific nations, and clusters of nations whose research agendas might align with little regard for geographic location. However, all steps of the CIMULACT process subsequent to the visioning workshop took place on European level, or if conducted on a national level, revised and elaborated only parts of the intermediary which were than aggregated as a European result. Therefore, CIMULACT’s final results can hardly be investigated for national properties. Furthermore, CIMULACT results are products of group processes and therefore cannot be investigated regarding different gender or social lines of inquiry.

Method—comparing citizen- and expert-based topics for R&I agenda setting

Foresight studies employ a large diversity of methods, and neither reporting structure nor classification is standardized, which makes comparison challenging. While comparing national foresight reports, Gavigan and Scapolo [55] answered to these challenges with applying a national objectives analytical model, while Alsan and Atilla Oner [56] developed comparative matrices. Other authors have compared methodologies [57], or participants, organizers, and the difficulty to measure policy impacts [58]. While none of those comparative approaches showed the necessary specifications to be applied in comparing CIMULACT results to those of expert-based foresight studies, we applied some of their components. However, to provide a meaningful comparison, we developed a content-driven comparative analytical framework (Fig. 1), which includes five different analytical steps and calculation of three metrics.

Fig. 1
figure 1

Overview of workflow and method of comparison representing our method of comparison as well as the followed workflow. Five steps of analysis produce three different metrics. The analytical steps include preparation, comparative reading, qualitative assessment, aggregation, and review and discussion of differences, whereas metrics include degree of coverage, degree of alignment, and cross comparison. This represents our method of comparison as well as the followed workflow. Five steps of analysis produce three different metrics. The analytical steps include preparation, comparative reading, qualitative assessment, aggregation, and review and discussion of differences, whereas metrics include degree of coverage, degree of alignment and cross comparison


For selecting expert-based foresight reports for comparison, two primary selection criteria dominated the process, purpose, and scope of the respective report, with secondary criteria creating an additional filter. With regard to purpose, we selected studies that had a similar aim as CIMULACT: informing the development of research and innovation agendas and here mainly Horizon 2020, sourcing from databases, such as [59], The Publications Office of the European Union and the European Commission Joint Research Centre Publication Repository. In order to enable a proper comparison with regard to the to the highly integrated fields of research CIMULACT addressed, we selected foresight reports with most similar scope, i.e., broad ranging in terms of included research fields, geographical area, and socio-political analysis. We therefore excluded, for instance, foresight reports focusing on a specific sector or field of inquiry, or national in scope, even though they contained very detailed foresight knowledge. Categories were identified in other foresight mapping analysis [60], such as commissioning organizations, authoring organizations, publication date, time horizon, methodological approach, and the intended audience served as secondary selection criteria. The 16 reports finally selected can be found in Additional file 1: Table S1. As basis for comparison, we chose the 46 CIMULACT topics resulting from the last step of the participatory process, which include the final 23 topics, but also the larger set of topics that were not selected by EC officials during the project’s final conference. In order to prepare the CIMULACT topics for comparative reading, we identified and highlighted key words and critical concepts within the topics as comparative lenses. For more detailed information with regard to selection of reports and identification of key concepts, consult [61].

Comparative reading

Two independent readers used comparative reading templates, containing the highlighted text excerpts and thematic groupings prepared in the previous step, to match direct quotes from the 16 selected foresight report to the CIMUALCT’s corresponding critical concepts and keywords. Individual results were then aggregated into a single file for further processing and deriving synthesized findings.

Degree of coverage (DoC)

This score is derived from the aggregated comparative reading results as a basic unit of comparison, indicating whether an expert report did or did not mention an area of inquiry indicated within one of the CIMULACT research topics. Final DoC scores were calculated for each of the CIMULACT research topics by aggregating the results of the comparative readings across the 16 expert-based reports. For example, a DoC score of one (1/16) indicates that matching quotes from one expert-based foresight report were found for the respective CIMULACT topic.

Qualitative assessment

While the DoC score offers a good overview of how many reports cover the respective CIMULACT topic, it does not capture qualitative differences of how CIMULACT or the expert-based report cover the issue in question, i.e., direction of research, intention, or goals. To capture this, two independent assessors assigned scores measuring the degree of congruence or alignment between all direct quotes from the expert-based foresight reports and CIMULACT topics.

Degree of alignment (DoA)

Qualitative differences in alignment were assigned according to the following scoring scheme (0–3): zero alignment (0) no matching quotes found for CIMULACT topic; low (1) expert-based report discusses one or more aspects of the respective CIMULACT topic, but differs on goals, expected impact or perspective of analysis; medium (2) quotes generally match with CIMULACT topic, but miss key components; high (3) very high alignment between quotes and topic’s content.

Aggregation and review

Next, independent assessments were aggregated to calculate the alignment score for each topic. Average scores were calculated for cases where assessors’ scoring matched or differed by no more than one score. Cases of disagreement, with a difference in score by more than one, were revisited and discussed in depth with the aid of original texts until an agreement of no more than one score could be reached and a corresponding average score could be calculated.

Cross comparison (CrossComp)

By multiplying the first two metrics, DoC and DoA, this combinatory metric highlights the fall off points of similarity between expert- and citizen-based reports. This enables an efficient mode of quickly assessing each CIMULACT research topic’s comprehensive comparative score with regard to all of the surveyed reports. The maximum CrossComp score was calculated to be 16 (representing universal coverage and alignment), and final CrossComp percentages were calculated accordingly.

Discuss differences

Using CrossComp score, we sought to identify those research topics that were shared across citizen-based and expert-based projects, but whose comparative results shed light on the differentiating characteristics of citizen-based process results. Therefore, we have partitioned the CIMULACT research topics into six levels of relationality with regard to expert-based reports. Those CIMULACT research topics that have less than 15% CrossComp have been excluded from this analysis and are the topic of a forthcoming publication. The remaining five tiers of CIMULACT research topics show CrossComp scores and therefore comprehensive alignment of 15% < X < 20%, 20% < X < 25%, 25% < X < 35%, 35% < X < 45%, and X > 45% with the selected expert-based reports.

Results and discussion

In beginning to answer the question concerning the added benefit of large-scale participatory processes for R&I agenda setting, the comparative study results can be clustered in three primary groups of analysis. First and second of these groups are research topics that are unique to expert-based foresight studies and CIMULACT, respectively. These are topics that share little resonance across the two sources of foresight activities, and a detailed analysis of these results is the focus of a forthcoming publication. The third category of results then are those research topics that demonstrate a shared interest by both experts and citizens, and thus provide a fertile ground for determining the distinct benefits the participatory; citizen-based processes can provide policy crafting endeavors. Table 1 shows an overview of the finding regarding this category. Here, the question then becomes, not do they deliver, but what do they deliver that is sufficiently distinct to make the case for the continued or expanded use of citizen-based, large-scale participatory foresight processes? In coming to address this question, we sought to identify those research topics that were shared across citizen-based and expert-based projects, but whose comparative results shed light on the differentiating characteristics of citizen-based process results.

Table 1 Overview of results

General differences

Overall, the displayed data suggests a general misalignment between CIMULACT topics and the compared expert-based reports. Considering that all CIMULACT topics showed CrossComp scores of lower than 50% indicates severe differences in coverage and/or qualitative alignment, speaking to notable differences between citizen-based and expert-based results. Given that Qualitative Alignment scores were much lower than Degree of Coverage scores—we believe that these differences can be classified as “uniqueness” or “novelty” of citizen-sourced results.

Further supporting this argument, consider the following calculations. If a topic had only 50% DoC and 50% QA, its CrossComp score would be 25%. This means that if only half of the expert reports mentioned a research topic and discussed it in such a way that semi-aligned the expert and CIMULACT recommendations, a CrossComp score of 25% would be achieved. Though this is a seemingly low threshold, expert reporting achieved this for only 12 of the CIMULACT research topics. This means that nearly three-quarters of all CIMULACT research topics offered significantly differentiated content in a qualitatively distinct fashion.

Discussing exemplary topics as demonstrative of comprehensive alignment and differences

While each research topic could be the focus of a more in-depth analysis, we have selected one research topic from each of the five tiers to examine as demonstrative of the comprehensive alignment metric and a general understanding of the differences between citizen- and expert-based foresight towards policy craft (see Table 1, in bold). All selections are research topics from the final 23 topics chosen and elaborated in the final policy conference of CIMULACT (see step six, section “CIMULACT—Citizen and Multi-actor Consultation on Horizon 2020”), except the example for level five—data for all—as neither of the topics from this tier were included in the final 23 research topics.

Level 1: technology as a means of well-being

Given the technocratic emphasis of expert reports as outlined in our original research [61], the low overall alignment score for this topic was initially surprising, especially with so much emphasis placed on technological panacea for economic development, healthcare, energy, security, etc. It is hard to imagine how citizens could seemingly contribute something focused on technology writ large and yet so drastically out of line with expert foresight reports. Could this input be valid, and, if so, is it then a unique contribution?

In examining the citizen definition of well-being, however, hints towards an explanation for this seeming anomaly begin to emerge. In the CIMULACT research topic definition, we find terms such as: “we govern them [technologies],” “end-user centric,” “negative consequences [of technology],” “screen addiction,” “shifting relationships,” “thinner boundaries between virtual and real actions,” and “exploitation at the workplace.” Through this citizen-generated rubric for understanding “well-being” as a product of technological development, the qualitative differences between experts and citizens become clearer. While experts focus on the regional or global impacts of a technological field, citizens are concerned with the impacts of such technologies with respect to their personal agency, their relationships with devices and data, and their social contracts regarding work and community.

Level 2: good food research

The necessity of food to human (and other) life begins to explain why this research area was more widely shared between experts and citizens, but this same necessity also brings even improved CrossComp scores into question. Since the future prospects of food would be an essential component of any scenario, how can this CIMULACT research topic vary so widely from expert notions of future food research policy? Indeed, this question itself betrays part of its solution.

The explicitly detailed goals that citizens proposed for future food research demarcate CIMULACT research topics from expert foresight reports. While it must be acknowledged that over two-thirds of the expert report mentioned food as a research area, this only compounds the qualitative differences between these reports and citizen recommendations. According to our analysis, citizen recommendations were far more focused on the specific goals that research should be pointed towards—a combination of ends, as opposed to a singular field.

For example, the ambiguous term “biotechnology” was deployed widely across expert reports with respect to food research and was sometimes linked to more specific pursuits—genetic engineering, “bioeconomy,” or synthetic proteins. In many cases, research in these technological fields is linked to increasing food yield and land-use efficiency, often linked to changing global demographics. In a similarly ambiguous fashion, CIMULACT calls for research into “biotechnology,” but aims such research at “impact of new research driven paradigms on health, economy, environment, and sustainability in a comprehensive and systemic way.” While there is a clear shared interest in researching biotechnologies, there are differences concerning to what ends such research should be pursued, and what concerns shall drive that research.

CIMULACT further distinguishes its set of recommendations for Good Food Research by calling for the following: “Applied research can provide knowledge and information,” “dissemination of outcomes,” “respective regulations and policies,” “Implementation of educational programmes,” and “use new food in schools.” Again, the specificity of the modes of research to be conducted, the generation and dissemination of information, and the spaces in which research can interface with society remain unique to CIMULACT. These are more than just qualitative differences, but also reflect methodological approaches, participant selection, and multi-faceted awareness raising campaigns.

Level 3: evidence-based personalized healthcare

This topic area was mentioned in some fashion or another by 75% of the analyzed expert reports, clearly indicating that healthcare, and personalized healthcare specifically, is a shared priority area for future R&I policy. However, despite the broad expert coverage of elements mentioned in this CIMULACT research field, there remained overarching low alignment—again pointing to unique additions that wide-spread citizen-sourced foresight can provide.

In the instance of this CIMULACT research topic, content similarities clustered around the broad mention of “personalized health care monitors,” in particular “wearable” like smart watches and health trackers, and “personalized, holistic, data-based health services.” Both the means for increased personal data collection, and the ends to which such data could be deployed, were hot topics for future-oriented studies. However, these shared endorsements by both citizens and experts did not indicate that research should be conducted in the same way or be inclusive of the same participants.

Several key components of the citizen-based recommendations were consistently omitted from expert-based reports: bilateral technology skills training for both doctors and citizens, with social psychology for doctors, health and lifestyle for patients, and digital literacy skills for both. These were core demands within the citizen recommendations—technological and scientific development must go hand in hand with skills training for both sides of the health care equation. And yet rarely do we find emphasis on learning or co-learning of skills when these research fields are addressed by experts. It is clear that citizens see the benefit in these technologies, while simultaneously acknowledging the need for a shared knowledge and skills base to minimize the potential drawbacks such technologies can have (for instance, unsecured personal medical data).

Level 4: making dense and growing urban areas more sustainable and liveable

This topic was also covered by 75% of the analyzed expert studies and had medium level of qualitative alignment according to our comparison. With regard to its CrossComp score, it is among the top four CIMULACT research topics. Urbanization, “smart cities,” and urban design were among the terms frequently used by expert foresight work to address issues in line with this CIMULACT topic area. Many reports discussed the global trend of increasing urban populations, particularly with regard to the additional resources and infrastructure such migrations require. Some reports addressed conditions that can turn cities into hubs of innovation and creativity and ultimately drive economic growth. The difficulties of governing urban areas, particularly from a logistical standpoint, were often, were often addressed as sites for technological developments: big-data analytics, semi-autonomous digital agents (and the “internet of things”), system-wide real-time monitoring, and drone transportation, to name a few.

Many of these topics and issues are also recognized within the CIMULACT research topic, though qualitative differences emerge in the framing of research programs to address them. For instance, CIMULACT calls for “challenges of density, diversity, ecology, population development, and financial addressing [them] in combination, not on their own, using different forms of citizen consultation....” Through this prescription, CIMULACT explicitly deviates from expert reports in two important ways.

Firstly, it calls for research not within possible technological solutions, but comprehensive mapping and monitoring of the complex relationships that shape urban “systems” (logistics, transport, energy, community, culture, etc.). While expert-based reports acknowledged the inherent complexity of urban areas, few made direct calls for research into these systems, and fewer still inclusive of “social” or “soft” systems. Citizens explicitly called for research that reaches across layers of the urban “fabric,” and holistically looks at interwoven urban systems, to specifically discover what “sustainability” means for urban living and how it can be achieved. Second, it calls for research that deploys the tacit knowledge of the urban citizenry by calling for multiple mode of participatory engagement. This was a rarity in most expert reports and yet is the critical mode of engagement outlined in CIMULACT. While expert reports often refer to research being developed by public and/or private technological development labs as pointing to the way forward, CIMULACT advocates that co-generative process—between citizens, researchers, and policy makers—should be foundational to all urban research. Again, these differences are more than just subjective whimsy. Rather, they mark clear, unique desires of citizens both in what they propose as research goals, and the methodological approach they see as most beneficial.

Level 5: data for all: sharing the power of data

The growing role of “Data,” and the affordances that technological leaps in communication technology have lent to the process of data collection and analysis, have had a growing role in shaping the conversation around the relationship between data and society. This trend made the issue almost universally addressed by expert foresight reports and the CIMULACT project. Thirteen of the 16 reports in our comparison study made mention of “big data” or a related variation of the term.

This topic also had one of the strongest qualitative alignment scores between CIMULACT and expert reports, primarily due to the shared concern of data security and to a lesser degree data validation. Concerns with how personal data was collected, stored, and used were seen as core areas for future research towards ensuring that data could be put to use in a fashion that protected the citizens it was meant to serve. Additionally, both the citizens and experts shared the idea that research and development of modes of data analysis (advanced computing, algorithm design, etc.) were priorities for future R&I policy.

The unique citizen contribution to this discussion, and also the core reason for a CrossComp score below 50%, was the demand for “data of data, data access (open), and ethical data use...” These concepts were not widely shared with expert foresight reports, and serve to differentiate the citizen view of whom data should be used by, and the purpose big-data analysis should serve. While citizens called for increase data literacy skills, both in self-protection of privacy and in learning the skills needed to put data to use in their own communities, expert discussion of big data often assumed that both data and its analysis would fall under a proprietary framework of governance. Discussion of open access data was far from the norm in expert-sourced foresight, as was explicit research into securing such access and providing skills training to citizens.

A final deviation comes in the explicit call by CIMULACT for research into utilizing data and analysis as a mode of “participating in collective decisions.” Citizens it seemed were not just interested in accessing data, and learning how to conduct analysis, but they wanted this mode of input to foster agency in social and political processes. This research strand found little resonance within the expert studies, and yet serves to orient the rest of the CIMULACT research topic towards a broader, and perhaps more meaningful, goal with respect to the relationship between the citizenry and governance.


In this contribution, we provided an overview of method, results, and theoretical positioning of a large-scale participatory agenda setting process (CIMULACT). We then described our developed analytical approach, applied to compare these citizen-based results—research topics—to those of expert-based foresight studies with similar aim of informing EU R&I policy. For comparison, we chose those research topics from CIMULACT that were sufficiently covered by expert-based reports. In so doing, we provide findings to answer our research question: are CIMULACT results sufficiently differentiated from expert-based foresight to justify a continued or expanded use of citizen-based, large-scale participatory foresight processes for R&I agenda setting?

Firstly, it is important to acknowledge the limitations of this study and the methodology employed during our research. Comparative analysis of texts, particularly with respect to the hermeneutics involved within each text’s passages, is an inherently qualitative effort and demands a high level of human resources—time and cognitive processing necessary to read and deliberate productively. These expenditures placed restrictions on the number of expert reports we could include in our survey, and the overall number of textual analyses we could perform. Additionally, with respect to the mandatory qualitative approach, there can be no definitively correct assessment scores on which to answer our research question. We have done our best to create the comparative data and interpret its meaning towards a wider field, but inevitably differences of opinion will arise within the community (as they already have within out small research team). In addressing both the issue of resource restrictions and human inconsistency, we have already begun work on methods that can provide an additional baseline of analytical precision through natural language processing algorithms (research forthcoming). While we do not see this as a cure-all solution to the limits of this research, we do see a value in providing such methods to future researchers engaging in comparative foresight research.

Secondly, according to our findings, we assert that given a robust method, time, and sufficient resources, large-scale, inclusive processes for R&I policy-oriented foresight certainly provide unique results, in a form applicable to serve as input for agenda setting. As outlined above, from the thousands of pieces of citizen input, 46 distinct research topics were synthesized—each a narrative of multiple social needs coming into conjunction with opportunities for experimentation-driven innovation. Each research topic outlines both specific fields for R&I investment, while simultaneously outlining how those fields support one another in working towards more solutions for more complex social challenges. Clearly therefore, CIMULACT does deliver results in line with the project’s stated goal of informing EU R&I policy.

More importantly, however, when looking at research topics shared by both experts and citizens, our research concludes that citizen-based policy advice qualitatively differs from that elicited by expert-based reports, to a considerable degree, in terms of both direction and focus pursued. For instance, our research points to citizen-based topics’ focus on impacts of technological developments on personal well-being, relationships, work ethics or community life—all focal points that expert-based foresight studies seldom mention or emphasize. Additionally, while broad and inclusive in approach to certain challenges, citizen-based topics were often directed at very specific goals, often with distinct everyday life relevance, portraying clear societal needs.

Furthermore, if directions for research were commonly endorsed in the compared studies, citizen-based and expert-sourced advice to R&I policy differed with regard to how research should be conducted and whom should participate. Overall, CIMULACT topics frequently demanded multi-perspective and inter- and transdisciplinary approaches to stated challenges, while expert-based reports more often suggested traditional research formats in singular fields.

In light of these findings, we believe that a strong case can be made for the continued development and utilization of large-scale participatory methods deployed to inform future-oriented policy drafting processes. While acknowledging the increased costs that inclusive methods like those of CIMULACT are likely to incur, we believe that this comparative study stands as firm evidence of the methods’ value—supporting claims that citizens will create novel and pertinent results when given the opportunity.