Policy Sciences

, Volume 45, Issue 1, pp 23–46 | Cite as

Diversity and constructive conflict in stakeholder dialogue: considerations for design and methods

Open Access


Diversity is generally recognized as a key issue for learning in stakeholder dialogue on wicked sustainability issues. Yet the question on how design of stakeholder dialogue and supporting methods actually enhance learning in stakeholder dialogue deserves more attention. This paper presents constructive conflict as a central design issue for stakeholder dialogue. This means that a dialogue entails the articulation of a diversity of perspectives and the confrontation of claims and ideas based on these perspectives. Building on three properties of diversity (variety, balance and disparity), the methodological implications of constructive conflict as a central design issue will be derived. These implications are structured according to three design steps: stakeholder identification and selection, articulation of perspectives and confrontation of claims and ideas. It is argued that social scientific methods are needed to support design of stakeholder dialogue. Q methodology is presented as an example that was used in a stakeholder dialogue on sustainable biomass in the Netherlands to identify stakeholder perspectives, to select stakeholders and to structure the dialogue. The paper wraps up with conclusions on constructive conflict as a design issue.


Constructive conflict Diversity Participatory method Learning Stakeholder dialogue Q methodology 


The notion of diversity is generally acknowledged as a key issue for participatory processes. Related to the rationale for participation (Fiorino 1990), different motivations to support diversity exist. The focus of this paper is on participation in policy processes on wicked sustainability issues. Wicked problems (Rittel and Webber 1973; also referred to as ‘messy’: Ackhoff 1974; as ‘wicked problems of organized complexity’: Mason and Mitroff 1981; as ‘unstructured’: Hisschemöller and Hoppe 2001; and as ‘ill-structured’: Simon 1973; Dunn 1988; Mitroff and Sagasti 1973) involve high and divergent societal stakes and (scientific) uncertainties. Stakeholders often disagree on the question what the goal of policy should be, as well as what the relevant means are for attaining that goal (e.g. which policy measures). Diversity is considered critical for dealing with wicked problems. Diversity is needed to enhance the quality of knowledge for policy decisions on wicked issues. The wicked character of sustainability issues means that they can often be framed in different ways and that they involve many (scientific) uncertainties and different values. As a consequence, there is a legitimate need to ‘integrate’ and ‘assess’ complex and uncertain expert knowledge (Gough et al. 2003; Funtowicz and Ravetz 1993). A technocratic orientation is thus too narrow for policymaking on complex societal issues related to sustainability (Fiorino 1990; see also Voss et al. 2009). Diversity is needed in order to enrich the policy process with different types of knowledge, expertise and values. One way to do this is through stakeholder dialogue. A stakeholder dialogue is defined as an organized meeting of stakeholders with different knowledge, values and backgrounds, who would otherwise not meet (or not all together), structured to a greater or lesser extent by means of specific methods, tools or techniques. A stakeholder is defined as someone involved in, affected by, knowledgeable of, or having relevant expertise or experience on the issue at stake (based on Van Asselt and Rijkens-Klomp 2002).

Diversity in stakeholder dialogue is strongly linked to the concept of ‘learning’. Learning is a frequently used concept in studies on participation in relation to innovation processes and sustainability (e.g. Webler et al. 1995; Grin and Van de Graaf 1996; De Marchi et al. 2000; The Social Learning Group 2001; Robinson 2003; Van de Kerkhof and Wieczorek 2005; Kamp 2007; Blackstock et al. 2007). In a stakeholder dialogue, learning takes place through the interaction with other actors. This has been referred to as ‘social learning’ in social psychology (Bandura 1971; Bandura 1986). The idea that interaction between people with different perspectives can lead to the emergence of new insights is generally accepted in literature (Jehn et al. 1999; Levine and Resnick 1993; Hoffman 1959; Hoffman and Maier 1961; Hisschemöller et al. 2001; Hisschemöller 2005; Webler 1995; Gibbons et al. 1994; Funtowicz and Ravetz 1993). Yet the question what the aim of stimulating learning through the diversity implies for dialogue design and choice of supporting methods is often not explicitly addressed. There are two main reasons for which the answer to this question is not obvious. The first reason relates to the question how to ensure the inclusion of diversity in a stakeholder dialogue. This question immediately raises another question: what does it mean to have a diverse dialogue, i.e. what is diversity? This question will be explicitly dealt with later in this paper (section “Constructive conflict as a design issue for stakeholder dialogue”). The second reason relates to the question how a dialogue, assuming that it includes a diversity of stakeholders and stakeholder perspectives, makes use of its diversity. Although diversity is considered an essential characteristic for stakeholder dialogues, groups often fail to systematically explore diverse issues and thus to benefit from their diversity (Stasser and Titus 1985). Groups rather tend to engage in consensus confirming discussions and to avoid conflict (Stasser and Titus 1985; Schweiger et al. 1986; Stasser et al. 1989; Stasser and Stewart 1992; Wittenbaum et al. 1999; Wittenbaum 2000). This has been labelled the ‘diversity paradox’ (Joldersma 1997).

The issue of dialogue design and method choice to support learning through diversity in stakeholder dialogue thus deserves more attention. This paper elaborates upon the question of how stakeholder dialogue can benefit from diversity by means of constructive conflict. It illuminates a number of methodological considerations for the design of stakeholder dialogue on wicked policy problems and illustrates these with an example. The example concerns a stakeholder dialogue that was set-up with constructive conflict as a central design issue, with particular attention to Q methodology to stimulate constructive conflict. The paper is structured as follows. The next section discusses two issues relating to dialogue design that are derived from the wicked character of sustainability issues. These issues concern the aim of stakeholder dialogue and procedures for stakeholder selection. The section thereafter goes into more detail into diversity and constructive conflict in stakeholder dialogue. It explicates the concept of diversity and analyses, on the basis of a multidisciplinary literature review, how diversity can stimulate learning through constructive conflict in stakeholder dialogue. From this review, a number of methodological implications are derived in the next section. These implications are structured along three design steps: stakeholder identification and selection, articulation of perspectives and confrontation of claims and ideas. In the following chapter, an example is discussed. The Biomass Dialogue is a stakeholder dialogue on energy from biomass in the Netherlands that was developed with these methodological implications in mind. Q methodology was applied in this dialogue to support stakeholder selection and to structure the dialogue. The final section wraps up with conclusion and reflection.

Wicked problems and stakeholder dialogue design

Two issues relating to the design of stakeholder dialogue need elaboration, as they are critical for the methodological implications of constructive conflict that will be discussed in the next section. These issues follow from the wicked nature of sustainability issues. The first issue concerns learning as the aim of stakeholder dialogue, the second issue concerns procedures for stakeholder selection that are congruent with the wicked nature of sustainability issues.

Aim of stakeholder dialogue: learning through constructive conflict

The nature of wicked problems suggests that problem boundaries are not given but need to be probed. Different stakeholders have different perspectives on a problem and its potential solutions and accordingly different problem definitions. This emphasizes the importance of stakeholder dialogue for problem definition rather than problem solving. There have been several contributions to the policy sciences field that highlight the pitfall of trying to solve a problem without exactly knowing what the problem is (e.g. Cockerill et al. 2009; Hisschemöller and Hoppe 2001; Van de Kerkhof 2006a, b). This pitfall has been referred to as ‘type III error’, which means ‘solving the wrong problem’ (Mitroff 1974; Raiffa 1970: p264; cited in: Dunn 2001).1 A strong focus on problem solving wrongfully suggests that participants are well aware of others and their own position. In the case of wicked problems, especially, this is often not the case. Reflecting the argumentative turn in policy analysis and planning (Fischer and Forester 1993), a focus on problem definition rather than problem solving means that a stakeholder dialogue is a problem structuring activity: “socio-political interaction aimed at becoming aware of a problem through generating, using, exchanging, confronting, evaluating and integrating as much (contradictory) information as possible, which is enclosed in causal, normative and means-end presumptions about the problem and its solution”(Hisschemöller 1993).

The aim of stakeholder dialogue as a problem structuring is thus to learn about a problem and its solutions (see Dunn 2004). More specifically, learning in stakeholder dialogues on wicked issues means gaining an improved understanding of the diversity of perspectives on the problem and its potential solutions (see also Renn 1999). Diversity of perspectives is critical for enhancing learning processes on complex issues (Schweiger et al. 1986; Jehn et al. 1999 who also showed that this is not the case for routine tasks). Learning in this sense starts with realizing that not only one unique perspective exists, but at least two (Müller-Merbach 2004). A perspective is defined as the integrated whole of beliefs, values and presumptions that people use to get to grips with a particular problem (similar to a frame: Rein 1986; Schön and Rein 1994; and to a policy theory: Hoogerwerf 1990). A perspective shapes a person’s perceptions and determines how someone perceives a particular problem and its solution. It represents a way of making sense of and acting upon reality.

Yet diversity of perspectives is no guarantee for learning. All kinds of mechanisms exist that hamper an open exploration of divergent perspectives (Cuppen 2009; Stasser and Titus 1985; Greitemeyer and Schulz-Hardt 2003). For instance, people tend to take up information that underlines their initial ideas, rather than information that conflicts with their initial ideas. Also, the influence of jargon may lead to situations in which some people fully understand what is being discussed and others not at all (this is for instance even the case when different scientific disciplines interact). This paper suggests constructive conflict as a mechanism to enhance learning in stakeholder dialogue. Constructive conflict refers to an open exploration and evaluation of competing ideas and knowledge claims in order to achieve new ideas, insights and options for problem solving. It takes place through a process in which participants confront each other’s claims with their own claims, unravel argumentations, make (implicit) assumptions explicit, and jointly develop new ideas that are more robust. The term has been introduced before by Amason et al. (1995), and it has also been referred to as productive conflict (De Dreu 1997; Jehn 1997) and creative conflict (Müller-Merbach 2004). Although the word ‘conflict’ may have negative connotations, diversity of perspectives is actually the reason why stakeholder dialogues are thought to be effective in the first place (Amason et al. 1995). Critical for stakeholder dialogue is the question how conflict remains constructive rather than destructive. This question will be addressed in the next section.

A stakeholder dialogue enhances learning through deliberation rather than consensus building (see Van de Kerkhof 2006a, b). This does not mean that a stakeholder dialogue cannot result in consensus. Rather, it means that participants are not forced to reach a consensus, as this may put impediments to the creation of useful insights for policy-makers and stakeholders and may lead to the adoption of invalid assumptions and/or inferior (policy) choices. Mitroff and Emshoff (1979: p10) state that: “the danger is not in reaching compromise, but in reaching it too soon and for the wrong reasons, e.g. because of the inability to tolerate conflict as a sometimes necessary and valuable tool for policy making”. Instead of focusing on consensus, a focus on confrontation of competing viewpoints prevents shifting away from reaching a quality decision towards reaching an agreeable one (Coglianese 1999 cited in Van de Kerkhof 2006a, b). A stakeholder dialogue is thus open-ended, in the sense that it does not necessarily result in one consensus policy option or strategy. With learning as the aim, closure in the problem structuring process ideally results in a synthesis of divergent perspectives. This means a new, robust understanding of the problem and its potential solutions. Synthesis contains an assessment of the problem and divergent ways for dealing with the problems, with the broad range of perspectives as input. So, rather than resulting in one consensus or compromise outcome, it results in an overview of divergent ways for dealing with the problem, taking into account congruency (Grin and Van de Graaf 1996) between perspectives.

Congruency of stakeholder selection procedures

The fact, that boundaries of wicked problems are not given but need to be probed, suggests that the design of stakeholder dialogue needs to avoid imposing predefined boundaries. Dunn (1997) refers to this as ‘estimating the boundaries of ignorance’. He uses the notion of ‘congruency’ to indicate that methods to support problem structuring should fit the type of problem under investigation. Congruency is especially important with relation to stakeholder selection procedures. A congruent method for stakeholder selection is based on the empirical identification of perspectives. Random sampling for example is not congruent, as it is based on invalid problem perceptions, such as an even distribution of relevant perspectives among the stakeholder population. Also, the often-used selection procedure that is based on actor type (invite stakeholders from industry, academia, NGOs, policymakers, etc.) is not necessarily congruent as it is based on the assumption that diverse stakeholder types reflect diverse perspectives. This is, however, an empirical question (the assumption was shown to be incorrect for the case of sustainable biomass: Cuppen et al. 2010). The design of stakeholder dialogue should thus foster an open and unbiased process without (implicit) assumptions on questions such as who can deliver facts to the process, who can deliver values to the process, which variables need to be included, which facts are (un)contested, etc. Rather than assumed, these questions need to be empirically investigated. This reflects the constructivist notion that facts and values are strongly interwoven for (policy) problems. What is a fact to one person may be a value to another person and as a consequence conflicts can arise through conflicting values as well as through conflicting knowledge claims (Hisschemöller 2005). This paper will investigate how and which (participatory) methods can support such an open and unbiased process of problem definition.

The constructivist notion implies that methods that are based on a distinction between facts and values are not helpful for structuring wicked problems in stakeholder dialogue. To clarify this claim, we can look at some public participation methods that implicitly make this distinction. Methods such as consensus conference (Joss 2000; Einsiedel and Eastlick 2000; Oughton and Strand 2004), planning cell (Dienel 1978), cooperative discourse (Renn 1999, 2004) and citizen’s jury (Crosby 2003; Huitema et al. 2007) are based on the idea that participatory methods are a way to combine technical expertise and rational decision-making with public values and preferences (see e.g. Renn et al. 1993). This presumption reflects a distinction between experts and laypeople and attributes ‘technical expertise’ to the domain of experts, and ‘values and preferences’ to the domain of laypeople. The distinction between experts and laypeople seems to reflect a perception of certainty on the relevant expertise for dealing with the problem, although no such certainty may really exist. Moreover, as many differences as there may be between experts and laypeople, just as many differences may there be among experts and among laypeople. Rather that presuming which groups can contribute in what ways, the question ‘who has relevant expertise and would be willing to contribute?’ is more interesting. In that sense, Collins and Evan’s suggestion to abandon the oxymoron ‘lay expertise’ is quite appealing (Collins and Evans 2002).2 They argue that—as the dictionary definition of ‘layman’ includes the meaning ‘someone who is not an expert’—it makes more sense to talk about ‘experts’ than ‘lay experts’. Lay people are experts with respect to their own problems (Mitroff et al. 1983), although the expertise of these (former lay-) experts has not been recognized by certification (Collins and Evans 2002). Expertise is not restricted to technical or scientific expertise, but it can be experience-based or related to particular values or interests. The notion that laypeople have particular expertise to offer will probably not be denied by the scholars working on the methods mentioned above. The point is, however, that stakeholder dialogues on wicked problems should go beyond the lay-expert distinction. Method and design for stakeholder dialogue should treat the identification of stakeholders with relevant expertise (regardless of whether he is an ‘expert’ or a ‘layperson with expertise’) as an empirical question rather than a (implicit) assumption.

This question can be tackled by using congruent methods for stakeholder selection. That is, if we say that stakeholders have particular expertise becauseof their perspective, we need to find methods that can empirically investigate perspectives and select stakeholders on the basis of those perspectives. This approach allows for involving different types of expertise without defining them beforehand.

Constructive conflict as a design issue for stakeholder dialogue

Conflict is not in any form constructive. The concept of diversity helps us to define how stakeholder dialogue can benefit from conflict, or in other words, when conflict is constructive.

What is diversity?

The idea that diversity is an important merit of stakeholder dialogue is obviously not new. However, what is meant with diversity often lacks definition. In order to understand how exactly stakeholder dialogue can benefit from specific forms of diversity, three properties of diversity need to be distinguished. These three properties are the following (Stirling 1998): variety, balance and disparity. ‘Variety’ refers to the number of categories into which the elements can be divided. ‘Balance’ refers to how the elements are distributed among the categories. ‘Disparity’ refers to the degree and nature to which the categories themselves are different from each other. As an illustration of these three properties of diversity, let us suppose we have to evaluate a policy for the diversification of neighbourhoods. One thing we could do is assess the diversity of neighbourhoods in terms of nationalities. The three properties of diversity can be used for operationalization of diversity; in other words, when is a neighbourhood diverse in terms of nationalities? The bigger the variety of nationalities is, the bigger is the diversity of the neighbourhood. But the neighbourhood will be also more diversified when the number of people per nationality is balanced. For instance, a Dutch neighbourhood in which many Dutch people live and only a few Turkish people is not very diversified. Finally, a neighbourhood will also be more diversified if there is a certain extent of disparity between the nationalities. For instance, a Dutch neighbourhood with only Western nationalities is less diversified than a Dutch neighbourhood in which also Asian nationalities live. Disparity is a qualitative dimension of diversity. So, whereas anyone would probably agree that a Dutch and a German person are more similar in terms of nationality than a Dutch and a Chinese person, disparity will often involve a (personal) evaluative judgment. However, the same can be said for variety, as the categorization of elements is not fixed. It will depend on someone’s particular perspective how variety and disparity are constituted.

Variety, balance and disparity in stakeholder dialogue

From research on working groups, it appears that variety can improve the quality of group decision-making: heterogeneous groups produce higher-quality outcomes (better strategies) than homogeneous groups (Hoffman 1959; Hoffman and Maier 1961). Although the working groups in these two studies were heterogeneous with regard to personality (measured through a standard survey to assess personality traits) rather than perspective on problem and solutions, this effect also seems to hold for groups that are heterogeneous with regard to perspectives. The inclusion of a variety of perspectives induces more divergent thinking, consideration of multiple perspectives and consideration of higher proportions of unshared information (Brodbeck et al. 2002). According to Dunn (2001), marginal viewpoints, or rarely mentioned hypotheses, have more probative value than hypotheses mentioned more frequently—those on which there is substantial consensus. Highly probable or predictable hypotheses do not challenge accepted knowledge claims. In other words, one will probably learn more from something never heard before than from something already familiar with.

So, not only is it important that a variety of perspectives is included in a stakeholder dialogue, but also the inclusion of marginal perspectives is critical for learning. This refers to the disparity property of diversity: the more different an idea is, the larger its learning effect will be. This idea is supported by research by Brodbeck et al. (2002) who state that “minority influence facilitates open-mindedness towards alternative solutions”. They showed that the likelihood that a group identifies new and qualitatively superior decisional alternatives increases as a result of “minority-induced divergent thinking”. This idea can also be found in innovation systems literature and institutional analyses, in which it is argued that it is important to involve, e.g. small entrepreneurs because incumbent system players are usually not the first to initiate successful system innovations (Agterbosch 2006; Breukers and Wolsink 2007; Hekkert et al. 2007). Marginality refers to the newness of perspectives for stakeholders: a marginal perspective is a perspective that is not often heard in the dominant debate about the issue under study. This does not necessarily mean that there are only a small number of stakeholders adhering to the perspective, neither does it mean that a perspective that only a small number of stakeholders adhere to cannot be dominant.

In addition, the balance property of diversity is important. It appears that groups in which variety is balanced are more likely to disseminate unshared information than unbalanced groups (Brodbeck et al. 2002). Furthermore, it has been argued that the balanced inclusion of perspectives reduces groupthink (Janis 1972; Dryzek and Niemeyer 2008). This suggests that learning in stakeholder dialogue can be enhanced by the participation of an equal number of stakeholders from each perspective. This means that, regardless how dominant or marginal the perspective is in terms of percentage of the whole stakeholder population, every perspective is equally present in the dialogue.

When is conflict constructive?

Conflict is not in every form beneficial to group processes (see e.g. De Dreu and Van de Vliert 1997; Jehn and Mannix 2001; Jehn et al. 1999; Jehn 1995; Kruglanski and Webster 1991). Three particular issues appear to be important when it comes to constructive conflict in groups. First, when it comes to learning from minority dissent, learning seems to benefit more from authentic conflict than from artificial conflict, as is the case in (some instances of) the devil’s advocate approach (Nemeth et al. 2001; Nemeth et al. 2004). Nemeth et al. (2001) investigated in their study whether groups in which someone was assigned the role of devil’s advocate and who was known to believe that position (the so-called consistent devil’s advocate condition) produce similar results as groups in which someone was not assigned a particular role in the group (the so-called authentic minority condition). In both conditions, the person advocated a dissenting minority viewpoint. Their study shows that groups in the consistent devil’s advocate condition produce fewer solutions to a specific problem than groups in the authentic minority condition. In addition, the number of quality solutions (as judged by two independent raters) is higher for groups in the authentic minority condition than for groups in the consistent devil’s advocate condition. This means that the mere fact that someone is assigned the role of devil’s advocate makes groups less productive in the sense that these groups produce fewer solutions and fewer good solutions to a specific problem than groups in which the dissenter is not role-playing. This may be very specific for minority dissent, as other role-playing and gaming literature suggest that role-playing is a powerful tool for learning (see for an overview of literature Crookall 1995). For stakeholder dialogues, this is an important finding, as it suggests that devil’s advocate methods are not most suitable for stimulating constructive conflict.

Second, conflict seems to enhance learning only when it is cognitive rather than affective (Jehn 1997; Amason et al. 1995). Cognitive conflict “pertains to conflict about ideas in the group and disagreement about the contents and issues of the task” (Jehn 1997), whereas affective conflict “exists when personal and relationship components within the group are characterized by friction, frustration and personality clashes within the group” (Jehn 1997). Affective conflict typically includes tensions, animosity and annoyance among members within a group (Jehn 1995). The distinction between cognitive and affective conflict is not based on the object of a discussion (for instance facts versus values), but on the nature of a discussion. Following the constructivist notion that it does not make sense to distinguish facts from values, conflicts can arise on both, taking the form of cognitive as well as affective conflict. It appears that in groups performing non-routine tasks (which a stakeholder dialogue is too), cognitive conflict is beneficial for task performance (Jehn 1995; Jehn 1997). Groups that debate and criticize each other’s ideas are more creative than groups that do not allow conflict (Nemeth et al. 2004). However, affective conflict appears to be detrimental for group performance (Jehn 1995; Jehn 1997). It is of course the question to what extent in practice cognitive and affective conflict can be separated. Persistent cognitive conflict may very well lead to affective conflict, including all kind of affective reactions such as annoyance and animosity (De Dreu 1997).

Third, conflict needs to be manageable, in the sense that people do not feel overwhelmed by it (Jehn 1995). It is likely that there is an optimum relation between conflict intensity and group performance (De Dreu 1997). Similarly, there is an optimum relation between “newness” and “familiarity” of information, which is referred to as ‘optimal cognitive distance’ (Nooteboom et al. 2007). The concept ‘optimal cognitive distance’ suggests that there is an inverted U-shaped relation between ‘cognitive distance’ and ‘innovation performance’: “In first instance, as cognitive distance increases, it has a positive effect on learning by interaction. When people with different knowledge and perspectives interact, they stimulate and help each other to stretch their knowledge for the purpose of bridging and connecting diverse knowledge. […], However, at a certain point cognitive distance becomes so large as to preclude sufficient mutual understanding needed to utilize those opportunities.” (Nooteboom et al. 2007). The concept ‘optimal cognitive distance’ is also important in relation to the learning potential of new or marginal perspectives. It was argued above that one would learn more from something never heard before than something already familiar with. The concept ‘optimal cognitive distance’ suggests that this is only the case when the information is not too different or new, but rather ‘sufficiently’ different or new. For example, I could have difficulties in understanding, and therefore learning from, an advanced MSc astronomy course, because this information is too new or unfamiliar to me. A way to deal appropriately with the newness of information in a dialogue might be to include people who can bridge disparate perspectives or claims and as such can serve as mediators or translators of specific information.

Methodological implications for dialogue design

What can we learn from the above review on the role and value of the three properties of diversity and constructive conflict for the design of stakeholder dialogue? The methodological implications are structured according to three suggested design steps: (1) stakeholder identification and selection, (2) articulation of perspectives and (3) confrontation of claims and ideas.3

Stakeholder identification and selection

Stakeholder identification and selection is a critical step in the design of stakeholder dialogue, as it is decisive for the validity of the dialogue. That is, no matter how well the dialogue itself is designed, if the stakeholder selection procedure was not able to identify and include the variety of perspectives, problem structuring in the dialogue is seriously hampered. There is a plethora of methods to facilitate participatory processes (see for an overview e.g. Rowe and Frewer 2005). However, methods to facilitate stakeholder identification and selection remain rather underexposed. Much attention is paid in literature to the representativeness of participatory processes, “offering criticism of their frequent failure to reflect the characteristics of wider society in the groups used” (Martin 2008). It is, however, rarely clear which criteria are used to judge whether a selection procedure is representative (Crawford et al. 2003: p 46; cited in Martin 2008; see for a more detailed discussion of representativeness and types of sampling Cuppen et al. 2010).

Building on the three properties of diversity and their contributions to constructive conflict, representativeness is understood here as a stakeholder sample reflecting a balanced inclusion of the variety of perspectives that exists within the stakeholder population. Dryzek and Niemeyer (2008) refer to this as ‘discursive representation’. A balanced inclusion of the variety of perspectives means that the share of marginal, or disparate, perspectives in the dialogue is equal to that of other perspectives (in terms of number of participants). This implies that the distribution of perspectives in the dialogue, i.e. the sample, does not reflect the distribution in the wider stakeholder population. Importantly, it means that stakeholders are being selected because of their perspective on the issue under consideration rather than their affiliation or the fact that they belong to a particular group (this reflects the notion of congruency of selection procedures introduced in “Wicked problems and stakeholder dialogue design”). So rather than using actor categories as a guideline for stakeholder selection (e.g. stakeholders from industry, (local/national) government, NGOs, academia), perspectives are used as a guideline for stakeholder selection. This raises the need for the identification of perspectives before stakeholder selection can take place.

In the second section of this paper, it was discussed that the wicked character of the problems under study implies that problem boundaries need to be probed. It was argued there that the design of stakeholder dialogue needs to avoid imposing predefined boundaries and that it should avoid (implicit) assumptions on questions such as who owns what kind of relevant expertise. This means that bottom-up methods are needed for identifying the diversity of perspectives, which make it possible to link stakeholders to perspectives. Methods should assist the empirical identification of the classes within the diversity (i.e. perspectives), rather than assuming diversity through some kind of category system4 (such as stakeholder selection based on socio-demographic variables, actor types, personality traits, or value orientations such as in cultural theory: Douglas and Wildavsky 1982; Thompson et al. 1990).

In order to be able to identify the diversity of perspectives, it is important to cut across stakeholder networks. Within networks, people know each other and each other’s ideas; hence, knowledge and perspectives are likely to be more redundant than diversified (Granovetter 1973). It is therefore less likely that people from the same network will be confronted with new ideas or perspectives than when they discuss with people from other networks. This refers to the disparity property of diversity. Hence, in order to make sure that disparate perspectives are included, it is important to identify stakeholders from different networks.

Articulation of perspectives

Identification of perspectives and selection of stakeholders are strongly related in the design of stakeholder dialogue. Both take place in the preparation phase of the actual dialogue. Then, in the dialogue itself, learning through constructive conflict continues. In order to learn from the variety of perspectives present in the dialogue, perspectives need to be articulated. This can be done by clarifying argumentations on specific problem issues and relating these to the stakeholder perspectives (see for an example next section). Emphasis is on the implicit or taken-for-granted elements of perspectives, such as implicit assumptions. People are often unaware of their own perspectives, and their behaviour is often guided by tacit, implicit or taken-for-granted beliefs and presumptions. By making implicit elements explicit, perspectives can be unravelled, and as such learning about how different people look at reality and how they define a problem can be enhanced. This helps to understand what a particular perspective encompasses and how perspectives differ from, and relate to one another.

In the dialogue, articulation of perspectives can take place in ‘like-minded’ subgroups (in line with assumptional analysis, see Mitroff et al. 1979; Mitroff and Emshoff 1979) to stimulate deliberation. That is, average speech time in smaller groups is higher, and shy or cautious persons may be more inclined to speak in a small group than in a large group. This means that participants with similar perspectives can jointly work out argumentations for specific claims based on their particular perspective. The operationalization of perspectives depends on the specific aim, context and topic of the participatory process one is engaging in.

Confrontation of claims and ideas

The knowledge claims, ideas and presumptions that resulted from the articulation of perspectives are compared and confronted in the dialogue. Prepared by the first two steps, this is the step in which constructive conflict should now fully evolve. It involves the challenging of claims, ideas and presumptions, and it should stimulate participants to engage in a process of reflection, rethinking or frame reflection (Schön and Rein 1994). The presumption is that when challenged to consider other, maybe even conflicting claims, participants will develop a better understanding of other ideas and the relation to their own ideas. This may either strengthen or change their own ideas.

In the third section of this paper, three issues were discussed that appear important for constructive conflict in stakeholder dialogue. Implications for methods and design can be derived from these issues. First, conflict appears to be more constructive when it is authentic than when it is artificial. This is ‘good news’ for stakeholder dialogue, in which ‘real’ stakeholders, with ‘real’ expertise, interests and values participate. Confrontation between these ‘real’ stakeholders and their ‘real’ perspectives enhances constructive conflict (more than e.g. role-play would do).

Second, conflict is only constructive when it is cognitive rather than affective. In terms of methods and design, cognitive conflict can be stimulated by the confrontation on a specific level. This means that the objects of discussion are specific technological or policy options, and that confrontation takes place on the level of knowledge claims rather than perspectives. Confrontation of perspectives will not result in a fruitful dialogue; there is no possibility of falsifying perspectives, as perspectives are the lenses through which people make sense of social reality (Schön and Rein 1994). Confrontation of perspectives will therefore result in conflict, but not in constructive conflict. Rather, it will create a deadlock, because no one perspective is ‘better’ or ‘more true’ than another. Confrontation of perspectives will more likely result affective conflict, which has been shown to be detrimental for group processes. It is only at the level of specific technological or policy options that people with different perspectives can ‘find’ each other (Grin and Van de Graaf 1996; Guba and Lincoln 1989). Yet the concrete level of discussion is no guarantee for cognitive conflict. As was mentioned above, there may always be a risk that cognitive conflict turns into affective conflict. This needs to be avoided and should be an important point of attention for the facilitator of the dialogue. It needs to be emphasized in the dialogue that participants are not required to reach a consensus or an agreement and that they are allowed to disagree.

Third, conflict needs to be manageable, i.e. there is an optimum in the relation between the distance between perspectives and the learning potential. So, whereas the inclusion of disparate perspectives enhances constructive conflict and thus problem structuring, it may be necessary to include stakeholders with ‘intermediate’ perspectives who can serve as ‘translators’ or ‘bridges’ between disparate perspectives. Another way to manage conflict is by dividing participants into smaller subgroups, each of which is chaired by an external chairperson. This chairperson should be someone who is perceived as being knowledgeable and independent, in the sense that he or she can openly engage in an exploration of competing viewpoints, and when necessary, can bridge disparate perspectives or ideas.

An example: the Biomass Dialogue

The aim of the Biomass Dialogue was to develop ideas about sustainable biomass chains for the Netherlands and to identify what is needed in order to realize these chains. Sustainable energy from biomass can be labelled a wicked problem, as there are many uncertainties and disagreements with regard to the knowledge and values at stake. Values, or goals for policy, include for example security of energy supply, climate change, sustainability and independence from geopolitically instable regions. (Scientific) uncertainties relate for instance to the CO2 balance of biomass chains, (socio-)economic impacts and land use changes.

The Biomass Dialogue ran from May 2007 to 2008 and consisted of a preparation phase and three workshops. It was organized under the umbrella of two different research projects.5 Researchers from these projects formed a project team and were responsible for design and facilitation of the dialogue. For some parts of the dialogue, external chairmen were invited. The first workshop was aimed at analysing the current situation in order to identify the problems that need to be tackled. In the second workshop, a desirable future vision was developed on the basis of specific biomass chains that were proposed by participants. This was taken in the third workshop as a starting point to ‘back cast’ (see e.g. Robinson 1982; Quist and Vergragt 2006) the implementation trajectory that is needed to get there.

In addition to the substantive aim of the dialogue mentioned above (develop ideas about sustainable biomass chains for the Netherlands and identify what is needed in order to realize these chains), there was a methodological aim. This methodological aim concerned the application and evaluation of a methodology for problem structuring in stakeholder dialogue that was developed in another research project (labelled as Constructive Conflict Methodology).6 This methodology follows the rationale as set out in the previous sections of this paper. The aim of the methodology is to enhance learning in stakeholder dialogue through constructive conflict. In this section, the methodological implications that were suggested in the previous section will be discussed in the context of the Biomass Dialogue.

Of crucial importance for the discussion on the role of constructive conflict in the Biomass Dialogue is the use of Q methodology. Therefore, before explaining how constructive conflict was enhanced throughout the Biomass Dialogue, a short introduction to Q methodology and the way it was used within the Biomass Dialogue is given.

Q methodology

Q methodology (Stephenson 1935; Brown 1980; McKeown and Thomas 1988) was developed as a method to study subjectivity. It can be used to identify the range of perspectives on a particular issue. Q methodology—a combination of qualitative and quantitative method—has been applied in various fields of social science. It has for instance been used to identify views regarding citizenship, the public interest, environmental policy and the quality of participation processes (Clarke 2002; Dryzek and Berejikian 1993; Ellis et al. 2007; Van Eeten 2001; Webler et al. 2001; Barry and Proops 1999; Davies and Hodge 2007; Swedeen 2006; Wolsink 2004; Breukers 2006).

Three characteristics of Q methodology make it especially relevant for stakeholder dialogue. Firstly, Q methodology is open with respect to the boundaries of the problem under consideration. Q methodology empirically constructs perspectives without imposing predefined categories. The merit of Q methodology is that “by allowing the categories of the analysis to be manipulated by respondents, the researcher loses the exclusive power to signify the reality of the researched” (Robbins and Krueger 2000, p. 645: 645). This in contrast to for example surveys or questionnaires which include specific items supposed to measure specific pre-defined variables. Q methodology identifies respondents’ views on statements in relation to all statements presented (see e.g. Dryzek et al. 1993), whereas in surveys and questionnaires respondents are asked to express views on isolated statements. Secondly, Q methodology allows for diversity in stakeholder perspectives. It enables addressing the three properties of diversity: variety, balance and disparity. Q Methodology traditionally aims to give a picture of the variety of perspectives among the population, rather than analysing the level of support for those perspectives among the population (the balance of perspectives) as in traditional surveys and questionnaires. This implies that the procedure for sampling respondents is usually different from that in surveys and questionnaires. Rather than random sampling and large sample sizes, Q methodology relies on purposive sampling and smaller sample sizes. The fact that there is a person who can be assumed to have a different point of view is enough reason to include him or her in the sample. Thirdly, Q methodology allows for statistically linking stakeholders to perspectives. This characteristic makes it possible to identify for example the stakeholders who most strongly represent a particular perspective, or who are a ‘mix’ of multiple perspectives. As a consequence, Q methodology is very useful as an empirical method to select stakeholders for participation in a dialogue. It makes it possible to select participants on the basis of perspectives rather than for example affiliation.

The central task in Q methodology is the Q sort. Each respondent ranks a set of statements on a (usually normally distributed) scale that represents significance or salience (Brown 1980 : p198), such as most agree to most disagree. Statements are selected in such a way as to reflect the diversity of the ‘concourse’. The concourse is the full range of discussions and discourses on the particular issue under study (identified through all kinds of written and/or spoken sources). Factor analysis of the Q sorts results in clusters of Q sorts (factors) that can be interpreted as perspectives. The Q sorting task is often accompanied by an interview, in which respondents are asked to explain their sort—this helps to interpret the factors (see for a detailed description of the procedural steps: Davies and Hodge 2007).

Q methodology in the Biomass Dialogue

Q methodology was employed in the dialogue with a threefold aim. First, the aim was to empirically construct the variety of stakeholder perspectives on sustainable biomass in a bottom-up way. Second, the aim was to select stakeholders for participation in the dialogue accordingly. Stakeholder selection was supposed to result in the balanced inclusion of the variety of perspectives, including disparate or marginal perspectives (a detailed account of this application of Q methodology can be found in: Cuppen et al. 2010). Third, the aim was to structure the dialogue on the basis of the perspectives.

To cover the broad range of perspectives, 200 statements were gathered from reports, websites, news items and transcripts from other stakeholder projects on biomass. This was relatively easy, as the issue of biomass was heavily discussed in the media and at several public debates at the time of this study. As a consequence, the range of ideas and opinions was relatively well articulated. The transcripts from a stakeholder dialogue organized by the same project leader were furthermore very useful for identifying marginal perspectives, as it included people and ideas that were not all very well represented in the dominant debate on biomass.7 The set of 200 statements needed to be reduced to a more manageable size, while maintaining diversity. To validate the diversity within the reduced set of statements, three things were done. First of all, the set of statements was individually categorized by three members of the project team who then identified unique statements within categories. The three categorizations and sets of unique statements were compared and discussed, and iteratively, this process continued until sixty-two statements remained. Second, the set of statements was piloted with five people (who were all well acquainted with the biomass debate) to check whether any relevant statements were missing. This resulted in a definite set of sixty statements. Third, during the Q interview, the interviewer asked the respondent whether he or she thought statements were missing. This was in general not the case. Of the few respondents who chose to add statements, it concerned emphasizing an issue that was already covered by other statements, for instance by rephrasing a statement into its positive or negative counterpart.

The goal of respondent selection was to reflect the broad range of perspectives. Criteria for selection were that the respondent was a stakeholder who was somehow professionally involved in the biomass issue and who could potentially bring new issues or points of view to the process (which can be found in statements made in interviews, on websites, in reports or in earlier projects). Respondents were identified through newspaper articles, news–websites, earlier contacts of the project team (e.g. via the earlier stakeholder project mentioned above) and snowball-sampling technique (“please mention someone with a different, and someone with a similar perspective on energy from biomass”). Identification of respondents with diverse perspectives was also relatively easy due to the broad attention to the biomass issue. Eventually, seventy-five stakeholders performed as respondents in the Q sorting task. These respondents represented different sectors and organizations: knowledge institutes and academia, (energy) companies, branch/sector organizations (e.g. sector organization for oil and fat), small-/medium-sized enterprises (including energy consultants, e.g. working on cultivation or treatment of biomass, or on energy/heat/fuel production), NGOs, national government, regional and local government.

Respondents ranked the sixty statements according to a normally distributed 11-point scale, ranging from ‘most in line with my opinion’ to ‘least in line with my opinion’. Factor analysis of the seventy-five Q sorts resulted in six factors that were interpreted based on the statements and the interviews as six distinct perspectives on sustainable biomass (see Table 1). These six perspectives are sufficiently different in order to reflect the range of perspectives on sustainable biomass. Yet there are also issues of overlap between perspectives. For instance, both perspectives 4 and 5 see biomass as a commodity in a market in which it will eventually compete with fossil fuels, but on the condition that biomass applications have a positive energy balance. Perspective 4 is, however, more positive about the question as to whether this will be feasible and sees the solution in 2nd generation, certified biomass. Perspective 5 doubts the feasibility of a positive energy balance, as well as the potential availability of biomass. Perspective 5 is, however, not as critical as perspective 2, according to which all developments need to be stopped, as we cannot prevent negative impacts for developing countries. Perspectives 3 and 6 are both entrepreneurial perspectives. Yet perspective 6 is more pragmatic and less idealistic than perspective 3, according to which the focus should be on small-scale, decentralized applications in the Netherlands. Perspective 6 does not want to make a choice for a specific scale and type of application. Another difference between the two perspectives is that perspective 3 is very critical about the role of the Dutch government and policy.
Table 1

Six perspectives on sustainable biomass

Perspective 1: Keep all options open

This perspective focuses on knowledge development. Generic claims about the sustainability of biomass applications are not possible, because the sustainability of an application is very much dependent on the specific situation. Therefore, it does not make sense to exclude specific options in advance, or to embrace others. Biorefinery (refining biomass in order to use all valuable elements within the biomass) is seen as a promising development.

Perspective 2: Hit the brakes

This perspective is very sceptical about the possibilities of sustainable biomass applications and calls for a halt. A growing international biomass market increases the risks for developing countries, with regard to environment, socio-economic situation, human rights and food supply. At the moment, there is no biomass that is sustainable for people, planet and profit. As long as we cannot guarantee sustainable biomass, we should halt the development of new applications.

Perspective 3: Support small-scale innovative initiatives

The third perspective focuses mainly on small-scale and decentralized applications in the Netherlands. Initiatives by small innovative entrepreneurs are hard to get off the ground, because the Dutch government mainly has an eye for the large companies. However, we should not expect innovations from these companies, because they benefit from maintaining the existing fossil fuel–based system. We should not keep putting money in research, but rather in implementation.

Perspective 4: Security of supply with global, certified, 2nd generation biomass

This perspective has a strong market orientation. The most important incentive for the development of biomass applications is the replacement of fossil fuels, i.e. security of supply. This perspective is optimistic about the potential of biomass, especially the 2nd generation biomass8, and states as a condition that the sustainability of biomass should be guaranteed by means of a certification system.

Perspective 5: Efficiency the goal, biomass a means?

According to this perspective, we should not overestimate the potential of biomass. In the future, other renewable sources (e.g. solar, wind) will be better suited for our energy supply, because the availability of those sources is larger. We should be critical about the sustainability of biomass applications: the whole chain should be taken into account when assessing whether there is a positive energy balance. Energy-efficiency is a key. Technology and market have not sufficiently been developed.

Perspective 6: Just do it, step by step

This perspective is pragmatic. It underlines that we cannot know at this moment what will be the best option in the future. This means that we should act now with the knowledge that we have, instead of postponing actions. All options should be kept open; there should be a broad range of applications. The role of entrepreneurs is very important in this perspective.

Stakeholder identification and selection

In line with the methodological implications discussed in the previous section, the aim of this step was to identify forty out of the seventy-five respondents for participation in the dialogue, who reflect the variety of perspectives in a balanced way. The number of forty was chosen for practical reasons. Thirty stakeholders were seen as a manageable number in the context of the Biomass Dialogue and its resources; taking into account that some stakeholders would not be willing or able to participate, forty were invited. The forty stakeholders were selected on the basis of the results of the factor analysis. Factor analysis shows for each factor (perspective) a loading for each respondent. The higher a loading, and the lower the loadings on the other factors, the more a respondent’s perspective is in line with the perspective represented by that factor. This makes it possible to identify for each perspective, the respondents that most strongly represent that perspective. For each perspective, an equal number of respondents loading most strongly on the particular perspective was invited to take part in the dialogue (balanced inclusion of the variety of perspectives). This means that the more marginal and the more mainstream perspectives on sustainable biomass are included on an equal footing in the dialogue. Perspectives 1, 2, 4 and 5 could very well be recognized in the public debate on biomass at that time in the Netherlands; this was not so much the case for perspectives 3 and 6, which can be considered marginal for the Netherlands at that time. Factor analysis also shows disparity of the factors. It results in a statistical account of similarity and difference between perspectives.

Overall, about 30 people participated in the dialogue. Not all participants visited all three workshops: fifteen participants attended the first workshop, seventeen participants the second and eleven participants were present at the third workshop.

It was argued in the previous section that stakeholder selection should not only include a diversity of perspectives, but also cut across networks in order to increase the probability that people meet new people with new ideas. To check whether the stakeholder selection procedure for the Biomass Dialogue succeeded in doing this, participants were asked in an evaluation form after workshop 1 whether they had met people at the workshop, with whom they had never discussed about the biomass issue before. All participants affirmed to this question. They were furthermore asked with the ideas of whom of the other participants they were already familiar. There was one participant (an entrepreneur in pure vegetable oil) with whose ideas eight of twelve participants9 were familiar. In general, the picture was very mixed, and most participants were familiar with some of the other participants and their ideas and unfamiliar with some of others. Hence, stakeholder selection in the Biomass Dialogue appears to have been successful in cutting across networks.

Articulation of perspectives

In the dialogue, further articulation of the perspectives took place. The project team wrote a report documenting the results of the Q analysis. This report contained an extensive description of the six perspectives. In workshop 1, this report was presented, as well as the analysis of participant’s positions with regard to the six perspectives. In general, participants recognized the perspectives. They indicated that the presentation of perspectives clarified the complexity of opinions, claims and arguments. It helped them to better understand other people’s statements and ideas. They indicated, in the evaluation after the workshops, that the perspectives contributed to a constructive and open-dialogue, because the diversity of perspectives was made explicit and more understandable. Most people also recognized their ‘analysed’ perspective, but some also indicated that they felt being a mix of two perspectives. The persons, who indicated so, were in general indeed statistically loading on more than one perspective according to the Q analysis.

Based on the Q perspectives, homogeneous subgroups were formed to further articulate the perspectives by evaluating the sustainability of a series of specific biomass chains in workshop 1. The idea behind the formation of ‘like-minded’ subgroups was that, by means of deliberation, argumentations for specific ideas could be clarified and worked out, without engaging in discussions with different-minded people (see also Mitroff et al. 1979; Mitroff and Emshoff 1979). Argumentations and assumptions were made explicit by using an argumentation model as a support tool (added by the use of ‘repertory grid technique’, see Cuppen, 2009). The specific chains were selected in order to cover a variety of existing biomass chains, differing in terms of scale, biomass source, technology, end product and organization of the chain. The chains were presented with help of cartoons visualizing the chains and a short text of one A4 page at most to explain the chain.

Confrontation of claims and ideas

Confrontation is aimed at making explicit the assumptions and beliefs that may form the basis of divergent perspectives and to stimulate participants to consider and evaluate alternative options and viewpoints. In the context of the Biomass Dialogue, this refers to assumptions and beliefs about the sustainability of biomass options. The confrontation of claims and ideas was prepared in workshop 1 by the ‘like-minded’ subgroups who worked out the argumentation models. For the confrontation in workshop 1 also subgroups were used, but now in such a way that each subgroup consisted of participants with different perspectives. These subgroups were asked to reflect on the evaluations made by the ‘like-minded’ subgroups. They could do so by adding or changing elements in the argumentation model that was produced by the like-minded subgroup.

It was discussed in the previous section that conflict is more likely to be constructive when it is authentic rather than artificial, cognitive rather than affective, and when it is manageable (i.e. based on not ‘too divergent’ perspectives). The inclusion of ‘real’ stakeholders and their ‘real’ perspectives ensured authentic rather than artificial conflict. Cognitive conflict, rather than affective, was enhanced through the use of specific existing (workshop 1) and innovative (workshop 2) biomass chains that served as a tool to derive claims and ideas about the sustainability of biomass (workshop 1) and a desirable biomass vision (workshop 2). It was furthermore stressed by the facilitators of the dialogue and the invited chairmen that participants were allowed to disagree, which contributed to an atmosphere in which participants felt open to explore divergent, even conflicting, points of view. External persons were invited for chairing the subgroups. These persons were invited by the project team because they were thought to be perceived as knowledgeable and independent, in the sense that they could engage openly in an exploration of competing viewpoints. Importantly, they were thought to be able to bridge disparate perspectives or ideas when necessary; for example to serve as a mediator between stakeholders with different perspectives or conflicting ideas. This was supposed to contribute to the manageability of conflict.

Results and conclusion

Each of the three workshops of the Biomass Dialogue resulted in a specific outcome. After each workshop, the project team produced a report describing the outcome of the workshop, which was sent to all participants. Each workshop started with a discussion of the report, and participants were asked to reflect on the report and to indicate whether they agreed or wanted to make amendments. The outcome of the first workshop was a set of criteria for judging the sustainability of biomass chains. All participants in the workshop indicated agreement on these criteria, despite the diversity of perspectives and despite the finding that different reference situations were used to judge sustainability. The second workshop resulted in three partial visions of the future about the role of biomass in a sustainable energy supply in the Netherlands. These three partial visions reflected the different ideas about the desirability of specific types of chains among participants. The third workshop resulted in scenarios describing the pathway to each of the partial visions. These three outcomes of the dialogue can be seen as a synthesis of divergent perspectives. As was noted earlier, participants were allowed to disagree, and there was no focus on consensus building. Yet the first outcome of the dialogue was a consensus outcome: agreement on a list of criteria to judge the sustainability of biomass chains. The other two outcomes, the visions and the pathways to those visions, showed divergence between stakeholders’ desired future situations and between the strategies and actions that were considered necessary and desirable. In working out the strategies and actions, it was tried to identify congruency of perspectives and visions: can we identify specific strategies and actions that fit with more than one perspective? The dialogue, however, was not able to reach this level of specificity in the time that was given for the project.

After each workshop, participants filled in an evaluation form asking for example what they learned and which parts of the workshop they did (not) appreciate. From our observations and participant responses, it became clear in the dialogue that the process of articulation and confrontation stimulated participants to reflect on their initial ideas and assumptions. Articulation and confrontation for instance resulted in the finding in workshop 1 that people arrived at different sustainability judgments because they used different reference situations. For different aspects of the chain, participants looked for different reference situations, such as an existing fossil alternative, a renewable alternative, an alternative usage of the biomass or a future situation. As such, confrontation further clarified the divergent perspectives and gave a better understanding of why people arrive at different conclusions with regard to the sustainability of biomass chains. Furthermore, the extent to which participants in the dialogue learned about the diversity of perspectives was evaluated in a quasi-experimental design (with ex post Q sorts and a control group), added by qualitative data from the evaluation forms filled in after each workshop and observations made during the dialogue (see Cuppen, forthcoming). The evaluation showed that participants in general increased their understanding of the diversity of perspectives. The perspectives were found to be helpful for participants to get grip on the complexity of the biomass issue and to understand how people with different perspectives arrive at different conclusions about biomass chains.


Elaborating on constructive conflict as a central design issue for stakeholder dialogue on wicked problems, we observe a need for (both theoretical and empirical) research on methods to support design of stakeholder dialogue. Especially stakeholder selection procedures that are based on the empirical identification of diversity of perspectives require more attention. A lot of valuable research has been done on participatory methods, procedures and tools. This mainly involves empirical applications of specific participatory procedures (e.g. Einsiedel et al. 2001; Burgess et al. 2007; Gregory et al. 2003; Guston 1999; Joss and Durant 1995; Lahdelma et al. 2000; Mayer et al. 2005; Renn et al. 1991, 1993; Stave 2002; Stirling and Mayer 2001; Street 1997; Van de Kerkhof 2006a, b) and some very useful typologies of participatory procedures (see e.g. Van Asselt and Rijkens-Klomp 2002; Rowe and Frewer 2005). Yet, many of these procedures do not necessarily fit well with the focus on wicked problems and derived from that diversity and constructive conflict as central design issues. In most participatory methods, diversity of perspectives is not explicitly addressed as a design issue. Exceptions are for example ‘deliberative mapping’ (Burgess et al. 2007) and ‘multicriteria mapping’ (not to be mistaken with multicriteria approaches that are used in a prescriptive rather than a heuristic way, see Stirling and Mayer 2001). A large part of participatory methods aims at consensus building, conflict-resolution or negotiation. In this type of methods, confrontation is something that should be avoided rather than encouraged (Guba and Lincoln 1989 state for instance that “it is engagement not confrontation that leads to reconstruction” of claims, concerns and issues). The dialectical approach (Mason 1969; Mason and Mitroff 1981) is probably the only example that explicitly assigns a central role for confrontation.

This paper showed how constructive conflict was used as a design issue in the Biomass Dialogue. Supported by specific methods and tools, constructive conflict was enhanced through stakeholder selection, articulation of perspectives and confrontation of claims. Q methodology played an important role in the Biomass Dialogue. It was presented in this paper as an example of a social scientific method that can support the design of stakeholder dialogue on wicked problems. Due to its integrated qualitative and quantitative nature, Q methodology is able to empirically construct the variety of stakeholder perspectives in a bottom-up and open-ended way. Q results were used to select stakeholders on the basis of these perspectives, in such a way that the variety of Q perspectives, including marginal or disparate perspectives, was included in a balanced way in the dialogue. The perspectives were furthermore used as a basis for articulation and confrontation. As such, Q methodology facilitated problem structuring.

This paper stresses the importance of the first phases of a dialogue: identification of perspectives and stakeholder selection. It does hardly touch upon the synthesis of a dialogue. Although it may sometimes be relevant (or tempting) to evaluate the quality of a dialogue on the basis of its outcomes (usefulness of results, agreement on courses of action, uptake in actual policymaking, etc.), type III errors can ensue from a strong focus on outcomes as it may mean a neglect of input to the dialogue. The wicked character of the problems under consideration and the subsequent aim of problem structuring legitimate a strong focus on input to the dialogue, i.e. bottom-up identification of perspectives and stakeholder selection. Yet obviously, as a next step, synthesis is an important part of dialogue that needs attention as well. We should not pretend that synthesis will naturally follow from articulation and confrontation when all previous steps have been carefully taken. Further empirical research can shed light on how synthesis can be attained in a dialogue where diversity is at the core. How to reach closure when participants need not to reach consensus? How to implement insights that result from problem structuring in stakeholder dialogue in policy process and decisions? This paper casted a quick glance on these questions based on the outcomes of the Biomass Dialogues, yet the issue of synthesis in stakeholder dialogue deserves more attention.


  1. 1.

    Type III errors are different from type I and type II errors, which involve setting the significance level too high or too low in testing the null hypothesis (Dunn 2001: p434).

  2. 2.

    Their proposal was part of a proposal for a ‘third wave of science studies’ which evoked a number of critical reactions by Jasanoff (2003), Rip (2003), and Wynne (2003).

  3. 3.

    These steps are part of a participatory approach for stakeholder dialogue on wicked issues labelled as Constructive Conflict Methodology (Cuppen, 2009). Constructive Conflict Methodology furthermore includes a synthesis step, in which closure takes place.

  4. 4.

    This is not to say that it is possible to avoid imposing some structure or frame on participants in a dialogue. A facilitator, researcher or organizer for instance imposes a frame by deciding that there needs to be a dialogue on a particular issue (and not on another issue), by choosing a particular method, and by selecting stakeholders in a particular way.

  5. 5.

    BSIK (Investing in knowledge infrastructure) Climate Changes Spatial Planning project “An integrated framework to assess spatial and related implications of increased implementation of biomass delivery chains” and the project “Strategies for implementing sustainable transition trajectories in the transport sector” funded by Netherlands Organisation for Scientific Research (NWO).

  6. 6.

    BSIK (Investing in knowledge infrastructure) Climate Changes Spatial Planning project “Probing a method to facilitate the interactive linking of expert knowledge to stakeholder assessment”.

  7. 7.

    This concerned the project “Concrete Steps to a sustainable Eemsmond”, coordinated by Hisschemöller (Vrije Universiteit Amsterdam) and Stokman (Groningen University), in which stakeholders from the northern region in the Netherlands interactively explored investment opportunities for energy options, in particular from biomass.

  8. 8.

    There is no commonly shared definition of 2nd generation biomass. An often-used definition in the Netherlands originates from the Dutch GAVE programme (support for technological development of 2nd generation technologies) and is based on the amount of CO2 emission reduction: 1st generation biofuels achieve CO2 emission reductions of around 30–50% compared to fossil fuels and 2nd generation biofuels achieve a CO2 reduction of 80% or more. However, others define 2nd generation on the basis of the biomass source that is being used: 2nd generation biofuels are made from residuals, and 1st generation from cultivated biomass. Others define 2nd generation biomass as lignocellulose (e.g. woody material), whether or not combined with a specific type of technology to convert the biomass into a fuel.

  9. 9.

    Not all participants filled in the evaluation form; there were fifteen participants in workshop 1.



The research on which this paper is based was funded by the research programme Climate changes Spatial Planning. I am very grateful to Matthijs Hisschemöller for his intellectual contribution to the work on which this paper is based. I would like to thank Emmy Bergsma and Sylvia Breukers for the pleasant teamwork in the organisation of the Biomass Dialogue. I would also like to thank Udo Pesch and 2 anonymous reviewers for their valuable comments on earlier versions of this paper.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.


  1. Ackhoff, R. (1974). Redesigning the future. New York: John Wiley & Sons.Google Scholar
  2. Agterbosch, S. (2006). Empowering wind power. On social and institutional conditions affecting the performance of entrepreneurs in the wind power supply market in the Netherlands. Utrecht: Utrecht University.Google Scholar
  3. Amason, A. C., Hochwarter, W. A., Thompson, K. R., & Harrison, A. W. (1995). Conflict: An important dimension in successful management teams. Organizational Dynamics, 24, 20–35.CrossRefGoogle Scholar
  4. Bandura, A. (1971). Social learning theory. New York: General Learning Press.Google Scholar
  5. Bandura, A. (1986). Social foundations of thought and action. Englewood Cliffs, NJ: Prentice Hall.Google Scholar
  6. Barry, J., & Proops, J. (1999). Seeking sustainability discourses with Q methodology. Ecological Economics, 28, 337–345.CrossRefGoogle Scholar
  7. Blackstock, K. L., Kelly, G. J., & Horsey, B. L. (2007). Developing and applying a framework to evaluate participatory research for sustainability. Ecological Economics, 60, 726–742.CrossRefGoogle Scholar
  8. Breukers, S. (2006). Changing institutional landscapes for implementing wind power. A geographical comparison of institutional capacity-building: The Netherlands. England and North Rhine Westphalia, Amsterdam: Amsterdam University Press.CrossRefGoogle Scholar
  9. Breukers, S., & Wolsink, M. (2007). Wind power in changing institutional landscapes: An international comparison. Energy Policy, 35, 2737–2750.CrossRefGoogle Scholar
  10. Brodbeck, F. C., Kerschreiter, R., Mojzisch, A., Frey, D., & Schulz-Hardt, S. (2002). The dissemination of critical, unshared information in decision-making groups: the effects of pre-discussion dissent. European Journal of Social Psychology, 32, 35–56.CrossRefGoogle Scholar
  11. Brown, S. R. (1980). Political Subjectivity: Applications of Q methodology in political science. New Haven and London: Yale University Press.Google Scholar
  12. Burgess, J., Stirling, A., Clark, J., Davies, G., Eames, M., Staley, K., et al. (2007). Deliberative mapping: A novel analytic-deliberative methodology to support contested science-policy decisions. Public Understanding of Science, 16, 299–322.CrossRefGoogle Scholar
  13. Clarke, A. H. (2002). Understanding sustainable development in the context of other emergent environmental perspectives. Policy Sciences, 35, 69–90.CrossRefGoogle Scholar
  14. Cockerill, K., Daniel, L., Malcynski, L., & Tidwell, V. (2009). A fresh look at a policy sciences methodology: Collaborative modeling for more effective policy. Policy Sciences, 42, 225.CrossRefGoogle Scholar
  15. Coglianese, C. (1999). The limits of consensus. The environmental protection system in transition: Toward a more desirable future. Environment, 41, 1–6.CrossRefGoogle Scholar
  16. Collins, H. M., & Evans, R. (2002). The third wave of science studies: Studies of expertise and experience. Social Studies of Science, 32, 235–296.CrossRefGoogle Scholar
  17. Crawford, M., Rutter, D., & Thelwall, S. (2003). User involvement in change management London: NCCSDO.Google Scholar
  18. Crookall, D. (1995). A guide to the literature on simulation/gaming. In D. Crookall & K. Arai (Eds.), Simulation and gaming across disciplines and cultures: ISAGA at a watershed (pp. 151–177). London: Sage Publications.Google Scholar
  19. Crosby, N. (2003). Health democracy: empowering a clear and informed voice of the public. Edina MN: Beaver’s Pond Press.Google Scholar
  20. Cuppen, E. (2009). Putting perspectives into participation: Constructive conflict methodology for problem structuring in stakeholder dialogues. Oisterwijk: Boxpress.Google Scholar
  21. Cuppen, E. (forthcoming). A quasi-experimental evaluation of learning in a stakeholder dialogue on bio-energy. Submitted to Research Policy.Google Scholar
  22. Cuppen, E., Breukers, S., Hisschemöller, M., & Bergsma, E. (2010). Q Methodology to select participants for a stakeholder dialogue on energy options from biomass in the Netherlands. Ecological Economics, 69, 579–591.CrossRefGoogle Scholar
  23. Davies, B. B., & Hodge, I. D. (2007). Exploring environmental perspectives in lowland agriculture: A Q methodology study in East Anglia, UK. Ecological Economics, 61, 323–333.CrossRefGoogle Scholar
  24. De Dreu, C. (1997). Productive conflict: The importance of conflict management and conflict issue. In C. De Dreu & E. Van de Vliert (Eds.), Using conflict in organizations (pp. 9–22). London: Sage.Google Scholar
  25. De Dreu, C., & Van de Vliert, E. (1997). Using conflict in organizations. London: Sage.Google Scholar
  26. De Marchi, B., Funtowicz, S. O., Lo Cascio, S., & Munda, G. (2000). Combining participative and institutional approaches with multicriteria evaluation. An empirical study for water issues in Troina, Sicily. Ecological Economics, 34, 267–282.CrossRefGoogle Scholar
  27. Dienel, P. C. (1978). Die Planungszelle. Opladen: Westdeutscher Verlag.Google Scholar
  28. Douglas, M., & Wildavsky, A. (1982). Risk and culture. Berkeley, CA: University of California Press.Google Scholar
  29. Dryzek, J. S., & Berejikian, A. (1993). Reconstructive democratic theory. American Political Science Review, 87, 48–60.CrossRefGoogle Scholar
  30. Dryzek, J. S., & Niemeyer, S. (2008). Discursive representation. American Political Science Review, 102, 481–492.CrossRefGoogle Scholar
  31. Dunn, W. N. (1988). Methods of the second type: Coping with the wilderness of conventional policy analysis. Policy Studies Review, 7, 720–737.CrossRefGoogle Scholar
  32. Dunn, W. N. (1997). Probing the boundaries of ignorance in policy analysis. American Behavioral Scientist, 40, 277–298.CrossRefGoogle Scholar
  33. Dunn, W. N. (2001). Using the method of context validation to mitigate type III error in environmental policy analysis. In M. Hisschemöller, R. Hoppe, W. N. Dunn, & J. R. Ravetz (Eds.), Knowledge, power and participation in environmental policy analysis (pp. 417–436). New Brunswick and London: Transaction Publishers.Google Scholar
  34. Dunn, W. N. (2004). Public policy analysis (3rd ed.). New Jersey: Pearson Prentice Hall.Google Scholar
  35. Einsiedel, E. F., & Eastlick, D. L. (2000). Consensus conferences as deliberative democracy: A communications perspective. Science Communication, 21, 323–343.CrossRefGoogle Scholar
  36. Einsiedel, E. F., Jelsoe, E., & Breck, T. (2001). Publics at the technology table: The consensus conference in Denmark, Canada, and Australia. Public Understanding of Science, 10, 83–98.CrossRefGoogle Scholar
  37. Ellis, G., Barry, J., & Robinson, C. (2007). Many ways to say ‘No’, different ways to say ‘Yes’: Applying q-methodology to understand public acceptance of wind farm proposals. Journal of Environmental Planning and Management, 50, 517–551.CrossRefGoogle Scholar
  38. Fiorino, D. J. (1990). Citizen participation and environmental risk: A survey of institutional mechanisms. Science Technology & Human Values, 15, 226–243.CrossRefGoogle Scholar
  39. Fischer, F., & Forester, J. (1993). The argumentative turn in policy analysis and planning. London: UCL Press Limited.Google Scholar
  40. Funtowicz, S. O., & Ravetz, J. R. (1993). Science for the post-normal age. Futures, 25, 739–755.CrossRefGoogle Scholar
  41. Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P., & Trow, M. (1994). The new production of knowledge. London: SAGE Publications Ltd.Google Scholar
  42. Gough, C., Darier, E., De Marchi, B., Funtowicz, F., Grove-White, R., Guimarães Pereira, A., et al. (2003). Contexts of citizen participation. In B. Kasemir, J. Jäger, C. C. Jaeger, & M. T. Gardner (Eds.), Public participation in sustainability science (pp. 37–61). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  43. Granovetter, M. S. (1973). The strength of weak ties. The American Journal of Sociology, 78, 1360–1380.CrossRefGoogle Scholar
  44. Gregory, R., Fischhoff, B., Thorne, S., & Butte, G. (2003). A multi-channel stakeholder consultation process for transmission deregulation. Energy Policy, 31, 1291–1299.CrossRefGoogle Scholar
  45. Greitemeyer, T., & Schulz-Hardt, S. (2003). Preference-consistent evaluation of information in the hidden profile paradigm: Beyond group-level explanations for the dominance of shared information in group decisions. Journal of Personality and Social Psychology, 84, 322–339.CrossRefGoogle Scholar
  46. Grin, J., & Van de Graaf, H. (1996). Technology assessment as learning. Science, Technology & Human Values, 21, 72–99.CrossRefGoogle Scholar
  47. Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park, California: Sage Publications.Google Scholar
  48. Guston, D. H. (1999). Evaluating the first US consensus conference: The impact of the citizens’ panel on telecommunications and the future of democracy. Science Technology & Human Values, 24, 451–482.CrossRefGoogle Scholar
  49. Hekkert, M. P., Suurs, R. A. A., Negro, S. O., Kuhlman, S., & Smits, R. E. H. M. (2007). Functions of innovation systems: A new approach for analysing technological change. Technological Forecasting and Social Change, 74.Google Scholar
  50. Hisschemöller, M. (1993). De democratie van problemen. Amsterdam: VU Uitgeverij.Google Scholar
  51. Hisschemöller, M. (2005). Participation as knowledge production and the limits of democracy. In S. Maasen & P. Weingart (Eds.), Democratization of expertise? Exploring new forms of scientific advice in political decision-making (pp. 189–207). Dordrecht: Kluwer Academic Publishers.Google Scholar
  52. Hisschemöller, M., & Hoppe, R. (2001). Coping with intractable controversies: The case for problem structuring in policy design and analysis. In M. Hisschemöller, R. Hoppe, W. N. Dunn, & J. R. Ravetz (Eds.), Knowledge, power and participation in environmental policy analysis (pp. 47–72). New Brunswick and London: Transaction Publishers.Google Scholar
  53. Hisschemöller, M., Hoppe, R., Dunn, W. N., & Ravetz, J. R. (2001). Knowledge, power and participation in environmental policy analysis: An introduction. In M. Hisschemöller, R. Hoppe, W. N. Dunn, & J. R. Ravetz (Eds.), Knowledge, power and participation in environmental policy analysis (pp. 1–26). New Brunswick (U.S.A) and London: Transaction Publishers.Google Scholar
  54. Hoffman, L. R. (1959). Homogeneity of member personality and its effect on group problem-solving. Journal of Abnormal and Social Psychology, 58, 27–32.CrossRefGoogle Scholar
  55. Hoffman, L. R., & Maier, N. R. (1961). Quality and acceptance of problem solutions by members of homogeneous and heterogeneous groups. Journal of Abnormal and Social Psychology, 62, 401–407.CrossRefGoogle Scholar
  56. Hoogerwerf, A. (1990). Reconstructing policy theory. Evaluation and Program Planning, 13, 285–291.CrossRefGoogle Scholar
  57. Huitema, D., Van de Kerkhof, M., & Pesch, U. (2007). The nature of the beast: Are citizen’s juries deliberative or pluralist? Policy Sciences, 40, 287–311.CrossRefGoogle Scholar
  58. Janis, I. L. (1972). Victims of groupthink: Psychological studies of foreign policy decisions and fiascoes. Boston: Houghton-Mifflin.Google Scholar
  59. Jasanoff, S. (2003). Breaking the waves in science studies: Comment on H.M Collins and Robert Evans, ‘The third wave of science studies’. Social Studies of Science, 33, 389–400.CrossRefGoogle Scholar
  60. Jehn, K. A. (1995). A multimethod examination of the benefits and detriments of intragroup conflict. Administrative Science Quarterly, 40, 256–282.CrossRefGoogle Scholar
  61. Jehn, K. A. (1997). Affective and cognitive conflict in work groups: Increasing performance through value-based intragroup conflict. In C. De Dreu & E. Van de Vliert (Eds.), Using conflict in organizations (pp. 87–100). London: Sage Publications.Google Scholar
  62. Jehn, K. A., & Mannix, E. A. (2001). The dynamic nature of conflict: A longitudinal study of intragroup conflict and group performance. Academy of Management Review, 44, 238–251.CrossRefGoogle Scholar
  63. Jehn, K. A., Northcraft, G. B., & Neale, M. A. (1999). Why differences make a difference: a field study of diversity, conflict and performance in workgroups. Administrative Science Quarterly, 44, 741–763.CrossRefGoogle Scholar
  64. Joldersma, C. (1997). Participatory policy making: Balancing between divergence and convergence. European Journal of Work and Organizational Psychology, 6, 207–218.CrossRefGoogle Scholar
  65. Joss, S. (2000). Die Konsensuskonferenz in Theorie und Anwendung. Stuttgart.Google Scholar
  66. Joss, S., & Durant, J. (1995). The UK national consensus conference on plant biotechnology. Public Understanding of Science, 4, 195–204.CrossRefGoogle Scholar
  67. Kamp, L. M. (2007). The importance of learning processes in wind power development. European Environment, 17, 334–346.CrossRefGoogle Scholar
  68. Kruglanski, A. W., & Webster, D. M. (1991). Group member’s reactions to opinion deviates and conformists at varying degrees of proximity to decision deadline and environmental noise. Journal of Personality and Social Psychology, 61, 212–225.CrossRefGoogle Scholar
  69. Lahdelma, R., Salminen, P., & Hokkanen, J. (2000). Using multicriteria methods in environmental planning and management. Environmental Management, 26, 595–605.CrossRefGoogle Scholar
  70. Levine, J. M., & Resnick, L. B. (1993). Social foundations of cognitions. Annual Review of Psychology, 41, 585–612.CrossRefGoogle Scholar
  71. Martin, G. P. (2008). ‘Ordinary people only’: knowledge, representativeness, and the publics of public participation in healthcare. Sociology of Health & Illness, 30, 35–54.CrossRefGoogle Scholar
  72. Mason, R. O. (1969). A dialectical approach to strategic planning. Management Science, 15, B403–B414.CrossRefGoogle Scholar
  73. Mason, R. O., & Mitroff, I. I. (1981). Challenging strategic planning assumptions: Theory, cases and techniques. New York: Wiley.Google Scholar
  74. Mayer, I. S., Van Bueren, E. M., Bots, P. W. G., & Van der Voort, H. (2005). Collaborative decisionmaking for sustainable urban renewal projects: A simulation-gaming approach. Environment and Planning B: Planning and Design, 32, 403–423.CrossRefGoogle Scholar
  75. McKeown, B., & Thomas, D. (1988). Q methodology. Newbury Park, Beverly Hills, London, New Delhi: Sage Publications.Google Scholar
  76. Mitroff, I. I. (1974). The subjective side of science: A philosophical inquiry into the psychology of the Apollo moon scientists. New York: Elsevier.Google Scholar
  77. Mitroff, I. I., & Emshoff, J. R. (1979). On strategic assumption-making; A dialectical approach to policy and planning. The Academy of Management Review, 4, 1–12.Google Scholar
  78. Mitroff, I. I., Emshoff, J. R., & Kilmann, R. H. (1979). Assumptional analysis: A methodology for strategic problem solving. Management Science, 25, 583–593.CrossRefGoogle Scholar
  79. Mitroff, I. I., Mason, R. O., & Barabba, V. P. (1983). The 1980 census: policymaking amid turbulence. Lexington.Google Scholar
  80. Mitroff, I. I., & Sagasti, F. (1973). Epistemology as general systems theory: An approach to the design of complex decision-making experiments. Philosophy of the Social Sciences, 3, 117–134.CrossRefGoogle Scholar
  81. Müller-Merbach, H. (2004). Creative conflict. Knowledge Management Research & Practice, 2, 129–130.CrossRefGoogle Scholar
  82. Nemeth, C., Brown, K., & Rogers, J. (2001). Devil’s advocate versus authentic dissent: stimulating quantity and quality. European Journal of Social Psychology, 31, 707–720.CrossRefGoogle Scholar
  83. Nemeth, C., Personnaz, B., Personnaz, M., & Goncalo, J. A. (2004). The liberating role of conflict in group creativity: A study in two countries. European Journal of Social Psychology, 34, 365–374.CrossRefGoogle Scholar
  84. Nooteboom, B., Van Haverbeke, W., Duysters, G., Gilsing, V., & Van den Oord, A. (2007). Optimal cognitive distance and absorptive capacity. Research Policy, 36, 1016–1034.CrossRefGoogle Scholar
  85. Oughton, D. H., & Strand, P. (2004). The Oslo consensus conference on protection of the environment. Journal of Environmental Radioactivity, 74, 7–17.CrossRefGoogle Scholar
  86. Quist, J., & Vergragt, P. (2006). Past and future of backcasting: The shift to stakeholder participation and a proposal for a methodological framework. Futures, 38, 1027–1045.CrossRefGoogle Scholar
  87. Raiffa, H. (1970). Decision analysis: Introductory lectures on choices under uncertainty. Reading, Massachusetts: Addison-Wesley Publishing Company.Google Scholar
  88. Rein, M. (1986). Frame-reflective policy discourse. Inaugural address, 17th Oct, 1986 (extended version).Google Scholar
  89. Renn, O. (1999). A model for an analytic-deliberative process in risk management. Environmental Science and Technology, 33, 3049–3055.CrossRefGoogle Scholar
  90. Renn, O. (2004). The challenge of integrating deliberation and expertise: Participation and discourse in risk management. In T. L. MacDaniels & M. J. Small (Eds.), Risk analysis and society: An interdisciplinary characterization of the field (pp. 289–366). Cambridge: Cambridge University Press.Google Scholar
  91. Renn, O., Webler, T., & Johnson, B. B. (1991). Public participation in hazard management: The use of citizen panels in the US. Issues in Health & Safety, 19, 197–226.Google Scholar
  92. Renn, O., Webler, T., Rakel, H., Dienel, P., & Johnson, B. (1993). Public participation in decision making: A three-step procedure. Policy Sciences, 26, 189–214.CrossRefGoogle Scholar
  93. Rip, A. (2003). Constructing expertise: In a third wave of science studies. Social Studies of Science, 33, 419–434.CrossRefGoogle Scholar
  94. Rittel, H., & Webber, M. (1973). Dilemmas in general theory of planning. Policy Sciences, 4, 155–169.CrossRefGoogle Scholar
  95. Robbins, P., & Krueger, R. (2000). Beyond bias? The promise and limits of Q method in human geography. Professional Geographer, 52, 636–648.CrossRefGoogle Scholar
  96. Robinson, J. (1982). Energy backcasting: A proposed method of policy analysis. Energy Policy, 10, 337–344.CrossRefGoogle Scholar
  97. Robinson, J. (2003). Future subjunctive: backcasting as social learning. Futures, 35, 839–856.CrossRefGoogle Scholar
  98. Rowe, G., & Frewer, L. J. (2005). A typology of public engagement mechanisms. Science Technology & Human Values, 30, 251–290.CrossRefGoogle Scholar
  99. Schön, D. A., & Rein, M. (1994). Frame reflection: Towards the resolution of intractable policy controversies. New York: Basic Books.Google Scholar
  100. Schweiger, D. M., Sandberg, W. R., & Ragan, J. W. (1986). Group approaches for improving strategic decision making: A comparative analysis of dialectical inquiry, devil’s advocacy and consensus. The Academy of Management Journal, 29, 51–71.CrossRefGoogle Scholar
  101. Simon, H. (1973). The structure of ill-structured problems. Artifical Intelligence, 4, 181–201.CrossRefGoogle Scholar
  102. Stasser, G., & Stewart, D. D. (1992). Discovery of hidden profiles by decision-making groups: Solving a problem versus making a judgment. Journal of Personality and Social Psychology, 63, 426–434.CrossRefGoogle Scholar
  103. Stasser, G., Taylor, L. A., & Hanna, C. (1989). Information sampling in structured and unstructured discussion of three- and six-person groups. Journal of Personality and Social Psychology, 48, 1467–1478.CrossRefGoogle Scholar
  104. Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48, 1467–1478.CrossRefGoogle Scholar
  105. Stave, K. A. (2002). Using system dynamics to improve public participation in environmental decisions. System Dynamics Review, 18, 139–167.CrossRefGoogle Scholar
  106. Stephenson, W. (1935). Correlating persons instead of tests. Character and Personality, 4, 17–24.Google Scholar
  107. Stirling, A. (1998). On the economics and analysis of diversity. SPRU Electronic working paper series, 28.Google Scholar
  108. Stirling, A., & Mayer, S. (2001). A novel approach to the appraisal of technological risk: A multicriteria mapping study of a genetically modified crop. Environment and Planning C: Government and Policy, 19, 529–555.CrossRefGoogle Scholar
  109. Street, P. (1997). Scenario workshops: A participatory approach to sustainable urban living? Futures, 29, 139–158.CrossRefGoogle Scholar
  110. Swedeen, P. (2006). Post-normal science in practice: A Q study of the potential for sustainable forestry in Washington State, USA. Ecological Economics, 57, 190–208.CrossRefGoogle Scholar
  111. The Social Learning Group. (2001). Learning to manage global environmental risks. Cambridge USA: MIT Press.Google Scholar
  112. Thompson, M., Ellis, R., & Wildavsky, A. (1990). Cultural theory. Boulder: Westview Press.Google Scholar
  113. Van Asselt, M., & Rijkens-Klomp, N. (2002). A look in the mirror: Reflection on participation in integrated assessment from a methodological perspective. Global Environmental Change, 12, 167–184.CrossRefGoogle Scholar
  114. Van de Kerkhof, M. (2006a). A dialogue approach to enhance learning for sustainability: A Dutch experiment with two participatory methods in the field of climate change. The Integrated Assessment Journal, 6, 7–34.Google Scholar
  115. Van de Kerkhof, M. (2006b). Making a difference: On the constraints of consensus building and the relevance of deliberation in stakeholder dialogues. Policy Sciences, 39, 279–299.CrossRefGoogle Scholar
  116. Van de Kerkhof, M. & Wieczorek, A. J. (2005). Learning and stakeholder participation in transition processes towards sustainability: Methodological considerations. Technological Forecasting and Social Change, 72, 733–747.Google Scholar
  117. Van Eeten, M. (2001). Recasting intractable policy issues: The wider implications of the Netherlands civil aviation controversy. Journal of Policy Analysis and Management, 20, 391–414.CrossRefGoogle Scholar
  118. Voss, J. P., Smith, A., & Grin, J. (Eds.). (2009). Designing long-term policy. Policy Sciences, 42 (Special issue).Google Scholar
  119. Webler, T. (1995). “Right” discourse in citizen participation: An evaluative yardstick. In O. Renn, T. Webler, & P. Wiedemann (Eds.), Fairness and competence in citizen participation (pp. 35–86). Dordrecht: Kluwer Academic Publishers.CrossRefGoogle Scholar
  120. Webler, T., Kastenholz, H., & Renn, O. (1995). Public participation in impact assessment: A social learning perspective. Environmental Impact Assessment Review, 15, 443–463.CrossRefGoogle Scholar
  121. Webler, T., Tuler, S., & Krueger, R. (2001). What is a good public participation process? Five perspectives from the public. Environmental Management, 27, 435–450.CrossRefGoogle Scholar
  122. Wittenbaum, G. M. (2000). The bias toward discussing shared information: Why are high-status group members immune? Communication Research, 27, 379–400.CrossRefGoogle Scholar
  123. Wittenbaum, G. M., Hubbel, A. P., & Zuckerman, C. (1999). Mutual enhancement: Toward an understanding of the collective preference for shared information. Journal of Personality and Social Psychology, 5, 967–978.CrossRefGoogle Scholar
  124. Wolsink, M. (2004). Policy beliefs in spatial decisions: Contrasting core beliefs concerning space making for waste infrastructure. Urban Studies, 41, 2669–2690.CrossRefGoogle Scholar
  125. Wynne, B. (2003). Seasick on the third wave? Subverting the hegemony of propositionalism: Response to Collins and Evans (2002). Social Studies of Science, 33, 401–417.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2011

Authors and Affiliations

  1. 1.Faculty of Technology, Policy and ManagementDelft University of TechnologyDelftThe Netherlands

Personalised recommendations