FormalPara Learning Objectives

By studying this chapter, you should be able to:

  • Understand the main rationale behind Multi-Method Research in the social sciences.

  • Be aware of different ontological and epistemological assumptions and their consequences for conducting multimethod research.

  • Grasp the concept of mechanistic heterogeneity analytically.

  • Critically discuss different sources of causal heterogeneity at the level of mechanisms, and their repercussions for causal inference in multimethod research.

  • More consciously generate generalization strategies for their own research projects, and critically examine the external validity of existing multimethod research.

10.1 Introduction

Over the last decades, multimethod research (MMR) has gained considerable popularity in the analysis of public policy (see Fielding, 2010; Hendren et al., 2018; Wolf, 2010 for overviews about MMR studies in public policy), echoing a general trend in the political and social sciences (seminally, Lieberman, 2005; for up-to-date discussions, see Beach and Kaas 2020; Goertz, 2017; Humphreys & Jacobs, 2015; Seawright, 2016). Many texts define MMR as any research design which uses two or more methods to analyze the same research topic, often involving cross-case analysis of patterns of association between causes and outcomes and within-case analysis of how the causal linkage(s) work (see Creswell & Plano Clark, 2018; Schoonenboom & Burke Johnson, 2017; Tashakkori & Teddlie, 2021 for various definitions).

The most common type of MMR in political science involves the combination of some form of cross-case analysis, e.g., using regression-based methods (see Chaps. 4 and 5), or some variant of mediation analysis (see Chap. 6) or Qualitative Comparative Analysis (see Chap. 7), and one or several within-case studies using methods like congruence analysis or process tracing (see Chap. 8).Footnote 1 The cross-case analysis enables the identification of the net causal effects or invariant association between X and Y, i.e., does X make a difference for Y? The within-case analysis, on the other hand, focuses on the causal linkage aka mechanism(s), i.e., how does X work to bring about Y? The core logic behind this variant of MMR, in a nutshell, is that combining methods that allow for different kinds of inferences bears the potential to use the particular strengths of one technique to cancel out the other’s weaknesses, and vice versa (e.g., Beach, 2020, 163; Clarke et al., 2014, 341; Goertz, 2017, 5–6; Lieberman, 2005, 436; Weller & Barnes, 2016, 426–27). In doing so, the promise of MMR is that its design ultimately yields more robust inferences by shedding light on social phenomena or substantiating our understanding of policy problems from different analytical perspectives.

The question of whether MMR can deliver on this promise – whether different methods can efficiently complement each and strengthen overall causal inferences “because taken on their own each sort of evidence has significant limitations” (Clarke et al., 2014, 341) – has not gone uncontested. In fact, there is a notable strand within the methodological literature reflecting upon the notion of mutual complementarity in MMR. The core of this debate deals with whether different methods that make different types of causal claims and use different types of evidence can really be merged as seamlessly as is frequently portrayed (Beach and Kaas, 2020). Among other things, it has been highlighted that MMR can involve the problem of conceptual stretching or might even introduce conceptual incongruity if specific causal properties are added/dropped from concepts when moving between the cross-case and the within-case level of an analysis (Ahmed & Sil, 2009; Ahram, 2013). Similarly, while case studies can be used to check for measurement errors or to develop context-sensitive indicators (e.g., Seawright, 2016, 50–53), it can be that translating within-case observations into comparable cross-case data, and the other way around, is neither intuitive nor straightforward (Ahram, 2013; Kuehn & Rohlfing, 2009). Finally, it has been frequently mentioned that case studies can be used in MMR to check for under- and/or overspecification of the explanatory model at the cross-case level (Lieberman, 2005; Seawright, 2016: 67–74). Yet, Rohlfing (2008) convincingly shows that model misspecifications can travel between different levels of analysis because residuals and effect sizes might point towards the wrong cases for further within-case study, hence aggravating the situation, since an incorrect model is corroborated by looking at the wrong cases. In short, numerous pitfalls can complicate the effective integration of different approaches and methods in MMR designs.

This chapter concentrates on another significant problem: How can insights about causal mechanisms gained by studying how they work in one case be generalized to cases that we have not studied using case studies but look similar at the cross-case level? The issue of generalization has so far largely been ignored in the political science literature on MMR. As we will show below, generalizing about mechanisms is particularly difficult in settings that exhibit mechanistic heterogeneity. We define mechanistic heterogeneity as a scenario where multiple different mechanisms link the same explanatory factor(s) X to the same outcome Y (Álamos-Concha et al., 2021; Beach et al., 2019). For instance, we might find out that epistemic authorities (aka experts) gained influence over a policy in one case through a mechanism involving a process where the experts gained access to decision-makers by joining the bureaucracy itself (Löblová, 2018). However, in another case, influence might have been achieved through other processes, such as experts or lobbies’ framing of the debates from the outside.

This form of heterogeneity and complexity at the level of mechanisms is widely discussed in the literature on case-based methodology (Beach & Pedersen, 2016, 2019; Bennett & Checkel, 2015; Blatter & Haverland, 2012; Falleti & Lynch, 2009; George & Bennett, 2005; Rohlfing, 2012). However, it is largely neglected in most accounts that deal with the integration of cross-case and within-case analysis (but see Beach et al., 2019; Goertz, 2017; Weller & Barnes, 2016), which is why we do not yet have a good understanding of how to deal with the issue of making cross-case and within-case analysis communicate in MMR. To put it simply, the cross-case analysis tells us about differences and similarities at the level of X’s and Y’s; in contrast, the within-case analysis tells us about linkages (if any) between X and Y. In fact, we are making different types of causal claims, using very different types of empirical material (Clarke et al., 2014).

Addressing this question in the context of a volume on causation in policy studies is important for several reasons. First, we can observe an apparent ‘mechanistic turn’ in the social sciences which gradually expands across its subfields, including the field of public policy analysis (e.g., Capano et al., 2019; Capano & Howlett, 2021; Fontaine, 2020; Kay & Baker, 2015; Lindquist & Wellstead, 2019; van der Heijden et al., 2019). For instance, Fontaine (2020, 274) stresses that there is an emerging consensus on the fact that producing evidence about mechanisms via process tracing bears a significant “potential contribution to comparative policy analysis.” Capano and Howlett (2021, 142 italics in the original) go one step further, arguing that “[p]olicy-makers [..] need a realistic causal theory about what occurs when policy tools are deployed and how it occurs if they want to design something that will actually happen more often than not, and to escape the trap of poorly conceived and related tacit knowledge, experience, and heuristics.” Yet, secondly, if we accept that producing comprehensive causal explanations requires both robust evidence that a probable cause X is correlated/associated with Y as well as sound evidence for the causal mechanisms linking X and Y, the ability to generalize mechanistic claims from one studied case to other cases belonging to the same population becomes a significant issue. In one case study, we might have found that the linkage worked in one way, but how would we know whether the linkage (if any) is similar in other cases if we have not also investigated them? For instance, can we assume that a particular strategy used by a political entrepreneur that worked during a crisis would work in other situations? Assuming that mechanisms work in similar ways in other, non-studied cases is in effect generalizing based on hope instead of evidence. If researchers and policymakers need to know what works, how, and under what conditions, a well-informed mapping of the underlying mechanisms operative within a population of cases is crucial to generalize how X and Y are linked in different cases within a population.

The chapter is structured as follows: Section 10.2 outlines the basic ideas behind MMR designs, introduces the main templates, and discusses key ontological and epistemological differences when combining cross-case and within-case analysis. Section 10.3 addresses the problem of mechanistic heterogeneity by illustrating what heterogeneity at the level of mechanisms means. After that, Sect. 10.4 presents a selected set of potential sources to which researchers should turn to check for mechanistic heterogeneity in MMR. In Sect. 10.5, we discuss a stepwise generalization strategy that is sensitive to mechanistic heterogeneity and whose primary goal is to progressively update the confidence in the external validity of mechanisms by gradually expanding the knowledge about how mechanisms work in different (sets of) cases. The chapter closes with some final remarks.

10.2 Basic Ideas Behind MMR

The main rationale behind combining cross-case and within-case methods in MMR is that it allows researchers to make different types of causal inferences (e.g., Beach, 2020; Beach & Rohlfing, 2018; Goertz, 2017; Lieberman, 2005; Rohlfing & Schneider, 2018; Seawright, 2016; Weller & Barnes, 2016). On the one hand, cross-case analyses are particularly good at identifying cause–effect relationships by examining regular associations in the form of controlled experiments, correlations, or set-relations across a sample of cases. On the other hand, within-case analyses can establish the causal linkages between one or several causes and the respective contributions by tracing the underlying causal mechanism(s). By integrating both analytic perspectives and using methods in combination to address a shared research theme, it is argued that one can strengthen the soundness and robustness of the inferences since each mode of analysis has particular strengths that can make up for the other’s blind spots (Cartwright, 2011; Clarke et al., 2014; Steel, 2008).

But how does this division of labor work in research practice? The literature on MMR has produced numerous taxonomies and typologies of different designs (see Bryman, 2006; Creswell & Plano Clark, 2018; Schoonenboom & Burke Johnson, 2017; Tashakkori & Teddlie, 2021, among others). One common defining element is whether the methods are applied in parallel or sequentially. In parallel designs, two or more methods are applied simultaneously; in sequential designs, one is used after the other. A different feature is whether the parts of an MMR study depend on each other or are performed independently. In the former scenario, insights from one study inform the data collection and/or analysis of the other; in the latter scenario, data collection and/or analysis are performed separately within each method.

The sequential research strategy is probably the most common in political science research. Two variants are typically distinguished (e.g., Beach & Rohlfing, 2018, 11–18; Lieberman, 2005; Rohlfing, 2008; Rohlfing & Schneider, 2018, 44–45; Seawright, 2016). In ‘cross-case first/within-case second’ designs, the researcher starts with some form of cross-case analysis to identify robust connections between a (set of) explanatory factor(s) X and an outcome of interest Y. This is followed by one or several case studies based on the findings of the first analytic step. On the other hand, ‘within-case first/cross-case second’ designs follow the opposite logic. Here, the analysis starts at the within-case level to uncover some causal connection and/or mechanisms and then continues with the cross-case analysis to explore whether the identified relationship also holds across a population of cases.

While one of the original motivations behind the methodological work on MMR was to (at least partially) overcome the divide between qualitative and quantitative methods, recent debates have again emphasized the ontological and epistemological differences between research approaches and the challenges they create for integrating methods from the different cultures into an (at least somewhat coherent) MMR design. At least two types of approaches can be differentiated: variance-based and case-based (for the following, see Beach & Kaas, 2020).

Variance-based approaches to MMR build on a counterfactual understanding of causation as developed in the Potential Outcome framework. Counterfactual causation is defined as the claim that a cause produced an outcome because its absence would result in the absence of the outcome, all other things being held equal. Without evaluating the difference that a cause can make between the actual and the counterfactual, no causal inference is possible. Therefore, the main causal inference is established at the cross-case level using controlled comparisons. Put it more bluntly, the cross-case method is in the inferential driver’s seat, while the within-case serves as an adjunct method.Footnote 2 This does not mean that the within-case study is not important. It fulfills crucial functions such as validating measurement, establishing a case’s counterfactual, reconstructing the causal pathways, or searching for confounders (Seawright, 2016; Weller & Barnes, 2016). Causal evidence, however, lies across cases.

In case-based approaches to MMR, multiple understandings of causation exist side-by-side (Baumgartner & Falk, 2019; Beach & Pedersen, 2019; Rohlfing & Schneider, 2018). They usually have in common that the inferential workhorse in MMR designs is located at the within-case level instead. To establish a causal relationship, it must be checked whether the identified explanatory factors indeed exert some causal power over the outcome in a case, and if so, how exactly the causal mechanism plays out (e.g., Beach et al., 2019; Schneider & Rohlfing, 2016, 2019). Here, the analysis at the cross-case level plays an adjunct role, e.g., by establishing an X/Y relation in the first place, guiding the case selection for the within-case study, or mapping the population of cases for further generalization (Box 10.1).

Box 10.1: The Variance-Based and the Case-Based Approach to MMR

The question of variance-based and case-based approaches to MMR needs to be located in the broader discussions within the philosophy of sciences (e.g., Cartwright, 2011; Russo & Williamson, 2011) and political science. In this sense, it connects to the seminal readings like King et al. (1994), which argued in favor of a shared understanding of causal inferences across quantitative and qualitative (i.e., empirically oriented case-based methods). This has been challenged in recent debates, which (again) points out the ontological and epistemological differences between the qualitative and quantitative methods (Brady & Collier, 2010). Consequently, there has been a rise of methodological guidelines for different MMR designs depending on the research tradition in which it is grounded (see Beach & Kaas, 2020 for an overview).

Variance-based approaches to MMR (e.g., Lieberman, 2005; Seawright, 2016; Weller & Barnes, 2014, 2016), as pointed out in the main text, usually are grounded in the potential-outcomes framework (aka counterfactual causation). It applies a top-down perspective where the main goal is to identify robust causal effects in a population of cases, or a sample thereof (with randomized controlled trials as a gold standard). This is followed by an assessment at the within-case level of whether the causal relationship holds or not. The cross-case analysis using controlled comparisons is the main workhorse for causal inference, focusing on difference-making. To align cross-case and within-case analysis, variance-based approaches often understand causal mechanisms as intervening variables whose difference-making can be assessed using controlled comparisons between cases.

For case-based approaches to MMR, the ontological underpinnings are varied, relying on regulatory theory (e.g., QCA) or mechanisms (e.g., process tracing) (Beach & Rohlfing, 2018; Goertz, 2017; Rohlfing & Schneider, 2018; Schneider & Rohlfing, 2016; see also Chaps. 1, 2, 6, and 7). However, what is shared by all existing frameworks is that the main causal inference happens at the within-case level through case study methods like process tracing. In this regard, case-based approaches are bottom-up in their focus on causation as it plays out within single cases, after which generalizations might be made to other cases. As regards the understanding of causal mechanisms, there is an emerging consensus on a productive account of mechanisms – which we also subscribe to in this chapter – that understands mechanisms in the form of actors engaging in activities that link a cause and outcome together in a productive causal relationship. Nevertheless, epistemological discussions are still ongoing about how to identify the working of mechanisms (see also Chaps. 2, 6 and 8).

10.3 The Problem of Mechanistic Heterogeneity for External Validity in MMR

Making generalizations about the working of mechanisms from one studied case to other cases which are not studied is a crucial problem in the social sciences and beyond (e.g., Cartwright, 2011; Khosrowi, 2019; Steel, 2008; Wilde & Parkkinen, 2019). Knowing how a policy intervention works in one case does not necessarily tell us how it would work in other, non-studied cases.

The relevance of this issue is evident in case-based approaches, where the examination of mechanisms is the main inferential workhorse. But the ability to make generalizable claims about mechanisms is also essential for the variance-based approach. For instance, Weller and Barnes (2014, 21) argue that one goal of within-case analysis is “to understand substantive relationships at the level of individual cases and to use those insights to learn something about the population of cases that feature that substantive relationship.” Therefore, large-N mediation analysis (see Chap. 6) is often used to study mechanisms. However, by studying many cases using variance-based methods, one learns about the average causal effects of X (or the intervening variable) on the values of Y. An average does not tell us how the linkage works in any given case. In Cartwright’s words, average causal effects tell us that “it works somewhere” while leaving us in the dark about how it actually works in any given case (Cartwright, 2011).

Once we find a causal mechanism in a studied case using within-case analysis, the key question asks whether we can infer that a similar – nota bene: not exactly the same (!) – mechanism also connects X and Y in other cases. In other words, how do we ensure the external validity of findings about causal mechanisms? The answer heavily depends on the degree of causal heterogeneity at the within-case level.

We speak of mechanistic homogeneity if two or more sufficiently similar mechanisms are operative in all the cases that exhibit the same relationship between X and Y. Mechanistic heterogeneity, on the other hand, refers to two situations: (1) the same X and Y are linked together through different mechanisms (mechanistic equifinality), or (2) the same X triggers different mechanisms leading to a different Y (mechanistic multifinality) (Beach, 2020; Beach et al., 2019; Beach & Rohlfing, 2018; Falleti & Lynch, 2009; George & Bennett, 2005; Gerring, 2010; Goertz, 2017; Sayer, 2000; Weller & Barnes, 2016).

It is important to note that we do not understand causal mechanisms as chains of events, but instead as process-level causal explanations that provide an account of what actors are doing. This account explains why the actors’ activities are linked together and how they contribute to producing the outcome in the case. Of course, these process-level explanations can have varying levels of detail (aka abstraction). At the most abstract level are schematic theories that focus on the most critical interactions, describing actors and what they are doing in very abstract terms (e.g., “a political entrepreneur engages in speeches that attempt to frame a debate”). At the other extreme are very detailed, case-specific accounts that use formal nouns to describe actors, include many different parts, and where activities are specified in great detail (Box 10.2).

Box 10.2: Causal Heterogeneity

The term causal heterogeneity includes a range of phenomena linked to complex causal patterns that can characterize any X/Y relationship. In the statistical literature, the problem of causal heterogeneity plays a significant role, for example, when considering whether different subgroups in a given population react differently to a specific treatment, e.g., an administered policy instrument (e.g., Seawright, 2016; Pearl, 2017; Xie, Xie et al., 2012). Issues of causal heterogeneity are also prominent in the context of QCA, where they are discussed concerning conjunctural causation, equifinality, and asymmetry (Ragin, 2008; see also Chap. 7). Yet, researchers must be aware that causal heterogeneity not only pertains to X/Y relations but also to the level of mechanisms (e.g., Beach et al., 2019; Beach & Rohlfing, 2018; Goertz, 2017; Weller & Barnes, 2016).

Figure 10.1 illustrates the issue of mechanistic homogeneity and heterogeneity using causal diagrams in a stylized form for a simple X/Y relationship.

Fig. 10.1
4 block diagrams for different scenarios of mechanical homogeneity and heterogeneity depict different x y relationships.

Abstract examples of mechanistic homogeneity and heterogeneity. Own depiction

Full size image

The first scenario displays one variant of mechanistic homogeneity where X and Y are connected via the same mechanism (CM1) in both cases. In contrast, the next situations all refer to different forms of mechanistic heterogeneity.

In the second scenario, two single but different mechanisms connect the same X to the same Y, CM1 in one case and CM2 in another case.Footnote 3

The situation turns more complex in the third scenario. Here, the same X triggers multiple mechanisms in two cases, i.e., mechanistic multifinality, yet there is only one mechanism that is shared by both cases (CM1), whereas the two cases differ on the second mechanism triggered by X, namely, CM2 versus CM3.

Finally, the fourth scenario shows how different mechanisms might interact with each other in different ways across cases – CM1 and CM2 in one case, and CM1, CM2, and CM3 in the second case.

These illustrations are, of course, very simple scenarios. More frequently, explanatory models do not involve one individual factor, but instead several factors X1, X2, X3…, Xi. Here, patterns can become much more complex. Causal mechanisms can work additively or interact with each other, appear in a different sequential order, show complementary instead of conflicting effects (among others, see Beach & Rohlfing, 2018, 18–25; Goertz, 2017, 53–57; Mikkelsen, 2017, 429–34; Weller & Barnes, 2016, 433–37 for further illustrations). For instance, X1 and X2 might trigger two mechanisms, CM1 and CM2, but in one case, this happens simultaneously, whereas in other contexts X1 happens before X2, or even that X1 triggers CM1, which then leads to X2 triggering CM2 – highlighting temporal or causal ordering as reflections of mechanistic heterogeneity. Another example is discussed under the label of ‘masking’ (Clarke et al., 2014; Steel, 2008, 68; see also George & Bennett, 2005, 145–47). Masking means that a given X might be linked to the same Y through multiple mechanisms with opposite effects on the Y. For instance, a crisis might trigger a process where some actors engage in a frantic search for solutions and advocate for them. At the same time, the same crisis can push other actors to become risk-averse, thereby starting a process of resistance to any change. In the case, both processes might be operative, and the outcome is a compromise on some modest change that either group did not desire.

10.4 Sources of Mechanistic Heterogeneity in MMR

When combining cross-case analysis and within-case analysis in MMR to identify causal mechanisms and make generalizable claims about them, a crucial problem is that the information utilized at the cross-case level is usually uninformative about what is going on at the within-case level of mechanisms. Let us revisit the abstract example displayed in Fig. 10.1: there is simply no way to establish how exactly the mechanisms connecting X and Y play out just by looking at the X/Y relations. Against this backdrop, examining how a mechanism works by studying how it works within one case and generalizing to other unstudied cases is extremely risky. Very different mechanistic scenarios might lurk underneath the same X/Y relationship.

Before we sketch out a generalization strategy sensitive to mechanistic heterogeneity in the next section, we discuss three primary potential sources of mechanistic heterogeneity so that researchers are informed about where to look for heterogeneity pitfalls when generalizing mechanistic claims (Box 10.3).

Box 10.3: Potential Sources of Mechanistic Heterogeneity

As in cross-case analysis, the assumption of causal homogeneity at the level of mechanisms is usually too heroic to be met in the social sciences. We, therefore, argue that mechanistic heterogeneity should be the default assumption when conducting within-case analysis in general and MMR in particular (Beach et al., 2019). Instead of simply assuming that things work in the same way in different cases, researchers should engage in empirical testing of whether mechanistic heterogeneity is present in a population if they want to avoid making flawed generalizations about the working of causal mechanisms.

A non-exhaustive list of non-exclusive sources of mechanistic heterogeneity includes, inter alia, complex concepts and measures based on multiple attributes with particular causal properties, qualitative hedges within concepts and measures triggering multiple different mechanisms, omitted causal factors and confounders, varying contexts and differences in scope conditions, factors which are identified as redundant or insignificant at the cross-case level, but still have a causal impact at the level of mechanisms, or different forms of temporal and/or causal dynamics which underlie an X/Y relationship.

10.4.1 Complex Concepts or Measures

The first source of mechanistic heterogeneity is that concepts and measures used at the cross-case analysis capture more than one causal property and can trigger multiple mechanisms. Concepts in the social sciences are usually thought of as multidimensional constructs that have several analytical levels, i.e., attributes and indicators (Adcock & Collier, 2001; Goertz, 2020). The literature on concepts and concept formation has developed various strategies for systematizing the constitutive properties of a concept so that they can be fruitfully applied in empirical research.

In the so-called classical approach to concept formation, the constitutive attributes of a concept are individually necessary and jointly sufficient (Goertz, 2020; Sartori, 1970). The Venn diagram in Fig. 10.2a illustrates the underlying logic, whereby we start from three constitutive attributes (A, B, C). For a case to be captured by a concept using the classical approach, all three properties must be present – i.e., A and B and C. If only one of the three attributes is missing, the given social phenomenon does not qualify as a manifestation of the concept.

Fig. 10.2
4 Venn diagrams for different concepts of heterogeneity. All diagrams have circles A, B, and C overlapping each other with B at the top.

Concept formation strategies and conceptual heterogeneity. Own depiction based on Barrenechea and Castillo (2019)

Full size image

On the other hand, the family resemblance approach offers an alternative strategy to concept formation. In contrast to the classic approach, concepts only have sufficient attributes without a specific feature being individually necessary. Under family resemblance, a case is described by a concept when it has at least one of the constituent attributes, regardless of which one. The Venn diagram in Fig. 10.2d illustrates this approach: the presence of either A or B or C – or any combination of the three – is sufficient for the concept to be present (Barrenechea & Castillo, 2019; Goertz, 2020).Footnote 4

Beyond these two standard approaches to concept formation, mixed types can also be possible.

In a variant, for instance, there is no single sufficient attribute for having a concept; instead, several conceptual properties must be present, none of which is necessary. To witness, if we require that two out of three attributes need to be present for a concept, this may mean that the concept describes any case showing A and B, or A and C, or B and C, or A and B and C. Figure 10.2c exemplifies this logic based on three (‘n’) conceptual attributes out of which at least two (‘m’) must be given for the concept to apply.

Another mixed type of the two standards approaches is based on the idea that one or more constitutive properties of a concept are necessary, but additional attributes are required but not necessary. For example, thinking again of a concept made up of three attributes A, B, C, we can envisage that A is necessary, but either B or C must be added for a case to be described by the respective concept. As demonstrated in Fig. 10.2b, the concept only applies if another attribute is fulfilled in addition to A.Footnote 5

What does this have to do with mechanistic heterogeneity? The point is that these structures can introduce different levels of (causal) heterogeneity into concepts (Barrenechea & Castillo, 2019; Beach et al., 2019; Collier & Mahon Jr, 1993; Goertz, 2020). As Figure 10.2a highlights, concepts based on necessary and jointly sufficient conditions are very homogeneous since cases are described by this concept only if they show all three attributes. On the other end of the spectrum, concepts that follow a family resemblance logic show a high degree of potential heterogeneity because a total of seven characteristic combinations lead to the presence of the concept – i.e., all combinations except ~A* ~ B* ~ C (Fig. 10.2d). The two mixed types can be located in between. Since different attributes have different causal properties and can trigger different causal mechanisms, it does not need much imagination to envisage that this also leads to mechanistic heterogeneity.

A study by Binder (2015) on the conditions for robust UN interventions in international conflicts illustrates this. Here, the factor ‘spillover effects’ is conceptualized via three attributes that capture different spillover aspects. The three aspects are, first, refugee flows; second, transnationally operating rebel groups; and third, other negative effects such as drug traffic, terrorism, and economic downturns. To count as a conflict with spillover effect, any of the three factors is sufficient following a family resemblance approach. In such a situation, the cases included in the cross-cases analysis which are coded as experiencing spillover effects contain mechanistic heterogeneity by design: some suffer from only one of these factors, i.e., refugee flows or transnationally operating rebels or economic downturns, others from a combination of two or even all three factors. But the causal mechanisms triggered by each attribute are most probably very different even though they all are coded as cases of ‘spillover effect’.

In situations like these, we do not know which mechanism is actually present in a given case just by looking at the relationship between X (here, spillover effects) and Y (here, UN intervention). Hence, we cannot generalize from one case to any other since it is unclear whether cases that only show high refugee flows trigger the same mechanism(s) as cases with only transnationally operating rebels or all three attributes present. At best, we might generalize to cases that share the same configuration of conceptual attributes. But even this is difficult, as we highlight below, since there might still be different dynamics at play among cases that share the same attributes.

The problem of (causal) heterogeneity pertains to various concept formation strategies and complex measures. It also occurs if subtypes are constructed and then used in the form of a ranked scale (Collier & Levitsky, 1997; Møller & Skaaning, 2010). It is inherent to index building which rests on the assumption of homogeneity at different levels of the index (Barrenechea & Castillo, 2019). It may also apply to lexical scales where the defining attributes are hierarchically arranged so that the attribute at the lower level is necessary to the next higher level (Skaaning et al., 2015).

All in all, we should expect that causal heterogeneity, and consequently mechanistic heterogeneity, is pervasive when studying public policy phenomena, especially against the backdrop of the widespread use of complex concepts in cross-case analysis. While this might not be a problem if one is only interested in establishing X/Y relations, it becomes a crucial pitfall in MMR if the aim is to generalize the insights gained at the within-case level to a larger sample of unstudied cases. Simply assuming that causal mechanisms play out in similar ways across all cases would not be warranted in this situation.

10.4.2 Known and Unknown Omitted Conditions

The second source of mechanistic heterogeneity comes from known and/or unknown omitted conditions in cross-case analysis. The problem of unknown omitted conditions, i.e., contextual or explanatory factors that are not part of the original model, is frequently discussed in the methodological literature as a problem for MMR (Kuehn & Rohlfing, 2009; Radaelli & Wagemann, 2018; Seawright, 2016; Weller & Barnes, 2016). Known omitted conditions, i.e., factors that are not considered in the within-case analysis because they do not make a difference in the cross-case analysis, are less frequently problematized in the literature (but seeÁlamos-Concha et al., 2021 ; Beach et al., 2019 ; Schneider & Rohlfing, 2019).

Conditions omitted in cross-case analysis can substantially impact the within-case level as they can introduce additional mechanisms or interact with existing mechanisms. The problem is straightforward with factors omitted from explanatory models and is widely discussed, for instance, in the literature as potential confounders (e.g., Goertz, 2017; Radaelli & Wagemann, 2018; Seawright, 2016; Weller & Barnes, 2014). Yet, contextual (aka, scope) conditions that are omitted can also play an important role because they can impact how mechanisms operate (i.e., Bunge, 1997; George & Bennett, 2005; Gerring, 2010; Goertz & Mahoney, 2009; Sayer, 2000). This line of thinking also fits nicely into the context-mechanism-outcome (CMO) framework developed by Pawson and Tilley (1997) concerning realistic evaluations. In a nutshell, the framework posits that mechanisms underlying any cause–effect relationship need to be properly contextualized, and whether they work in similar or different ways across varying contexts remains an empirical issue. Returning to the above example of spillover effects and the strength of UN intervention (Binder, 2015), one question concerning the generalizability from one case to another would ask whether the mechanisms differ according to the temporal duration of the conflict. For instance, during a protracted conflict, the intensity of violence might ebb and flow, and there might be several waves of refugees where each wave builds up more and more pressure for international action. A different dynamic might be observed during a short but extremely violent conflict. Of course, whether this is meaningful for treating mechanisms as different depends on the theoretical perspective.

While conditions that are not considered in the analysis can play a crucial role in mechanistic heterogeneity and the generalizability of mechanisms across cases, they are not the only source. One problem we might think of when integrating within-case and cross-case analysis to make generalizations about mechanisms is that explanatory factors might turn out as redundant, irrelevant, or insignificant at the cross-case level, but still have an important causal role to play at the within-case level. This is because, strictly speaking, the level at which causes are operative is always within a single case. Therefore, establishing patterns of difference-makers using statistical techniques or QCA tells us nothing about what is going on within cases. Instead, they only allow us to observe patterns of (in)variation across cases.

For instance, a QCA model might show that condition C is irrelevant since the outcome Y appears together with the presence of C (e.g., ABC) and its absence (e.g., AB~C). In short, C is not a difference-maker from a cross-case perspective (Baumgartner & Falk, 2019; see also Chap. 7). However, once we move down to the case level, the presence or absence of C might be causally relevant for the operation of the mechanism as it still constitutes an analytically important context in which the causal mechanism is embedded (Álamos-Concha et al., 2021; Beach et al., 2019; Schneider & Rohlfing, 2019). The same holds for variables that turn out as (in)significant in regression analyses. All that regressions say is that X has, on average, a particular effect Y, or that it does not; but whether a given factor impacts how the mechanism operates within a given case is an entirely different question that can only be addressed through means of within-case analysis, as this information cannot be derived from the statistical effects (Goertz, 2017; Seawright, 2016; Weller & Barnes, 2014).

In sum, issues like context-sensitivity, proper scoping, or omitted factors as a source of causal heterogeneity are widely acknowledged in the literature discussing various forms of cross-case and within-case methods. From the perspective of MMR and the task of generalizing causal mechanisms, the problem is aggravated since researchers need to be aware of the limited homogeneity beneath the effect of X on Y and the possibility of multiple mechanisms connecting X and Y across sub-sets of cases.

10.4.3 Causal and Temporal Dynamics

A third problem when generalizing insights about the working of mechanisms in MMR is that an X/Y relation identified at the cross-case level usually tells us (next to) nothing about the underlying causal and/or temporal dynamics. A look at the literature on within-case studies and MMR discusses a variety of different dynamics that can lurk underneath the same X/Y relationship (Beach & Rohlfing, 2018, 18–25; Beach et al., 2019, 125–28; Blatter & Haverland, 2012, 94; Falleti & Mahoney, 2015, 217; Goertz, 2017, 123–69; Grzymala-Busse, 2011, 1275; Mikkelsen, 2017, 429–34; Weller & Barnes, 2016, 434–35). If unnoticed, they can have a tremendous impact on the generalizability of mechanistic claims since the researcher would assume that the same patterns are linking X in Y in all cases while, in reality, they differ across cases.

One example of mechanistic heterogeneity that can hide behind the same X/Y relation is the temporal sequence of conditions and mechanisms. For instance, a cross-case analysis based on QCA or standard regression techniques might indicate that three factors A, B, C are associated with Y. For illustrational purposes, we use the example of large refugee flows, transnationally operating rebel groups, and other negative effects such as an increase in drug traffic, terrorism, and economic downturns that provoke a robust UN intervention. We can envisage a case where the three factors follow a temporal sequence, according to which the rise of transnational rebel groups (B) first causes an increase in refugee flows (A), which then leads to economic downturns and other negative consequences (C), which finally causes a robust UN humanitarian intervention. Can we now assume that the same sequence is present in all cases? This would probably be a pretty heroic assumption, since many other sequences can still be plausibly theorized. For instance, it might be the case that all three factors appear simultaneously, or the ordering of conditions might be different.

Interaction patterns might be another way that mechanistic heterogeneity manifests itself. For instance, mechanisms might work independently versus conjointly in different cases. Revisiting the example again, the increase in refugee flows, the rise of transnational rebel groups, and negative effects such as an increase in drug traffic, terrorism, and economic downturns might each trigger separate causal mechanisms through different actors and venues that ultimately lead to UN interventions. In other words, A leads to Y, B leads to Y, and C leads to Y through three independent causal mechanisms CM1, CM2, and CM3. However, in other cases, we might find a different situation. One reasonable alternative might be that the three factors do not show an independent effect, but instead work conjointly, so that each causal mechanism adds or reinforces each other until the UN decides on a robust humanitarian intervention.

It is important to note that these challenges cannot merely be fixed by including interaction terms in regressions or using configurational methods like QCA.Footnote 6 Regarding the latter, conjunctions in QCA only tell us that two or more conditions are jointly associated with an outcome; however, they do not tell us anything about the interactions present among the individual conditions within the configuration. Yet the same applies to interaction terms in a regression analysis where we learn that a factor’s average causal effect depends on the level of another factor; however, this contains no information on what dynamics and interplays we should expect at the level of mechanisms.

10.5 Taking Mechanistic Heterogeneity in MMR More Seriously

In all the situations described in the previous section, generalizing from one studied case to other cases that have not been studied risks making flawed inferences about which causal mechanisms are operative in different cases. Strictly speaking, we can only know which mechanisms are operative in a given case by investigating that case. This means that researchers are confronted with an inherent trade-off when establishing the external validity of mechanistic claims: examine all cases within a given population at tremendous analytical costs, or make a mechanistic generalization based on hope, with no empirical evidence to substantiate it (Khosrowi, 2019). The trade-off is of special relevance to public policy, where the complexity of processes in different contexts (both across space and time) makes mechanistic heterogeneity likely pervasive.

To engage with this inherent trade-off, we propose a generalization strategy that pays close attention to mechanistic heterogeneity using a sequential, ‘cross-case analysis first/within-case analysis second’ design. Building on the work by Weller and Barnes (2014, 2016), we advise engaging in multiple follow-up case studies that assess which causal mechanisms are present in strategically selected cases within a population, thereby gradually establishing the boundaries of the external validity of our mechanistic claims. In situations where we find mechanistic heterogeneity, we should map the different causal mechanisms operating in various sub-sets of the population to clarify why different mechanisms are operative in different sub-sets of cases (see Beach et al., 2019, 133–54 for a more detailed discussion) (Box 10.4).

Box 10.4: Strategy for Testing the Generalizability of Mechanisms Under the Assumption of Mechanistic Heterogeneity

The rationale behind the suggested snowballing-outwards procedure is to use findings from within-case analysis to revise the knowledge of the boundaries in which particular mechanisms are operative and progressively update the confidence in the external validity of the mechanistic claims which can and which cannot be made.

The proposed strategy consists of the six steps, starting after the cross-case analysis has produced a robust X/Y relationship:

  1. (i)

    Theoretical unpacking of all potential plausible mechanisms that could link X and Y.

  2. (ii)

    Mapping of the potential population of cases.

  3. (iii)

    Initial process tracing of most-similar with population positive case.

  4. (iv)

    Second process tracing of the positive case that is as similar to initial case as possible.

  5. (v)

    Gradually probing more dissimilar positive cases, paying close attention to potential sources of mechanistic heterogeneity.

  6. (vi)

    Concluding with a mechanism-focused comparison of the deviant case(s) to explore potential necessary factors by tracing the breakdown of the mechanism(s) previously identified.

After a robust X/Y relationship is identified at the cross-case level via statistical or configurational methods, the first step of the proposed generalization strategy starts with theoretically unpacking various potential mechanistic explanations. Unpacking mechanisms involves disaggregating causal processes into parts composed of actors doing things.Footnote 7 What is necessary at this stage is that researchers make the causal logic underlying the linkages in a mechanism explicit. Doing so also sheds light on all kinds of factors (causal and contextual) that we might expect to be relevant for whether and/or how a given mechanism works. For instance, one pathway might include a part where, to table a proposal that frames a debate, the expert needs to be a trusted epistemic authority by the policymakers. In fact, by theorizing and empirically tracing how a mechanism works, we also shed light on the conditions required for it to work in a particular way.

Of course, throughout the next steps, one should still cast the net widely and be open for further evidence about causal mechanisms which have not been hypothesized at this early stage; however, the first step should include a theoretical mapping of the most plausible different mechanistic scenarios and the respective settings in which they might occur.

In the next step, a cross-case mapping of the potential population of cases is undertaken. This involves scoring cases based on values of the explanatory factors X and the outcome Y and potential contextual and causal conditions that might affect how mechanisms work. Here it is crucial to go beyond the identified X/Y relations and to include all analytically relevant (causal or contextual) conditions. In principle, it should be the goal of this mapping to identify clusters of cases as causally homogeneous as possible to minimize the a priori risk of mechanistic heterogeneity.Footnote 8

Based on this mapping, we can select a case for tracing the underlying mechanisms between X and Y. At the initial stage, all positive cases that are members of the X(s), Y, and the given context are potential candidates for process tracing since mechanisms can only be observed in cases where X and Y are present. Ideally, this process tracing identifies one or several mechanisms linking X and Y in a given context C.

However, it might also be the case that no mechanism is identified in the chosen case. Here, we would advise proceeding to another similar case study and checking whether there is also no mechanism linking X and Y. If this is the case, the evidence points towards a mere correlation. Additionally, it could also be that the process tracing reveals one or more contextual factors that impact the working of the mechanism(s), but have not been considered so far. These new contextual features should then be added to revise the mapping of the cases and define more homogeneous subsets.

Based on this initial process tracing of one case, if resources allow it, we should conduct a second study of a case that is as similar as possible on as many relevant causal and contextual factors with the initially studied case. Finding a similar mechanism(s) operative in the second case increases our confidence that the process works similarly across cases. This way, we reduce the risk of missing important factors that might impact how the mechanism works. If, on the other side, we find a different (or no) mechanism(s) operative in a similar case, we would need to look for omitted conditions that differ between the two cases and which explain the difference in the underlying mechanism.

The exploration of mechanistic heterogeneity then continues by strategically selecting more and more different cases to identify the boundaries within which the mechanism operates. When we find different mechanisms operative, we would then want to assess what conditions differ between the cases to understand under which conditions different mechanisms are operative.

This exercise of empirically testing for mechanistic heterogeneity should be done with an eye to those sources which seem particularly problematic for the research design. For instance, if one of the main explanatory factors is operationalized via a complex concept, one should check whether different causal attributes impact the unfolding of a mechanism. Similarly, researchers should pay close attention to potential interactions, sequencing, and other dynamics among mechanisms that are hidden behind simple X/Y relationships if there is some theoretical or empirical argument that would lead researchers to expect this. In other words, instead of assuming that the same causal mechanism is present in all cases showing X and Y, we encourage researchers to look beyond the results of the cross-case analysis and leverage additional theoretical and empirical insights and probe whether the mechanistic homogeneity or heterogeneity is present in their MMR design.

10.6 Concluding Remarks

One reason for the popularity of MMR is that its main objective coalesces with the evolving consensus in the social sciences that strong causal explanations require evidence of an association between X and Y and evidence for the underlying causal mechanisms between X and Y. The main objective of this chapter was to familiarize researchers with the notion of mechanistic heterogeneity and the challenges this causes when conducting MMR based on some type of cross-case analysis in combination with some form of within-case method. After discussing some basic logics of MMR, we introduced the idea of mechanistic heterogeneity. We highlighted several sources that can bring about causal heterogeneity at the mechanism level in MMR designs. We contend that mechanistic homogeneity is typical when conducting social science research. Starting from the assumption that the social world is characterized by causal complexity, which might be present both at the cross-case level and the within-case level, we must pay more attention to mechanistic heterogeneity when making generalizations about mechanisms. Otherwise, we risk ending up with flawed inferences about the working of causal mechanisms across a sample of cases.Footnote 9

Assuming causal homogeneity at the level of mechanisms makes MMR designs considerably easier. But, as tempting as it might sound, we simply do not know a priori whether this assumption is correct in any given MMR design which strives to integrate insights derived through within-case studies and results from a cross-case analysis. To put it more bluntly, “[...] merely assuming that populations are similar at lower levels would amount to an extrapolation based on hope” (Khosrowi, 2019, 48). Against this backdrop, we call upon researchers to do better than assuming mechanistic homogeneity. Instead, we engage in empirically testing the limits to which we can generalize mechanistic claims, transparently map out the presence of mechanistic heterogeneity, and establish the proper boundaries for the generalization.

The debate about how to achieve this goal is just beginning. We hope that the guidelines and insights presented in this chapter help to improve research practices and encourage more explicit guidelines on how to address mechanistic heterogeneity while deploying different combinations of methods.

Suggested Readings

  1. 1.

    Beach, Derek, and Rasmus Brun Pedersen. 2019. Process-Tracing Methods: Foundations and Guidelines. Second Edition. Ann Arbor: University of Michigan Press.

  2. 2.

    Beach, Derek, and Ingo Rohlfing. 2018. Integrating Cross-Case Analyses and Process Tracing in Set-Theoretic Research: Strategies and Parameters of Debate. Sociological Methods & Research 47(1): 3–36.

  3. 3.

    Lieberman, Evan S. 2005. Nested Analysis as a Mixed-Method Strategy for Comparative Research. American Political Science Review 99 (03): 435–52.

  4. 4.

    Seawright, Jason. 2016. Multi-Method Social Science: Combining Qualitative and Quantitative Tools. Strategies for Social Inquiry. Cambridge: Cambridge University Press.

  5. 5.

    Schneider, Carsten Q., and Ingo Rohlfing. 2016. Case Studies Nested in Fuzzy-Set QCA on Sufficiency: Formalizing Case Selection and Causal Inference. Sociological Methods & Research 45 (3): 526–68.

  6. 6.

    Weller, Nicholas, and Jeb Barnes. 2016. Pathway Analysis and the Search for Causal Mechanisms. Sociological Methods & Research 45 (3): 424–57.

Review Questions

  • What are the primary purposes of multimethod research? Can you illustrate the main strengths?

  • What pitfalls and trade-offs come with multimethod research?

  • How do variance-based and case-based approaches of multimethod research differ?

  • Define mechanistic heterogeneity and homogeneity in your own words. Can you give one or two examples of mechanistic heterogeneity from your field of research?

  • Discuss how serious you think the problem of mechanistic heterogeneity is in political science? For instance, is it common or only seldom? Does it depend on the understanding of the mechanism, or is mechanistic heterogeneity a problem irrespective of the existing variants?

  • Can you illustrate how mechanistic heterogeneity complicates the task of making generalizations in multimethod research?

  • Complex concepts, omitted conditions, and causal/temporal dynamics can be seen as major sources for mechanistic heterogeneity when linking cross-case and within-case analysis. Can you think of examples from your field of research which illustrate the described problems?

  • Make a list of advantages and disadvantages that come with the strategy that maps and tests the boundaries for generalization in multimethod research. Discuss whether the additional efforts justify the proposed gains. Is generalizing mechanisms based on hope a better strategy from your perspective?