Several characteristics distinguish synthesis efforts, particularly meta-analyses, in LCS from those in other disciplines, such as medical research, ecology, and economics. First, inputs are often a mix of quantitative and qualitative data. This mix is often determined by the degree of compatibility of data and sampling methods across case studies, as well as the extent to which the interpretation of case-study authors’ findings is integrated into the meta-analysis. Second, some meta-analytic techniques used in LCS, such as QCA, use the structure of interactions between variables within a case as the unit of analysis (i.e., case-oriented meta-analysis), which yields a qualitatively different explanation of variation across cases than meta-analyses of quantitative data only. Finally, and perhaps most important, standards for conducting a land change case-study are even less agreed upon in LCS than in other contexts due to the diversity of contributing disciplines, which can lead to substantial inconsistencies in data types and observational instruments across case studies.
Interpretability and standardization
Synthesis methods require standardization—or at least harmonization—of relevant data across cases. In the context of LCS, where the blending of quantitative and qualitative data is common, standardization often entails identifying a robust array of possible cause–effect relationships involving the land change phenomenon of interest and providing operational definitions for each variable that can be identified in the articles and implemented as codes. If the number of case studies in a meta-analysis is large, the researcher may have to train others to perform the coding based on a predefined set of variables. When disagreements occur in the coding of a variable in a particular case, a discussion between the coders ensues about ways to classify or measure the presence or absence of a particular variable in a case. Alternatively, these discussions may expose an ambiguity in the variable as it is formulated and/or reported. A kind of ‘progressive contextualization’ (Vayda 1983) occurs in which the analyst explores and then explicates the links between land changes and the larger contexts of the change. Through these iterative procedures, the analyst may become aware of new patterns in the data, often having to do with the contexts of the case studies.
Generally, the more vaguely defined or reported a variable, the more likely it is that coders interpret a particular case differently and data standardization across cases will be difficult to establish. Interpretability and standardization challenges arise from the inherent difficulties in comparing case studies that are conducted with different objectives, use diverse approaches, and methodologies, and rely on varying levels of information and empirical evidence for results. Unambiguous empirical evidence for cause–effect relationships in land change studies is difficult to establish as feedback mechanisms and multiple drivers easily confound relationships, necessitating highly contextualized explanation. Thresholds of solid evidence and forceful argumentation are not clear-cut, and although quantitative methods often appear more convincing in describing relationships, qualitative methods are often more powerful for analyzing causality (Rudel 2008). Substantial variability may exist across case-study findings, which can only be disentangled if the primary data are available.
The spatial scale and extent of analysis within case studies are additional sources of interpretation and standardization issues, and can bring into question the spatial validity of synthesis and meta-study findings. This stems from incomplete or ambiguous geographic descriptions of the study sites used in case studies. A high-quality geographic description of a case-study site depends not only on the precision of the geographic details provided (e.g., geo-referenced maps, geographic coordinates, and/or text descriptions), but also on the clarity of the relationship between the geographic site and the reported data, and the degree to which subsequent users of the case-study are able to accurately interpret the geography and global context of the study site in a geographic information system (e.g., to map study site coverage across world regions). Frequently, geographic descriptions of case-study sites are missing at least one of these elements. Geographic descriptions are commonly provided in the form of maps, but are often not represented with the precision needed to make direct connections between reported data and the boundaries of the study area. For example, local land change case studies are often conducted at the village level and represented as points on a map. While the inhabitants of a village (or county, municipality, or other administrative area) and their land-use decisions might be central to the research, the land change patterns represented in such studies might precisely conform to a village boundary, extend across village boundaries, or represent some undocumented subset of land-use patterns within these boundaries. In general, for case studies of geographic entities above the size of small field plots, geographic point locations are incapable of capturing the geographic context and variability of land change patterns or processes typical in most land change case studies. These issues of geographic representation make it difficult to assess which part or parts of Earth’s land are actually represented by the results of a given synthesis and cast doubt on how generally applicable findings may be to the global or regional patterns of local land change processes.
The aggregation of individual case-study data required for synthesis may therefore suffer from the inherent problem of attempting to compare the incomparable. As a first priority, strengthening the case-study data reporting standards in LCS to create forms useful for cross-study synthesis (metadata) is essential, especially if this can be supported by enhanced tools for data sharing, searching, and synthesis across studies (Wolkovich et al. 2012; Agarwal et al. 2010; Ellis 2012). The geographic location and extent of individual LCS studies are especially critical to understanding global context and relevance of case studies for synthesis research and should be described using standard geographic data, such as Google Earth.kml, which has already become an option at Elsevier journals (e.g., Jetz et al. 2012; Karl et al. 2013; Martin et al. 2012; Van Vliet et al. 2012). A complementary approach—revisiting older case-study sites to create longitudinal datasets—would provide more empirically solid data compared to ‘snapshot’ studies that rely on recall and/or predictions. A recent example, following on a global meta-study on swidden cultivation (van Vliet et al. 2012), is a compilation of 8 longitudinal case studies on the consequences of swidden change (van Vliet et al. 2013).
Adoption of other non-spatial standardized observational instruments and data reporting standards, like those prescribed by the diagnostic institutional analysis and development framework for analyzing socio-ecological systems developed by Ostrom (2007) or the FAO’s Land-Cover Classification System (Jansen and Di Gregorio 2002), would also facilitate cross-comparison and analysis of local land change case studies. However, the utility of such metadata descriptors tends to depend on the user and specific synthesis effort. While greater metadata detail would appear to offer greater opportunity to support synthesis, we acknowledge that more detailed meta-knowledge systems are time-consuming for data producers to apply to their studies, but firmly believe their use is crucial to the advancement of LCS. It is worth noting that Ostrom’s well-known work on the governance of forest commons—the International Forestry Resources and Institutions (IFRI) project (Wollenberg et al. 2007)—was supported by FAO, following on the success of her prior cross-site work on irrigation systems (Tang1992). Programs underwriting comparative work, such as the US NSF’s Research Coordination Network program, may build on the IFRI model to support the sorts of standardized studies necessary for rigorous testing and development of land change theory.
The desire for sufficiently compatible data to support comparative land change analysis is not new, extending back at least two decades, including an edited volume on the comparative analysis of human societies (Moran 1995) and the science/research plan of the Land-Use and Land-Cover Change (LUCC) Project (Turner et al. 1995). Lack of progress toward this goal may be attributable to a number of factors, perhaps primordially the absence of a disciplinary home. By its nature, LCS is an interdisciplinary endeavor, with practitioners axiomatically operating at the edges of fields whose professional societies may not be motivated to develop or encourage the use of standard research protocols. Likewise, while journals devoted to land change topics have established footholds in the rapidly growing universe of peer-reviewed publications, none has yet developed procedures for describing, much less archiving, the data on which submissions are based. Some progress has been achieved through increasingly stringent requirements on the part of funding agencies (e.g., US NSF) that grant recipients make their data available, e.g., through the Inter-University Consortium for Political and Social Research (ICPSR); however, formal requirements have generally been limited to US examples. The development and widespread use of optimal meta-knowledge systems for LCS synthesis will require concerted efforts on the parts of professional societies, journals, and funding agencies and particularly the global change research community, including the Global Land Project.
Case acquisition and selection bias
The first task of a meta-analyst is to define the phenomenon under study, a set of keywords that bound the population of case studies, the languages within which s/he wants to search, and a search and selection strategy. While English is often the default option, particular topics merit searches across a series of languages. A search of case studies of tropical deforestation, for example, will be more complete if it includes journals published in French, Portuguese, and Spanish. Key word searches in the Web of Science®, and other databases can establish the population of case studies for analysis. Additionally, a scoping search prior to a full search helps to select appropriate sources and refine research questions and keywords. Using search engines and literature databases certainly identifies a population of scientifically reputable studies, but it can miss studies in the gray literature, such as government reports. This is often referred to as the ‘file drawer problem’ in which the population of studies used in meta-analyses tend to be those that are readily accessible, which biases the selection of studies by neglecting ‘gray literature’ and non-English publications. Incorporation of this gray literature into meta-studies has become increasingly unlikely due to the ease of generating the population of cases for study through search engines that focus on peer-reviewed literature. Additional case studies outside of the coverage of search engines can be included through expert recommendation. Although this may increase the number of relevant case studies considered, it may also introduce other biases into the case selection process and reduce the repeatability of the meta-study.
Another source of bias is introduced because case-study topics and locations often follow ‘fashion trends,’ and researchers tend to study negative rather than positive developments. An example is the Sahelian region of sub-saharan Africa, where the vast majority of case studies described processes, drivers, and impacts of desertification as being mainly human-driven (e.g., as outlined in Mortimore and Turner 2005) despite a counter literature that challenged this notion (Nicholson et al. 1990) and others providing evidence that desertification was not as widespread as previously thought and a ‘greening’ of the region was even observed (e.g., Olsson et al. 2005; Rasmussen et al. 2001). The focus on desertification was rooted in the fact that it was an issue high on the policy agenda, and hence, funding was available to study it and case-study sites were selected in areas where desertification was likely to be found (Mortimore and Turner 2005; Rasmussen et al. 2001). This example is not unique and can only be addressed if local case-study selection is done such that the full range of variation possible in the land change phenomenon of interest is represented. Defining and explicitly presenting precise case selection criteria will make clear the intended scope of the meta-study and allow reselection of the case set used for analysis.
It can also be difficult to obtain—or even know—the full range of variation in land changes because locales without the land changes of interest do not attract investigator interest and, as a result, the dynamics that contribute to no change may not be well represented in the published literature. This problem can be overcome, to some degree, by searching for detailed ethnographic studies of peoples who inhabit the areas of interest. While these studies may mention land-use change almost as an aside, they can provide valuable contrasts, when coded, to those cases that report an abundance of change. Similarly, case studies which did not find significant effects of land change are less likely to be published and thus might cause a bias in meta-analyses of effect sizes (e.g., Gurevitch and Hedges 1999). However, it is possible to detect this publication bias and quantify its impact on the validity of the results (Gurevitch and Hedges 1999; Nakagawa and Santos 2012).
Prospects for enhancing the availability of case studies lie in the data from local case studies being recorded and stored in a more accessible manner. Many valuable case-study results remain in unpublished theses and gray literature for a variety of reasons: They are never submitted for peer-reviewed publication because language barriers or lack of incentives to publish internationally; they are difficult to publish because they replicate other studies and/or lack significant findings of change (i.e., ‘no change’ case studies); and their data may have been produced for specific development projects, and access is therefore restricted. Efforts to make such studies available would require considerable efforts in obtaining access to reports and theses in multiple locations and languages. Translating published case-study literature into English and sharing it online would also vastly expand the geographic coverage and amount of case-study research available. Infrastructure that addresses these potentially large and rich data sources is highly valuable (e.g., the Inter-university Consortium for Political and Social Research, http://www.icpsr.umich.edu).