Introduction

Changes to landscapes such as habitat fragmentation and large spatial-scale restoration practices have many effects on biodiversity and key ecosystem processes [1–3]. The effects of landscape modification can be difficult to quantify for a range of reasons including: (1) the number of complex (and often interacting) spatial and temporal processes and patterns affecting landscapes [4, 5], (2) difficulties in replicating at large (landscape) spatial scales as well as controlling variability, and (3) the potential for substantial colinearity between landscape explanatory variables [6].

The not inconsiderable challenges associated with establishing large spatial-scale experiments and other kinds of studies in landscape ecology have led some workers to establish small spatial-scale experiments that are highly controlled and well replicated. These are sometimes called experimental model systems [7] or microcosm experiments, and they have been used to test predictions from a range of theories associated with landscape alteration (e.g. [8]). Although these small-scale experiments are undoubtedly valuable, the ability to accurately extrapolate results from these to a large spatial scale remains unknown [9], especially given the complex mosaic of different kinds of vegetation that comprises most spatial patterns of cover in the majority of landscapes [10, 11]. That is, with a small spatial-scale inferential framework, the external validity of results will be a serious issue.

The potential major limitations of small spatial-scale experiments underscore the importance of large spatial-scale studies in landscape ecology. Many types of studies can be used to quantify the effects of landscape change on biodiversity and ecological processes [9, 12], including observational studies, true experiments, quasi-experiments and natural experiments. In this review we briefly outline some of the strengths and weaknesses of each of these four classes of studies in a large spatial-scale landscape ecology context. Our insights are broadly based on our collective ecological and statistical expertise and experiences in establishing different kinds of landscape studies in several parts of south-eastern Australia over the past 35 years.

Definitions and Features of Four Broad Types of Studies in Landscape Ecology

Observational Studies

The majority of studies of the effects of landscape modification are observational investigations that do not exploit active interventions (e.g. manipulation of sites) to study biotic response (or other response variables) to landscape change (reviewed in [5, 9]). The basis for observational studies is the observation and measurement of a given ecosystem or ecological process. Conclusions are determined on the basis of these observations. Variables can be (although not always) relatively easy to measure and are an intrinsic property of the units (i.e. endogenous or internal to the ecosystem, such as natural regeneration of trees and other vegetation) rather than being assigned to them (i.e. exogenous or originating outside the ecosystem, such as a pre-determined intervention involving the planting of vegetation) (see Fig. 1).

Fig. 1
figure 1

Key differences in the broad types of studies in landscape ecology, namely, true experiments, quasi-experiments, natural experiments and observational studies. The strength of inference increases from top to bottom (i.e. from observational studies to true experiments). The primary difference between strong vs. weak quasi-experiments is whether interventions are exogenous or endogenous.

Full size image

The main advantage of observational studies is that it they do not interfere with the ecosystem under study. The major disadvantage is that they are mensurative, and therefore the inferential status of the results is weak in that they cannot answer causal questions because it is often difficult to separate the effects attributed to different variables and therefore associate a variable with a particular outcome. Explanatory variables in observational studies are endogenous and are determined by the processes that also influence the outcome. The design of observational studies typically does not include well-defined ecological contrasts, and there are few constraints on site selection, although stratification of sites can be employed. No method of analysis can overcome the difficulties associated with confounding effects of unknown or unmeasured variables.

Despite the inherent problems of observational studies, they can nevertheless produce important insights into the impacts of landscape modification on biota. As an example of an observational study we have established, cross-sectional statistical relationships were quantified between the occurrence of the vulnerable gliding marsupial, the Yellow-bellied Glider (Petaurus australis) [14], and spatial patterns of landscape-level, forest age-class cover in the wet montane ash forests of Victoria (south-eastern Australia). This work found evidence for the presence/absence of the species being significantly more likely to be associated with large patches of old-growth forest in the landscape. However, there was some confounding between explanatory variables, with old-growth patches being on flatter and wetter parts of the landscape [15], making it difficult to determine if patterns of animal occurrence were influenced more strongly by stand age, forest productivity or a combination of both potential explanatory variables—a problem that is still unresolved.

True Experiments

True experiments [sensu 16–18] allow the determination of causal links between manipulated factors and their measured effects on an ecosystem. This makes them particularly useful in landscape ecology where relationships between actions and outcomes can be established and then incorporated into management practices. An intentionally designed (true) experiment represents an organised and planned inquiry conducted under at least partially controlled conditions [17, 18]. Such true experiments involve artificially altering or manipulating a landscape or ecosystem in some way to force the system to yield information about the effects of variables that have been manipulated. In true experiments, experimental treatments (interventions) are therefore intentional and designed to answer specific questions. The salient attributes of a true experiment include random assignment of treatments, external replication, control of variability and the presence of a control treatment. Random assignment of treatments to experimental units is important, ensuring that observed responses are actually caused by the treatments themselves and not by the way in which they were allocated. Replication is necessary to establish the degree of reproducibility of responses to treatments and ensure that the investigation has internal validity. The blocking or grouping of units that are similar in some respect allows for the control of possible sources of variability and therefore increases the precision with which treatment comparisons can be made.

Fisher [17, 18] was responsible for much of the development of experimental design, initially within the field of agricultural science. Cox [19] and Mead [20] provide comprehensive accounts of statistical design of experiments, and Johnstone [21] discusses some practical issues in planning and managing experiments in ecology and agricultural research. There are relatively few true experiments in landscape ecology, although some of the fragmentation experiments established over the past 5–30 years are an exception [3, 22–25]. We established a true experiment in the wet forests of Victoria to answer the question: What is the response of vertebrate biota to different spatial patterns of logging? The experimental treatments were four kinds of logging: clearfelling, variable retention harvesting with small islands left unlogged, variable retention harvesting with large islands left unlogged and unlogged (control) areas. The experiment was implemented in blocks, with each block comprising one of each of the four treatments. Faunal occurrence was documented before and after the implementation of treatments, and the work to date suggests that variable retention harvesting has positive benefits for small mammals and birds, with the size of retention islands making little difference to animal response (e.g. [26]).

Whilst true experiments are the most powerful form of study to support strong inference, they can have some important limitations when used in landscape ecology. For example, it is often too expensive and not practically possible to implement random assignment of treatments, a key pillar of an orthodox 'Fisherian' true experiment [18]. As outlined above, adequate replication of large experimental units (such as entire landscapes) for true experiments is often difficult, and it is even more of a challenge when the target species for study are wide-ranging, highly mobile taxa, such as birds and bats [27]. In other cases, the strict statistical design constraints that underpin true experiments may limit the kinds of potentially important factors and/or management interventions that can be tested, including complex interactions among factors (but see [28]). For example, the Suitability of Altered Forest Ecosystems Project (SAFE) in Borneo is quantifying a range of drivers of effects of fragmentation, but the effects of matrix conditions surrounding fragments are not being examined despite their importance being increasingly recognised [29]. Finally, some simplified treatments may not be relevant to on-the-ground landscape management. As an example, Russell-Smith et al. [30] reported the results of an elegant experiment quantifying the effects of fire on biodiversity in the tropical savannas of northern Australia. However, they acknowledged that the outcomes were of limited value for fire management because the constraints on the kinds of treatments that could be implemented in the experiment meant that the work “failed the test of management relevance” [30].

Quasi-Experiments and Natural Experiments

The term quasi-experiment was first coined by Campbell and Stanley [31] and is used widely in many contexts, particularly the social sciences. Shadish et al. [32] use the term quasi-experiment to describe an “experiment that lacks random assignment but otherwise possesses many of the structural features and purposes of a randomised experiment”. This is the definition adopted in this review. In a quasi-experiment within a natural context, a management treatment such as tree planting or plantation establishment (e.g., see [33, 34]) is compared with one or more contrasting treatments but there can be little or no random assignment of areas to interventions (treatments), as they already exist. In these situations, random selection may have to be a substitute for random assignment. However, a formal process of random selection is essential, as is effective spatial replication. Thus, quasi-experiments lack random assignment but this deficiency can be partly compensated for by extensive replication and random selection.

Natural experiments typically take advantage of uncontrolled (usually exogeneous) events. That is, an intervention has occurred because of some naturally occurring event and the researcher takes advantage of this. An example is wildfire (e.g. [35]). In this case, the selection of experimental units is typically restricted to one or two large contiguous areas, and hence there can be no replication of experimental units and therefore no random selection. This means that for natural experiments, intervention effects may be confounded with other factors, and the design may therefore not be internally valid. Furthermore, pseudoreplicates will not be statistically independent, and so the design does not provide a valid estimate of experimental error.

True experiments are manipulative and created, but quasi-experiments and natural experiments are found; investigators therefore need to be alert to opportunities to establish them as chances to do so are not common in landscape ecology. For example, decisions by the Government of New South Wales to expand the plantation estate in south-eastern Australia provided a unique opportunity to establish a major landscape-level quasi-experiment in 1998. This work in the Nanangroe region of southern New South Wales was designed to quantify temporal dynamics in vertebrate biota within woodland patches as the surrounding landscape was being transformed from grazing-dominated paddocks to a landscape dominated by a maturing Radiata Pine (Pinus radiata) plantation [33]. A total of 56 patches was selected from a 'population' of 70 woodland patches, and a stratified random selection process was implemented so that patches represented different size and vegetation classes [33]. The vertebrate fauna of the woodland patches was documented prior to the clearing of the surrounding paddocks (primarily the removal of scattered paddock trees) and then on a yearly or biannual basis thereafter. A set of 55 woodland patches were selected on neighbouring farms where there has been no landscape transformation to act as de facto controls for appropriate contrast with the woodland patches in the pine plantation. Surveys of all patches has been ongoing since 1998, and a range of novel responses to plantation establishment has been documented [36, 37].

Although the inferential status of results of quasi-experiments is weaker than that of a 'true' experiment (i.e. quasi-experiments cannot establish causality in the way that a true experiment can), they are an attractive strategy in landscape ecology because they have fewer practical constraints and often fewer ethical constraints (such as the partial clearing of areas of native vegetation). Furthermore, they can be conducted in a realistic way and on a larger spatial scale in a natural setting and so often provide a wider general inferential framework than a true experiment (e.g. see [38]). Apart from this important and fundamental distinction, quasi-experiments have truly comparable experimental units, including controls, and share many other important design and data structural attributes of a true experiment. Importantly, quasi-experiments usually have both internal and external validity; that is, the design provides appropriate data to allow valid estimates of experimental error with adequate precision, a basic requirement for inference. Some of the shortcomings of quasi-experiments may, in part, be compensated for by establishing these experiments on a large spatial scale.

Statistical Checklist for the Design of True Experiments and Quasi Experiments

In this section we present a checklist of important statistical design issues that should be considered in developing and implementing true experiments and quasi-experiments. Many involve sound experimental design principles that can be applied in landscape ecology where manipulations are often constrained and have effects at different spatial and temporal scales. The insights outlined in this section are based on our collective experiences in designing several landscape studies that we have established over the past 25 years.

Background Ecological Issues

Posing good and important questions lies at the heart of all robust scientific studies, including landscape-scale studies [39]. However, ecologists have often been poor at setting substantive questions [40]. A paucity of good questions is a serious problem because it often results in studies being poorly focussed and incapable of delivering effective outcomes [39]. Setting substantive questions will often demand a deep understanding of the ecosystem in question as well as data and statistical considerations. Setting good questions can also be guided by a good conceptual model of the ecosystem in which an experiment or other kind of study is planned [39].

An important practical consideration in study design is whether there is a natural system in a landscape suitable for experiments that offers appropriate interventions or ecological contrasts and are these interventions or ecological contrasts exogenous or endogenous (as defined in the preceding section). A related practical issue is whether the resources and expertise are available to seize the opportunity to study that system or set of landscapes.

Structural Features

A number of structural issues underpin a good design of landscape studies. For example, researchers need to be aware of the added value of factorial treatment structures in preference to a sequence of smaller studies. Factorial treatment structures provide information on interactions as well as increased precision in quantifying main effects [41]. Additional issues include: Are there distinct natural spatial units and are there identifiable units at multiple (nested) levels? Which spatial units are relevant for inferences about interventions or ecological contrasts? What is the extent of the framework for inference (scope of inductive basis) and does the scope of the investigation reflect inherent complexity of the landscape? That is, is inference spatially limited or extensive? The large array of sites, farms and landscapes in our quasi-experiment on the South West Slopes of New South Wales represent the range of environments across the entire bioregion [42, 43], and the study therefore has extensive inference over several million hectares. Conversely, the scope of inference of our landscape studies in the unique range of habitats at Booderee National Park is relevant only to that 6600 ha in that reserve [44, 45]. What are the resultant data structures arising from a particular statistical design and what is an appropriate statistical model for analysis and its implications for data analysis?

Size

Good design is about obtaining high-quality data at minimum cost. A key issue is that the size of a study is important, particularly at the level relevant for inference relating to treatment effects. Important questions are therefore: How many units, such as sites or transects, are needed? Observational units are usually relevant to measurement, whereas experimental units are relevant to inference about interventions. Is there sufficient existing data and relevant parameters for formally calculating the statistical power of the study? If estimation of detection probabilities (as distinct from occupancy probabilities) is an important consideration, it may be necessary to repeatedly survey some sites. Is the design internally valid so that replicates of treatments or ecological contrasts are statistically independent (i.e. avoiding or limiting pseudoreplication)? Does the design provide precise/efficient estimates of experimental error at all relevant levels? Do researchers understand the consequences of too many or too few replicates at each of the multiple levels of the ‘experiment’? In a nested experimental design, it is important to have greater replication at a higher level in the design as replication percolates down the design. Useful insights into the adequacy of replication at all levels can be obtained by constructing a 'dummy' analysis of variance table.

An example of some of the principles outlined in the preceding paragraph is the large-scale study quasi-experiment which we established to quantify the effects of planting on biodiversity. The hierarchical design encompasses 23 landscapes (with and without restoration plantings, with 12 landscapes in one region and 11 in another), two farms within those 23 landscapes (farms with and without restoration plantings) and four sites located on the 46 farms within the 23 landscapes (comprising plantings and remnant woodland patches) [38]. This design provided a powerful basis from which to make inferences about the impacts of planting on biodiversity at the landscape, farm and site levels [42, 43]. A final key question is: Is there inefficient use of limited resources? Rather than one study with more replication than is essential, it may be possible to conduct additional experiments.

Statistical Control of Variation

There are at least three key considerations associated with the statistical control of a landscape study. First: Is blocking or pairing of spatial units possible? Here the aim is to select blocks and pairs so that units within blocks as are similar as possible. Intra-block correlations are therefore high. Local control of variation by blocking can be beneficial as natural systems can be highly variable. Second: Can stratifying variables be identified to account for identifiable heterogeneity (e.g. vegetation cover, agricultural practice)? Third: Are there measurable, potentially useful covariates?

Assignment

Random assignment is often not possible in landscape-scale studies. Furthermore, the random selection of sites (or other units) may be compromised by practical logistics in landscape ecology. However, there may be circumstances where it is useful to develop formal approaches for random selection. Such formal approaches may involve enumerating all eligible units, thereby clearly defining the population being studied and the framework for inference. Identifying stratifying variables to further define the selection process can add value to the study. The large spatial-scale quasi-experiment established by Lindenmayer et al. [46], with the aim to quantify the effects of forest fragmentation and landscape context effects on birds, is an example of the application of these principles.

Other Considerations

The preceding checklist outlines some of the statistical aspects of the design and implementation of large-scale studies in landscape ecology. In the following sections we discuss some additional considerations underpinning the design and implementation of observational studies, quasi-experiments and true experiments.

Long-Term Statistical Planning Prior to Implementation

Design is an inherently statistical process and seeks to maximise information at minimal cost. Indeed, it has been fundamental to all the studies that we (RBC and DBL) have established in the past three decades. In each case, detailed planning for a prolonged period (often exceeding 6 months) has occurred prior to the establishment of a study, and this has entailed close collaboration between a professional statistical scientist (RBC) and a landscape ecologist (DBL). In conducting such prolonged pre-implementation planning, we have attempted to avoid RA Fisher’s [17] concerns about asking a statistician for advice after a study has been done; that is: “To consult a statistician after an experiment is often merely to ask him [her] to conduct a post mortem examination. He [she] can perhaps say what the experiment died of”.

Identifying Meaningful Ecological Contrasts and Controls

It can be a challenge to define valid controls in landscape ecology studies for a range of reasons. For example, many studies in landscape ecology attempt to contrast the biodiversity of human-modified environments with that in 'intact' areas that have not been heavily disturbed by humans. However, such relatively 'intact' areas are often undisturbed for a reason—they have not been considered suitable for agriculture, urbanisation, forestry or some other human use. This may have been because they are steep, low in productivity or have limited accessible water. Hence, their environmental characteristics are often fundamentally different from those of areas which have been subject to extensive human disturbance. There is therefore potential inherent confounding between landscape change and environmental conditions, and this confounding can be important as the environmental conditions can be a critical determinant of the distribution and abundance of many elements of the biota [47]. We suggest that detailed analysis of the environmental conditions that characterise a proposed study region can be an important part of the planning of landscape ecology studies and, for example, determine the extent of environmental differences between ‘intact’ areas and those supporting ‘fragments’ of original vegetation cover.

Other studies in landscape ecology do not require a contrast between controls in ‘intact’ landscapes and human-disturbed landscapes. Rather, of greater interest in such studies is the impact of a given management landscape intervention on biodiversity, a landscape process or some other response variable. Agri-environment schemes [48] are an example of policies where farmers are paid to implement particular management practices to improve agricultural environments for biodiversity, vegetation condition, soil stability or some other attribute of farmland environments [49, 50]. In these cases, ‘control’ sites may be areas where ‘business as usual farming’ is maintained and used for contrast with places where altered management practices (sponsored through an agri-environment scheme) are implemented. This was the case in a large spatial scale agri-environment scheme in eastern Australia spanning more than 2000 km south–north where temperate woodland patches on 158 farms were subject to enhanced conservation management through livestock grazing control, weed management, revegetation, and cessation of firewood and bushrock harvesting. The effectiveness of such agri-environment scheme practices were contrasted with temperate woodland ‘control’ patches on the same farm where there were no changes in pre-existing management practices [51, 52]. For that study, it was important for the ‘control’ and intervention sites to be on the same farm because farm-level management practices, such as prevalence of chemical spraying and the control of exotic predators, can have significant impacts on biodiversity [53], and these often vary markedly between farms.

In cases where an intervention is to be studied, taking baseline measurements provide a natural ‘control’; that is, each site becomes its own control. As an example, we established a prospective longitudinal quasi-experiment to quantify landscape context effects (sensu [54]) on vertebrate biota at Nanangroe in south-eastern Australia. Relevant contrasts in this comparative quasi-experiment were provided by the woodland patches on farmland, pine stands surrounding the woodland patches and cleared paddocks surrounding the woodland patches on farms [33, 55]. Each one of the 131 sites in this study essentially acts as its own control for quantifying patterns of temporal change in biota as the surrounding landscape undergoes change (associated with the maturation of adjacent stands of plantation pine) or for farmland sites where the surrounding grazed landscape remains unchanged.

Have Sites (or Other Experimental Units) in Reserve for Contingencies

The planning of studies should make provision for the potential loss of sites to ensure the work can validly continue if such losses occur. This issue is important because large spatial-scale landscape ecology studies will almost always entail working on land that is not under the direct control of a researcher. Sites (or other experimental or observation units) in large spatial-scale studies can therefore be vulnerable to being lost, which can compromise the strength of inferences. For example, the logging experiment we describe in the section True Experiments that quantified the use by vertebrate biota of islands of retained forest within otherwise clearcut stands in south-eastern Australia was potentially compromised when a retention island which was one of the treatments was mistakenly cut down by harvesting contractors. The structure of the experiment included four treatments in a given block; consequently, three other treatments in the block were lost. Fortunately, each block was replicated eight times, meaning that sufficient replicates (seven blocks, each with four treatments) remained, thereby facilitating the continuation of the experiment [26, 56].

Have the Capacity to Respond Quickly

As outlined in the preceding text, major natural disturbances, such as wildfires, can create important opportunities to establish natural experiments, such as to document the response of biodiversity to fire severity and landscape heterogeneity created by wildfire [57] or other kinds of natural disturbances (e.g. volcanic eruptions [58]). Starting work as soon as possible after such natural disturbances can be important. For example, such work can help quantify if particular species remain on disturbed sites and, in turn, influence the kind of recovery mechanism (e.g. nucleated internal recovery vs. dispersal from unburned areas beyond the boundary of a disturbed area [59]). However, researchers need to be cognisant of the statistical limitations of natural experiments as mentioned previously.

‘Experiments’ Provide a Secure Foundation for Monitoring Studies

Well-designed large spatial-scale true experiments and quasi-experiments provide a powerful framework for repeated measurement studies, thus becoming longitudinal studies or spatio-temporal studies in landscape ecology. The inference can then focus on changes taking place over time as well as on cross-sectional differences. Studies of this nature can clearly distinguish changes over time from differences between sites; each site becomes its own control and so provides a powerful opportunity for assessing direct effects that are not obscured by cross-sectional differences (see [42, 43]).

The many advantages of long-term studies have been examined elsewhere [60, 61], but their maintenance requires ongoing funding and logistical support, both of which represent considerable challenges, especially at large spatial scales. For example, meeting this challenge may require chief investigators to continue to pose new and appropriate questions to ensure the work is ‘novel’, thereby making it attractive to funders. Maintenance of the long-term integrity of time-series data is a key issue in long-term studies, and field measurement protocols may need to retained throughout the duration of a long-term study to limit confounding between changes in field protocols and temporal changes in the entities being measured [39]. If new field measurement methods emerge, they may need to be calibrated against pre-existing ones [62].

Discussion

There is a growing recognition of the need for decisions associated with the management of landscapes to be evidence-based [63]. Different kinds of studies can be employed to gather such evidence. In this paper, we outline the characteristics of four broad types of study undertaken in landscape ecology, namely, observational studies, true experiments, quasi-experiments and natural experiments. It is important to be aware of their respective strengths and limitations, in part to be aware of the strength of inference, and hence the veracity of evidence, associated with each approach. The type and validity of any inferences that might result from an analysis of data depend very much on the method of data collection and quality of the data.

The application of the principles of good experimental design are critical for each kind of study. Indeed, statistical science is concerned not only with informative methods of data analysis but also with improved design. Good experimental design is an inherently statistical process, and this highlights the importance of collaborative partnerships between statisticians and ecologists in landscape ecology. No analysis or theory of statistical inference can compensate for fundamental flaws in design.