Introduction

Economists routinely use treatment effects models to evaluate government programs’ causal impacts on observed outcomes (Imbens and Wooldridge 2009; Athey and Imbens 2017). Identifying causal relationships requires addressing various potential sources of bias, including non-random selection, confounding, and spillover effects. Economists have developed a variety of quasi-experimental impact evaluation methods to address these biases, including matching, instrumental variables, regression discontinuity, and difference-in-differences. These methods implicitly assume that the economic outcomes of interest are observable. In some settings, however, economists only observe outcomes measured in physical terms, which might lack well-defined economic interpretations. Although impact evaluations using physical outcomes might be feasible, their results do not necessarily serve as useful proxies for programs’ economic impacts. This paper presents a method for inferring the economic impact of a program when outcomes are observed in physical terms.

We present this method in the context of forest conservation and demonstrate its empirical application by evaluating a forest zoning system in Thailand that aimed at protecting mangrove forests. Mangroves are coastal ecosystems located in the tropics and subtropics that provide a wide range of benefits to local and global populations. In a series of papers, Mäler and colleagues listed these benefits as including the production of shellfish, firewood, timber, and game; feeding and nursery areas for commercially important fish; shoreline and storm protection; nesting, resting, and feeding sites for birds; and carbon sequestration (Mäler et al. 1996, 2008, 2009; Arrow et al. 2000). A large literature confirms the economic importance of these benefits (Himes-Cornell et al. 2018), including in Thailand (Barbier 2007).Footnote 1

Private land owners, however, often have incentives to convert forests, including mangroves, to more profitable land uses. Since the free market likely fails to supply the socially optimal amount of forest, governments often intervene by implementing forest conservation programs, including protected areas, payments for ecosystem services, and community-based forest management. Most impact evaluations of these programs use forest cover as the outcome variable and aim to measure a program’s causal effect on avoided deforestation (Blackman 2013; Lan and Yin 2017). An impact evaluation of this type defines a program’s average treatment effect on the treated (ATT) as the difference between the realized forest cover change in treated areas—i.e., sites where the program was implemented—and the counterfactual forest cover change that would have occurred in the same areas had they not been treated.

The ATT for avoided deforestation is a good proxy for a forest conservation program’s economic impact only if the benefits and costs of conservation are homogeneous (Vincent 2016). Environmental economics theory has recognized that environmental values can instead vary spatially and that government programs to protect the environment must account for this variation since at least Mäler’s (1974) seminal treatise. There is abundant evidence that both the benefits (e.g., carbon sequestration, Asner et al. 2010; biodiversity habitat, Gibson et al. 2011; Le Saout et al. 2013; watershed services, Pattanayak and Kramer 2001; Brauman et al. 2007) and the costs (Naidoo et al. 2006; Polasky 2008) of conservation are highly heterogeneous. Mäler et al. (2008; 2009) emphasize that the economic benefits of maintaining mangroves are “very case sensitive” and can vary greatly by location due to differences in ecological, economic, and institutional conditions. A program that increases forest cover by, for instance, 20% thus does not necessarily increase forest value by 20% too. Depending on the distribution of benefits and costs across the sites included in a program, the ATT for avoided deforestation can either overestimate or underestimate the ATT for a program’s economic impact.

We develop an indirect way to estimate the ATT for a forest conservation program’s economic impact by building on work by Heckman (2010), who emphasized the structural framework embedded in treatment effects models. We employ a discrete-choice Roy (1951) model to describe the process of selection into treatment. One can view this model as a random utility model (RUM; McFadden 1973; Manski 1977) and parametrically estimate the effects of forest attributes on behavior in response to a conservation program. A byproduct of this procedure is the estimated propensity score. We show that under certain conditions, a transformation of the propensity score has a utility interpretation, where utility is as perceived by the government agency that decides which forests to include in the program. We use this transformation as a weight to construct a utility-weighted ATT. This new metric can be viewed as a generalized form of the ATT for avoided deforestation, and it offers a better measure of the ATT for a forest conservation program’s economic impact, at least as perceived by the government.

Our method connects the impact evaluation literature to the non-market valuation literature. Propensity score matching is a common impact evaluation technique, and random utility models are widely applied in valuing non-market goods (Freeman et al. 2014). We show how to combine these two approaches and exploit their respective advantages. In doing so, we contribute to filling a gap in the environmental policy analysis literature, in which valuation and evaluation are disconnected (Ferraro et al. 2012).

Our empirical analysis focuses on mangrove protection in Thailand. In 1987, recognizing both the high value of Thailand’s mangroves and their rapid loss (Charuppat and Charuppat 1997), the Thai government introduced a zoning system for managing the nation’s mangroves (Havanond 2008). This system assigned mangroves to different categories that varied in terms of restrictions on forest use, including conversion to non-forest land uses. We evaluate the zoning system using both the ATT for avoided deforestation and the utility-weighted ATT that we develop, and we compare the results. The utility-weighted ATT indicates that the system avoided the loss of 12.8% of the economic value associated with the protected mangrove area. This impact is about a quarter smaller than the conventional ATT for avoided deforestation, 17.3 percentage points. The difference between the two measures indicates that the zoning system tended to be less effective at reducing deforestation in locations where the government perceived the net benefits of protection as being greater. This relationship is the opposite of what one would expect for a maximally effective program.

The rest of the paper is organized as follows. Section 2 presents the theoretical foundations and econometric aspects of our method. Section 3 applies the method to the mangrove zoning system in Thailand. The last section discusses the interpretation of the utility-weighted ATT and its limitations as an evaluation metric.

Deriving and Estimating a Utility-Weighted ATT

Impact evaluations are program-specific, and the empirical strategy one should use to identify program impacts depends on program features and the data structure. Although our proposed method can in principle be applied to non-environmental programs, to explain its derivation and estimation we refer to the evaluation of a hypothetical forest protection program, with propensity score matching as the evaluation strategy. We begin with an intuitive explanation of the method before presenting details on its derivation and estimation.

The Intuition for a Utility-Weighted ATT

Avoided deforestation (Q), a binary variable that equals 1 if deforestation does not occur (i.e., forest is retained) and 0 otherwise, is the most widely used outcome variable in the impact evaluation literature on forest protection programs. To estimate the ATT for avoided deforestation (\(\tau _{ATT^Q}\)), economists first observe the realized forest outcomes for a sample of protected (“treated”) sites, then use a technique such as matching to construct for each treated site a counterfactual (i.e., the hypothetical forest outcome had the site not been treated), and finally compute an unweighted average of the differences between the realized outcomes and the counterfactuals across all treated sites. In this approach, all treated sites have the same weight; they are not weighted by the utility gains from protecting them. Hence, the total amount of effective protection–protection that actually results in avoided deforestation–affects the ATT, but which sites are protected effectively does not. Table 1 presents a simple numerical example of the evaluation of a program that protects three sites. We consider three hypothetical cases in which protection is effective at only one of the sites. When all sites have the same weight, \(\tau _{ATT^Q}\) equals one-third in each case, regardless of which site is protected effectively.

Even if the sites have the same area, the utility gains from protecting them could vary. We assume that effective protection generates the least gain at Site A and the greatest gain at Site C. We can incorporate these heterogeneous values into the evaluation by constructing a utility-weighted ATT (\(\tau _{ATT^W}\)), which is lowest when only Site A is protected effectively (Case 1) and highest when only Site C is protected effectively (Case 3). Compared to the conventional evaluation metric \(\tau _{ATT^Q}\), this utility-weighted metric \(\tau _{ATT^W}\) conveys more information on the economic performance of protection. As Table 1 illustrates, \(\tau _{ATT^W}\) can be larger than, smaller than, or equal to \(\tau _{ATT^Q}\), depending on which sites are protected effectively and how utility gains vary across them.

Site-level protection values are often not directly observable. We develop a new approach to overcome this data limitation within an impact evaluation framework by inferring the values in an indirect way. If the government protection agency (the “regulator”) makes rational protection decisions, then we can model their decision process by a RUM and estimate each site’s propensity score for being protected. A higher propensity score implies a higher expected utility gain as viewed by the regulator. Below, we formalize this approach and show how to derive site-level utility weights based on the propensity score and use them to estimate \(\tau _{ATT^W}\).

Behaviors of the Land User and the Regulator

We use the Roy model to explain forest-related behavior first for a private land user and then for a regulator. Assume there is a collection of forest sites, with individual sites distinguished by i, and for simplicity a single land user who decides whether to deforest each site. \(Q_i=0\) if deforestation occurs and \(Q_i=1\) otherwise. Unless necessary for clarity, we suppress the site index i henceforth to simplify notation. X denotes a set of site characteristics known to both the land user and the regulator. In the absence of regulation, the land user’s private utilities with and without deforestation are given respectively by \(Y^L_0\) and \(Y^L_1\),

$$\begin{aligned} Y^L_0&= \mu ^L_0(X) + U^L_0 \end{aligned}$$
(1)
$$\begin{aligned} Y^L_1&= \mu ^L_1(X) + U^L_1 \end{aligned}$$
(2)

\(U_0^L\) and \(U_1^L\) denote utility effects known only to the land user; both have expected values of zero. These effects could include the land user’s idiosyncratic knowledge of market conditions for products extracted from mangroves or supplied by non-mangrove land uses, or non-market uses of mangroves unknown to the regulator.

In the presence of regulation, a rational deforestation decision implies

$$\begin{aligned} Q = \mathbf {1}(Y^L_1 > Y^L_0) \end{aligned}$$
(3)

where

$$\begin{aligned} Q = D Q_{1} + (1 - D) Q_{0} \end{aligned}$$
(4)

with D being the treatment status (\(D=1\) if the site is protected, \(D=0\) otherwise) and Q1 and Q0 being deforestation outcomes with and without protection, respectively. The regulator (and econometricians) can observe Q but not \(Y^L_0 - Y^L_1\), the private utility gain if deforestation occurs. As written, Eq. (3) includes no penalty for noncompliance with the protection regulation. Absent such penalty, the land user has no incentive to modify their behavior compared to the unregulated market setting.

Turning to the behavior of the regular, we initially follow the mangrove management example in Arrow et al. (2000) by assuming the regulator aims to increase social welfare, which mirrors Mäler’s (1974) assumption of an environmental management agency with knowledge of society’s preferences. Recognizing the existence of a social benefit from retaining forest that the land user ignores, the regulator has an incentive to intervene and correct the market failure by protecting sites. The regulator is rational and protects a site if and only if the social benefit from retaining forest exceeds the expected private gain from deforestation (i.e., the expected opportunity cost of protection),

$$\begin{aligned} D = \mathbf {1} \Big (\mu ^S(X) + U^S > k^R\big (\mu ^L_0(X) - \mu ^L_1(X) \big )\Big ) \end{aligned}$$
(5)

The social benefit is on the left side of the inequality and has two components: \(\mu ^S(X)\), which is determined by observed site characteristics; and \(U^S\), which is the regulator’s private knowledge about the social benefit and has an expected value of zero. The right side shows the expected private gain from deforestation, \(\mu ^L_0(X) - \mu ^L_1(X)\), multiplied by a scalar \(k^R\) that converts the expected private gain to the same units as the social benefit; note that it does not account for the unobserved (by the regulator and econometricians) component of private utility. We can rewrite this expression as

$$\begin{aligned} \begin{aligned} D&= \mathbf {1} \Big ( \mu ^S(X) - k^R\big (\mu ^L_1(X) - \mu ^L_0(X) \big )> - U^S \Big )\\&\equiv \mathbf {1} \Big ( \mu ^R(X) > U^R \Big ) \end{aligned} \end{aligned}$$
(6)

where the second line uses more compact notation to relabel terms on the first line. We refer to this expression as the selection equation.

To induce compliance, regulation imposes a penalty, such as a fine or lost profits, that the land user must pay if they are caught deforesting (\(Q=0\)) a protected site (\(D=1\)). We assume the land user can form an unbiased estimate of these costs, \(C>0\), by multiplying their subjective probability of being caught by a prediction of the penalty. Because C is estimated by the land user, it is not observed by the regulator (or econometricians). Its addition changes Eq. (3) to

$$\begin{aligned} Q = \mathbf {1} \Big ( Y^L_{1} > Y^L_{0} - D C \Big ) \end{aligned}$$
(7)

which we can rewrite as

$$\begin{aligned} Q = D \times \mathbf {1} \Big ( \mu ^L(X)> U^L - C \Big ) + (1 - D) \times \mathbf {1} \Big ( \mu ^L(X) > U^L \Big ) \end{aligned}$$
(8)

We group U and C on the right hand side of the inequalities to highlight that they pertain to the land user’s private information.

Evaluating the Program’s Net Economic Impact

We now shift the perspective from the regulator’s ex ante protection decisions to ex post evaluation of the protection program. If the outcome variable is forest status Q, then the individual treatment effect can easily be defined based on Eq. (8),

$$\begin{aligned} \tau _i = Q_{1i} - Q_{0i} = \mathbf {1} \Big ( \mu ^L(X_i)> U^L_i - C \Big ) - \mathbf {1} \Big ( \mu ^L(X_i) > U^L_i \Big ) \end{aligned}$$
(9)

To interpret \(\tau _i\), we categorize all treated sites into three groups:

S1::

sites that will not be deforested regardless of being treated or not, i.e., \(Q_{1i} = Q_{0i} = 1\);

S2::

sites that will be deforested regardless of being treated or not, i.e., \(Q_{1i} = Q_{0i} = 0\);

S3::

sites that will not be deforested only if they are treated, i.e., \(Q_{1i} = 1\) and \(Q_{0i} = 0\).Footnote 2

From Eq. (5), the regulator views the net economic gain from protecting site i as the difference between the site’s social benefit, which is lost if the site is deforested, and the expected private gain from deforestation. Denote this difference by \(\Delta Y_i\). For sites in group S1, the land user voluntarily chooses not to deforest even in the absence of regulation, and so protection causes neither a gain in social benefits nor a loss in land user utility:

$$\begin{aligned} \Delta Y_i (i \in \mathbf {S1}) = 0 \end{aligned}$$
(10)

For sites in group S2, the land user always deforests. Protection still does not cause a gain in social benefits, but now it causes a loss in land user utility due to the additional cost of deforestation, C, in the presence of regulation:

$$\begin{aligned} \Delta Y_i (i \in \mathbf {S2}) = - k^R C \end{aligned}$$
(11)

As noted earlier, the regulator does not observe this cost, a point to which we will return. Finally, for sites in group S3, protection causes both a gain in social benefits and a loss in land user utility:

$$\begin{aligned} \Delta Y_i (i \in \mathbf {S3}) = \mu ^S(X_i) + U_i^S - k^R \Big ( \mu ^L(X_i) - U_i^L \Big ) \end{aligned}$$
(12)

The program’s overall economic treatment effect is determined by summing the impacts across all sites in all three groups:

$$\begin{aligned} \tau _{\Delta Y} = \sum _{i \in \mathbf {S2}, D_i=1} (-k^R C) + \sum _{i \in \mathbf {S3}, D_i=1} \Big ( \mu ^S(X_i) + U_i^S - k^R \mu ^L(X_i) \Big ) \end{aligned}$$
(13)

We have used here the assumption that the expected value of \(U^L_i\) is zero.

\(\tau _{\Delta Y}\) can be interpreted as the social welfare gain if the regulator is benign, but it lacks this welfare interpretation if the regulator is not benign. In the latter case, it measures the net economic gain according to the regulator’s preferences, which could differ from society’s preferences. We will offer some insights into this discrepancy for mangrove protection in Thailand by comparing the government’s stated criteria for protection to the factors that our estimated selection equation indicates actually influenced protection decisions.

Empirical Identification of the Utility-Weighted ATT

Empirical identification of \(\tau _{\Delta Y}\) faces four challenges. The first is that C is private information known to the land user but not econometricians. The version of \(\tau _{\Delta Y}\) that can be estimated will consequently omit the first sum in Eq. (13),

$$\begin{aligned} \tau _{\Delta Y} = \sum _{i \in \mathbf {S3}, D_i=1} \Big ( \mu ^S(X_i) + U_i^S - k^R \mu ^L(X_i) \Big ) \end{aligned}$$
(14)

This expression represents an upper bound on the true value of \(\tau _{\Delta Y}\). The difference between the estimated and true values of \(\tau _{\Delta Y}\) is smaller if \(k^R C\) is smaller, which occurs if the regulator places lower weight on land user utility, enforcement is weaker, or the penalty is lower. It is also smaller if C is a fine whose revenue is returned to the public (a transfer payment) or funds another government program that generates an economic gain.

The second challenge is that \(\tau _{\Delta Y}\) has a range of \(\mathbb {R}^{+}\). This range prevents an ATT based on it from being compared to the ATT for avoided deforestation. The latter is the expectation of Eq. (9) conditional on treatment,

$$\begin{aligned} \tau _{ATT^Q} = \mathbf {E} [Q_1 - Q_0 | D = 1] \end{aligned}$$
(15)

which is just the share of S3 among the treated sites,

$$\begin{aligned} \tau _{ATT^Q} = \frac{\sum _{i \in \mathbf {S3}, D_i=1} 1}{\sum _{D_i = 1} 1} \end{aligned}$$
(16)

\(\tau _{ATT^Q}\) thus ranges from zero to one. We rescale \(\tau _{\Delta Y}\) to the same range by applying the following expression,

$$\begin{aligned} \tau _{ATT^W} = \frac{\sum _{i \in \mathbf {S3}, D_i=1} \Big ( \mu ^S(X_i) + U_i^S - k^R \mu ^L(X_i) \Big )}{\sum _{D_i=1} \Big ( \mu ^S(X_i) + U_i^S - k^R \mu ^L(X_i) \Big )} \end{aligned}$$
(17)

The denominator represents the maximal utility gain that can possibly be achieved, while the numerator represents the actual gain. The “W” in \(\tau _{ATT^W}\) indicates that this economic ATT weights treated sites differently from each other, unlike the physical \(\tau _{ATT^Q}\) (see Table 1).

The third challenge is that \(\tau _{\Delta Y}\) and its rescaled version \(\tau _{ATT^W}\) include direct measures of utility, which econometricians do not observe. Empirical identification of \(\tau _{ATT^W}\) requires identifying the conditional expectation of the expression in parentheses in it, which we label \(W_i\):

$$\begin{aligned} W_i \equiv \mathbb {E} [\mu ^S(X_i) + U^S_i - k^R \mu ^L(X_i) | X_i = x, D_i = 1] \end{aligned}$$
(18)

Despite its inclusion of direct utilities, \(W_i\) can be identified as follows. In the selection equation (Eq. (6)), we assume that \(\mu ^R(X)\) can be modeled as a linear function of X, \(\theta X\), and that \(U^R\) follows the standard logistic distribution. These assumptions enable us to identify \(\theta\) by estimating the selection equation as a logistic regression model. With \(\theta\) identified, we can identify \(W_i\) by applying the expression \(W_i = \mathbb {E} [\theta X_i + U^S_i | \theta X_i + U^S_i > 0]\), which we show in Appendix 1 is equivalent to

$$\begin{aligned} W_i = \frac{e^{\theta X_i}}{1 + e^{\theta X_i}} \ln (1 + e^{\theta X_i}) \end{aligned}$$
(19)

\(W_i\) is monotonically increasing in \(\theta X_i\), which implies that, compared to \(\tau _{ATT^Q}\), \(\tau _{ATT^W}\) gives more weight to treated sites associated with higher expected utility gains. It follows that \(W_i\) is also monotonically increasing in the propensity score from the logistic model,

$$\begin{aligned} p(X_i) = \frac{\exp {(\theta X_i)}}{1 + \exp {(\theta X_i)}} \end{aligned}$$
(20)

as substituting this expression into Eq. (19) yields

$$\begin{aligned} W_i = p(X_i) \ln \Big ( \frac{1}{1 - p(X_i)} \Big ) \end{aligned}$$
(21)

The final challenge is the familiar one with impact evaluations: the econometrician observes treated sites only in their treated state, not their counterfactual untreated state, and faces an associated selection problem because both treatment status and the treatment effects are functions of X. We address this challenge by using the propensity score to select a matched set of untreated observations as controls for the treated observations. Propensity score matching eliminates selection bias if: (i) selection is only on observables, \((Q_1, Q_0) \perp D|X\) (the Conditional Independence Assumption, CIA; Rosenbaum and Rubin 1983); and (ii) the characteristics of treated and untreated sites overlap sufficiently so that the treated sites and their matched controls are statistically indistinguishable, \(\mathbb {P}(D=1|X=x) \in (0,1)\) for all x in the support of X (the common support assumption; Ho et al. 2007). The CIA is satisfied if \(U^S \perp U^L\), which requires including all variables that influence selection and outcomes in the estimated selection equation, and full support can be checked empirically by examining the balance of the treatment and control groups.

Empirical identification of \(\tau _{ATT^W}\) thus involves first estimating the selection equation (Eq. (6)), then using Eq. (21) to estimate \(W_i\), and finally using propensity score matching to estimate

$$\begin{aligned} \hat{\tau }_{ATT^W} = \sum _i \frac{D_i \hat{W}_i}{\sum _i D_i \hat{W}_i} (Q_i - Q_j) \end{aligned}$$
(22)

where \(Q_j\) is the matched counterpart of \(Q_i\). Compared to the estimated version of the ATT for avoided deforestation (Eq. (16)), which is

$$\begin{aligned} \hat{\tau }_{ATT^Q} = \sum _i \frac{D_i}{\sum _i D_i} (Q_i - Q_j) \end{aligned}$$
(23)

the only difference is the inclusion of the estimated utility weights.

We close with three associated empirical points. First, Appendix 2 shows how the expressions for \(\hat{\tau }_{ATT^Q}\) and \(\hat{\tau }_{ATT^W}\) given by Eqs. (22) and (23) can be extended to cases in which Q is a continuous, not binary, variable. Second, Appendix 3 shows how to estimate the standard error of \(\hat{\tau }_{ATT^W}\). Third, Eqs (22) and (23) imply that the magnitude of \(\hat{\tau }_{ATT^W}\) relative to \(\hat{\tau }_{ATT^Q}\) depends on the correlation between the estimated individual treatment effects (\(Q_i - Q_j\)) and the estimated propensity scores (\(\hat{p}(X_i)\)). \(\hat{\tau }_{ATT^W}\) is larger (smaller) than \(\hat{\tau }_{ATT^Q}\) if the correlation is positive (negative), and the two measures are identical if the correlation is zero. This relationship enables one to predict the potential range of \(\hat{\tau }_{ATT^W}\) for a particular empirical case, by using the estimated individual treatment effects and varying the correlation from −1 to +1. One can therefore not only estimate \(\hat{\tau }_{ATT^W}\) but also determine if it is closer to the minimal or maximal values it potentially could have had.

Empirical Application: Mangrove Protection in Thailand

Policy Context

In the 1970s and 1980s, Thailand experienced a boom in shrimp farming, which entailed the complete or near-complete conversion of affected mangrove areas into shrimp ponds. Mean annual mangrove deforestation rose sharply, from 45 km2/yr during 1961–1979 to 130 km2/yr during 1979–1986 (Charuppat and Charuppat 1997). By 1986, nearly half (47%) of the country’s original mangrove area had been converted to other land uses, with shrimp farms accounting for a third of the cumulative historical loss (Charuppat and Charuppat 1997). In response to this rapid and large loss, the country’s highest governmental authority, the Prime Minister’s Cabinet, issued a resolution on December 15, 1987 that established a new national mangrove management system. This system assigned the country’s mangroves to three management zones: (1) Protection Zone, which prohibited deforestation, defined as the conversion of mangroves to other land uses, and all resource extraction; (2) Economic Zone A, which prohibited deforestation but allowed sustainable harvesting of wood products by local communities and commercial companies; and (3) Economic Zone B, which allowed both deforestation and extractive uses. The government implemented this system for 13 years. A Cabinet resolution on August 22, 2000 marked its end, by prohibiting the renewal of harvesting contracts in Economic Zone A and any additional deforestation or resource extraction in Economic Zone B. In 2002, the government shifted responsibility for mangrove management from the Royal Forestry Department (RFD) to a new Department of Marine and Coastal Resources (DMCR).

We evaluate Thailand’s zoning system using both the conventional ATT for avoided deforestation and the utility-weighted ATT, with propensity score matching as the impact evaluation method and satellite data on deforestation during 1987–2000. Five features of the zoning system make it attractive for illustrating the application of the utility-weighted ATT. First, consistent with the description of the “regulator” in Sect. 2.2, the Cabinet can be viewed as a single agent that made a discrete protection intervention. This situation differs from more complex ones with protected areas established by multiple bodies (e.g., national, subnational) at multiple points in time. Second, the system had well-defined start and end dates. The delineation of the pre-treatment and treatment periods and the appropriate endpoint for measuring the system’s outcomes are both clear, which is necessary for an impact evaluation.

Third, the system included a zone—the Protection Zone—that prohibited deforestation and thus represents a suitable “treatment” for an impact evaluation with avoided deforestation as the physical outcome measure. Although Economic Zone A prohibited deforestation too, it is not suitable for evaluation because it allowed sustainable wood harvesting. Distinguishing harvesting from deforestation is difficult in satellite imagery before a forest regenerates from harvesting. Including Economic Zone A in the treatment group would risk misclassification of tree-cover loss and biased estimates of treatment effects.

Fourth, using the Protection Zone as the treatment requires information on the spatial locations of mangroves assigned to it, and such information exists because the 1987 Cabinet resolution approved and endorsed a specific map prepared by the RFD. This map delineated the boundaries of all three zones in the entire national area estimated as originally having been mangrove. Cabinet resolutions from 1982 and 1984 had called for the development of a zoning system (Mangroves for the Future 2011). The RFD’s mapping effort stemmed from these resolutions and was completed before the 1987 resolution. The zone assignments remained unchanged during 1987–2000, which implies that using a single selection model—i.e., the first-stage logistic regression model in propensity score matching—to analyze the selection process is appropriate.

Finally, a suitable set of control locations exists because the RFD’s 1987 map omitted some mangroves. We interviewed two senior officers in the DMCR who were members of the RFD team that prepared the map,Footnote 3 and they reported that these omissions were inadvertent. A likely reason is the relatively low resolution of the satellite data that the RFD used for mapping mangroves in the 1980s (1:250,000; Charuppat and Charuppat 1997). Lying entirely outside the zoning system, these omitted mangroves represent a purer control group than do mangroves in Economic Zone B, where deforestation was allowed but required the RFD’s permission.

To our knowledge, the zoning system has never been evaluated using a causal-inference method. In fact, some literature mischaracterizes it as consisting of two zones, not three (Aksornkoae 2004). Case studies of small numbers of villages dominate the literature on mangrove management in Thailand (e.g., two villages, Sudtongkong and Webb 2008; four villages, Barbier 2008 and Kongkeaw et al. 2019). They lack the representativeness necessary for evaluating the national performance of the zoning system, but they offer reasons to doubt the system’s effectiveness at reducing deforestation. They highlight a gap between its progressive aspects, which on paper signaled a policy shift away from conversion toward conservation and gave local communities a greater role in managing mangroves than did corresponding policies for terrestrial forests (Kongkeaw et al. 2019), and its implementation, which by 1996 included communities in the management of barely 5% of the total zoned area (Charuppat and Charuppat 1997). They document disputes between communities and the government over ownership of mangroves, which created de facto open access (Sudtongkong and Webb 2008; Barbier 2008), and a lack of response by the government when communities reported illegal conversion or harvesting (Sudtongkong and Webb 2008). Consistent with this last point, the DMCR officers we interviewed reported that monitoring and enforcement of the system were weak due to inadequate staffing and budget and that penalties, if assessed at all, were typically low.

Despite these deficiencies, the system might not have been completely ineffective: (Charuppat and Charuppat 1997) reported that mean annual mangrove deforestation fell by nearly 80% during the decade after the 1987 Cabinet resolution compared to the decade before. There is thus merit in determining the system’s effectiveness during its entire implementation period, in both biophysical and economically more relevant terms.

Data and Covariate Construction

We highlight key features of our data here, with full details on sources and processing reported in Appendix 4. We obtained a digital map showing mangrove presence/absence for Thailand in 2000 from Giri et al. (2005, 2011). We created a corresponding 1987 map using the same type of satellite data (Landsat), same techniques, and same resolution (30 meter) used to create the 2000 map. Using these maps, we drew a random sample of 50,000 points from locations where mangroves were present in at least one of the two years (see Fig. 1). We set the minimum distance between any two sampled points at 50 meters to prevent drawing multiple points from a single pixel. Given our focus on avoided deforestation, we dropped all points where mangroves were not present in 1987.Footnote 4

We classified the treatment status of the sampled points using a shapefile from the DMCR that showed the zone boundaries from RFD’s 1987 map. The map delineated the zones at a fine scale (Fig. 2). The Protection Zone included 524 distinct polygons (noncontinguous areas), which were on average very small (mean and median areas = 0.28 km2 and 0.026 km2, respectively). It accounted for only 4% of the total zoned area. Overlaying the 1987 mangrove cover map on the zone map revealed that the total area of the unzoned mangrove area omitted from the zone map was equivalent to 26% of the total zoned area. Overlays also revealed that the cumulative 1987–2000 deforestation rate was much lower in the Protection Zone (17%) than in the unzoned area (48%). Although the difference in these rates suggests that assignment to the Protection Zone might have cut deforestation by nearly two thirds, determining the actual impact requires controlling for selection bias through estimation of the ATT for avoided deforestation.

We retained points in the Protection Zone as the treatment group and points in the unzoned area as a pool of potential controls. The 1987 Cabinet resolution stated specific criteria for assigning mangroves to the Protection Zone. Table 2 lists these criteria, which reflect various environmental public goods associated with mangroves and thus imply a social welfare foundation for the criteria. We based our choice of covariates for the selection model on these criteria, the interview with the DMCR officers, and the matching literature on protected areas (PAs). The officers reported that the RFD relied most heavily on criteria 1.6 (existing PAs) and 1.10 (proximity to river banks and coastlines) in preparing the zone map. This reliance is understandable for two reasons. The first is access to spatial information: according to the DMCR officers, the RFD used official PA maps included in the Royal Gazette notices that established the PAs, and it used high-quality military topographical maps to measure distances. Second, these two criteria overlap with several others, with 1.6 subsuming 1.4, 1.5, 1.8, and 1.9 and 1.3 providing the coastal-protecton rationale for 1.10. We dropped points that were in PAs established before the Cabinet resolution (criterion 1.6), as they did not represent incremental treatment by the zoning system. These points accounted for 49% of the treatment group. We incorporated criterion 1.10 by constructing two distance variables, one for distance to the nearest river and the other for distance to the coast.Footnote 5

Regarding the remaining criteria, we incorporated 1.7 by constructing two dummy variables, which distinguished points with higher and lower average wind speeds, respectively, from points with intermediate average wind speeds, and a dummy for points located outside the monsoon climate zone.Footnote 6 We incorporated 1.1 and 1.2 by constructing an index of the number of mangrove-resident animal species whose ranges encompassed each point.

The 1987 Cabinet resolution did not state comparable criteria for Economic Zones A and B, but its authorization of these two zones implies that assignment of mangroves to the Protection Zone was also influenced by the opportunity cost of protection, with protection being less likely in locations where resource extraction and conversion to non-mangrove land uses were more valuable to local communities or industries. Consideration of the opportunity cost of protection is consistent with a net social welfare concept that accounts for more than just the public good benefits of protection reflected in the stated criteria in Table 2. It is also consistent with the PA matching literature, which provides ample evidence that various socioeconomic variables can affect either protection or avoided deforestation and thus are important to include in a selection model. We included a dummy variable to identify points on the Gulf of Thailand coast, where the shrimp farm industry was more heavily concentrated (Charuppat and Charuppat 1997). Although Thailand’s original mangrove area was nearly evenly split between the two coasts (Charuppat and Charuppat 1997), our 1987 mangrove map revealed that the Gulf coast’s share had fallen to barely a third (36%) by that year. This concentration implies a higher opportunity cost of protection on the Gulf coast and thus a lower likelihood of protection there. We also included distance variables of the types commonly used to measure proximity to features that affect the opportunity cost of protection (Pfaff et al. 2009, 2015): distances to the province capital and district (amphoe) seat, distance to the nearest road, and a pair of dummy variables that distinguished points nearer to and farther from the forest edge from points at an intermediate distance from the edge. Following Miteva et al. (2015), we also included population density for the subdistrict (tambon) where a point was located and the fraction of subdistrict area that was in mangrove in 1987,Footnote 7 both of which can likewise be expected to affect the opportunity cost of protection. Most of these socioeconomic variables can be given a political interpretation in addition to a purely economic one (i.e., opportunity cost). For example, the Gulf dummy could proxy for the political influence of the shrimp farm industry. For this reason, statistical significance of these variables in the estimated selection model could reflect political influence over zoning decisions, which could cause the decisions to deviate from welfare-maximizing ones.

We did not include two variables commonly used in the PA evaluation literature, slope and elevation, because mangroves occur primarily in intertidal zones where variation in these factors is negligible.Footnote 8 The total number of covariates in our selection model, 14, is typical for the literature (Miteva et al. 2015, 13 covariates; Ahmadia et al. 2015), 15 covariates).

Matching requires the common support assumption: covariates must have sufficient overlap between the treatment group and the untreated group. For each covariate, we identified untreated points with values more than 5% beyond the range of treated points and dropped them from the sample. Our final data set included 7,790 points, 671 of them treated. The pool of potential controls was thus large relative to the number of treatment points, which is advantageous for matching. Table 3 lists the covariates and their means and standard deviations for the treated and untreated points. Substantial differences can be observed for many of the variables, which highlights the need for matching to construct a balanced sample before estimating the treatment effects.

Evaluation Based on ATT for Avoided Deforestation

We used matching with replacement, so that each untreated point could be used as the best match (“nearest neighbor”) for multiple treated points. We set the caliper—the maximum permissible difference between the propensity score for a treated point and its matched control—at 0.01. Application of the caliper reduced the number of treated points by a small amount, from 671 to 653 if we matched each treated point with a single control point and 646 if we matched it with two.

We evaluated covariate balance using the two most common diagnostics in the matching literature: the standardized mean difference between the treatment and control groups, and the ratio of variances for the two groups (Austin 2009). Formal statistical tests of these diagnostics are not valid in propensity score matching (Imai et al. 2008), so we instead applied guidelines recommended in the literature. We required the absolute value of the standardized mean difference to be below 0.1 (Normand et al. 2001; Austin 2009) and the variance ratio to be within the 2.5th and 97.5th percentiles of an F-distribution with degrees of freedom implied by the number of matched treatment-control pairs (Austin 2009). These percentiles were 0.86 and 1.17 in our sample. With the number of matched controls per treated point set initially at one, one covariate, Wind speed: high, was unbalanced according to the standardized mean difference (0.148) and barely balanced according to the variance ratio (1.165) (see Appendix 5). In response, we increased the number of matched controls to two, which eliminated the imbalance (Table 4). We used two matches in all subsequent analyses.

The literature on PA evaluation emphasizes the need to address spillover effects, which can bias estimates of treatment effects (Robalino et al. 2017; Herrera et al. 2019). There are two types of spillovers, leakage and blockage (Fuller et al. 2019). In the case of leakage, PA establishment displaces deforestation that would have occurred inside a PA to an unprotected location where deforestation would not have occurred in the absence of protection. Conversely, in the case of blockage, PA establishment increases the cost of deforestation in unprotected locations and reduces the amount of deforestation that would have otherwise occurred in those locations. Failure to control for leakage and blockage can result in overestimation and underestimation, respectively, of PAs’ effectiveness in reducing deforestation.

When investigating local spillovers, the definition of “local” depends on the geographical scale of the PA in question. Because mangroves naturally occur in narrow coastal strips, Thailand’s zoning system delineated the polygons for the Protection Zone at a much finer scale (see Fig. 2) than, say, the scale of PAs commonly analyzed in countries such as Brazil (Herrera et al. 2019) and Costa Rica (Robalino et al. 2017). As noted earlier, the mean and median areas of Protection Zone polygons were substantially smaller than 1 \(\mathrm {km}^2\). We therefore expect spillover effects to occur primarily within a few kilometers of the corresponding polygons. To investigate the magnitude of spillovers, we first estimated \(\tau _{ATT^Q}\), the treatment effect for avoided deforestation, using all 646 matched pairs as a baseline. Then, we progressively excluded untreated points that were within a 1 km, 2 km, 3 km, 4 km, or 5 km buffer of a Protection Zone polygon and, at each step, reestimated the selection model, formed a new set of controls in view of the reduced pool of potential controls, and reestimated \(\tau _{ATT^Q}\). Table 5 displays the results. Imposing the buffers only slightly affected the estimates of \(\tau _{ATT^Q}\), and the confidence intervals overlap for all the estimates. We therefore conclude that spillover effects were negligible and that the no-buffer case yields the preferred estimate of \(\tau _{ATT^Q}\).

In addition to the lack of evidence of spillovers, we prefer the no-buffer case because it includes the largest number of treated points, which falls to only 437 for the 5 km buffer. It also has the best balance: the number of covariates that violate the balance conditions rises from zero for this case to 3, 2, 2, 6, and 7, respectively, for the 1 km, 2 km, 3 km, 4 km, and 5 km buffers. This deterioration in match quality is due to the shrinking pool of potential controls.

From Table 5, the estimated \(\tau _{ATT^Q}\) for the no-buffer case has a highly significant (p < 0.01) value of 0.173. This estimate indicates that 17.3% of the Protection Zone prevented deforestation that would have occurred in the absence of protection. It is barely half of the 31-percentage point difference between the observed 1987–2000 deforestation rates for the Protection Zone and the unzoned area reported earlier, which underscores the need to control for selection bias when evaluating forest protection programs: due to selection bias, the observed difference in deforestation rates exaggerates the Protection Zone’s impact on avoided deforestation. Nevertheless, the positive and significant value of the estimated \(\tau _{ATT^Q}\) confirms that creation of the Protection Zone reduced mangrove deforestation during 1987-2000.Footnote 9

Evaluation Based on Utility-Weighted ATT

To implement our new evaluation approach based on \(\tau _{ATT^W}\), we estimated individual propensity scores for all treated points using the same logistic regression that served as the first stage of our matching estimator. We used the no-buffer sample given its superior balance and the lack of evidence of spillover effects. Table 6 presents the regression results. As discussed in Sect. 3.2, the first six covariates pertain to assignment criteria for the Protection Zone stated in the 1987 Cabinet resolution. Most of them are not significant at even p < 0.1. Wind speed: low is significant at that level and has the expected negative sign. Animal species is highly significant (p < 0.01) but is unexpectedly negatively signed: mangroves were less likely to be assigned to the Protection Zone if they had more species. In sum, we found little evidence that the stated criteria influenced treatment.

We found more evidence that socioeconomic covariates mattered. As expected, the Gulf dummy had a negative effect (p < 0.1). The probability of being treated was higher for interior mangroves and lower for mangroves near the forest edge (p < 0.01 for both) and lower for locations with higher population density (p < 0.05), which mirrors results in previous matching studies on PAs (e.g., Joppa and Pfaff 2009; Miteva et al. 2015). We cannot determine whether the significance of these variables reflects rational consideration of the opportunity costs of protection, which would be consistent with the general design of the zoning system, or instead reflects political influences that caused treatment to deviate from social welfare maximization. The lack of evidence that the stated criteria for the Protection Zone mattered inclines us to believe that treatment decisions were not purely apolitical, however.

Using these results, we followed the approach explained in Sect. 2.3 to construct the utility weights for all treated observations in the sample. As explained in Sect. 2.4, the magnitudes of \(\tau _{ATT^W}\) and \(\tau _{ATT^Q}\) differ when predicted individual treatment effects and propensity scores have a non-zero correlation. When predicted individual treatment effects are positively (negatively) correlated with propensity scores, \(\tau _{ATT^W}\) is larger (smaller) than \(\tau _{ATT^Q}\). Because \(\tau _{ATT^Q}\) does not depend on this correlation, we can readily determine the potential range of \(\tau _{ATT^W}\) for the Protection Zone. In effect, we can simulate a more complex, and more realistic, case than the simple example given in Table 1. Holding \(\tau _{ATT^Q}\) constant at its estimated value and using the utility weights we constructed, we found that \(\tau _{ATT^W}\) attains a maximum value of 0.749 if the two series are perfectly positively correlated and a minimum value of -0.514 if they are perfectly negatively correlated. If we rule out protection causing deforestation of mangroves that would not be deforested in the absence of protection (see Sect. 2.3), then the minimum value becomes zero. \(\tau _{ATT^W}\) therefore has a potential range from zero to nearly four times \(\tau _{ATT^Q}\) (0.749 vs. 0.173). This wide range, straddling \(\tau _{ATT^Q}\), demonstrates that the ATT for avoided deforestation can be a highly misleading proxy for the economic impact of a forest conservation program.

Our estimate of the utility-weighted ATT turned out to be near the lower end of the potential range, 0.128. This estimate is approximately a quarter (26%) less than the estimated ATT for avoided deforestation and is highly significant (p < 0.01). Figure 3, which plots quantiles of the predicted individual treatment effects for points in the treatment group against the corresponding predicted propensity scores, illustrates that the estimate of \(\tau _{ATT^W}\) is smaller than the estimate of \(\tau _{ATT^Q}\) for the reason just given: i.e., the two series are negatively correlated.Footnote 10 The two measures have different interpretations: while the estimate of \(\tau _{ATT^Q}\) indicates that approximately 17% of the Protection Zone prevented deforestation that would have occurred in the absence of protection, the estimate of \(\tau _{ATT^W}\) indicates that the Protection Zone protected approximately 13% of the economic value, as perceived by the government, that would have been lost if all mangroves in the Protection Zone had been deforested. They are not alternative measures of the same quantity.

The difference in magnitude between the two measures suggests that interpreting \(\tau _{ATT^Q}\) as a proxy for the Protection Zone’s economic impact would overestimate the impact by roughly a third (35%). The difference is not significant, however (p = 0.16). The lack of significance is due primarily to the relatively large standard errors of \(\tau _{ATT^Q}\) and \(\tau _{ATT^W}\), 0.024 and 0.041, respectively, which in turn has two causes. One cause that is common to both estimates is the relatively small number of treated points in the sample. Tripling the number, from 646 to 1,938, would cause the test statistic to become significant at p < 0.04. The other cause, which pertains to \(\tau _{ATT^W}\), is that unequal weights can yield less precise estimates than equal weights (see Appendix 3). The weights included in the estimate of \(\tau _{ATT^W}\) are unequal, having a coefficient of variation of 1.46.

Conclusion

Propensity score matching is one of the most widely applied methods for identifying and estimating treatment effects models, especially in the forest conservation literature. In its standard application, the first stage of a propensity score matching study serves only as a statistical step for predicting an index, the propensity score, that is subsequently used to match treated observations to controls. We have argued that when the outcome variable is a binary variable that characterizes behavioral responses by agents to a government program, an economic interpretation is embedded in the first-stage analysis as it reflects the regulator’s preferences in assigning treatment. We exploit this economic interpretation to develop a method, based on utility weights derived from the predicted propensity scores for treated observations, that indirectly evaluates the economic impact of a program whose observed outcomes are measured in physical, not economic, terms. In so doing, we connect the impact evaluation literature to the non-market valuation literature.

If the regulator’s preferences align with society’s, then the utility-weighted ATT generated by our method has a social welfare interpretation. Of course, such alignment does not necessarily occur in practice. For the case of mangrove protection in Thailand during 1987–2000 that we use to demonstrate the method’s empirical application, we show how one can gain insight into the degree of alignment by first reviewing the stated criteria for treatment and then comparing those criteria to estimation results for the selection model. In the Thailand case, the stated criteria for protecting mangroves plausibly aligned with social welfare, as they accounted for both a range of environmental public goods supplied by mangroves and the opportunity costs of protection. On the other hand, selection into treatment was generally not significantly influenced by covariates representing environmental public goods; instead, it was mainly influenced by covariates that represented opportunity costs and possibly political influences. We would thus argue that the utility-weighted ATT in this case measures the economic gain from treatment valued according to government preferences that did not align perfectly with social preferences.

The utility-weighted ATT expands the toolkit that applied economists have for evaluating the impacts of government programs, forest conservation programs in particular. Although we have focused on the evaluation of programs whose observed physical outcomes are binary, we have shown how it can be extended to continuous outcomes (Appendix 2). Nonetheless, it has limitations. First, in the absence of information on enforcement actions against noncompliance with a program, it is best interpreted as an upper bound on economic impacts. This bias is probably small in the context of mangrove protection in Thailand during 1987-2000, as enforcement was reportedly weak and penalties low. Second, derivation of the utility weights assumes that treatment is assigned at a single point in time by a single government regulator. This assumption fits well the empirical context of the intervention we evaluate—the 1987 Cabinet resolution in Thailand—but we acknowledge that interventions in other contexts can be implemented at different times and by different regulators. Future research could work on relaxing this limitation by extending the method to multiple treatments, as has been done for example with difference-in-differences models. We note that mangrove management policy in Thailand continued to evolve after 2000 (Beresnev et al. 2016) and that our findings for protection during 1987-2000 should not be assumed to apply to the subsequent period. Future research could also work on relaxing a third limitation: the method models a situation where the utility impacts of treatment are spatially independent. The benefits of forest protection are instead often spatially interdependent, with protection of site i supplying greater benefits if site j is also protected. Examples include watershed conservation and wildlife habitat. Non-environmental treatments might similarly have spatially interdependent impacts.

The Thailand application reveals a fourth limitation: distinguishing estimates of the utility-weighted ATT and the ATT for avoided deforestation from each other is less feasible in smaller samples. Both estimates were highly significant individually, thus indicating that protection reduced deforestation and had a positive economic impact (as valued by the government). Moreover, the utility-weighted ATT was substantially smaller than the ATT for avoided deforestation. Yet, the two estimates were not significantly different. To estimate the difference between them more precisely, we would have needed a larger number of treated observations to overcome the large standard errors that can result from the variation in the utility weights.

We close by noting a difference in how a government might respond to a low estimate of the utility-weighted ATT compared to a low estimate of the physical ATT, when the government’s objective is to maximize social welfare. The response to a low estimate of the physical ATT could include shifting program targets to alternative sites (or recipients), in order to increase a program’s impact as measured in physical terms. The theoretical framework for the utility-weighted ATT does not imply a similar option to retarget a program, because it assumes that the government made the socially optimal choice of program targeting when it selected sites (or recipients) for treatment. If a program’s impact as measured by the utility-weighted ATT is low, then increasing its impact is purely a matter of strengthening enforcement, not retargeting. Viewed from another angle, this difference implies that retargeting a program in response to a low estimate of the physical ATT might not be welfare-improving, particularly when the estimates of the physical ATT and the utility-weighted ATT are very different.

Table 1 \(\tau _{ATT^Q}\) and \(\tau _{ATT^W}\) in a simple numerical example
Table 2 Protection zone criteria–1987 cabinet resolution
Table 3 Definitions and summary statistics of the covariates
Table 4 Covariate balance with and without matching (match number \(=\) 2)
Table 5 Propensity score matching estimates with controls for spillovers
Table 6 Results of the logistic regression on treatment decision
Fig. 1
figure 1

Map of mangrove sampling frame. Note: Sampling frame (i.e., areas with mangrove cover in either 1987 or 2000) is shown in black. Gray boundaries demarcate Thai provinces

Fig. 2
figure 2

Local illustration of the mangrove zoning system (portion of Krabi Province on Thailand’s Andaman Sea coast)

Fig. 3
figure 3

Distribution of the average treatment effect for avoided deforestation (\(\tau _{ATT^Q}\)) by quantiles of the predicted propensity score (20 quantiles)