Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

An important problem in causal inference and estimation of treatment effects is identifying a reliable comparison group (control observations) against which to compare those that have been exposed to the treatment (treated observations). It is common knowledge that the estimate obtained by the difference in the values of the indicator of interest associated with the two groups could be biased due to lack of overlap in the covariate distributions or common support between the treated and control observations (Dehejia and Wahba 2002; Imbens and Wooldridge 2009). This is especially problematic with non-experimental control observations (Dehejia and Wahba 2002) in which case combining propensity score matching and regression methods has been suggested to yield more consistent estimates of the treatment effect than using either method alone (Imbens and Wooldridge 2009). Matching removes self-selection bias due to any correlation between the observable (pre-treatment) covariates and the dependent variable, while regression isolates the effect of change in the covariates on change in the dependent variable over the period of the treatment. Using the combined approach, this paper discusses the effect of using different sets of control groups on estimates of treatment effects of the agricultural extension system in Uganda, the National Agricultural Advisory Services (NAADS) program.

The goal of the NAADS program, which was initiated in 2001, is to increase incomes through increased adoption of profitable agricultural enterprises and improved technologies and practices, agricultural productivity, and marketed output. The program aims at targeting the economically-active poor―those with limited physical and financial assets, skills and knowledge―through farmer groups based on specific enterprises identified by farmers (NAADS 2007). Although the program is a public intervention, farmers have to decide whether to participate in the program or not. When a farmer decides to participate, he or she has to do so through membership of a NAADS-participating farmer group. Then, together with the members of the group, and with members of other NAADS-participating groups, they request for specific technologies and advisory services associated with their preferred enterprises and obtain grants to procure those technologies and related advisory services. The grant is initially used to finance the establishment of a technology development site (TDS) for demonstrations and training, and proceeds (outputs or sale of outputs) from the TDS become a revolving fund for members of the group. The main channel of impact of the program is thus via farmers’ access to this grant. Knowledge and skills gained from the activities surrounding the TDS, as well as from select farmers trained to provide follow-up advisory services [community-based facilitators (CBFs)], are also very important.

The program is expected to generate indirect or spillover effects to the extent that the TDSs, NAADS service providers and CBFs are accessible as sources of knowledge and skills to other farmers in the community where the program is implemented. Estimating these indirect effects involves identifying farmers that have benefited from the program in such a manner, which is potentially challenging due to possible misclassification of service providers. For example, the government’s regular extension service and NGOs operated in the same areas as the NAADS program. Since some ex-government extension workers and NGOs are occasionally contracted to provide NAADS services, it is possible for them to be wrongly associated with the NAADS program even when they are operating outside the NAADS framework. Spillovers across program boundaries or communities through information flow among farmers and from non-NAADS service providers using the NAADS framework are also possible. We discuss the implications of these from using different controls groups. Next, we present the data and evaluation method, followed by the results, conclusions, and implications.

1 Data and Methods

1.1 Data

The data are from two rounds of household surveys conducted in 2004 and 2007. The 2004 survey served as the baseline on which a stratified sample was based according to the year when the NAADS program was first implemented in the community (sub-county) where the program: began in 2001/02; began in 2002/03; began between 2005 and 2007; or had never been implemented at the time of the 2007 survey. This was done to account for the effect of the rollout of the program that may result in a modified treatment among later entrants to the program due to learning from previous treatments among earlier entrants of the program (supply-side effects of the program), as well as from nonrandom preparedness of later entrants prior to receiving the treatment (demand-side effects). About 400, 300, 100, and 100 households were surveyed from each of the four strata (see Benin et al. 2011 for details). This paper is based on the panel of 719 household observations. The indicator of interest for estimating the treatment effect is agricultural income (INC)—details of this and other variables used are presented later.

1.2 Estimation Approach

What we are interested in is the average treatment effect of the treated (ATT j ):

$$ {ATT}_j=E\left[{INC}_{1j}|{NAADS}_j=1\right]-E\left[{INC}_{0j}|{NAADS}_j=1\right] $$
(1)

where INC 1j is agricultural income of farm household j due to participation in the program and INC 0j is agricultural income of the same farm household j if it did not participate in it. Although, we cannot observe the counterfactual, the underlying estimation problem can be represented as a treatment-effects model of the form:

$$ {INC}_{jt}={\alpha}_j+{\tau}_t+{\delta NAADS}_j+\boldsymbol{\beta} \boldsymbol{\hbox{'}}{\boldsymbol{x}}_{jt}+{\upepsilon}_{jt} $$
(2)
$$ {NAADS}_j^{\ast }=\boldsymbol{\gamma} \boldsymbol{\hbox{'}}{w}_j+{u}_j $$
(3)
$$ {NAADS}_j=\left\{\begin{array}{c}\hfill 1,\kern0.5em \mathrm{if}\ {NAADS}_j^{\ast }>0\hfill \\ {}0,\kern0.5em \mathrm{otherwise}\hfill \end{array}\right. $$
(4)

where: \( {NAADS}_j^{\ast } \) is a latent unobserved variable whose counterpart, NAADS j , is observed in dichotomous form; x j and w j are vectors of variables determining agricultural income and the decision to participate in the program, respectively; NAADS j  = 1 and NAADS j  = 0 represent participation (or treatment) and non-participation (or control), respectively; α and τ capture the individual and time specific effect, respectively; β and γ are the vectors of parameters measuring the relationships between the dependent and independent variables; ϵ and u are the random components of the equations with joint normal distribution of means (μ, 0) and covariance matrix \( \left[\begin{array}{cc}\hfill {\sigma}_{\varepsilon}^2\hfill & \hfill {\sigma}_{\upepsilon u}\hfill \\ {}\hfill {\sigma}_{\upepsilon u}\hfill & \hfill 1\hfill \end{array}\right] \).

We apply a two-stage weighted regression (2SWR) method (e.g. Robins and Rotnitzky 1995). In the first stage, we estimate Eq. (3) by probit to obtain propensity scores, which are used in selecting a matched sub-sample of treatment and control observations. In the second stage, the propensity scores are used as weights in a weighted least squares regression of Eq. (2) on the matched sub-sample according to:

$$ \Delta {INC}_j=\widehat{\alpha}+{\widehat{\delta}}_B{NAADS}_j+{INC}_{jt0}+{e}_j $$
(5)
$$ \Delta {INC}_j=\widehat{\alpha}+{\widehat{\delta}}_F{NAADS}_j+{INC}_{jt0}+\widehat{\boldsymbol{\beta}}{\boldsymbol{\hbox{'}}}_{2 SWRT}\Delta {\boldsymbol{x}}_j+{e}_j $$
(6)

where: ΔINC = INC t1 – INC t0, and INC t0 and INC t1 are the incomes in the initial (2004) and later (2007) periods, respectively; Δx = x t1 – x t0, and x t0 and x t1 are the initial and later period values of the covariates, respectively. Equations (5) and (6) represent specifications without and with the covariates, and the impact of the program is measured by \( {\widehat{\delta}}_B \) and \( {\widehat{\delta}}_F \) for the two model specifications, respectively. In any two-stage estimation procedure, it is important to address the identification of the second-stage regression or endogeneity of the first-stage regression. A common procedure used is excluding some of the explanatory variables used in estimating the first-stage probit from the second-stage regression (i.e. having x ⊂ w or x ≠ w and corr (w, ε/x) = 0). In general, nonlinearity of the first-stage probit model renders exclusion restrictions unnecessary (Wilde 2000). Further, since we apply a fixed-effect or difference estimator in the second-stage regression, the condition is satisfied in the sense that Δx ≠ w.

Participation is measured using the status observed in 2007 to avoid crossover in different years so that a treatment household is always a treatment household and cannot switch status; the same for a control household. Of the 719 observations, 66 are treated and 653 are controls, which we split into three. The first control sub-group is made up of those in the same area where the program is implemented and claimed to have benefited indirectly from the program, labeled NAADSNON-1. The second sub-group also is made up those in the area where the program is implemented but did not claim any benefits (labeled NAADSNON-2), while the third sub-group is made up those in areas where the program was never implemented (labeled NAADSNON-3). These three sub-groups make up 256, 284, and 113 observations, respectively. Because matching with the nearest neighbor only can limit any potential gain from matching participants with more than one non-participant with similar attributes, we consider and report estimated treatment effects based on matching with one, three, and five nearest neighbors.

1.2.1 Variables

Agricultural income (INC) is agricultural income per adult equivalent and measured as the total gross value of households’ crop, livestock, beekeeping and aquaculture output (or agricultural gross revenue) divided by the total number of adult equivalents in the household. The choice of covariates was guided by the principles and design of the NAADS program as well as the literature on agricultural household models (e.g. Singh et al. 1986) and adoption of agricultural technologies (e.g. Feder et al. 1985). The variables used include: human capital (gender, age, education and size structure of household); financial capital (livelihood and income strategies); physical capital [land owned and value of agricultural productive assets (e.g. equipment, livestock, etc.)]; social capital (membership in other organizations); access to infrastructure and services (distance to nearest financial services, road, market); location in the four administrative regions of Uganda (Central, Eastern, Northern and Western); and dummy variables representing the year when the NAADS program was introduced in the sub-county. Physical capital may be potentially endogenous and so we estimate the second-stage regression with and without them to analyze the effect of this problem. All monetary values were converted into year 2000 constant prices to help exclude the influence of inflation and other temporal monetary and fiscal trends.

To improve matching, it is common practice to try different variables and transformations of the variables such as logarithms and higher order and interaction terms, because matching is a nonparametric method of preprocessing data to reduce imbalance between treated and control groups (Imbens and Wooldridge 2009). We follow this practice and use: histograms of the propensity scores between the two groups to select the sub-sample with adequate common support; and balancing tests to check the extent to which any differences that existed between the two groups prior to matching have been reduced in the matched sample.

2 Results

2.1 Determinants of Participation in the Program: Overlap in Covariate Distributions

Selected results on common support and balancing tests for different combinations and transformations (squared and interaction terms) of the covariates using matching with three nearest neighbors are shown in Fig. 1 and Table 1, respectively. We find that different covariates and transformations yield different outcomes of common support and balance between the two groups after matching. The general pattern is a skewness of the propensity scores toward one for participants and zero for non-participants. The situation is most perverse when no transformations of the variables are included or when the covariates on the length of program presence are included (Fig. 1a–c). Regarding the latter, different propensity scores are generated for different controls who are identical in all aspects except location in a NAADS sub-county and several treated observations have to be dropped to improve common support. The models associated with the probits when we include transformations of the covariates and exclude the covariates on the length of program presence are preferred because their results show that there is greater common support and only up to six treated observations have to be dropped in any sub-sample (Fig. 1d–f). The sample characteristics in Table 1 also show that any statistically significant differences that existed between the treated and control groups prior to the matching were eliminated or reduced. Together, the results suggest that pooling observations for the different unique control groups as done in Benin et al. (2011) could lead to different policy implications and, as we shall see next, limit any potential gain in knowledge from matching each participant with multiple non-participants that are similar in several attributes but different in others.

Fig. 1
figure 1

Histogram of estimated propensity scores between participants and non-participants. Source: Authors’ illustration based on probit results. (a) to (c) excludes squared or interaction terms of the variables in the probits; (d) to (f) includes them

Full size image
Table 1 Sample characteristics between NAADS participants (treated observations) and non-participants (control observations)
Full size table

2.1.1 Estimated Treatment Effects of the Program on Agricultural Income (INC)

Estimates of the treatment effect are summarized in Table 2 (detailed selected second stage regression results are shown in the annex Table 3). The results show that the NAADS program has had positive impact on agricultural revenue per AE, particularly when participants are compared with those who did not claim any benefits (NAADSNON-2) or with those located where the program was never implemented (NAADSNON-3). The estimated impacts are statistically weak for the former and insignificant for the latter, however. The positive effect on agricultural revenue per AE is consistent with the estimated effects on other outcomes such as adoption of crop and livestock improved varieties, crop and livestock productivity, and sale of output; although the statistical significance of the estimates are reversed for NAADSNON-2 and NAADSNON-3 however, which is surprising (see annex Table 4). The estimated effects when direct participants are compared with NAADSNON-1 were consistently negative for the different outcomes analyzed, suggesting that the impacts of the program on direct participants were not as large as the change observed for indirect participants. Because farmers in this group are not very familiar with the NAADS program, they may have confused NAADS service providers with agents of other programs, leading to an overestimation of NAADS program effects for this group, as was likely the results in Benin et al. (2011).

Table 2 Estimated treatment effects (% difference between participants and non-participants in 2004–2007 change in agricultural revenue per adult equivalent)
Full size table
Table 3 2SWR results of Δ Ln agricultural revenue per adult equivalent
Full size table
Table 4 Estimated treatment effects in other selected outcomes
Full size table

The estimates from the model specification without the covariates are generally larger, suggesting that changes in other factors have been important, particularly changes in sources of income and access to infrastructure and services, particularly roads and markets (see annex Table 3).

The lower estimates associated with the model specification with the covariates including change in physical capital suggest that the impact of the program was also via its effect on these assets. The u-shaped or inverted u-shaped relationship between the estimates and increasing number of nearest neighbor matches is consistent with the literature that greater number of matches generally increases precision, but at the cost of increasing bias (Dehejia and Wahba 2002).

3 Conclusions and Implications

In this paper we used different sets of control groups and different propensity score matching specifications combined with regression to estimate the average treatment effect of the agricultural extension system in Uganda on households’ agricultural revenue. By breaking up the control observations into sub-groups reflecting likely differences in potential contamination with the treatment, we show how matching each treatment observation with multiple controls that are similar in several attributes but different in others can yield more insights on estimates of average treatment effects. Unfortunately, the results were mixed and weak, in terms of consistent sign and statistical significance across the different methods, model specifications, and outcomes analyzed, making it difficult to draw definitive conclusions regarding the direct impact of the program and, particularly, its indirect impact. Our underlying assumption was that participation in the NAADS program confers benefits via material inputs that will lead to subsequent outcomes. But this assumption was not consistently validated in the results obtained. While changes in other factors (sources of income and access to road and market) are important in raising agricultural revenue, a major limitation with the study is our inability to capture the separate effect of access to non-NAADS extension services.