Abstract
This chapter outlines statistical and econometric procedures that can be applied to the analysis of meta-data . Particular attention is paid to ensuring robustness of the insights from a meta-regression . Specific detail is paid to the fine econometric details to sharpen the insights of practitioners when deciding which tools to use for a meta-analysis for literature assessment of policy prescriptions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
An important question is what level of aggregation should be considered for identification of sample selection. The state designations used by Hoehn (2006) and Boyle et al. (2013) arise from the availability of data. This may or may not be the best approach to identifying sample selection. We do not address this important question in this chapter, but leave it for future research.
- 2.
Note that we do not use the between correction employed in Wooldridge’s actual procedure. This is due to the fact that he suggests estimating a probit for each time period, in doing so this would remove the inverse Mill’s ratio from the regression.
- 3.
With a small sample it may prove useful to deploy a bootstrap procedure. However, the bootstrapping mechanism is not exactly clear since there are two key issues, the selection that is present and the fact that most likely the meta-analyst will have an unbalanced panel in the value-estimate dimension. One approach could be to re-sample based on individual values as opposed to studies. However, this treats values from common studies as independent.
- 4.
Individual effect sizes can be grouped according to a number of criteria. Here we consider one obvious group, by study, because observations for a particular study could be unique due to the application, design and conduct of the study, data analysis, and effect size reporting. We do not imply that other data groupings should not be investigated if such analyses seem appropriate for any specific meta-analysis.
- 5.
An alternative measure of overall influence of an observation is the change in the OLS residuals from leaving a single observation out. This index can be calculated as \(DF_{i} = \hat{\omega }_{i} - \hat{\omega }_{(i)} = \frac{{h_{ii} \hat{\omega }_{i} }}{{1 - h_{ii} }} = D_{i} \frac{{s^{2} (1 - h_{ii} )k}}{{\hat{\omega }_{i} }},\) where \(\hat{\omega }_{(i)}\) is the ith residual, constructed leaving the ith observation out of the analysis. We elect to use D i since in general DF i and D i yield similar results but D i has a more intuitive interpretation of Eq. (17.11).
- 6.
See Davidson and MacKinnon (2004, Sect. 2.6) for more details on influence and leverage.
- 7.
This approach is common in the time-series literature where \(n_{k}\) consecutive observations would be left out (Poirretti 2003).
- 8.
See Appendix A for a derivation.
- 9.
See Appendix B for full derivation.
- 10.
A heteroscedasticity robust version of this statistic could also be constructed consistent with the concerns of Nelson and Kennedy (2009).
- 11.
Another issue to consider is the measurement of the regressors. Measurement is affected by documentation in original studies and choices made by the analyst. We do not address issues of variable measurement here, but this is another area of fruitful future research.
- 12.
A model-averaging approach is also feasible. Chapter 22 provides more discussion of these methods.
- 13.
We have used P and A to signify “potentially included” and “always included,” respectively.
- 14.
Other authors have used entirely different methods, such as model averaging.
- 15.
- 16.
An alternative, but important issue is how the meta-analyst chooses to specify the categories across regressors within the meta-data. The meta-analyst could elect to have fewer categories to minimize the number of cells. However, this draws into question the commonality of categories and their impact on effect sizes. This is another investigator assumption that we do not address in this chapter, but it warrants empirical investigation to determine the consequence on estimated meta-equations.
- 17.
While it is likely that the majority of the meta-data will be discrete in nature, it is certainly reasonable to have variables that are continuous in nature, such as the sample size used in an empirical analysis, or the standard deviation of the estimate of interest. In this case we can smooth over both the discrete and continuous variables in the meta data with only slightly more complex notation. Our kernel weighting takes the same approach, introducing a weight function geared towards continuous data to pair with our product kernel for discrete data. For a single continuous covariate, a kernel function takes the form \(h^{ - 1} k\left( {\frac{{x_{i} - x}}{h}} \right)\), where h is our smoothing parameter and \(k( \cdot )\) is commonly chosen to be a probability density function, such as the normal probability density, \(k\left( {\frac{{x_{i} - x}}{h}} \right) = (\sqrt {2\pi } )^{ - 1} e^{{\left( { - (x_{i} - x)^{2} /(2h^{2} )} \right)}}\). To be as general as possible we now allow for q regressors with \(q^{c}\) continuous regressors, \(q^{u}\) unordered discrete variables and \(q^{o}\) ordered discrete variables (\(q = q^{c} + q^{o} + q^{u}\)) Our generalized product kernel in the mixed, continuous-discrete data setting is
$$\begin{aligned} W_{h\lambda ,ix} & = K_{h} \left( {x_{i}^{c} ,x^{c} } \right)L_{{\lambda^{u} }}^{u} \left( {x_{i}^{u} ,x^{u} } \right)L_{{\lambda^{o} }}^{o} \left( {x_{i}^{o} ,x^{o} } \right) \\ & = \prod\limits_{s = 1}^{{q_{c} }} k\left( {\frac{{x_{is}^{c} - x_{s}^{c} }}{{h_{s} }}} \right)\prod\limits_{s = 1}^{{q_{u} }} \left( {\lambda_{s}^{u} } \right)^{{1\left( {x_{is}^{u} \ne x_{s}^{u} } \right)}} \prod\limits_{s = 1}^{{q_{u} }} \left( {\lambda_{s}^{o} } \right)^{{\left| {x_{is}^{o} - x_{s}^{o} } \right|}} . \\ \end{aligned}$$The smoothed meta-function is estimated as
$$\hat{m}(x) = \frac{{\sum\nolimits_{i = 1}^{n} y_{i} W_{h\lambda ,ix} }}{{\sum\nolimits_{i = 1}^{n} W_{h\lambda ,ix} }}.$$ - 18.
This speed of convergence is important given the dramatically different behavior of the estimator in the presence of irrelevant discrete variables.
- 19.
The whisker plots mimic the 45° plots earlier, except confidence intervals are added in the form of lines extending vertically from each estimate such that the length of the lines is the length of the confidence interval. Any confidence level can be used and when 0 is contained in the interval the point estimate is statistically insignificant for that response effect.
- 20.
Or, with mixed continuous-discrete data from \(x = (x^{c} ,x^{u} ,x^{o} )\) into \(x^{r} = (x^{{c_{r} }} ,x^{{u_{r} }} ,x^{{o_{r} }} )\) and \(x^{ir} = (x^{{c_{ir} }} ,x^{{u_{ir} }} ,x^{{o_{ir} }} )\).
- 21.
In this general setup q need not be equal to p. For example, including an unordered discrete variable with three different values \(\left\{ {0,1,2} \right\}\) would require two dummy variables in a parametric framework but only a single categorical regressor in our nonparametric framework.
- 22.
See Hsiao et al. (2007) for a description of the test statistic when continuous data are present as well.
- 23.
This bootstrap procedure ensures that the first three moments of the bootstrapped residuals are identical to the first three moments of the actual residuals.
- 24.
Weighting the dependent and independent variables is commonly used to account for heterogeneity. In environmental economics, weights are often based on the sample size of primary studies, since the number of estimates to report is an arbitrary decision of each primary investigator. Thus, the observations on y and X used to estimate Eq. (17.1) are down-weighted by the number of observations on effect size provided by each study. This procedure helps to mitigate the influence of studies that potentially influence the results by providing more observations than others studies (Nelson and Kennedy 2009). Bergstrom and Taylor (2006) discuss weighting by the variance of the effect size estimate or weighting by the effect size significance probabilities. These approaches contain assumptions. First, sample size weighting implies that each study should have equal weight and it is the observations that are of importance. This may not be the case in studies with split samples that effectively carry out multiple independent experiments where any single experiment could have been published independently. Second, weighting by variance implies that precision is the key feature. However, focusing on variance overlooks accuracy; it is possible to have a very precise measure that is not accurate. Third, weighting by p-values implies that significant effect sizes carry the most important information. This overlooks the consideration that insignificant effect sizes may be real and should be given equal consideration in the analysis. Thus, while there are logical reasons for using weights in meta-analyses, these considerations are not without concerns. Thus, it is important to investigate estimation with and without weights, and perhaps using a variety of plausible weighting schemes to evaluate the robustness of parameter estimates.
References
Bateman, I. J., & Jones, A. P. (2003). Contrasting conventional with multi-level modeling approaches to meta-analysis: Expectation consistency in U.K. woodland recreation values. Land Economics, 79(2), 235–258.
Belsley, D., Kuh, E., & Welsch, R. E. (1980). Regression diagnostics. New York: Wiley.
Bergstrom, J. C., & Taylor, L. O. (2006). Using meta-analysis for benefit transfers: Theory and practice. Ecological Economics, 60(2), 351–360.
Boyle, K. J., Parmeter, C. F., Boehlert, B. B., & Paterson, R. W. (2013). Due diligence in meta-analyses to support benefit transfers. Environmental and Resource Economics, 55(3), 357–386.
Brander, L., van Beukering, P., & Cesar, H. (2007). The recreational value of coral reefs: A meta-analysis. Ecological Economics, 63(1), 209–218.
Brouwer, R., Langford, H., Bateman, I. J., & Turner, R. K. (1999). A meta-analysis of wetland contingent valuation studies. Regional Environmental Change, 1(1), 47–57.
Cook, R. D., & Weisberg, S. (1982). Residuals and influence in regression. New York: Chapman and Hall.
Davidson, R., & MacKinnon, J. G. (2004). Econometric Theory and Methods. New York: Oxford University Press.
Delgado, M. S., Henderson, D. J., & Parmeter, C. F. (2013). Does education matter for economic growth? Oxford Bulletin of Economics and Statistics, 76(3), 334–359.
Der Simonian, R., & Kacker, R. (2007). Random-effects model for meta-analysis of clinical trials: an update. Contemporary Clinical Trials, 28, 105–114.
Der Simonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trails, 7, 177–188.
Florax, R. J. G. M., Travisi, C. M., & Nijkamp, P. (2005). A meta-analysis of the willingness to pay for reductions in pesticide risk exposure. Review of Agricultural Economics, 32(4), 441–467.
Ghermandi, A., & Nunes, P. A. L. D. (2013). A global map of coastal recreation values: Results from a spatially explicit meta-analysis. Ecological Economics, 86, 1–15.
Ghermandi, A., van den Bergh, J. C. J. M., Brander, L. M., de Groot, H. L. F., & Nunes, P. A. L. D. (2010). Values of natural and human-made wetlands: A meta-analysis. Water Resources Research, 46, 1–12.
Guggenberger, P. (2010). The impact of a Hausman pretest of the size of a hypothesis test: The case. Journal of Econometrics, 156, 337–343.
Hall, P., Li, Q., & Racine, J. (2007). Nonparametric estimation of regression functions in the presence of irrelevant regressors. The Review of Economics and Statistics, 89(4), 784–789.
Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161.
Hedges, L. V., & Vevea, J. L. (1998). Fixed-and random-effects models in meta-analysis. Psychological Methods, 3(4), 486–504.
Henderson, D. J., Kumbhakar, S. C., & Parmeter, C. F. (2012a). A simple method to visualize results in nonlinear regression models. Economics Letters, 117(3), 578–581.
Henderson, D. J., Papageorgiou, C., & Parmeter, C. F. (2012b). Growth empirics without parameters. The Economic Journal, 122(559), 125–154.
Higgins, J. P. T., & Thompson, S. G. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558.
Hoehn, J. P. (2006). Methods to address selection effects in the meta-regression transfer of ecosystem values. Ecological Economics, 60(2), 389–398.
Hsiao, C., Li, Q., & Racine, J. (2007). A consistent model specification test with mixed discrete and continuous data. Journal of Econometrics, 140(2), 802–826.
Hurvich, C. M., Simonoff, J. S., & Tsai, C. (1998). Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society, 60, 271–293.
Johnston, R. J., & Rosenberger, R. S. (2010). Methods, trends and controversies in contemporary benefit transfer. Journal of Economic Surveys, 24, 479–510.
Kaul, S., Boyle, K. J., Kuminoff, N. V., Parmeter, C. F., & Pope, J. C. (2013). What can we learn from benefit transfer errors? Evidence from 20 years of research on convergent validity. Journal of Environmental Economics and Management, 66(1), 90–104.
Li, Q., & Racine, J. S. (2007). Nonparametric econometrics: Theory and practice. Princeton, NJ: Princeton University Press.
Londoño, L. M., & Johnston, R. J. (2012). Enhancing the reliability of benefit transfer over heterogeneous sites: A meta-analysis of international coral reef values. Ecological Economics, 78(1), 80–89.
Leamer, E. (1983). Lets take con out of econometrics. American Economic Review, 73(1), 31–43.
Manski, C. F. (2007). Identification for prediction and decision. Cambridge: Harvard University Press.
Moeltner, K., Boyle, K. J., & Paterson, R. W. (2007). Meta-analysis and benefit transfer for resource valuation: Addressing classical challenges with Bayesian modeling. Journal of Environmental Economics and Management, 53(2), 250–269.
Moeltner, K., & Rosenberger, R. S. (2008). Predicting resource policy outcomes via meta-regression: Data space, model space, and the quest for ‘Optimal Scope’. The B.E. Journal of Economic Analysis and Policy, 8(1), 1–31.
Nelson, J. P., & Kennedy, P. E. (2009). The use (and abuse) of meta-analysis in environmental and natural resource economics: an assessment. Environmental and Resource Economics, 42, 345–377.
Ouyang, D., Li, Q., & Racine, J. (2009). Nonparametric estimation of regression functions with discrete regressors. Econometric Theory, 25(1), 1–42.
Proietti, T. (2003). Leave-K-out diagnostics in state-space models. Journal of Time Series Analysis, 214(2), 221–236.
Racine, J. S. (1997). Feasible cross-validatory model selection for general stationary processes. Journal of Applied Econometrics, 12, 169–179.
Rose, A. K., & Stanley, T. D. (2005). A meta-analysis of the effect of common currencies on international trade. Journal of Economic Surveys, 19(3), 347–375.
Rosenberger, R. S., & Johnston, R. J. (2009). Selection effects in meta-analysis and benefit transfer: Avoiding unintended consequences. Land Economics, 85(3), 410–428.
Rosenberger, R. S., & Loomis, J. B. (2000). Panel stratification in meta-analysis of economic studies: an investigation of its effects in the recreation valuation literature. Journal of Agricultural and Applied Economics, 32(3), 459–470.
Sala-I-Martin, X. X. (1997). I just ran two million regressions. The American Economic Review, 87, 178–183.
Wooldridge, J. M. (1995). Selection corrections for models under conditional mean independence assumptions. Journal of Econometrics, 68(1), 115–132.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendices
Appendix A: Derivation of (17.12)
We have
where \(P^{(t)} = X_{(t)} (X^{\prime} X)^{ - 1} X_{(t)}^{\prime}\). The first equality follows by definition of the OLS estimator, the second by our definitions for X (−t), X (t), y (−t) and y (t), and the third by Racine (1997, Eq. 9).
Appendix B: Derivation of (17.13)
The last equality here follows from the fact that
both of which can be discerned from the matrix equality \(A - A(A + B)^{ - 1} A = B - B(A + B)^{ - 1} B\), when (A + B)−1 exists (let A = I and B = − P (t)).
Rights and permissions
Copyright information
© 2015 Springer Science+Business Media Dordrecht
About this chapter
Cite this chapter
Boyle, K.J., Kaul, S., Parmeter, C.F. (2015). Meta-analysis: Econometric Advances and New Perspectives Toward Data Synthesis and Robustness. In: Johnston, R., Rolfe, J., Rosenberger, R., Brouwer, R. (eds) Benefit Transfer of Environmental and Resource Values. The Economics of Non-Market Goods and Resources, vol 14. Springer, Dordrecht. https://doi.org/10.1007/978-94-017-9930-0_17
Download citation
DOI: https://doi.org/10.1007/978-94-017-9930-0_17
Published:
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-017-9929-4
Online ISBN: 978-94-017-9930-0
eBook Packages: Business and EconomicsEconomics and Finance (R0)