A propensity score approach in the impact evaluation on scientific production in Brazilian biodiversity research: the BIOTA Program
- 386 Downloads
Evaluation has become a regular practice in the management of science, technology and innovation (ST&I) programs. Several methods have been developed to identify the results and impacts of programs of this kind. Most evaluations that adopt such an approach conclude that the interventions concerned, in this case ST&I programs, had a positive impact compared with the baseline, but do not control for any effects that might have improved the indicators even in the absence of intervention, such as improvements in the socio-economic context. The quasi-experimental approach therefore arises as an appropriate way to identify the real contributions of a given intervention. This paper describes and discusses the utilization of propensity score (PS) in quasi-experiments as a methodology to evaluate the impact on scientific production of research programs, presenting a case study of the BIOTA Program run by FAPESP, the State of São Paulo Research Foundation (Brazil). Fundamentals of quasi-experiments and causal inference are presented, stressing the need to control for biases due to lack of randomization, also a brief introduction to the PS estimation and weighting technique used to correct for observed bias. The application of the PS methodology is compared to the traditional multivariate analysis usually employed.
KeywordsQuasi-experiment Propensity score Impact evaluation Biota program Bibliometrics
Mathematics Subject Classification62P25
This work was supported by the São Paulo Research Foundation (FAPESP) [Grant number 2008/58628-7]; and Coordination for the Improvement of Higher Level Personnel (CAPES) [Grant AUX-PE-PNPD- 1945/2008].
Conflict of interest
Authors declare none conflict of interest. FAPESP played no role in the design of the study, data analysis, or in manuscript preparation.
- Carney, J., Smith, W., Parsad, A., Johnston, K., & Millsap, M. (2008). Evaluation of the Faculty Early Career Development (CAREER) Program. Bethesda, MD: Abt Associates Inc.Google Scholar
- Chen, S., Mu, R., & Ravallion, M. (2008). Are there lasting impacts of aid to poor areas? Evidence for rural China. Policy Research Working Paper 4084, World Bank, Washington, DC.Google Scholar
- Ferraro, P. J. (2009). Counterfactual thinking and impact evaluation in environmental policy. New Directions for Evaluation 122 (Special Issue: Environmental Program and Policy Evaluation: Addressing Methodological Challenges).Google Scholar
- Ihaka, R., & Gentleman, R. (1996). R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics, 5(3), 299–314.Google Scholar
- Lynch, L., Gray, W., & Geoghegan, J. (2007). Are farmland preservation program easement restrictions capitalized into farmland prices? What can a propensity score matching analysis tell us? Applied Economic Perspectives and Policy, 29(3), 502–509.Google Scholar
- Ravallion, M. (2008). Evaluating anti-poverty programs. In T. P. Schultz & J. Strauss (Eds.), Handbook of development economics (4th ed., pp. 3787–3846). Amsterdam: North-Holland.Google Scholar
- Ridgeway, G. (1999). The state of boosting. Computing Science and Statistics, 31, 172–181.Google Scholar
- Ridgeway, G. (2005). GBM 1.5 package manual. http://cran.r-project.org/doc/packages/gbm.pdf.
- Ridgeway, G., McCaffrey, D., & Morral, A. (2006). twang: Toolkit for weighting and analysis of nonequivalent groups. Software for using matching methods in R. Available at http://cran.r-project.org/web/packages/twang/index.html.
- Robins, J. M., & Rotnitzky, A. (2001). Comment on “inference for semiparametric models: Some questions and an answer” by P. Bickel and J. Kwon. Statistica Sinica, 11, 920–936.Google Scholar
- Rodrigues, R. R., & Bonomi, V. L. R. (Eds.). (2008). Diretrizes para conservação e restauração da biodiversidade no estado de São Paulo. São Paulo: SMA-SP & FAPESP.Google Scholar
- Rubin, D. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701.Google Scholar
- Salles-Filho, S., Bonacelli, M. B. M., Zackiewicz, M., Castro, P. F. D., & Bin, A. (2007). Development and application of a methodology for evaluating S,T&I Programs: The decomposition method. In Workshop Internacional sobre Inovações Metodológicas na Avaliação de Impacto da pesquisa Agropecuária, 2008, Brasília, XII Seminário Latino-Iberoamericano de Gestión Tecnológica - ALTEC 2007, Buenos Aires, 2007 (Vol. 1, pp. 2–6).Google Scholar
- Salles-Filho, S., Castro, P. F. D., Bonacelli, M. B. M., Zeitoum, C., & Sá, F. P. (2010b). Indicators for evaluation of results and impacts of research program in conservation and sustainable use of biodiversity, the case of BIOTA/FAPESP. In International conference: Getting post 2010 biodiversity targets right, Bragança Paulista.Google Scholar