Abstract
The precarious state of the educational system existing in the inner-cities of the U.S., including its potential causes and solutions, has been a popular topic of debate in recent years. Part of the difficulty in resolving this debate is the lack of solid empirical evidence regarding the true impact of educational initiatives. For example, educational researchers rarely are able to engage in controlled, randomized experiments. The efficacy of so-called “school choice” programs has been a particularly contentious issue. A current multi-million dollar evaluation of the New York School Choice Scholarship Program (NYSCSP) endeavors to shed some light on this issue. This study can be favorably contrasted with other school choice evaluations in terms of the consideration that went into the randomized experimental design (a completely new design, the Propensity Matched Pairs Design, is being implemented) and the rigorous data collection and compliance-encouraging efforts. In fact, this study benefits from the authors’ previous experiences with the analysis of data from the Milwaukee Parental Choice Program, which, although randomized, was relatively poorly implemented as an experiment.
At first glance, it would appear that the evaluation of the NYSCSP could proceed without undue statistical complexity. However, this program evaluation, as is common in studies with human subjects, suffers from unintended, although not unanticipated, complications. The first complication is non-compliance. Approximately 25% of children who were awarded scholarships decided not to use them. The second complication is missing data: some parents failed to complete fully survey information; some children did not take pre-tests; some children failed to show up for post-tests. Levels of missing data range approximately from 3 to 50% across variables. Work by Frangakis and Rubin (1999) has revealed the severe threats to valid estimates of experimental effects that can exist in the presence of noncompliance and missing data, even for estimation of simple intention-to-treat effects.
The technology we use to proceed with analyses of longitudinal data from a randomized experiment suffering from missing data and non-compliance involves the creation of multiple imputations for both missing outcomes and missing true compliance statuses using Bayesian models. The fitting of Bayesian models to such data requires MCMC methods for missing data. Our Bayesian approach allows for analyses that rely on fewer assumptions than standard approaches.
These analyses provide evidence of small positive effects of private school attendance on math test scores for certain subgroups of the children studied.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Angrist, J. D., Imbens, G. W., and Rubin, D. B. (1996), “Identification of Causal Effects Using Instrumental Variables,” Journal of the American Statistical Association 91, 444–472.
Ascher, C., Fruchter, N., and Berne, R. (1996), “Hard Lessons: Public Schools and Privatization,” Tech. rep., Century Foundation, New York, NY.
Barnard, J., Du, J., Hill, J. L., and Rubin, D. B. (1998), “A Broader Template for Analyzing Broken Randomized Experiments,” Sociological Methods and Research 27, 285–317.
Bishop, Y. M. M., Fienberg, S. E., and Holland, P. W. (1975), Discrete Multivariate Analyses: Theory and Practice, MIT Press.
Bonsteel, A. and Bonilla, C. A. (1997), A Choice for our Children: Curing the Crisis in America’s Schools, San Francisco, California: Institute for Contemporary Studies.
Brandi, J. E. (1998), Money and Good Intentions are not Enough, or Why Liberal Democrat Thinks States Need Both Competition and Community, Washington, D.C.: Brookings Institution Press.
Carnegie Foundation for the Advancement of Teaching (1992), School Choice: A Special Report, San Francisco, CA: Jossey-Bass, Inc. Publishers.
Chubb, J. E. and Moe, T. M. (1990), Politics, Markets and America’s Schools, Washington, D.C.: Brookings Institution Press.
Cobb, C. W. (1992), Responsive Schools, Renewed Communities, San Francisco, California: Institute for Contemporary Studies.
Cochran, W. G. and Rubin, D. B. (1973), “Controlling Bias in Observational Studies: A Review,” Sankhya 35, 417–446.
Coleman, J. S., Hoffer, T., and Kilgore, S. (1982), High School Achievement, New York: NY: Basic Books.
Cookson, P. W. (1994), School Choice: The Struggle for the Sould of American Education, New Haven, CT: Yale University Press.
Coulson, A. J. (forthcoming), “Market Education: The Unknown History.”
D’Agostino, Ralph B., J. and Rubin, D. B. (1999), “Estimating and Using Propensity Scores With Incomplete Data,” pending publication in JASA.
Derek, N. (1997), “The Effects of Catholic Secondary Schooling on Educational Achievement,” Journal of Labor Economics 15,1, 98–123.
Frangakis, C. E. and Rubin, D. B. (1999), “Addressing Complications of Intention-to-Treat Analysis in the Combined Presence of All-or-None Treatment-Noncompliance and Subsequent Missing Outcomes,” Biometrika 86, 365–380.
Fuller, B. and Elmore, R. F (1996), Who Chooses? Who Loses? Culture, Institutions, and the Unequal Effects of School Choice, New York: Teachers College Press.
Gelfand, A. E. and Smith, A. F. M. (1990), “Sampling-based Approaches to Calculating Marginal Densities,” Journal of the American Statistical Association 85, 398–409.
Glynn, R. J., Laird, N. M., and Rubin, D. B. (1986), “Mixture Modeling Versus Selection Modeling for Nonignorable Nonresponse,” in Drawing Inferences from Self-Selected Samples, ed. H. Wainer, pp. 115–142, Springer-Verlag.
Glynn, R. J., Laird, N. M., and Rubin, D. B. (1993), “Multiple Imputation in Mixture Models for Nonignorable Nonresponse With Follow-ups,” Journal of the American Statistical Association 88, 984–993.
Goldberger, A. S. and Cain, G. G. (1982), “The Causal Analysis of Cognitive Outcomes in the Coleman, Hoffer, and Kilgore Report,” Sociology of Education 55, 103–122.
Gu, X. S. and Rosenbaum, P. R. (1993), “Comparison of Multivariate Matching Methods: Structures, Distances, and Algorithms,” Journal of Computational and Graphical Statistics 2, 405–420.
Gutmann, A. (1987), Democratic Education, Princeton, NJ: Princeton University Press.
Haavelmo, T. (1943), “The Statistical Implications of a System of Simultaneous Equations,” Econometrica 11, 1–12.
Haavelmo, T. (1944), “The Probability Approach in Econometrics,” Econometrica 12, 1–115, (Supplement).
Hill, J. L., Rubin, D. B., and Thomas, N. (1999), “The Design of the New York School Choice Scholarship Program Evaluation,” in Donald Campbell’s Legacy, ed. L. Bickman, Sage Publications.
Hirano, K., Imbens, G. W., Rubin, D. B., and Zhou, A. (1999), “Estimating the Effect of an Influenza Vaccine in an Encouragement Design,” to appear in Biostatistics.
Holland, P. (1986), “Statistics and Causal Inference,” Journal of the American Statistical Association 81, 396, 945–970.
Imbens, G. W. and Angrist, J. D. (1994), “Identification and Estimation of Local Average Treatment Effects,” Econometrica 62, 467–476.
Imbens, G. W. and Rubin, D. B. (1997), “Bayesian Inference for Causal Effects in Randomized Experiments with Noncompliance,” The Annals of Statistics 25, 305–327.
Levin, H. M. (1998), “Educational Vouchers: Effectiveness, Choice, and Costs,” Journal of Policy Analysis and Management 17,3, 373–392.
Little, R. J. A. (1993), “Pattern-mixture models for multivariate incomplete data,” Journal of the American Statistical Association 88, 125–134.
Little, R. J. A. (1996), “Pattern-mixture models for multivariate incomplete data with covariates,” Biometrics 52, 98–111.
Little, R. J. A. and Rubin, D. B. (1987), Statistical Analysis With Missing Data, New York: John Wiley & Sons.
Meng, X.-L. and Rubin, D. B. (1993), “Maximum Likelihood Estimation Via the ECM Algorithm: A General Framework,” Biometrika 80, 267–278.
Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., and Teller, E. (1953), “Equations of state calculations by fast computing machines,” Chemical Physics 21, 1087–1091.
Mosteller, F. (1995), “The Tennessee Study of Class Size in the Early School Grades,” in The Future of Children, vol. 5, pp. 113–127.
Neyman, J. (1923), “On the Application of Probablity Theory to Agricultural Experiments Essay on Principles. Section 9,” translated in Statistical Science 5, 465–480, 1990.
Peterson, P. E. and Hassel, B. C., eds. (1998), Learning from School Choice, Washington, D.C.: Brookings Institution Press.
Peterson, P. E. and Howell, W. G. (1999), “What Happens to Low-Income New York Students When They Move from Public to Private Schools,” in City Schools: Lessons from New York, eds. D. Ravitch and J. Viteritti, Johns Hopkins University Press, forthcoming.
Peterson, P. E., Myers, D. E., Howell, W. G., and Mayer, D. P. (1999), “The Effects of School Choice in New York City,” in Earning and Learning; How Schools Matter, eds. S. E. Mayer and P. E. Peterson, Brookings Institution Press.
Rasell, E. and Rothstein, R., eds. (1993), School Choice: Examining the Evidence, Washington, D.C.: Economic Policy Institute.
Roseman, L. (1998), “Reducing Bias in the Estimate of the Difference in Survival in Observational Studies Using Subclassification on the Propensity Score,” Ph.D. thesis, Harvard University.
Rosenbaum, P. R. and Rubin, D. B. (1983), “The central role of the propensity score in observational studies for causal effects,” Biometrika 70,1, 41–55.
Rosenbaum, P. R. and Rubin, D. B. (1984), “Reducing Bias in Observational Studies Using Subclassification on the Propensity Score,” Journal of the American Statistical Association 79, 516–524.
Rosenbaum, P. R. and Rubin, D. B. (1985), “Constructing a control group using multivariate matched sampling methods that incorporate the propensity score,” The American Statistician 39, 33–38.
Rubin, D. B. (1973), “The Use of Matched Sampling and Regression Adjustment to Remove Bias in Observational Studies,” Biometrics 29, 185–203.
Rubin, D. B. (1974), “Estimating Causal Effects of Treatments in Randomized and Non-Randomized Studies,” Journal of Educational Psychology 66, 688–701.
Rubin, D. B. (1977), “Assignment to Treatment Groups on the Basis of a Covariate,” Journal of Educational Statistics 2, 1–26.
Rubin, D. B. (1978a), “Bayesian Inference for Causal Effects: The role of randomization,” The Annals of Statistics 6, 34–58.
Rubin, D. B. (1978b), “Multiple Imputations in Sample Surveys: A Phenomenological Bayesian Approach to Nonresponse (C/R: P29-34),” in ASA Proceedings of Survey Research Methods Section, pp. 20–28.
Rubin, D. B. (1979), “Using Multivariate Matched Sampling and Regression Adjustment to Control Bias in Observational Studies,” Journal of the American Statistical Association 74, 318–328.
Rubin, D. B. (1980), “Comments on “Randomization Analysis of Experimental Data: The Fisher Randomization Test”,” Journal of the American Statistical Association 75, 591–593.
Rubin, D. B. (1990), “Comment: Neyman (1923) and Causal Inference in Experiments and Observational Studies,” Statistical Science 5, 472–480
Rubin, D. B. and Thomas, N. (1992), “Characterizing the Effect of Matching Using Linear Propensity Score Methods With Normal Distributions,” Biometrika 79, 797–809.
Rubin, D. B. and Thomas, N. (1996), “Matching using estimated propensity scores: Relating theory to practice,” Biometrics 52, 249–264.
Schafer, J. L. (1997), Analysis of Incomplete Multivariate Data, New York: Chapman and Hall.
Wilms, D. J. (1985), “Catholic School Effect on Academic Achievement: New Evidence from the High School and Beyond Follow-Up Study,” Sociology of Education 58, 98–114.
References
Achieve, Inc. (1999). Home Page. Obtained from the World Wide Web at address http://www. achieve.org, September 1999.
Aikin, W. M. (1942). The story of the eight-year study, with conclusions and recommendations. Volume 1 in: Commission on the Relation of School and College, Progressive Education Association (ed.) (1942). Adventure in American Education. New York: Harper & Brothers.
Angrist, J. D., Imbens, G. and Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444–472.
Ascher, C., Fructer, N. and Berne, R. (1996). Hard lessons: public schools and privatization. New York: Twentieth Century Fund Press.
Balke, A. and Pearl, J. (1994). Counterfactual probabilities, computational methods, bounds and applications, in Proceedings of the Tenth Conference on Uncertainty in Artificial Intelligence, R. Lopez de Mantaras and D. Poole, eds. Morgan Kaufman.
Barnard, J., Frangakis, C., Hill, J., and Rubin, D. B. (1999). School choice in New York City: A Bayesian analysis of an imperfect randomized experiment. Invited Case Study, Fifth Workshop on Bayesian Statistics in Science and Technology. Pittsburgh, PA, September 24–25, 1999.
Bloom, B. S. (1984). The 2-sigma problem: the search for methods of group instruction as effective as one-to-one tutoring. Educational Researcher, 13, 4–16.
Bryk, A. S., Lee, V. E. and Holland, P. B. (1993). Catholic schools and the common good. Cambridge MA: Harvard University Press.
Bryk, A. S., Sebring, P. B., Kerbow, D., Rollow, S., and Easton, J. Q. (1998). Charting Chicago school reform: democratic localism as a lever for change. Boulder, CO: Westview Press.
Council of Chief State School Officers (CCSSO). (1995). State responsibility for student opportunity: commitment and issues. A statement of the Council of Chief State School Officers. Washington DC: Author. Obtained from the World Wide Web at address http://www.ccsso.org/oppol.html, August 1999.
D’Agostino, R. J., and Rubin, D. B. (1999). Estimating and using propensity scores with incomplete data. To appear, Journal of the American Statistical Association.
Friedman, M. (1962). Capitalism and Freedom. Chicago IL: University of Chicago Press.
Fullan, M. (1999). Change forces: the sequel (Educational Change and Development Series). Philadelphia: Falmer Press.
Fuller, B., Elmore, R. F. and Orfield, G. (1996). Policy-making in the dark. Chapter 1, pp. 1–21 in Fuller, B. and Elmore, R. F. with Orfield, G. (eds.) (1996). Who chooses? Who loses? Culture, institutions, and the unequal effects of school choice. New York: Teachers College Press.
Fuller, B. and Elmore, R. F. with Orfield, G. (eds.) (1996). Who chooses? Who loses? Culture, institutions, and the unequal effects of school choice. New York: Teachers College Press.
Gilles, S. G. (1998). Why parents should choose. Chapter 15, pp. 395–407 in Peterson, P. E. and Hassel, B. C. (eds.) (1998). Learning from school choice. Washington DC: Brookings Institution Press.
Gitelman, A.I. (1999). Treatment integrity concerns in comparative education studies. Unpublished Ph.D. Dissertation, Carnegie Mellon University, Pittsburgh PA.
Glennan, T. K. Jr. (1998). New American Schools after six years. Santa Monica CA: RAND Corporation.
Halloran, M.E. and Struchiner, C.J. (1995). Causal inference in infectious diseases, Epidemiology, 6,2 142–151.
Holland, P.W. (1988). Causal inference, path analysis, and recursive structural equation models, Sociological Methodology, 18, 449–484.
Imbens G. W., and Rubin D. B. (1997). Bayesian inference for causal effects in randomized experiments with noncompliance. Annals of Statistics, 25, 305–327.
Learning Research and Development Center (LRDC) (1999). Outreach and implementation. Obtained from the World Wide Web at address http://www.lrdc.pitt.edu/research/oai99.htm, September 1999.
Lieberman, M. (1993). Public education: an autopsy. Cambridge MA: Harvard University Press.
Manski, C. (1990). Nonparametric bounds on treatment effects, American Economic Review, Papers and Proceedings, 80, 319–323.
Mosteller, F. (1995). The Tennessee study of class size in the early school grades, in The Future of Children, Vol. 5, No. 2, Summer/Fall 1995. Obtained from the World Wide Web at address http: //www.futureof children. org/cri/08cri.htm, September 1999.
National Commission on Excellence in Education. (1983). A Nation at Risk: The Imperative for Educational Reform. Washington DC: author.
Peterson, P. E. (1998). School choice: a report card. Chapter 1, pp. 3–32 in Peterson, P. E. and Hassel, B. C. (eds.) (1998). Learning from school choice. Washington DC: Brookings Institution Press.
Peterson, P. E. and Hassel, B. C. (eds.) (1998). Learning from school choice. Washington DC: Brookings Institution Press.
Peterson, P. E., Myers, D. E and Howell, W. G. (1998). An Evaluation of the New York City School Choice Scholarships Program: The First Year. Cambridge MA and Washington DC: Harvard Program on Educational Policy and Governance and Mathematica Policy Research. Obtained from the World Wide Web at address http: //data.fas.harvard.edu/pepg/NewYork-First.htm, September, 1999.
Rangazas, P. (1997). Competition and private school vouchers. Education Economics, 5, 245–263.
Ravitch, D. (1995). National standards in American education: a citizen’s guide. Updated with a new introduction. Washington DC: Brookings Institution Press.
Robins, J.M. (1989). The analysis of randomized and non-randomized AIDS treatment trials using a new approach to causal inference in longitudinal studies, Health Service Research Methodology: A Focus on AIDS, L. Sechrest, H. Freeman and A. Mulley, eds. NCHSR, U.S. Public Health Service.
Schafer, J. L. (1997). Analysis of incomplete multivariate data. New York: Chapman and Hall.
School Choice Scholarships Foundation (SCSF) Inc. (1999). Home Page. Obtained from the World Wide Web at address http://nygroup.com/scs/, September, 1999.
Valverde, G. A. and Schmidt, W. H. (1997). Refocusing U.S. Math and Science Education. Issues in Science and Technology Online, Winter 1997 Obtained from the World Wide Web at address http://www.nap.edu/issues/14.2/schmid.htm, November, 1999.
References
Peterson, P. E., Myers, D. E., Howell, W. G., and Mayer, D. P. (1999), “The Effects of School Choice in New York City,” in Earning and Learning; How Schools Matter, eds. S. E. Mayer and P. E. Peterson, Brookings Institution Press.
Rosenthal, R. and Jacobson, L. (1968), Pygmalion in the Classroom: teacher expectation and pupils’ intellectual development, Holt, Rinehart and Winston.
Rubin, D. B. (1978), “Bayesian Inference for Causal Effects: The role of randomization,” The Annals of Statistics 6, 34–58.
Rubin, D. B. and Thomas, N. (2000), “Combining Propensity Score Matching with Additional Adjustments for Prognostic Covariates,” JASA.
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer Science+Business Media New York
About this paper
Cite this paper
Barnard, J., Frangakis, C., Hill, J., Rubin, D.B. (2002). School Choice in NY City: A Bayesian Analysis of an Imperfect Randomized Experiment. In: Gatsonis, C., et al. Case Studies in Bayesian Statistics. Lecture Notes in Statistics, vol 162. Springer, New York, NY. https://doi.org/10.1007/978-1-4613-0035-9_1
Download citation
DOI: https://doi.org/10.1007/978-1-4613-0035-9_1
Publisher Name: Springer, New York, NY
Print ISBN: 978-0-387-95169-0
Online ISBN: 978-1-4613-0035-9
eBook Packages: Springer Book Archive