Skip to main content
SpringerLink
Log in

School System Evaluation by Value Added Analysis Under Endogeneity

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

Value added is a common tool in educational research on effectiveness. It is often modeled as a (prediction of a) random effect in a specific hierarchical linear model. This paper shows that this modeling strategy is not valid when endogeneity is present. Endogeneity stems, for instance, from a correlation between the random effect in the hierarchical model and some of its covariates. This paper shows that this phenomenon is far from exceptional and can even be a generic problem when the covariates contain the prior score attainments, a typical situation in value added modeling. Starting from a general, model-free definition of value added, the paper derives an explicit expression of the value added in an endogeneous hierarchical linear Gaussian model. Inference on value added is proposed using an instrumental variable approach. The impact of endogeneity on the value added and the estimated value added is calculated accurately. This is also illustrated on a large data set of individual scores of about 200,000 students in Chile.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1.
Figure 2.
Figure 3.
Figure 4.

Similar content being viewed by others

References

  • Aitkin, M., & Longford, N. (1986). Statistical modelling issues in school effectiveness studies. Journal of the Royal Statistical Society. Series A, 149, 1–43.

    Article  Google Scholar 

  • Angrist, J.D., & Krueger, A.B. (2001). Instrumental variables and the search for identification: from supply and demand to natural experiments. The Journal of Economic Perspectives, 15, 69–85.

    Article  Google Scholar 

  • Blundell, R., & Bond, S. (1998). Initial conditions and moment restrictions in dynamic panel data models. Journal of Econometrics, 87, 115–143.

    Article  Google Scholar 

  • Bollen, K.A. (1989). Structural equations with latent variables. London: Wiley.

    Google Scholar 

  • Bollen, K.A. (2002). Latent variables in psychology and the social sciences. Annual Review of Psychology, 53, 605–634.

    Article  PubMed  Google Scholar 

  • Braun, H., Chudowsky N., & Koening, J. (2010). Getting value out of value-added: report of a workshop. Washington: The National Academics.

    Google Scholar 

  • Briggs, D.C., & Weeks, J.P. (2011). The persistence of school-level value-added. Journal of Educational and Behavioral Statistics, 36, 616–637.

    Article  Google Scholar 

  • Corvalán, J., Elaqua G., & de Wolf, I. (2010). El sector particular subvencionado en Chile. Tipologización y perspectivas frente a las nuevas regulaciones. In Ministry of Education of the Chilean Government (Ed.), Evidencias para políticas públicas en educación (pp. 11–40). Santiago: MINEDUC.

  • Ebbes, P., Böckenholt, U., & Wedel, M. (2004). Regressor and random-effects dependencies in multilevel models. Statistica Neerlandica, 58, 161–178.

    Article  Google Scholar 

  • Engle, R.E., Hendry, D.F., & Richard, J.F. (1983). Exogeneity. Econometrica, 51, 277–304.

    Article  Google Scholar 

  • EPI Briefing Paper (2010). Problems with the use of student test scores to evaluate teachers. Washington: Economic Policy Institute.

    Google Scholar 

  • Florens, J.P., Johannes, J., & Van Bellegem, S. (2012). Instrumental regression in partially linear models. Econometrics Journal, 15, 304–324.

    Article  Google Scholar 

  • Florens, J.P., & Mouchart, M. (1985). A linear theory for noncausality. Econometrica, 53, 157–176.

    Article  Google Scholar 

  • Florens, J.P., Mouchart, M., & Rolin, J.M. (1990). Elements of Bayesian statistics. New York: Dekker.

    Google Scholar 

  • Gansle, K.A., Noell, G.H., & Burns, J.M. (2012). Do student achievement outcomes differ across teacher preparation programs? An analysis of teacher education in Lousiana. Journal of Teacher Education. doi:10.1177/0022487112439894.

    Google Scholar 

  • Goldstein, H. (2001). Multilevel statistical models. London: Arnold.

    Google Scholar 

  • Gray, J., Hopkins, D., Reynolds, D., Wilcox, B., Farrell, S., & Jesson, D. (1999). Improving schools: performance and potential. Buckingham: Open University Press.

    Google Scholar 

  • Grilli, L., & Rampachini, C. (2011). The role of sample cluster mans in multilevel models. A view on endogeneity and measurement error issues. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 7, 121–133.

    Google Scholar 

  • Hanchane, S., & Mostafa, T. (2012). Solving endogeneity problems in multilevel estimation: an example using education production functions. Journal of Applied Statistics, 39, 1101–1114.

    Article  Google Scholar 

  • Hansen, L.P. (1982). Large sample properties of generalized methods of moments estimators. Econometrica, 50, 1029–1054.

    Article  Google Scholar 

  • Hanushek, E.A. (2006). School resources. In E.A. Hanushek, & F. Welch (Eds.), Handbook of the economics of education (pp. 865–908). Amsterdam: Elsevier.

    Google Scholar 

  • Hendry, D.F. (1995). Dynamic econometrics. Oxford: Oxford University Press.

    Book  Google Scholar 

  • Kim J.S., & Frees, E.W. (2006). Omitted variables in multilevel models. Psychometrika, 71, 659–690.

    Article  Google Scholar 

  • Kim J.S., & Frees, E.W. (2007). Multilevel modelling with correlated effects. Psychometrika, 72, 505–533.

    Article  Google Scholar 

  • Kyriakides, L. (2006). Using international comparative studies to develop the theoretical framework of educational effectiveness research: a secondary analysis of TIMSS 1999 data. Educational Research and Evaluation, 12, 513–534.

    Article  Google Scholar 

  • Lazarsfeld, P.F. (1950). The logical and mathematical foundation of latent structural analysis. In S.S. Stout (Ed.), Measurement and prediction (pp. 362–412). Princeton: Princeton University Press.

    Google Scholar 

  • Manzi, J., Martín, E., & Van Bellegem, S. (2011). School system evaluation by value-added analysis under endogeneity (Technical Report No. IT1-102). Measurement Center MIDE UC.

  • McPherson, A.F. (1992). Measuring added value in schools. London: National Commission of Education.

    Google Scholar 

  • Mundlak, Y. (1978). On the pooling of time-series and cross section data. Econometrica, 46, 69–85.

    Article  Google Scholar 

  • OECD (2008). Measuring improvements in learning outcomes. Best practices to assess the value-added of schools. Organisation for Economic Co-operation and Development OECD.

  • Paredes, R.D., & Paredes, V. (2009). Chile: academic performance and educational management under a rigid employment regime. CEPAL Review, 99, 117–129.

    Google Scholar 

  • Peng, W.J., Thomas, S.M., Yang, X., & Li, J. (2006). Developing school evaluation methods to improve the quality of schooling in China: a pilot value added study. Assessment in Education, 13, 135–154.

    Article  Google Scholar 

  • Picci, G. (1989). Parametrization of factor analysis models. Journal of Econometrics, 41, 17–38.

    Article  Google Scholar 

  • R Development Core Team (2011) R: a language and environment for statistical computing [computer software manual], Vienna, Austria. Available from http://www.R-project.org/. ISBN 3-900051-07-0.

  • Raudenbush, S.W. (2004). What are value-added models estimating and what does this imply for statistical practice? Journal of Educational and Behavioral Statistics, 29, 121–129.

    Article  Google Scholar 

  • Raudenbush, S.W., & Bryk, A.S. (2001). Hierarchical linear models: applications and data analysis method. London: Sage.

    Google Scholar 

  • Raudenbush, S.W., & Willms, J.D. (1995). The estimation of school effects. Journal of Educational and Behavioral Statistics, 20, 307–335.

    Article  Google Scholar 

  • Ray, A. (2006). School value added measures in England (Technical Report). Department for Education and Skills.

  • Ray, A., McCormack, T., & Evans, H. (2009). Value added in English schools. Education Finance and Policy, 4, 415–438.

    Article  Google Scholar 

  • Reardon, S.F., & Raudenbush, S.W. (2009). Assumptions of value-added models for estimating school effect. Education Finance and Policy, 4, 492–519.

    Article  Google Scholar 

  • Rosenbaum, P.R., & Rubin, D.B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70, 41–55.

    Article  Google Scholar 

  • San Martín, E., & Rolin, J.M. (2013). Identification of parametric Rasch-type models. Journal of Statistical Planning and Inference, 143, 116–130.

    Article  Google Scholar 

  • Sapelli, C., & Vial, B. (2002). The performance of private and public schools in the Chilean voucher system. Cuadernos de Economía, 39, 423–454.

    Google Scholar 

  • SIMCE (2009). Resultados nacionales SIMCE 2008. Unidad de Currículum y Evaluación (UCE), Ministerio de Educación, Gobierno de Chile. Available at http://www.simce.cl/index.php?id=430.

  • Skrondal, A., & Rabe-Hesketh, S. (2004). Generalized latent variable modeling: multilevel, longitudinal and structural equation models. Boca Raton: Chapman & Hall.

    Book  Google Scholar 

  • Snijders, T., & Bosker, R. (1999). Multilevel analysis: an introduction to basic and advanced multilevel modeling. London: Sage.

    Google Scholar 

  • Spencer, N.H., & Fielding, A. (2002). A comparison of modelling strategies for value-added analyses of educational data. Computational Statistics, 17, 103–116.

    Article  Google Scholar 

  • Tekwe, C.D., Carter, R.L., Ma, C.X., Algina, J., Lucas, M.E., Roth, J., et al. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29, 11–36.

    Article  Google Scholar 

  • Thomson, S. (2008). Examining the evidence from TIMSS: gender differences in year 8 science achievement in Australia. Studies in Educational Evaluation, 34, 73–81.

    Article  Google Scholar 

  • Timmermans, A., Doolaard, S., & de Wolf, I. (2011). Conceptual and empirical differences among various value-added models for accountability. School Effectiveness and School Improvement, 22, 393–413.

    Article  Google Scholar 

  • White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817–838.

    Article  Google Scholar 

  • Willms, J.D., & Raudenbusch, S.W. (1989). A longitudinal hierarchical linear model for estimating school effects and their stability. Journal of Educational Measurement, 26, 209–232.

    Article  Google Scholar 

  • Wooldridge, J. (2008). Introductory econometrics: a modern approach (4th ed.). Mason: South-Western College Pub.

    Google Scholar 

Download references

Acknowledgements

We thank Xavier Dumay, Jorge González, Jean Hindriks, Daniel Koretz, Michel Lubrano, Michel Mouchart, Erwin Ooghe, Amine Ouazad and Sally Thomas for helpful discussions. This research was partially supported by the FONDECYT Project No. 1110315 Schools Effectiveness and Value Added Models: From Quantitative Analysis to Qualitative Outcomes. Previous stages of this work have been presented during the conferences School Progress and Value Added Models (May 2010, MIDE UC, Chile), Efficiency Measurement of Educational Systems (January 2012, CORE, Belgium) and V European Congress of Methodology (July 2012, Spain), and at the Seminar of the Centrum voor Onderwijseffectiviteit en-Evaluatie (February 2013, KU Leuven, Belgium). The authors are grateful to the SIMCE Office of the Chilean Ministry of Education for providing access to the database. All opinions and conclusions expressed in this paper are those of the authors and not necessarily those of the Ministry of Education. Finally, we wish to thank three anonymous reviewers for their constructive comments which have helped to improve this manuscript.

Author information

Authors and Affiliations

  1. School of Psychology, Pontificia Universidad Católica de Chile, Vicuña Mackenna 4860, Macul, Santiago, Chile

    Jorge Manzi

  2. Measurement Center MIDE UC, Vicuña Mackenna 4860, Macul, Santiago, Chile

    Jorge Manzi & Ernesto San Martín

  3. Department of Statistics, Pontificia Universidad Católica de Chile, Vicuña Mackenna 4860, Macul, Santiago, Chile

    Ernesto San Martín

  4. Faculty of Education, Pontificia Universidad Católica de Chile, Vicuña Mackenna 4860, Macul, Santiago, Chile

    Ernesto San Martín

  5. Center for Operations Research and Econometrics, Université catholique de Louvain, 34 Voie du Roman Pays, 1348, Louvain-la-Neuve, Belgium

    Ernesto San Martín & Sébastien Van Bellegem

Authors
  1. Jorge Manzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ernesto San Martín
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sébastien Van Bellegem
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ernesto San Martín.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Manzi, J., San Martín, E. & Van Bellegem, S. School System Evaluation by Value Added Analysis Under Endogeneity. Psychometrika 79, 130–153 (2014). https://doi.org/10.1007/s11336-013-9338-0

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-013-9338-0

Key words

Search

Navigation