Standardized testing of public school students has been, and continues to be, a focal point of the political side of education reform. The use of standardized test results to set policies and identify problem schools requires some understanding of what factors contribute to high or low overall scores. Many large datasets at the state and national levels contain information about mean scores (i.e., average class size, poverty level, teacher salary, etc.); it is tempting to create statistical models, draw cause-and-effect conclusions, and perhaps set policy based on statistically significant relationships observed in these data; for example, National Assessment of Educational Progress (NAEP, US National Center for Education Statistics, n.d.); Trends in International Mathematics and Science Study, and Progress in International Reading Literacy Study (TIMSS & PIRLS International Study Center, n.d.); Programme for International Student Assessment (PISA, Organisation for Economic Co-operation and Development, n.d.). However, many examples of confounding, that is, the apparent associations between the variables that change depending on which covariates are selected, can be found in these associations. In this chapter, some results using standardized testing data are presented, which demonstrate by example the difficulties inherent in making conclusions or comparisons based on observational data and disentangling second- and third-order influences on these relationships. First, some statistical terminology is reviewed, and some simplified, fabricated examples are presented to illustrate concepts. Next, a dataset containing scores for the Illinois Standardized Achievement Test (ISAT, Illinois State Board of Education, n.d.), taken by Grade 8 students in Illinois public schools, is used to demonstrate confounding relationships. Finally, the Scholastic Achievement Test (SAT, College Board, n.d.) scores by state are used to show some misleading rankings of states' average scores.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
ACT. (n.d.). Homepage. Retrieved July 11, 2008, from http://www.act.org/
Bellamy, G. T.,&Goodlad, J. I. (2008). Continuity and change in the pursuit of a democratic public mission for our schools. Phi Delta Kappan, 89(8), 565–571.
Bock, D. E., Velleman, P. F.,&De Veaux, R. D. (2009). Intro stats (3rd ed.). Boston, MA: Pearson Education.
Bradley, R. H.,&Corwyn, R. F. (2002). Socioeconomic status and child development. Annual Review of Psychology, 53(1), 371–399.
College Board. (n.d.). About the SAT. Retrieved May 15, 2008, from http://www.collegeboard. com/student/testing/sat/about.html
Einstein, A. (n.d.). quotesmuseum Homepage. Retrieved July 11, 2008, from http://www.quotesmuseum.com/author/Albert%20Einstein/1925
Entwisle, D. R.,&Alexander, K. L. (1988). Factors affecting achievement test scores and marks of black and white first graders. The Elementary School Journal, 88(5), 449–471.
Glass, G. V. (1982). Meta-analysis: An approach to the synthesis of research results. Journal of Research in Science Teaching, 19(2), 93–112.
Illinois State Board of Education. (n.d.). Student assessment. Retrieved May 18, 2008, from http://www.isbe.net/assessment/isat.htm
Kao, G.,&Thompson, J. S. (2003). Racial and ethnic stratification in educational achievement and attainment. Annual Review of Sociology, 29(1), 417–442.
Organisation for Economic Co-operation and Development. (n.d.). PISA Homepage. Retrieved June 30, 2008, from http://www.pisa.oecd.org/pages/0,2987,en_32252351_32235731_1_1_1_ 1_1,00.html Petitti, D. B. (1998). Hormone replacement therapy and heart disease prevention: Experimentation trumps observation [Editorial]. Journal of the American Medical Association, 280(7), 650–652.
R Project for Statistical Computing. (n.d.). Homepage. Retrieved July 11, 2008, from http://www.r-project.org/
Smith, M. L.,&Glass, G. V. (1980). Meta-analysis of research on class size and its relationship to attitudes and instruction. American Educational Research Journal, 17(4), 419–433.
TIMSS&PIRLS International Study Center. (n.d.). Homepage. Retrieved July 11, 2008, from http://timss.bc.edu/
United States National Center for Education Statistics. (n.d.). NAEP: The nation's report card. Retrieved July 11, 2008, from http://nces.ed.gov/nationsreportcard/
Yore, L. D., Pimm, D.,&Tuan, H.-L. (2007). The literacy component of mathematical and scientific literacy. International Journal of Science and Mathematics Education, 5(4), 559–589.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer Science + Business Media B.V
About this chapter
Cite this chapter
Meyer, M.C. (2009). Confounding in Observational Studies using Standardized Test Data: Careful Disentanglement of Statistical Interpretations and Explanations. In: Shelley, M.C., Yore, L.D., Hand, B. (eds) Quality Research in Literacy and Science Education. Springer, Dordrecht. https://doi.org/10.1007/978-1-4020-8427-0_15
Download citation
DOI: https://doi.org/10.1007/978-1-4020-8427-0_15
Publisher Name: Springer, Dordrecht
Print ISBN: 978-1-4020-8426-3
Online ISBN: 978-1-4020-8427-0
eBook Packages: Humanities, Social Sciences and LawEducation (R0)