Skip to main content

Advertisement

Springer Nature Link
Log in

Content and alignment of state writing standards and assessments as predictors of student writing achievement: an analysis of 2007 National Assessment of Educational Progress data

  • Published:
Reading and Writing Aims and scope Submit manuscript

Abstract

We examined the degree to which content of states’ writing standards and assessments (using measures of content range, frequency, balance, and cognitive complexity) and their alignment were related to student writing achievement on the 2007 National Assessment of Educational Progress (NAEP), while controlling for student, school, and state characteristics. We found student demographic characteristics had the largest effect on between-state differences in writing performance, followed by state policy-related variables, then state and school covariates. States with writing tests that exhibited greater alignment with the NAEP writing assessment demonstrated significantly higher writing scores. We discuss plausible implications of these findings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
€32.70 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Finland)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Due to data security restrictions, unweighted NAEP sample data have been rounded in this paper.

  2. Because we use content from K-8 standards to predict NAEP 2007 grade 8 writing performance, we coded all substantive revisions of standards between 1999 and 2007; data from standards in force for a student at each grade from kindergarten through grade 8 were used (e.g., if a state had one set of standards in force for AY1999-2002 [students taking the 2007 NAEP would have been in grades K-3] and another in force for AY2002-2007 [students taking the 2007 NAEP would have been in grades 4–8], we used the data from the relevant set of state standards for those grades). Thus, we attempted to capture the totality of the content of states’ standards via coding all versions that would have been in effect throughout a student’s educational experience up to the point they took the 2007 NAEP. In essence, for each grade for each state, there was one set of content codes, but the policy-related documents used to derive those content codes may have changed over time if there were substantive revisions.

  3. As for standards, we coded all substantive revisions of writing assessments between 1999 and 2007 and the data from the assessment to which a student at each grade would have been administered from grades 3 through 8 were used.

References

  • Abrams, L. M., Pedulla, J. T., & Madaus, G. F. (2003). Views from the classroom: Teachers’ opinions of statewide testing programs. Theory into Practice, 42, 18–29.

    Article  Google Scholar 

  • Albertson, B. R. (2007). Organization and development features of grade 8 and grade 10 writers: A descriptive study of Delaware student testing program (DSTP) essays. Research in the Teaching of English, 41, 435–465.

    Google Scholar 

  • Applebee, A., & Langer, J. (2006). The state of writing instruction in America’s schools: What existing data tell us. Albany, NY: Center on English Learning and Achievement.

    Google Scholar 

  • Beard, R., Myhill, D., Riley, J., & Nystrand, M. (Eds.). (2009). The SAGE handbook of writing development. Thousand Oaks, CA: SAGE.

    Google Scholar 

  • Beaton, A. E., Rogers, A. M., Gonzalez, E., Hanly, M. B., Kolstad, A., Rust, K. F., et al. (2011). The NAEP Primer (NCES 2011-463). Washington, DC, USA: Department of Education, National Center for Education Statistics.

    Google Scholar 

  • Bourdin, B., & Fayol, M. (2000). Is graphic activity cognitively costly? A developmental approach. Reading and Writing: An Interdisciplinary Journal, 13, 183–196.

    Article  Google Scholar 

  • Carmichael, S. B., Martino, G., Porter-McGee, K., & Wilson, W. S. (2010). The state of the state standards—and Common Core—in 2010. Washington, DC: Thomas B. Fordham Foundation & Institute. Retrieved from ERIC database (ED516607).

  • Clarke, M., Shore, A., Rhoades, K., Abrams, L., Miao, J., & Li, J. (2003). Perceived effects of state-mandated testing programs on teaching and learning: Findings from interviews with educators in low-, medium-, and high-stakes states. Chestnut Hill, MA: National Board on Educational Testing and Public Policy. (ERIC Document Reproduction Service No. ED474867).

  • Cohen, D. (1995). What standards for national standards? Phi Delta Kappan, 77, 751–757.

    Google Scholar 

  • Cutler, L., & Graham, S. (2008). Primary grade writing instruction: A national survey. Journal of Educational Psychology, 100, 907–919.

    Article  Google Scholar 

  • Dean, D. (2008). Genre theory: Teaching, writing, and being. Urbana, IL: National Council of Teachers of English.

    Google Scholar 

  • Dutro, E., & Valencia, S. W. (2004). The relation between state and district literacy standards: Issues of alignment, influence, and utility. A research report. Seattle: Center for the Study of Teaching and Policy, University of Washington.

  • Faigley, L., & Witte, S. P. (1981). Coherence, cohesion, and writing quality. College Composition and Communication, 32(2), 2–11.

    Google Scholar 

  • Fraine, B. D., Damme, J., & Onghena, P. (2002). Accountability of schools and teachers: What should be taken into account? European Educational Research Journal, 1, 403–428.

    Article  Google Scholar 

  • Gershenson, S., & Langbein, L. (2015). The effect of primary school size on academic achievement. Educational Evaluation and Policy Analysis, 37, 135–155.

    Article  Google Scholar 

  • Gillespie, A., Graham, S., Kiuhara, S., & Hebert, M. (2014). High school teachers’ use of writing to support students’ learning: A national survey. Reading & Writing: An Interdisciplinary Journal, 27, 1043–1072.

    Article  Google Scholar 

  • Graham, S., Capizzi, A., Harris, K. R., Hebert, M., & Morphy, P. (2014). Teaching writing to middle school students: A national survey. Reading & Writing: An Interdisciplinary Journal, 27, 1015–1042.

    Article  Google Scholar 

  • Graham, S., McKeown, D., Kiuhara, S., & Harris, K. R. (2012). A meta-analysis of writing instruction for students in the elementary grades. Journal of Educational Psychology, 104, 879–896.

    Article  Google Scholar 

  • Graham, S., & Perin, D. (2007). A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology, 99, 445–476.

    Article  Google Scholar 

  • Graham, S., & Sandmel, K. (2011). The process approach: A meta-analysis. The Journal of Educational Research, 104, 396–407.

    Article  Google Scholar 

  • Grigorenko, E. L., Mambrino, E., & Preiss, D. D. (Eds.). (2012). Writing: A mosaic of new perspectives. New York, NY: Psychology Press.

    Google Scholar 

  • Grissmer, D. W., & Berends, M. (1994). Student achievement and the changing American family. Santa Monica, CA: RAND.

    Google Scholar 

  • Hamilton, L. S., & Berends, M. (2006). Instructional practices related to standards and assessments (RAND WR-37-EDU). Santa Monica, CA: RAND.

    Google Scholar 

  • Hamilton, L. S., Berends, M., & Stecher, B. M. (2005). Teachers’ responses to standards-based accountability. Santa Monica, CA: RAND.

    Google Scholar 

  • Hamilton, L. S., & Stecher, B. M. (2006, April). Measuring educators’ responses to high-stakes testing. In Paper presented at the annual meeting of the American Educational Research Association, San Francisco.

  • Hayes, J. R. (1996). A new model of cognition and affect in writing. In M. Levy & S. Ransdell (Eds.), The science of writing (pp. 1–27). Hillsdale, NJ: Erlbaum.

    Google Scholar 

  • Hillocks, G. (2002). The testing trap: How state writing assessments control learning. New York, NY: Teachers College Press.

    Google Scholar 

  • Hoffman, K., & Liagas, C. (2003). Status and trends in the education of blacks (NCES Publication No. 2003-034). Washington, DC: U.S. Department of Education.

  • Johnson, E. G. (1992). The design of the National Assessment of Educational Progress. Journal of Educational Measurement, 29, 95–110.

    Article  Google Scholar 

  • Johnson, E. G., & Rust, K. F. (1992). Population inferences and variance estimation for NAEP data. Journal of Educational Statistics, 17, 175–190.

    Google Scholar 

  • Juzwik, M. M., Curcic, S., Wolbers, K., Moxley, K. D., Dimling, L. M., & Shankland, R. K. (2006). Writing into the 21st century: An overview of research on writing, 1994 to 2004. Written Communication, 23, 451–476.

    Article  Google Scholar 

  • Kiuhara, S., Graham, S., & Hawken, L. (2009). Teaching writing to high school students: A national survey. Journal of Educational Psychology, 101, 136–160.

    Article  Google Scholar 

  • Klein, S. P., Hamilton, L. S., McCaffrey, D. F., & Stecher, B. M. (2000). What do test scores in Texas tell us? Education Policy Analysis Archives, 8(49), 1–20.

    Google Scholar 

  • Konstantopoulos, S. (2006). Trends of school effects on student achievement: Evidence from NLS:72, HSB:82, and NELS:92. Teacher College Record, 108, 2550–2581.

    Article  Google Scholar 

  • Koretz, D., Barron, S., Mitchell, K., & Stecher, B. (1996). The perceived effects of the Kentucky Instructional Results Information System (KIRIS) (MR-792-PCT/FF). Santa Monica, CA: RAND.

    Google Scholar 

  • Lee, J. (2013). Can writing attitudes and learning behavior overcome gender difference in writing? Evidence from NAEP. Written Communication, 30, 164–193.

    Article  Google Scholar 

  • Lee, V. E., & Bryk, A. S. (1989). A multilevel model of the social distribution of high school achievement. Sociology of Education, 62, 172–192.

    Article  Google Scholar 

  • Leithwood, K., & Jantzi, D. (2009). A review of empirical evidence about school size effects: A policy perspective. Review of Educational Research, 79, 464–490.

    Article  Google Scholar 

  • Linn, R. L., Baker, E. L., & Betebenner, D. W. (2002). Accountability systems: Implications of requirements of the No Child Left Behind Act of 2001. Educational Researcher, 31(6), 3–16.

    Article  Google Scholar 

  • Loveless, T. (2012). The 2012 Brown Center report on American education: How well are American students learning?. Washington, DC: Brown Center on Education Policy, Brookings Institution.

    Google Scholar 

  • Lubienski, S. T., & Lubienski, C. (2006). School sector and academic achievement: A multilevel analysis of NAEP mathematics data. American Educational Research Journal, 43, 651–698.

    Article  Google Scholar 

  • MacArthur, C. A., Graham, S., & Fitzgerald, J. (Eds.). (2006). Handbook of writing research. New York, NY: Guilford.

    Google Scholar 

  • Massell, D. (1994). Setting standards in mathematics and social studies. Education and Urban Society, 26, 118–140.

    Article  Google Scholar 

  • McCarthey, S. J. (2008). The impact of No Child Left Behind on teachers’ writing instruction. Written Communication, 25, 462–505.

    Article  Google Scholar 

  • McCarthey, S. J., & Ro, Y. S. (2011). Approaches to writing instruction. Pedagogies: An International Journal, 6, 273–295.

    Article  Google Scholar 

  • McLaughlin, M. W., & Shepard, L. A. (1995). Improving education through standards-based reform: A report by the National Academy of Education Panel on Standards-Based Reform. Stanford, CA: National Academy of Education.

    Google Scholar 

  • McMillan, J. H., Myran, S., & Workman, D. (1999). The impact of mandated statewide testing on teachers’ classroom assessment and instructional practices. In Paper presented at the Annual meeting of the American Educational Research Association., Montreal, Quebec, Canada. (ERIC Document Reproduction Service No. ED431041).

  • Moats, L., Foorman, B. R., & Taylor, P. (2006). How quality of writing instruction impacts high-risk fourth graders’ writing. Reading and Writing: An Interdisciplinary Journal, 19, 363–391.

    Article  Google Scholar 

  • National Center for Education Statistics. (2012). The nation’s report card: Writing 2011 (NCES 2012–470). Washington, DC: Institute of Education Sciences, U.S. Department of Education.

    Google Scholar 

  • National Commission on Writing for America’s Families, Schools, and Colleges. (2003). The neglected R: The need for a writing revolution. New York, NY: College Entrance Examination Board. Retrieved February 1, 2008, from http://www.writingcommission.org/prod_downloads/writingcom/neglectedr.pdf.

  • National Commission on Writing for America’s Families, Schools, and Colleges. (2004). Writing: A ticket to work…or a ticket out. A survey of business leaders. New York, NY: College Entrance Examination Board. Retrieved February 1, 2008, from http://www.writingcommission.org/prod_downloads/writingcom/writing-ticket-to-work.pdf.

  • O’Neill, P., Murphy, S., Williamson, M., & Huot, B. (2006). What teachers say about different kinds of mandated state tests. Journal of Writing Assessment, 2, 81–108.

    Google Scholar 

  • Persky, H. R., Daane, M. C., & Jin, Y. (2003). The nation’s report card: Writing 2002. U. S. Department of Education, Institute of Education Sciences. Washington, DC: National Center for Education Statistics.

  • Polikoff, M. S., & Fulmer, G. W. (2013). Refining methods for estimating critical values for an alignment index. Journal of Research on Educational Effectiveness, 6, 380–395.

    Article  Google Scholar 

  • Polikoff, M. S., Porter, A. C., & Smithson, J. (2011). How well aligned are state assessments of student achievement with state content standards? American Educational Research Journal, 48, 965–995.

    Article  Google Scholar 

  • Porter, A. C. (2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3–14.

    Article  Google Scholar 

  • Prior, P. (2006). A sociocultural theory of writing. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 54–66). New York: Guilford.

    Google Scholar 

  • Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage Publications.

    Google Scholar 

  • Roach, A., Elliott, S., & Webb, N. (2005). Alignment of an alternate assessment with state academic standards: Evidence for the content validity of the Wisconsin alternate assessment. Journal of Special Education, 38, 218–231.

    Article  Google Scholar 

  • Rothman, R., Slatterly, J. B., Vranek, J. L., & Resnick, L. B. (2002). Benchmarking and alignment of standards and testing. Los Angeles: Center for the Study of Evaluation, National Center for Research on Evaluation, Standards, and Student Testing.

  • Rowan, B., Camburn, E., & Correnti, R. (2004). Using teacher logs to measure the enacted curriculum: A study of literacy teaching in third-grade classrooms. The Elementary School Journal, 105, 75–101.

    Article  Google Scholar 

  • Rubin, D. B. (1987). Multiple imputation for non-response in surveys. New York, NY: Wiley.

    Book  Google Scholar 

  • Salahu-Din, D., Persky, H., & Miller, J. (2008). The nation’s report card: Writing 2007. U. S. Department of Education, Institute of Education Sciences. Washington, DC: National Center for Education Statistics.

  • Scherff, L., & Piazza, C. (2005). The more things change, the more they stay the same. A survey of high school students’ writing experiences. Research in the Teaching of English, 39, 271–304.

    Google Scholar 

  • Sharple, M., & Pemberton, L. (1992). Representing writing: External representations and the writing process. Oxford, UK: Kluwer Academic Publishers.

    Google Scholar 

  • Snijders, T. A. B., & Bosker, R. J. (2012). Multilevel analysis: An introduction to basic and advanced multilevel modeling (2nd ed.). London, UK: Sage Publications.

    Google Scholar 

  • Stecher, B. M., Barron, S. L., Chun, T., & Ross, K. (2000). The effects of the Washington state education reform on schools and classrooms (RAND report DRU-2263). Santa Monica, CA: RAND Corporation.

    Google Scholar 

  • Stecher, B. M., Barron, S. L., Kaganoff, T., & Goodwin, J. (1998). The effects of standards based assessment on classroom practices: Results of the 1996–1997 RAND survey of Kentucky teachers of mathematics and writing (CRESST Tech. Rep. No. 482). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing (CRESST).

  • Stecher, B., & Borko, H. (2002). Combining surveys and case studies to examine standards-based educational reform (CSE Tech. Rep. No. 565). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

  • Stecher, B., & Chun, T. (2002). School and classroom during two years of educational reform in Washington state (CSE Tech. Rep. No. 550). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing.

  • Stecher, B. M., Hamilton, L. S., & Gonzalez, G. (2003). Working smarter to leave no child behind. Santa Monica, CA: RAND.

    Google Scholar 

  • Taylor, G., Shepard, L., Kinner, F., & Rosenthal, J. (2002). A survey of teachers’ perspectives on high-stakes testing in Colorado: What gets taught, what gets lost. CSE technical report 588. Los Angeles, CA: Center for Research on Evaluation, Standards Student Testing. (ERIC Document Reproduction Service No. ED475139).

  • Troia, G. A. (2006). Writing instruction for students with learning disabilities. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 324–336). New York, NY: Guilford Press.

    Google Scholar 

  • Troia, G. A., & Graham, S. (2017). Use and acceptability of writing adaptations for students with disabilities: Survey of grade 3–8 teachers. Learning Disabilities Research & Practice, 32, 257–269.

    Google Scholar 

  • Troia, G. A., Harbaugh, A. G., Wolbers, K. A., & Lawrence, A. M. (2013). Relationships between writing motivation, writing activity, and writing achievement: Effects of grade, sex, and ability. Reading & Writing: An Interdisciplinary Journal, 26, 17–44.

    Article  Google Scholar 

  • Troia, G. A., Lin, S. C., Cohen, S., & Monroe, B. W. (2011). A year in the writing workshop: Linking writing instruction practices and teachers’ epistemologies and beliefs about writing instruction. The Elementary School Journal, 112, 155–182.

    Article  Google Scholar 

  • Troia, G. A., Lin, S. C., Monroe, B. W., & Cohen, S. (2009). The effects of writing workshop instruction on the performance and motivation of good and poor writers. In G. A. Troia (Ed.), Instruction and Assessment for Struggling Writers: Evidence-Based Practices (pp. 77–112). New York, NY: Guilford Press.

    Google Scholar 

  • Troia, G. A., Olinghouse, N. G., Wilson, J., Stewart, K. A., Mo, Y., Hawkins, L., et al. (2016). The Common Core writing standards: A descriptive study of content and alignment with a sample of former state standards. Reading Horizons Journal, 55(3), 98–141.

    Google Scholar 

  • Troia, G. A., Shankland, R. K., & Wolbers, K. A. (2012). Motivation research in writing: Theoretical and empirical considerations. Reading & Writing Quarterly: Overcoming Learning Difficulties, 28, 5–28.

    Article  Google Scholar 

  • U.S. Department of Education. (2015). Every Student Succeeds Act of 2015, Public Law No. 114-95. Retrieved from http://www.ed.gov/essa.

  • Webb, N. L., Alt, M., Ely, R., & Vesperman, B. (2005). Web alignment tool (WAT): Training Manual version 1.1. Retrieved from http://wat.wceruw.org/index.aspx.

  • Wixson, K. K., Frisk, M. C., Dutro, E., & McDaniel, J. (2002). The alignment of state standards and assessments in elementary reading (CIERA Report No. R-3-024). Ann Arbor, MI: Center for the Improvement of Early Reading Achievement.

    Google Scholar 

Download references

Acknowledgements

This research was supported in part by Grant #R305A100040 from the U.S. Department of Education, Institute of Education Sciences, to Michigan State University. Statements do not necessarily reflect the positions or policies of this agency, and no official endorsement by it should be inferred.

Author information

Authors and Affiliations

  1. Department of Counseling, Educational Psychology and Special Education, Michigan State University, 620 Farm Lane, 343 Erickson Hall, East Lansing, MI, 48824, USA

    Gary A. Troia & Mingcai Zhang

  2. Department of Educational Psychology, University of Connecticut, Storrs, CT, USA

    Natalie G. Olinghouse

  3. College of Education and Human Development, University of Delaware, Newark, DE, USA

    Joshua Wilson

  4. Department of Research, Evaluation, Assessment and Accountability, Minneapolis Public Schools, Minneapolis, MN, USA

    Kelly A. Stewart

  5. National Institute of Statistical Sciences, Washington, DC, USA

    Ya Mo

  6. Department of Elementary Education, Ball State University, Muncie, IN, USA

    Lisa Hawkins

Authors
  1. Gary A. Troia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Natalie G. Olinghouse
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingcai Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joshua Wilson
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kelly A. Stewart
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ya Mo
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lisa Hawkins
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gary A. Troia.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Troia, G.A., Olinghouse, N.G., Zhang, M. et al. Content and alignment of state writing standards and assessments as predictors of student writing achievement: an analysis of 2007 National Assessment of Educational Progress data. Read Writ 31, 835–864 (2018). https://doi.org/10.1007/s11145-017-9816-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11145-017-9816-3

Keywords

Search

Navigation