Advertisement

Reading and Writing

, Volume 21, Issue 1–2, pp 153–175 | Cite as

Evaluating fifth- and sixth-grade students’ expository writing: task development, scoring, and psychometric issues

  • Jonna M. KulikowichEmail author
  • Linda H. Mason
  • Scott W. Brown
Article

Abstract

Drawing from multiple theoretical frameworks representing cognitive and educational psychology, we present a writing task and scoring system for measurement of students’ informative writing. Participants in this study were 72 fifth- and sixth-grade students who wrote compositions describing real-world problems and how mathematics, science, and social studies information could be used to solve those problems. Of the 72 students, 69 were able to craft a cohesive response that not only demonstrated planning in writing structure but also elaboration of relevant knowledge in one or more domains. Many-facet Rasch Modeling (MFRM) techniques were used to examine the reliability and validity of scores for the writing rating scale. Additionally, comparison of fifth- and sixth-grade responses supported the validity of scores, as did the results of a correlational analysis with scores from an overall interest measure. Recommendations for improving writing scoring systems based on the findings of this investigation are provided.

Keywords

Assessment Knowledge Metacognition Self-regulation Writing 

References

  1. Alexander P. A. (1997). Mapping the multidimensional nature of domain learning: The interplay of cognitive, motivational, strategic forces. In M. L. Maehr, & P. R. Pintrich (Eds.), Advances in motivation and achievement (Vol. 10, pp. 213–250). Greenwich, CT: JAI Press.Google Scholar
  2. Alexander, P. A. (2003a). Profiling the developing reader: The interplay of knowledge, interest, and strategic processing. In C. M. Fairbanks, J. Worthy, B. Maloch, J. V. Hoffman, & D. L. Schallert (Eds.), The fifty-first yearbook of the national reading conference. Oak Creek, WI: National Reading Conference.Google Scholar
  3. Alexander, P. A. (2003b). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32, 10–14.CrossRefGoogle Scholar
  4. Alexander, P., Graham, S., & Harris, K. R. (1998). A perspective on strategy research: Prospect and progress. Educational Psychology Review, 10, 129–154.CrossRefGoogle Scholar
  5. Alexander, P. A., & Jetton, T. L. (2000). Learning from text: A multidimensional and developmental perspective. In M. L. Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. III, pp. 285–310). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  6. Alexander, P. A., & Judy, J. E. (1988). The interaction of domain-specific and strategic knowledge in academic performance. Review of Educational Research, 58, 375–404.CrossRefGoogle Scholar
  7. Alexander, P. A., Murphy, P. K., & Woods, B. S. (1996). Of squalls and fathoms: Navigating the seas of educational innovation. Educational Researcher, 25(3), 31–36, 39.Google Scholar
  8. Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561–573.CrossRefGoogle Scholar
  9. Bennett R. E., & Ward W. C. (Eds.). (1993). Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  10. Bereiter, C., & Scardamalia, M. (1982). From conversation to composition: The role of instruction in a developmental process. In R. Glaser (Ed.), Advances in instructional psychology (Vol. 2, pp. 1–64). Hillsdale, NJ: Erlbaum.Google Scholar
  11. Bereiter, C., & Scardamalia, M. (1987). The psychology of written composition. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  12. Brennan R. L. (1992). Elements of generalizability theory. Iowa City, IA: ACT.Google Scholar
  13. Brown, A. L. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In F. Weinert, & R. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 65–116). Hillsdale, NJ: Lawerence Erlbaum Associates.Google Scholar
  14. Camp, R. (1993). The place of portfolios in our changing views of writing assessment. In R. E. Bennett, & W. C. Ward (Eds.), Construction versus choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment (pp. 183–212). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  15. Cognition, Technology Group at Vanderbilt [CTGV]. (1991). Anchored instruction and its relationship to situated cognition. Educational Researcher, 19(6), 2–10.CrossRefGoogle Scholar
  16. CTGV (1992). The Jasper Series as an example of anchored instruction: Theory, program, description, and assessment data. Educational Psychologist, 27, 291–315.CrossRefGoogle Scholar
  17. Engelhard, G., Jr. (1992). The measurement of writing ability with a many-faceted Rasch model. Applied Measurement in Education, 5, 171–191.CrossRefGoogle Scholar
  18. Flower, L., & Hayes, J. (1980). The dynamics of composing: Making plans and juggling constraints. In L. Gregg, & R. Steinberg (Eds.), Cognitive process in writing (pp. 31–50). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  19. Graham S. (2006). Strategy instruction and the teaching of writing: A meta-analysis. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 187–207). New York: Guilford Press.Google Scholar
  20. Graham, S., & Harris, K. R. (2003). Students with learning disabilities and the process of writing: A meta-analysis of SRSD studies. In H. L. Swanson, K. R. Harris, & S. Graham (Eds.), Handbook of learning disabilities (pp. 323–344). New York: Guilford Press.Google Scholar
  21. Graham, S., Harris, K. R., & Reid, R. (1992). Developing self-regulated learners. Focus on Exceptional Children, 24, 1–16.Google Scholar
  22. Greenwald, E. A., Persky, H. R., Campbell, J. R., & Mazzeo, J. (1999). NAEP 1998 writing report card for the nation and the states. Education Statistics Quarterly, 1, 23–28.Google Scholar
  23. Harris, K. R., & Graham, S. (1999). Programmatic intervention research: Illustrations from the evolution of self-regulated strategy development. Learning Disabilities Quarterly, 22, 251–262.CrossRefGoogle Scholar
  24. Harris, K. R., Graham, S., & Mason, L. H. (2003) Self-regulated strategy development in the classroom: Part of a balanced approach to writing instruction for students with disabilities. Focus on Exceptional Children, 35, 1–16.Google Scholar
  25. Hout, B., & Neal, M. (2006). Writing assessment: A techno-history. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 417–432). New York: Guilford Press.Google Scholar
  26. Keating, D. P. (1990). Charting pathways to the development of expertise. Educational Psychologist, 25, 243–267.CrossRefGoogle Scholar
  27. Lawless, K. A., & Kulikowich, J. M. (2006). Domain knowledge and individual interest: The effects of academic level and specialization in statistics and psychology. Contemporary Educational Psychology, 31, 30–43.CrossRefGoogle Scholar
  28. Linacre, J. M. (1989). Multi-facet Rasch measurement. Chicago: MESA.Google Scholar
  29. Linacre, J. M. (1996). FACETS: A computer program for analysis of examinations with multiple facets. Chicago: MESA.Google Scholar
  30. Linacre, J. M. (1999). Investigation rating scale category utility. Journal of Outcome Measurement, 3, 103–122.Google Scholar
  31. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (pp. 13–105). New York: American Council on Education/Macmillan.Google Scholar
  32. Pressley, M., & Harris, K. (2006). Cognitive strategies instruction: From basic research to classroom instruction. In P. A. Alexander, & P. H. Winne (Eds.), Handbook of educational psychology (2nd ed., pp.265–286). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  33. Saenz, L. M., & Fuchs, L. S. (2002). Examining the reading difficulty of secondary students with learning disabilities: Expository versus narrative text. Remedial and Special Education, 23, 31–41.CrossRefGoogle Scholar
  34. Sanders, T. J., & Schilperoord, J. (2006). Text structure as a window on the cognition of writing: How text analysis provides insights in writing products, writing process. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 386–402). New York: Guilford Press.Google Scholar
  35. Scardamalia, M., & Bereiter, C. (1991). Literate expertise. In K. A. Ericsson, & J. Smith (Eds.), Toward a general theory of expertise: Prospects and limits (pp. 172–194). Cambridge, UK: Cambridge University Press.Google Scholar
  36. Shermis, M. D., Burstein, J., & Leacock, C. (2006). Applications of computers in assessment, analysis of writing. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 403–416). New York: Guilford Press.Google Scholar
  37. Sitko, B. M. (1998). Knowing how to write: Metacognition and writing instruction. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 93–115). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  38. Smith, E. V., Jr., & Kulikowich, J. M. (2004). An application of generalizability theory and many-facet Rasch measurement using a complex problem-solving skills assessment. Educational and Psychological Measurement, 64, 617–639.CrossRefGoogle Scholar
  39. Torrance, M., & Galbraith, D. (2006). The processing demands of writing. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 67–82). New York: Guilford Press.Google Scholar
  40. Walvoord, B. E., & Anderson, V. J. (1998). Effective grading: A tool for learning and assessment. San Francisco: Jossey-Bass.Google Scholar
  41. White, E. (1985). Teaching and assessing writing. San Francisco: Jossey-Bass.Google Scholar
  42. Wolfe, E. W., & Dozier, H. (2000). Development of a scale for measuring invasive plant environmentalism. Journal of Applied Measurement, 1, 219–237.Google Scholar

Copyright information

© Springer Science+Business Media, Inc. 2007

Authors and Affiliations

  • Jonna M. Kulikowich
    • 1
    Email author
  • Linda H. Mason
    • 1
  • Scott W. Brown
    • 2
  1. 1.Department of Educational PsychologyThe Pennsylvania State UniversityUniversity ParkUSA
  2. 2.University of ConnecticutConnecticutUSA

Personalised recommendations