Advertisement

Using Automated Indices of Cohesion to Evaluate an Intelligent Tutoring System and an Automated Writing Evaluation System

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7926)

Abstract

We present an evaluation of the Writing Pal (W-Pal) intelligent tutoring system (ITS) and the W-Pal automated writing evaluation (AWE) system through the use ofcomputational indices related to text cohesion. Sixty-four students participated in this study. Each student was assigned to either the W-Pal ITS condition or the W-Pal AWE condition. The W-Pal ITS includes strategy instruction, game-based practice, and essay-based practice with automated feedback. In the ITS condition, students received strategy training and wrote and revised one essay in each of the 8 training sessions. In the AWE condition, students only interacted with the essay writing and feedback tools. These students wrote and revised two essays in each of the 8 sessions. Indices of local and global cohesion reported by the computational tools Coh-Metrix and the Writing Assessment Tool (WAT) were used to investigate pretest and posttest writing gains. For both the ITS and the AWE systems, training led to the increased use of global cohesion features in essay writing. This study demonstrates that automated indices of text cohesion can be used to evaluate the effects of ITSs and AWE systems and further demonstrates how text cohesion develops as a result of instruction, writing, and automated feedback.

Keywords

Cohesion Intelligent Tutoring Systems Natural Language Processing Corpus Linguistics Computational Linguistics Writing Pedagogy 
This is a preview of subscription content, log in to check access.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    National Commission on Writing: The Neglected “R”. College Entrance Examination Board, New York (2003)Google Scholar
  2. 2.
    Kellogg, R., Raulerson, B.: Improving the Writing Skills of College Students. Psychonomic Bulletin and Review 14, 237–242 (2007)CrossRefGoogle Scholar
  3. 3.
    Graham, S., Perin, D.: A meta-analysis of writing instruction for adolescent students. Journal of Educational Psychology 99, 445–476 (2007)CrossRefGoogle Scholar
  4. 4.
    McGarrell, H., Verbeem, J.: Motivating revision of drafts through formative feedback. ELT Journal 61, 228–236 (2007)CrossRefGoogle Scholar
  5. 5.
    Devillez, R.: Writing: Step by Step. Kendall Hunt, Dubuque (2003)Google Scholar
  6. 6.
    Golightly, K., Sanders, G.: Writing and Reading in the Disciplines, vol. 2. Pearson, New Jersey (2000)Google Scholar
  7. 7.
    McNamara, D., Raine, R., Roscoe, R., Crossley, S., Jackson, G.T., Dai, J., Cai, Z., Renner, A., Brandon, R., Weston, J., Dempsey, K., Carney, D., Sullivan, S., Kim, L., Rus, V., Floyd, R., McCarthy, P., Graesser, A.: The Writing-Pal: Natural Language Algorithms to Support Intelligent Tutoring on Writing Strategies. In: McCarthy, P.M., Boonthum-Denecke, C. (eds.) Applied Natural Language Processing and Content Analysis: Identification, Investigation, and Resolution, pp. 298–311. IGI Global, Hershey (2012)Google Scholar
  8. 8.
    Graesser, A., McNamara, D., Louwerse, M., Cai, Z.: Coh-Metrix: Analysis of Text on Cohesion and Language. Behavioral Research Methods, Instruments and Computers 36, 193–202 (2004)CrossRefGoogle Scholar
  9. 9.
    McNamara, D., Crossley, S., Roscoe, R.: Natural Language Processing in an Intelligent Writing Strategy Tutoring System. Behavioral Research Methods, Instruments and Computers (2012) (Advance online publication)Google Scholar
  10. 10.
    McNamara, D., Kintsch, E., Songer, N., Kintsch, W.: Are Good Texts Always Better? Interactions of Text Coherence, Background Knowledge, and Levels of Understanding in Learning from Text. Cognition and Instruction 14, 1–43 (1996)CrossRefGoogle Scholar
  11. 11.
    Crossley, S., McNamara, D.: Cohesion, Coherence, and Expert Evaluations of Writing Proficiency. In: Ohlsson, S., Catrambone, R. (eds.) Proceedings of the 32nd Annual Conference of the Cognitive Science Society, pp. 984–989. Cognitive Science Society, Austin (2010)Google Scholar
  12. 12.
    Crossley, S., McNamara, D.: Text Coherence and Judgments of Essay Quality: Models of Quality and Coherence. In: Carlson, L., Hoelscher, C., Shipley, T.F. (eds.) Proceedings of the 29th Annual Conference of the Cognitive Science Society, pp. 1236–1241. Cognitive Science Society, Austin (2011)Google Scholar
  13. 13.
    Crossley, S., Weston, J., Sullivan, S., McNamara, D.: The Development of Writing Proficiency as a Function of Grade Level: A Linguistic Analysis. Written Communication 28, 282–311 (2011)CrossRefGoogle Scholar
  14. 14.
    Crossley, S., McNamara, D.: Predicting Second Language Writing Proficiency: The Roles of Cohesion and Linguistic Sophistication. Journal of Research in Reading 53, 115–136 (2012)CrossRefGoogle Scholar
  15. 15.
    Johnstone, K.M., Ashbaugh, H., Warfield, T.D.: Effects of repeated practice and contextual-writing experiences on college students’ writing skills. Journal of Educational Psychology 94, 305–315 (2002)CrossRefGoogle Scholar
  16. 16.
    Kellogg, R.T., Raulerson, B.A.: Improving the writing skills of college students. Psychonomic Bulletin & Review 13(2), 237–242 (2007)CrossRefGoogle Scholar
  17. 17.
    Graham, S., Harris, K.R.: The role of self-regulation and transcription skills in writing and writing development. Educational Psychologist 35, 3–12 (2000)CrossRefGoogle Scholar
  18. 18.
    Grimes, D., Warschauer, W.: Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. Journal of Technology, Learning, and Assessment 8, 4–43 (2010)Google Scholar
  19. 19.
    Roscoe, R., Kugler, D., Crossley, S., Weston, J., McNamara, D.: Developing pedagogically-guided threshold algorithms for intelligent automated essay feedback. In: McCarthy, P., Youngblood, Y. (eds.) Proceedings of the 25th International Florida Artificial Intelligence Research Society (FLAIRS) Conference, pp. 466–471. The AAAI Press, Menlo Park (2012)Google Scholar
  20. 20.
    Crossley, S., Roscoe, R., McNamara, D.: Using Natural Language Processing Algorithms to Detect Changes in Student Writing in an Intelligent Tutoring System. Manuscript Submitted to the 26th International Florida Artificial Intelligence Research Society Conference (2013)Google Scholar
  21. 21.
    Hempelmann, C., Dufty, D., McCarthy, P., Graesser, A., Cai, Z., McNamara, D.: Using LSA to Automatically Identify Givenness and Newness of Noun Phrases in Written Discourse. In: Bara, B.G., Bucciarelli, M. (eds.) Proceedings of the 27th Annual Conference of the Cognitive Science Society. Cognitive Science Society, Mahwah (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Department of Applied Linguistics/ESLGeorgia State UniversityAtlantaUSA
  2. 2.Learning Sciences InstituteArizona State UniversityTempeUSA

Personalised recommendations

Cite paper

Buy options