Combining Computational Models of Short Essay Grading for Conceptual Physics Problems

  • M. J. Ventura
  • D. R. Franchescetti
  • P. Pennumatsa
  • A. C. Graesser
  • G. T. Jackson X. Hu
  • Z. Cai
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3220)


The difficulties of grading essays with natural language processing tools are addressed. The present project investigated the effectiveness of combining multiple measures of text similarity to grade essays on conceptual physics problems. Latent semantic analysis (LSA) and a new text similarity metric called Union of Word Neighbors (UWN) were used with other measures to predict expert grades. It appears that the best strategy for grading essays is to use student derived ideal answers and statistical models that accommodate inferences. LSA and the UWN gave near equivalent performance in predicting expert grades when student derived ideal answers served as a comparison for student answers. However, if ideal expert answers are used, explicit symbolic models involving word matching are more suitable to predict expert grades. This study identified some computational constraints on models of natural language processing in intelligent tutoring systems.


Target Word Latent Semantic Analysis Intelligent Tutoring System Cognitive Tutor Student Answer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive tutors: Lessons learned. The Journal of the Learning Sciences 4, 167–207 (1995)CrossRefGoogle Scholar
  2. 2.
    Aleven, V., Koedinger, K.R.: An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science 26, 147–179 (2002)CrossRefGoogle Scholar
  3. 3.
    Burgess, C.: From simple associations to the building blocks of language: Modeling meaning in memory with the HAL model. Behavior Research Methods, Instruments, & Computers 30, 188–198 (1998)CrossRefGoogle Scholar
  4. 4.
    Chi, M.T.H., Siler, S.A., Jeong, H., Yamauchi, T., Hausmann, R.G.: Learning from human tutoring. Cognitive Science 25, 471–533 (2001)CrossRefGoogle Scholar
  5. 5.
    Foltz, P.W., Gilliam, S., Kendall, S.: Supporting content-based feedback in online writing evaluation with LSA. Interactive Learning Environments 8, 111–128 (2000)CrossRefGoogle Scholar
  6. 6.
    Graesser, A.C., Lu, S., Jackson, G.T., Mitchell, H., Ventura, M., Olney, A., Louwerse, M.M.: AutoTutor: A tutor with dialogue in natural language. In: Behavioral Research Methods, Instruments, and Computers (in press)Google Scholar
  7. 7.
    Graesser, A.C., McNamara, D.S., Louwerse, M.M., Cai, Z.: Coh-Metrix: Analysis of text on cohesion and language. In: Behavioral Research Methods, Instruments, and Computers (in press)Google Scholar
  8. 8.
    Graesser, A.C., Person, N., Harter, D., TRG: Teaching tactics and dialog in AutoTutor. International Journal of Artificial Intelligence in Education 12, 257–279 (2001)Google Scholar
  9. 9.
    Graesser, A.C., Person, N.K., Magliano, J.P.: Collaborative dialogue patterns in naturalistic one-on-one tutoring. Applied Cognitive Psychology 9, 359–387 (1995)CrossRefGoogle Scholar
  10. 10.
    Graesser, A.C., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N., Tutoring Research Group: Using latent semantic analysis to evaluate the contributions of students in AutoTutor. Interactive Learning Environments 8, 129–148 (2000)CrossRefGoogle Scholar
  11. 11.
    Hewitt, P.G.: Conceptual physics, 8th edn. Addison-Wesley, Reading (1998)Google Scholar
  12. 12.
    Landauer, T.K., Dumais, S.T.: A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review 104, 211–240 (1997)CrossRefGoogle Scholar
  13. 13.
    Landauer, T.K., Foltz, P.W., Laham, D.: An introduction to latent semantic analysis. Discourse Processes 25, 259–284 (1998)CrossRefGoogle Scholar
  14. 14.
    VanLehn, K., Jones, R.M., Chi, M.T.H.: A model of the self- explanation effect. Journal of the Learning Sciences 2(1), 1–60 (1992)CrossRefGoogle Scholar
  15. 15.
    VanLehn, K., Lynch, C., Taylor, L., Weinstein, A., Shelby, R., Schulze, K., Treacy, D., Wintersgill, M.: In: Cerri, S.A., Gouarderes, G., Paraguacu, F. (eds.) Intelligent Tutoring Systems 2002, pp. 367–376. Springer, Berlin (2002)CrossRefGoogle Scholar
  16. 16.
    Wiemer-Hastings, P., Wiemer-Hastings, K., Graesser, A.: Improving an intelligent tutor’s comprehension of students with Latent Semantic Analysis. In: Lajoie, S.P., Vivet, M. (eds.) Artificial Intelligence in Education, pp. 535–542. IOS Press, Amsterdam (1999)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • M. J. Ventura
    • 1
  • D. R. Franchescetti
    • 1
  • P. Pennumatsa
    • 1
  • A. C. Graesser
    • 1
  • G. T. Jackson X. Hu
    • 1
  • Z. Cai
    • 1
  1. 1.Institute for Intelligent SystemsMemphis

Personalised recommendations