Automatic Generation of Fine-Grained Representations of Learner Response Semantics

  • Rodney D. Nielsen
  • Wayne Ward
  • James H. Martin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5091)

Abstract

This paper presents a process for automatically extracting a fine-grained semantic representation of a learner’s response to a tutor’s question. The representation can be extracted using available natural language processing technologies and it allows a detailed assessment of the learner’s understanding and consequently will support the evaluation of tutoring pedagogy that is dependent on such a fine-grained assessment. We describe a system to assess student answers at this fine-grained level that utilizes features extracted from the automatically generated representations. The system classifies answers to indicate the student’s apparent understanding of each of the low-level facets of a known reference answer. It achieves an accuracy on these fine-grained decisions of 76% on within-domain assessment and 69% out of domain.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Briscoe, E., Carroll, J., Graham, J., Copestake, A.: Relational evaluation schemes. In: Proc. Beyond PARSEVAL Workshop at LREC (2002)Google Scholar
  2. 2.
    Callear, D., Jerrams-Smith, J., Soh, V.: CAA of short non-MCQ answers. In: Proc. CAA (2001)Google Scholar
  3. 3.
    Cohen, J.: A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 37–46 (1960)CrossRefGoogle Scholar
  4. 4.
    Gildea, D., Jurafsky, D.: Automatic labeling of semantic roles. CL 28(3), 245–288 (2002)Google Scholar
  5. 5.
    Glickman, O., Dagan, I., Koppel, M.: Web Based Probabilistic Textual Entailment. In: Proc. PASCAL RTE Workshop (2005)Google Scholar
  6. 6.
    Graesser, A.C., Hu, X., Susarla, S., Harter, D., Person, N.K., Louwerse, M., Olde, B.: AutoTutor: An Intelligent Tutor and Conversational Tutoring Scaffold. In: Proc. AIED (2001)Google Scholar
  7. 7.
    Jordan, P.W., Makatchev, M., VanLehn, K.: Combining competing language understanding approaches in an intelligent tutoring system. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220. Springer, Heidelberg (2004)Google Scholar
  8. 8.
    Kipper, K., Dang, H.T., Palmer, M.: Class-Based Construction of a Verb Lexicon. In: Proc. AAAI (2000)Google Scholar
  9. 9.
    Kohavi, R.: The power of decision tables. In: Lavrač, N., Wrobel, S. (eds.) ECML 1995. LNCS, vol. 912. Springer, Heidelberg (1995)Google Scholar
  10. 10.
    Lawrence Hall of Science: Assessing Science Knowledge (ASK), UC Berkeley, NSF-0242510 (2006)Google Scholar
  11. 11.
    Leacock, C.: Scoring free-response automatically: A case study of a large-scale Assessment. Examens 1(3) (2004)Google Scholar
  12. 12.
    Lin, D., Pantel, P.: Discovery of inference rules for Question Answering. Natural Language Engineering 7(4), 343–360 (2001)CrossRefGoogle Scholar
  13. 13.
    Mitchell, T., Russell, T., Broomhead, P., Aldridge, N.: Towards Robust Computerized Marking of Free-Text Responses. In: Proc. CAA (2002)Google Scholar
  14. 14.
    Nielsen, R.D.: Learner Answer Assessment in Intelligent Tutoring Systems. Ph.D. thesis, Dept. of Computer Science and Institute of Cognitive Science, CU Boulder (2008)Google Scholar
  15. 15.
    Nielsen, R.D., Ward, W., Martin, J., Palmer, M.: Annotating Students’ Understanding of Science Concepts. In: Proc. Language Resources and Evaluation Conference (2008)Google Scholar
  16. 16.
    Nivre, J., Scholz, M.: Deterministic Dependency Parsing of English Text. In: Proc. COLING (2004)Google Scholar
  17. 17.
    Nivre, J., Hall, J., Nilsson, J., Eryigit, G., Marinov, S.: Labeled Pseudo-Projective Dependency Parsing with Support Vector Machines. In: Proc. CoNLL (2006)Google Scholar
  18. 18.
    Palmer, M., Gildea, D., Kingsbury, P.: The proposition bank: An annotated corpus of semantic roles. CL (2005)Google Scholar
  19. 19.
    Peters, S., Bratt, E.O., Clark, B., Pon-Barry, H., Schultz, K.: Intelligent Systems for Training Damage Control Assistants. In: Proc. Interservice/Industry Training, Simulation, and Education Conference (2004)Google Scholar
  20. 20.
    Pulman, S.G., Sukkarieh, J.Z.: Automatic Short Answer Marking. In: Proc. Workshop on Building Educational Applications Using NLP, ACL (2005)Google Scholar
  21. 21.
    Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann (1993)Google Scholar
  22. 22.
    Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., Koedinger, K.R.: Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. In: Proc. User Modeling (2005)Google Scholar
  23. 23.
    Rosé, C.P., Roque, A., Bhembe, D., VanLehn, K.: A hybrid text classification approach for analysis of student essays. In: Dignum, F.P.M. (ed.) ACL 2003. LNCS (LNAI), vol. 2922. Springer, Heidelberg (2004)Google Scholar
  24. 24.
    Turney, P.D.: Mining the web for synonyms: PMI-IR versus LSA on TOEFL. In: Flach, P.A., De Raedt, L. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 491–502. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  25. 25.
    VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., Wintersgill, M.: The Andes physics tutoring system: Five years of evaluations. In: Proc. AIED (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Rodney D. Nielsen
    • 1
    • 2
  • Wayne Ward
    • 1
    • 2
  • James H. Martin
    • 1
  1. 1.Center for Computational Language and Education Research, CU Boulder 
  2. 2.Boulder Language Technologies 

Personalised recommendations