Abstract
This paper presents a process for automatically extracting a fine-grained semantic representation of a learner’s response to a tutor’s question. The representation can be extracted using available natural language processing technologies and it allows a detailed assessment of the learner’s understanding and consequently will support the evaluation of tutoring pedagogy that is dependent on such a fine-grained assessment. We describe a system to assess student answers at this fine-grained level that utilizes features extracted from the automatically generated representations. The system classifies answers to indicate the student’s apparent understanding of each of the low-level facets of a known reference answer. It achieves an accuracy on these fine-grained decisions of 76% on within-domain assessment and 69% out of domain.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Briscoe, E., Carroll, J., Graham, J., Copestake, A.: Relational evaluation schemes. In: Proc. Beyond PARSEVAL Workshop at LREC (2002)
Callear, D., Jerrams-Smith, J., Soh, V.: CAA of short non-MCQ answers. In: Proc. CAA (2001)
Cohen, J.: A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20, 37–46 (1960)
Gildea, D., Jurafsky, D.: Automatic labeling of semantic roles. CL 28(3), 245–288 (2002)
Glickman, O., Dagan, I., Koppel, M.: Web Based Probabilistic Textual Entailment. In: Proc. PASCAL RTE Workshop (2005)
Graesser, A.C., Hu, X., Susarla, S., Harter, D., Person, N.K., Louwerse, M., Olde, B.: AutoTutor: An Intelligent Tutor and Conversational Tutoring Scaffold. In: Proc. AIED (2001)
Jordan, P.W., Makatchev, M., VanLehn, K.: Combining competing language understanding approaches in an intelligent tutoring system. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds.) ITS 2004. LNCS, vol. 3220. Springer, Heidelberg (2004)
Kipper, K., Dang, H.T., Palmer, M.: Class-Based Construction of a Verb Lexicon. In: Proc. AAAI (2000)
Kohavi, R.: The power of decision tables. In: Lavrač, N., Wrobel, S. (eds.) ECML 1995. LNCS, vol. 912. Springer, Heidelberg (1995)
Lawrence Hall of Science: Assessing Science Knowledge (ASK), UC Berkeley, NSF-0242510 (2006)
Leacock, C.: Scoring free-response automatically: A case study of a large-scale Assessment. Examens 1(3) (2004)
Lin, D., Pantel, P.: Discovery of inference rules for Question Answering. Natural Language Engineering 7(4), 343–360 (2001)
Mitchell, T., Russell, T., Broomhead, P., Aldridge, N.: Towards Robust Computerized Marking of Free-Text Responses. In: Proc. CAA (2002)
Nielsen, R.D.: Learner Answer Assessment in Intelligent Tutoring Systems. Ph.D. thesis, Dept. of Computer Science and Institute of Cognitive Science, CU Boulder (2008)
Nielsen, R.D., Ward, W., Martin, J., Palmer, M.: Annotating Students’ Understanding of Science Concepts. In: Proc. Language Resources and Evaluation Conference (2008)
Nivre, J., Scholz, M.: Deterministic Dependency Parsing of English Text. In: Proc. COLING (2004)
Nivre, J., Hall, J., Nilsson, J., Eryigit, G., Marinov, S.: Labeled Pseudo-Projective Dependency Parsing with Support Vector Machines. In: Proc. CoNLL (2006)
Palmer, M., Gildea, D., Kingsbury, P.: The proposition bank: An annotated corpus of semantic roles. CL (2005)
Peters, S., Bratt, E.O., Clark, B., Pon-Barry, H., Schultz, K.: Intelligent Systems for Training Damage Control Assistants. In: Proc. Interservice/Industry Training, Simulation, and Education Conference (2004)
Pulman, S.G., Sukkarieh, J.Z.: Automatic Short Answer Marking. In: Proc. Workshop on Building Educational Applications Using NLP, ACL (2005)
Quinlan, J.R.: C4.5: Programs for Machine Learning. Morgan Kaufmann (1993)
Roll, I., Baker, R.S., Aleven, V., McLaren, B.M., Koedinger, K.R.: Modeling Students’ Metacognitive Errors in Two Intelligent Tutoring Systems. In: Proc. User Modeling (2005)
Rosé, C.P., Roque, A., Bhembe, D., VanLehn, K.: A hybrid text classification approach for analysis of student essays. In: Dignum, F.P.M. (ed.) ACL 2003. LNCS (LNAI), vol. 2922. Springer, Heidelberg (2004)
Turney, P.D.: Mining the web for synonyms: PMI-IR versus LSA on TOEFL. In: Flach, P.A., De Raedt, L. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 491–502. Springer, Heidelberg (2001)
VanLehn, K., Lynch, C., Schulze, K., Shapiro, J.A., Shelby, R., Taylor, L., Treacy, D., Weinstein, A., Wintersgill, M.: The Andes physics tutoring system: Five years of evaluations. In: Proc. AIED (2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Nielsen, R.D., Ward, W., Martin, J.H. (2008). Automatic Generation of Fine-Grained Representations of Learner Response Semantics. In: Woolf, B.P., Aïmeur, E., Nkambou, R., Lajoie, S. (eds) Intelligent Tutoring Systems. ITS 2008. Lecture Notes in Computer Science, vol 5091. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-69132-7_22
Download citation
DOI: https://doi.org/10.1007/978-3-540-69132-7_22
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-69130-3
Online ISBN: 978-3-540-69132-7
eBook Packages: Computer ScienceComputer Science (R0)