Abstract
One of the goals remaining in Intelligent Tutoring Systems is to create applications to evaluate open-ended text in a human-like manner. The aim of this study is to produce the design for a fully automatic summary evaluation system that could stand for human-like summarisation assessment. In order to gain this goal, an empirical study has been carried out to identify underlying cognitive processes. The studied sample is compound by 15 expert raters on summary evaluation with different professional backgrounds in education. Pearson’s correlation has been calculated to see inter-rater agreement level and stepwise linear regression to observe predicting variables and weights. In addition, interviews with subjects provided qualitative information that could not be acquired numerically. Based on this research, a design of a fully automatic summary evaluation environment has been described.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Aleven, V., Koedinger, K.R., Popescu, O.: A Tutorial Dialog System to Support Self- Explanation: Evaluation and Open Questions. In: Kay, J. (ed.) Artificial Intelligence in Education, pp. 35–46. IOS Press, Sydney (2003)
Foltz, P.W., Gilliam, S., Kendall, S.: Supporting content-based feedback in online writing evaluation with LSA. In: Interactive Learning Environments (2000)
Graesser, A., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N., the Tutoring Research Group: Using Latent Semantic Analysis to evaluate the contributions of students in Auto-Tutor. In: Interactive Leraning Environments, pp. 129–148 (2000)
Ikastolen_Elkartea. OSTADAR DBH-1 Euskara eta Literatura Irakaslearen Gida 3. zehaztapen maila. In: Ikastolen Elkartea (2003)
Kintsch, E., Steinhart, D., Stahl, G.: the LSA research group. Developing summarisation skills through the use of LSA-based feedback (2000)
Landauer, T.K., Dumais, S.T.: A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction and representation of knowledge. In: Psychological Review, pp. 211–240 (1997)
Landauer, T.K., Foltz, P.W., Laham, D.: Introduction to Latent Semantic Analysis. In: Discourse Processes, pp. 259–284 (1998)
Lin, C.-Y., Hovy, E.: Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics. In: Human Technology Conference. Edmonton-Canada, pp. 150–157 (2003)
Long, J., Harding-Esch, E.: Summary and recall of text in first and second languages. In: Gerver, D. (ed.) Language Interpretation and Communication, pp. 273–287. Plenum Press, New York (1978)
Manning, C., Schutze, H.: Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge (1999)
Rickel, J., Lesh, N., Rich, C., Sidner, C.L., Gertner, A.: Collaborative Discourse Theory as a Foundation for Tutorial Dialogue. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 542–551. Springer, Heidelberg (2002)
Robertson, J., Wiemer-Hastings, P.: Feedback on Children’s Stories Via Multiple Interface Agents. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, p. 923. Springer, Heidelberg (2002)
Rosé, C.P., Gaydos, A., Hall, B.S., Roque, A., VanLehn, K.: Overcoming the Knowledge Engineering Bottelneck for Understanding Student Language Input. In: Kay, J. (ed.) Artificial Intelligence in Education, IOS Press, Amsterdam (2003)
Sherrard, C.: Teaching students to summarize: Applying textlinguistics. In: Systems, pp. 1–11 (1989)
VanLehn, K., Jordan, P.W., Rose, C.P., Bhembe, D., Bottner, D., Gaydos, A., et al.: The Architecture of Why2 Atlas: A Coach for Qualitative Physics Essay Writing. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, p. 158. Springer, Heidelberg (2002)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zipitria, I., Elorriaga, J.A., Arruarte, A., de Ilarraza, A.D. (2004). From Human to Automatic Summary Evaluation. In: Lester, J.C., Vicari, R.M., Paraguaçu, F. (eds) Intelligent Tutoring Systems. ITS 2004. Lecture Notes in Computer Science, vol 3220. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30139-4_41
Download citation
DOI: https://doi.org/10.1007/978-3-540-30139-4_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-22948-3
Online ISBN: 978-3-540-30139-4
eBook Packages: Springer Book Archive