From Human to Automatic Summary Evaluation

  • Iraide Zipitria
  • Jon Ander Elorriaga
  • Ana Arruarte
  • Arantza Diaz de Ilarraza
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3220)


One of the goals remaining in Intelligent Tutoring Systems is to create applications to evaluate open-ended text in a human-like manner. The aim of this study is to produce the design for a fully automatic summary evaluation system that could stand for human-like summarisation assessment. In order to gain this goal, an empirical study has been carried out to identify underlying cognitive processes. The studied sample is compound by 15 expert raters on summary evaluation with different professional backgrounds in education. Pearson’s correlation has been calculated to see inter-rater agreement level and stepwise linear regression to observe predicting variables and weights. In addition, interviews with subjects provided qualitative information that could not be acquired numerically. Based on this research, a design of a fully automatic summary evaluation environment has been described.


Natural Language Processing Latent Semantic Analysis Text Comprehension Intelligent Tutor System Summary Evaluation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Aleven, V., Koedinger, K.R., Popescu, O.: A Tutorial Dialog System to Support Self- Explanation: Evaluation and Open Questions. In: Kay, J. (ed.) Artificial Intelligence in Education, pp. 35–46. IOS Press, Sydney (2003)Google Scholar
  2. 2.
    Foltz, P.W., Gilliam, S., Kendall, S.: Supporting content-based feedback in online writing evaluation with LSA. In: Interactive Learning Environments (2000)Google Scholar
  3. 3.
    Graesser, A., Wiemer-Hastings, P., Wiemer-Hastings, K., Harter, D., Person, N., the Tutoring Research Group: Using Latent Semantic Analysis to evaluate the contributions of students in Auto-Tutor. In: Interactive Leraning Environments, pp. 129–148 (2000)Google Scholar
  4. 4.
    Ikastolen_Elkartea. OSTADAR DBH-1 Euskara eta Literatura Irakaslearen Gida 3. zehaztapen maila. In: Ikastolen Elkartea (2003)Google Scholar
  5. 5.
    Kintsch, E., Steinhart, D., Stahl, G.: the LSA research group. Developing summarisation skills through the use of LSA-based feedback (2000)Google Scholar
  6. 6.
    Landauer, T.K., Dumais, S.T.: A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction and representation of knowledge. In: Psychological Review, pp. 211–240 (1997)Google Scholar
  7. 7.
    Landauer, T.K., Foltz, P.W., Laham, D.: Introduction to Latent Semantic Analysis. In: Discourse Processes, pp. 259–284 (1998)Google Scholar
  8. 8.
    Lin, C.-Y., Hovy, E.: Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics. In: Human Technology Conference. Edmonton-Canada, pp. 150–157 (2003)Google Scholar
  9. 9.
    Long, J., Harding-Esch, E.: Summary and recall of text in first and second languages. In: Gerver, D. (ed.) Language Interpretation and Communication, pp. 273–287. Plenum Press, New York (1978)Google Scholar
  10. 10.
    Manning, C., Schutze, H.: Foundations of Statistical Natural Language Processing. The MIT Press, Cambridge (1999)zbMATHGoogle Scholar
  11. 11.
    Rickel, J., Lesh, N., Rich, C., Sidner, C.L., Gertner, A.: Collaborative Discourse Theory as a Foundation for Tutorial Dialogue. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, pp. 542–551. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  12. 12.
    Robertson, J., Wiemer-Hastings, P.: Feedback on Children’s Stories Via Multiple Interface Agents. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, p. 923. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  13. 13.
    Rosé, C.P., Gaydos, A., Hall, B.S., Roque, A., VanLehn, K.: Overcoming the Knowledge Engineering Bottelneck for Understanding Student Language Input. In: Kay, J. (ed.) Artificial Intelligence in Education, IOS Press, Amsterdam (2003)Google Scholar
  14. 14.
    Sherrard, C.: Teaching students to summarize: Applying textlinguistics. In: Systems, pp. 1–11 (1989)Google Scholar
  15. 15.
    VanLehn, K., Jordan, P.W., Rose, C.P., Bhembe, D., Bottner, D., Gaydos, A., et al.: The Architecture of Why2 Atlas: A Coach for Qualitative Physics Essay Writing. In: Cerri, S.A., Gouardéres, G., Paraguaçu, F. (eds.) ITS 2002. LNCS, vol. 2363, p. 158. Springer, Heidelberg (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2004

Authors and Affiliations

  • Iraide Zipitria
    • 1
    • 2
  • Jon Ander Elorriaga
    • 2
  • Ana Arruarte
    • 2
  • Arantza Diaz de Ilarraza
    • 2
  1. 1.Developmental and Educational Psychology DepartmentUniversity of the Basque Country (UPV/EHU)Donostia
  2. 2.Languages and Information Systems DepartmentUniversity of the Basque Country (UPV/EHU)Donostia

Personalised recommendations