Abstract
This paper compares the accuracy of several variations of the Bleu algorithm when applied to automatically evaluating student essays. The different configurations include closed-class word removal, stemming, two baseline word-sense disambiguation procedures, and translating the texts into a simple semantic representation. We also prove empirically that the accuracy is kept when the student answers are translated automatically. Although none of the representations clearly outperform the others, some conclusions are drawn from the results.
This work has been sponsored by CICYT, project number TIC2001-0685-C02-01.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Page, E.B.: The use of computer in analyzing student essays. International review of education 14 (1968)
Whittington, D., Hunt, H.: Approaches to the computerized assessment of free text responses. In: Proceedings of the Int. CAA Conference (1999)
Mitchell, T., Russell, T., Broomhead, P., Aldridge, N.: Towards robust computerised marking of free-text responses. In: Proceedings of the 6th International Computer Assisted Assessment (CAA) Conference, Loughborough, UK (2002)
Laham, D.: Automated content assessment of text using Latent Semantic Analysis to simulate human cognition. Ph.D. thesis, University of Colorado, Boulder (2000)
Burstein, J., Kukich, K., Wolff, S., Chi, L., Chodorow, M.: Enriching automated essay scoring using discourse marking. In: Proceedings of the Workshop on Discourse Relations and Discourse Marking, ACL, Montreal, Canada (1998)
Burstein, J., Leacock, C., Swartz, R.: Automated evaluation of essay and short answers. In: Proceedings of the Int. CAA Conference (2001)
Valenti, S., Neri, F., Cucchiarelli, A.: An overview of current research on automated essay grading. Journal of I.T. Education 2, 319–330 (2003)
Papineni, K., Roukos, S., Ward, T., Zhu, W.: Bleu: a method for automatic evaluation of machine translation (2001)
Pérez, D., Alfonseca, E., Rodríguez, P.: Application of the BLEU method for evaluating free-text answers in an e-learning environment. In: Proceedings of the Language Resources and Evaluation Conference (LREC 2004) (2004)
Pérez, D., Alfonseca, E., Rodríguez, P.: Upper bounds and extension of the Bleu algorithm applied to assessing student essays. In: IAEA 2004 Conference (2004)
Lin, C.Y., Hovy, E.H.: Automatic evaluation of summaries using n-gram co-occurrence statistics. In: Proceedings of 2003 Language Technology Conference (HLT-NAACL 2003) (2003)
Lin, C.Y.: Rouge working note v. 1.3.1 (2004)
Fellbaum, C.: Analysis of a handtagging task. In: Proceedings of ANLP 1997 Workshop on Tagging Text with Lexical Semantics: Why, What, and How? Washington D.C., USA (1997)
Vossen, P.: EuroWordNet - A Multilingual Database with Lexical Semantic Networks. Kluwer Academic Publishers, Dordrecht (1998)
Alfonseca, E.: Wraetlic user guide version 1.0 (2003)
Foltz, P., Laham, D., Landauer, T.: The intelligent essay assessor: Applications to educational technology. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning (1999)
Rudner, L., Liang, T.: Automated essay scoring using bayes’ theorem. In: Proceedings of the annual meeting of the National Council on Measurement in Education (2002)
Carro, R.M., Pulido, E., Rodríguez, P.: Dynamic generation of adaptive internet-based courses. Journal of Network and Computer Applications 22, 249–257 (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Alfonseca, E., Pérez, D. (2004). Automatic Assessment of Open Ended Questions with a Bleu-Inspired Algorithm and Shallow NLP. In: Vicedo, J.L., Martínez-Barco, P., Muńoz, R., Saiz Noeda, M. (eds) Advances in Natural Language Processing. EsTAL 2004. Lecture Notes in Computer Science(), vol 3230. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-30228-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-540-30228-5_3
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-23498-2
Online ISBN: 978-3-540-30228-5
eBook Packages: Springer Book Archive