Computers and the Humanities

, Volume 37, Issue 4, pp 389–405 | Cite as

C-rater: Automated Scoring of Short-Answer Questions

  • Claudia Leacock
  • Martin Chodorow

Abstract

C-rater is an automated scoringengine that has been developed to scoreresponses to content-based short answerquestions. It is not simply a stringmatching program – instead it uses predicateargument structure, pronominal reference,morphological analysis and synonyms to assignfull or partial credit to a short answerquestion. C-rater has been used in two studies:National Assessment for Educational Progress(NAEP) and a statewide assessment in Indiana.In both studies, c-rater agreed with humangraders about 84% of the time.

automated scoring content-based scoring short answer scoring 

References

  1. Abney S. (1996). Partial Parsing via Finite-State Cascades. Proceedings of the ESSLLI '96 Robust Parsing Workshop.Google Scholar
  2. Burstein J., Wolff S., Lu C. (1999). Using Lexical Semantic Techniques to Classify Free-responses. In Ide N. and Veronis J. (eds.), The Depth and Breadth of Semantic Lexicons, Kluwer Academic Press.Google Scholar
  3. Burstein J. (2003). The E-rater® Scoring Engine: Automated Essay Scoring with Natural Language Processing. In Shermis M.D. and Burstein J. (eds.), Automated Essay Scoring: A Cross-Disciplinary Perspective, Lawrence Erlbaum, Mahwah, NJ.Google Scholar
  4. Cormen T.H., Leiserson C.E., Rivest R.L., Stein C. (2001). Introduction to Algorithms, Second Edition. The MIT Press, Cambridge, MA.Google Scholar
  5. Elliott S. (2003) Intellemetric: From Here to Validity. In Shermis M.D. and Burstein J. (eds.), Automated Essay Scoring: A Cross-Disciplinary Perspective, Lawrence Erlbaum, Mahwah, NJ.Google Scholar
  6. Fellbaum C. (1998). WordNet: An Electronic Lexical Database. MIT Press, Cambridge.Google Scholar
  7. Fleiss J.L. (1981). Statistical Methods for Rates and Proportions. John Wiley & Sons, New York. pp. 212–236.Google Scholar
  8. Landauer T.K., Laham D., Foltz P. (2003). Automated Scoring and Annotation of Essays with Intelligent Essay Assessor™. In Shermis M.D. and Burstein J. (eds.), Automated Essay Scoring: A Cross-Disciplinary Perspective, Lawrence Erlbaum, Mahwah, NJ.Google Scholar
  9. Landauer T.K., Laham D., Rehder B., Schreiner M.E. (1997). How Well Can Passage Meaning be Derived Without Using Word Order? A Comparison of Latent Semantic Analysis and Humans. In Shafto M.G. and Langley P. (eds.), Proceedings of the 19th Anual Meeting of the Cognitive Science Society, Lawrence Erlbaum, Mahwah, NJ, pp. 412–417.Google Scholar
  10. Kaplan R.M., Bennett R.E. (1994). Using the Free-response Scoring Tool to Automatically Score the Formulating-hypotheses Item. ETS Research Report 04–08.Google Scholar
  11. Kud J.M., Krupka G.R., Rau L.F. (1994). Methods for Categorizing Short Answer Responses. Proceedings of the Educational Testing Service Conference on Natural Language Processing Techniques and Technology in Education and Assessment. Princeton, NJ.Google Scholar
  12. Lin D. (1998). Automatic Retrieval and Clustering of SimilarWords. Proceedings of the 35th Annual Meeting of the Association for Computational Linguistic, Montreal, pp. 898–904.Google Scholar
  13. Morton T.S. (2000). Coreference for NLP applications. Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, Hong Kong.Google Scholar
  14. Salton G., Wong A., Yang C.S. (1975). A Vector Space Model for Automatic Indexing. Communications of the ACM, 18/11, pp. 613–620.Google Scholar
  15. Sandene B., Bennett R., Braswell J., Oranje A. (Forthcoming). Mathematics Online Study: Final Report. National Center for Education Statistics, Washington, DC.Google Scholar
  16. Vigilante R. (1999). Online Computer Scoring of Constructed-response Questions. Journal of Information Technology Impact, 1/2, pp. 57–62.Google Scholar

Copyright information

© Kluwer Academic Publishers 2003

Authors and Affiliations

  • Claudia Leacock
    • 1
  • Martin Chodorow
    • 2
  1. 1.Educational Testing ServicePrincetonUSA
  2. 2.Hunter CollegeCity University of New YorkUSA

Personalised recommendations