Journal of Science Education and Technology

, Volume 21, Issue 1, pp 183–196

Transforming Biology Assessment with Machine Learning: Automated Scoring of Written Evolutionary Explanations

Article

DOI: 10.1007/s10956-011-9300-9

Cite this article as:
Nehm, R.H., Ha, M. & Mayfield, E. J Sci Educ Technol (2012) 21: 183. doi:10.1007/s10956-011-9300-9

Abstract

This study explored the use of machine learning to automatically evaluate the accuracy of students’ written explanations of evolutionary change. Performance of the Summarization Integrated Development Environment (SIDE) program was compared to human expert scoring using a corpus of 2,260 evolutionary explanations written by 565 undergraduate students in response to two different evolution instruments (the EGALT-F and EGALT-P) that contained prompts that differed in various surface features (such as species and traits). We tested human-SIDE scoring correspondence under a series of different training and testing conditions, using Kappa inter-rater agreement values of greater than 0.80 as a performance benchmark. In addition, we examined the effects of response length on scoring success; that is, whether SIDE scoring models functioned with comparable success on short and long responses. We found that SIDE performance was most effective when scoring models were built and tested at the individual item level and that performance degraded when suites of items or entire instruments were used to build and test scoring models. Overall, SIDE was found to be a powerful and cost-effective tool for assessing student knowledge and performance in a complex science domain.

Keywords

Machine learning SIDE Text analysis Assessment Computers Evolution Explanation 

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  1. 1.School of Teaching and LearningThe Ohio State UniversityColumbusUSA
  2. 2.Language Technologies InstituteCarnegie Mellon UniversityPittsburghUSA

Personalised recommendations