Automated Essay Scoring System Based on Rubric

Abstract

In this paper, we propose an architecture of automated essay scoring system based on rubric, which combines automated scoring with human scoring. Rubrics are valid criteria for grading students’ essays. Our proposed rubric has five evaluation viewpoints “Contents, Structure, Evidence, Style, and Skill” and 25 evaluation items which are subdivided viewpoints. The system is cloud-based application and consists of several tools such as Moodle, R, MeCab, and RedPen. At first, the system automatically scores 11 items included in the Style and Skill such as sentence style, syntax, usage, readability, lexical richness, and so on. Then it predicts scores of Style and Skill from these items’ scores by multiple regression model. It also predicts Contents’ score by the cosine similarity between topics and descriptions. Moreover, our system classifies into five grades “A+, A, B, C, D” as useful information for teachers, by using machine learning techniques such as support vector machine. We try to improve automated scoring algorithms and a variety of input essays in order to improve accuracy of classification over 90%.

Appendices

Appendix 1: Proposed Rubric for Human Scoring

Evaluation Viewpoint	Achievement Level and Scoring
	D (0–1)	C (2–3)	B (4–5)	A (6–7)	A+ (8–9)
[Content] Understanding of the assigned tasks and validity of contents	Misunderstanding the assigned task, or the contents are not related to the topic at all	Understanding the assigned task, but includes some errors	Understanding the assigned task, but the contents are insufficient	Understanding the assigned task, but has some points to improve	Appropriate contents with relevant terms. No need for improvement
[Structure] Logical development	No structure or theoretical development	There is a contradiction in the development of the theory	Although developing theory in order, there are some points to be improved	Although developing theory in order, the theory is not compelling	The theory is compelling and conveying the writer’s understanding
[Evidence] Validity of sources and evidence	It does not show evidence	Demonstrates an attempt to support ideas	The sources to be referenced are inappropriate or unreliable	Uses relevant and reliable sources, but the way of reference is not suitable	Demonstrates the skillful use of high-quality and relevant sources
[Style] Proper usage of grammar and elaboration of sentences	There are some grammatical errors. Many corrections required	Not following the rules. Some corrections required	Almost follow the rules. A few corrections required	Although error-free, some improvement will be better	Virtually error-free and well elaborated. No point to improve
[Skill] Readability and writing skill	The sentences are hard to read. Writing skills are missing	There are several points to be improved, such as the length of sentences	Although sentences can be read generally, some improvement will be better	Easy to read. Rich in vocabulary	Easy to read. Skillfully communicates meaning to readers. Rich in vocabulary

Appendix 2: Proposed Rubric for Automated Scoring

Evaluation Viewpoints	Evaluation Items		Automated Scoring (0–9)
[Content]	1	Similarity between topic and description	Applicable
	2	Presence of keywords	Applicable
	3	Understanding of the writing task	Not applicable
	4	Comprehensive evaluation of contents
	5	Understanding of learning contents
[Structure]	6	Logic level	Not applicable
	7	Validity of opinions and arguments
	8	Division of facts and opinions
	9	Persuasiveness
[Evidence]	10	Quality level of reference material	Not applicable
	11	Relevance of reference material
	12	Validity of reference material
	13	Explanation about tables and figures
	14	Validity of the quantity of citations	Conditionally applicable
[Style]	15	Unification of stylistics	Applicable
	16	Eliminate misused or misspellings
	17	Validity of syntax
	18	Dependency of subject and predicate
	19	Proper punctuation
	20	Eliminate redundancy and double negation
	21	Eliminate notation variability and ambiguity
[Skill]	22	Kanji usage rate	Applicable
	23	Validity of sentence length
	24	Lexical richness
	25	Lexical level