Reducing Annotation Effort in Automatic Essay Evaluation Using Locality Sensitive Hashing

  • Tsegaye Misikir TashuEmail author
  • Dávid Szabó
  • Tomáš Horváth
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11528)


Automated essay evaluation systems use machine learning models to predict the score for an essay. For such, a training essay set is required which is usually created by human requiring time-consuming effort. Popular choice for scoring is a nearest neighbor model which requires on-line computation of nearest neighbors to a given essay. This is, however, a time-consuming task. In this work, we propose to use locality sensitive hashing that helps to select a small subset of a large set of essays such that it will likely contain the nearest neighbors for a given essay. We provided experiments on real-world data sets provided by Kaggle. According to the experimental results, it is possible to achieve good performance on scoring by using the proposed approach. The proposed approach is efficient with regard to time complexity. Also, it works well in case of a small number of training essays labeled by human and gives comparable results to the case when a large essay sets are used.


Locality Sensitive Hashing Automatic essay scoring Similarity search 


  1. 1.
    Heilman, M., Madnani, N.: The impact of training data on automated short answer scoring performance. In: Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 81–85 (2015)Google Scholar
  2. 2.
    Brooks, M., Basu, S., Jacobs, C., Vanderwende, L.: Divide and correct: using clusters to grade short answers at scale. In: The First ACM Conference on Learning @ Scale Conference, pp. 89–98. ACM, New York (2014)Google Scholar
  3. 3.
    Zesch, T., Heilman, M., Cahill, A.: Reducing annotation efforts in supervised short answer scoring. In: Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 124–132 (2015)Google Scholar
  4. 4.
    Slaney, M., Casey, M.: Locality-sensitive hashing for finding nearest neighbors [Lecture Notes]. IEEE Signal Process. Mag. 25, 128–131 (2008)CrossRefGoogle Scholar
  5. 5.
    Kim, Y.B., Reilly, U.O.: Large-scale physiological waveform retrieval via locality-sensitive hashing, pp. 5829–5833 (2015)Google Scholar
  6. 6.
    Horbach, A., Palmer, A., Wolska, M.: Finding a tradeoff between accuracy and rater’s workload in grading clustered short answers. In: The 9th Language Resources and Evaluation Conference (LREC), pp. 588–595 (2014)Google Scholar
  7. 7.
    Basu, S., Jacobs, C., Vanderwende, L.: Powergrading: a clustering approach to amplify human effort for short answer grading. Trans. ACL (2013)Google Scholar
  8. 8.
    Misikir Tashu, T., Horvath, T.: Pair-wise: automatic essay evaluation using word mover’s distance. In: 10th International Conference on Computer Supported Education, CSEDU, vol. 2, pp. 59–66. SciTePress (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Faculty of Informatics, Department of Data Science and Engineering, Telekom Innovation LaboratoriesELTE-Eötvös Loránd UniversityBudapestHungary
  2. 2.Faculty of Informatics, 3in Research GroupELTE-Eötvös Loránd UniversityMartonvásárHungary
  3. 3.Faculty of Science, Institute of Computer SciencePavol Jozef Šafárik UniversityKošiceSlovakia

Personalised recommendations