Abstract
In automated essay scoring (AES) systems, similarity techniques are used to compute the score for student answers. Several methods to compute similarity have emerged over the years. However, only a few of them have been widely used in the AES domain. This work shows the findings of a ten-year review on similarity techniques applied in AES systems and discusses the efficiency and limitations of current methods. In the final review, thirty-four (34) articles published between 2010 and 2020 were included. The metrics used to evaluate the performance of the AES systems are also elaborated. The review was conducted using the Kitchenham method, whereby three research questions were formulated and a search strategy was developed. Research papers were chosen based on pre-defined inclusion and quality assessment criteria. This review has identified two types of similarity techniques used in AES systems. In addition, several methods were used to compute the score for student answers in the AES systems. The similarity computation in AES systems is dependent on several factors, hence many studies have combined multiple methods in a single system yielding good results. In addition, the review found that the quadratic weighted kappa (QWK) was most frequently used to evaluate AES systems.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
Not applicable.
Code availability
Not applicable.
References
Alikaniotis, D., Yannakoudakis, H., & Rei, M. (2016). Automatic text scoring using neural networks. Berlin: Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-1068
Alves dos Santos, J., & Favero, E. (2015). Practical use of a latent semantic analysis (LSA) model for automatic evaluation of written answers. Journal of the Brazilian Computer Society, 21(21). https://doi.org/10.1186/s13173-015-0039-7
Beseiso, M., & Alzahrani, S. (2020). An empirical analysis of BERT embedding for automated essay scoring. International Journal of Advanced Computer Science and Application. https://doi.org/10.14569/IJACSA.2020.0111027
Blood, I. (2011). Automated essay scoring: A literature review. Working Papers in TESOL & Applied Linguistics, 11, 2, 40–64[online]. Retrieved May 25, 2020, from http://journals.tc-library.org/index.php/tesol/article/download/745/470
Britannica. (2021). Syntax. Encyclopedia Britannica. Retrieved June 6, 2021, from https://www.britannica.com/
Brück, T. V. D. B., & Pouly, M. (2019). Text similarity estimation based on word embeddings and matrixnorms for targeted marketing (pp. 1827–1836). Minneapolis: Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1181
Burrows, S., Gurevych, I., & Stein, B. (2014). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-014-0026-8
Cer, D., et al. (2018). Universal sentence encoder for english (pp. 169–174). Brussels: Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-2029
Chai, T., & Draxler, R. R. (2014). Root mean square error (RMSE) or mean absolute error (MAE)? –Arguments against avoiding RMSE in the literature. Geoscientific Model Development, 7, 1247–1250. https://doi.org/10.5194/gmd-7-1247-2014
Chandrasekaran, D., & Mago, V. (2020). Evolution of Semantic Similarity - A Survey. arXiv preprint arXiv:2004.13820, 1(1). https://doi.org/10.1145/3440755
Chen, Z., & Zhou, Y. (2019). Research on Automatic Essay Scoring of Composition Based on CNN and OR. IEEE. https://doi.org/10.1109/ICAIBD.2019.8837007
Citawan, R. S., Mawardi, V. C., & Mulyawan, B. (2018). Automatic Essay Scoring in E-learning System Using LSA Method with N-Gram Feature for Bahasa Indonesia. MATEC Web of Conferences. https://doi.org/10.1051/matecconf/201816401037
Cozma, M., Butnaru, A. M., & Lonescu, R. T. (2018). Automated essay scoring with string kernels and word embeddings. arXiv preprint arXiv: 1804.07954. https://doi.org/10.18653/v1/P18-2080
Driessens, K., & Dˇzeroski, S. (2005). Combining model-based and instance-based learning for first order regression. Bonn, Germany, s.n. https://doi.org/10.1145/1102351.1102376
Edwards, A., Camacho-Collados, J., Ribaupierre, H. d., & Preece, A. (2020). Go Simple and Pre-Train on Domain-Specific Corpora: On the Role of Training Data for Text Classification. Barcelona, Spain (online), s.n., 5522–5529. DOI: https://doi.org/10.18653/v1/2020.coling-main.481
Elalfi, A. E. E., Elgamal, A. F., & Amasha, N. A. (2019). Automated essay scoring using Word2vec and support vector machine. International Journal of Computer Applications (0975 – 8887), 177(25). https://doi.org/10.5120/ijca2019919707
Embeddings. (2020). Machine Learning Crash Course. [Online] Retrieved May 28, 2021, from https://developers.google.com/machine-learning/crash-course/embeddings/video-lecture
Farouk, M. (2019). Measuring sentences similarity: A survey. Indian Journal of Science and Technology, 12 (25). https://doi.org/10.17485/ijst/2019/v12i25/143977
Fauzi, M. A., et al. (2017). Automatic essay scoring system using N-Gram and cosine similarity for gamification based E-learning. Association for Computing Machinery. https://doi.org/10.1145/3133264.3133303
Filighera, A., Steuer, T., & Rensing, C. (2020). Fooling automatic short answer grading systems. Springer Nature Switzerland AG 2020 (pp. 177–190).https://doi.org/10.1007/978-3-030-52237-7_15
Gautam, D., & Rus, V. (2020). Using neural tensor networks for open ended short answer assessment. In: Artificial Intelligence in Education. AIED 2020. Lecture Notes in Computer Science (pp. 191–203). s.l.: Springer. https://doi.org/10.1007/978-3-030-52237-7_16
Ghosh, S., & Fatima, S. (2010). Design of an Automated Essay Grading (AEG) system in Indian context. International Journal of Computer Applications (0975 – 8887), 1(11). https://doi.org/10.5120/237-391
Gomaa, W. H., & Fahmy, A. A. (2013). A survey of text similarity approaches. International Journal of Computer Applications, 68(13). https://doi.org/10.5120/11638-7118
Gomaa, W. H. & Fahmy, A. A. (2014). Arabic short answer scoring with effective feedback for students. International Journal of Computer Applications (0975 –8887), 86(2). https://doi.org/10.5120/14961-3177
Gonzalez-Agirre, A. (2017). Computational models for semantic textual similarity, s.l.: University Of The Basque Country.
Guruji, P. A., Pagnis, M. M., Pawar, S. M., & Kulkarni, P. J. (2015). Evaluation of subjective answers using GLSA enhanced with contextual synonymy. International Journal on Natural Language Computing (IJNLC), 4(1). https://doi.org/10.5121/ijnlc.2015.4105
Gusenbauer, M. (2019). Google scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics, 118(1), 177–214. https://doi.org/10.1007/s11192-018-2958-5
Harispe, S., Sylvie, R., Stefan, J. & Jacky, M., (2015). Semantic similarity from natural language and ontology analysis. s.l.: Synthesis Lectures on Human Language Technologies, 8(1), 1–254. https://doi.org/10.2200/S00639ED1V01Y201504HLT027
Hastings, P., et al. (2012). Assessing the use of multiple sources in student essays. Behavior Research Methods, 44(3), 622–633. https://doi.org/10.3758/s13428-012-0214-0
He, T., (2020). Specificity Prediction for Sentences in Press Releases, s.l.: Uppsala University.
Hendre, M., Mukherjee, P. & Godse, M., (2020a). Utility of neural embeddings in semantic similarity of text data, Evolution in computational intelligence. s.l., Springer. https://doi.org/10.1007/978-981-15-5788-0_21
Hendre, M., Mukherjee, P., Preet, R., & Godse, M., (2020b). Efficacy of deep neural embeddings based semantic similarity in automatic essay evaluation. International Journal of Computing and Digital Systems. https://journal.uob.edu.bh:443/handle/123456789/4035
Hoblos, J. (2020). Experimenting with latent semantic analysis and latent dirichlet allocation on automated essay grading (pp. 1–7). IEEE. https://doi.org/10.1109/SNAMS52053.2020.9336533
Horbach, A., & Zesch, T., (2019). The influence of variance in learner answers on automatic content scoring. Frontiers in Education, 4(28). https://doi.org/10.3389/feduc.2019.00028
Huang, G., Liu, J., Fan, C. & Pan, T., (2018). Off-topic english essay detection model based on hybrid semantic space for automated english essay scoring system. s.l., EDP Sciences. https://doi.org/10.1051/matecconf/201823201035
Hussein, M. A., Hassan, H., & Nassef, M. (2019). Automated language essay scoring systems: A literature review. PeerJ Computer Science. https://doi.org/10.7717/peerj-cs.208
Ikram, A., & Castle, B. (2020). Automated Essay Scoring (AES); A Semantic analysis inspired machine learning approach: An automated essay scoring system using semantic analysis and machine learning is presented in this research (pp. 147–215). London: Association for Computing Machinery. https://doi.org/10.1145/3436756.3437036
Imai, S., Kawai, S., & Nobuhara, H. (2020). Stepwise pathnet: A layer-by-layer knowledge-selection-based transfer learning algorithm. Scientific Reports. https://doi.org/10.1038/s41598-020-64165-3
Islam, M. M., & Haque, A. S. L. (2010). Automated essay scoring using generalized latent semantic analysis. IEEE.
Jakobsson, E. (2019). On the effectiveness of handcrafted and learned features in automated essay scoring. Lund University.
Jurafsky, D., & Martin, J. H., (2018). Speech and Language Processing. In: s.l.:s.n.
Kaggle. (2012). The Hewlett Foundation: Automated Essay Scoring. Retrieved January 1, 2020, from https://www.kaggle.com/c/asap-aes/overview/evaluation
Kakkonen, T., Myller, N., & Sutinen, E., (2006). Applying latent dirichlet allocation to automatic essay grading. In: FinTAL 2006, LNAI 4139, 110–120, Springer-Verlag Berlin Heidelberg 2006. s.l.:s.n. https://doi.org/10.1007/11816508_13
Kakkonen, T., Myller, N., Sutinen, E., & Timonen, J. (2008). Comparison of dimension reduction methods for automated essay grading. Educational Technology & Society, 11(3), 275–288.
Ke, Z., & Ng, V., (2019). Automated essay scoring: A survey of the state of the art (pp. 6300–6308). International Joint Conferences on Artificial Intelligence. https://doi.org/10.24963/ijcai.2019/879
Kitchenham, B. & Charters, S., (2007). Guidelines for performing systematic literature reviews in software engineering (EBSE 2007-001), s.l.: Keele University and Durham University Joint Report.
Kohila, R., & Arunesh, D. K. (2016). Text Mining: Text Similarity measure for news articles based on string-based approach. Global Journal of Engineering Science and Research Management. https://doi.org/10.5281/zenodo.57373.
Kowsari, K. et al. (2019). Text classification algorithms: A survey. Information, 10(1). https://doi.org/10.3390/info10040150
Kulkarni, C., Socher, R., Bernstein, M. S., & Klemmer, S. R. (2014). Scaling short-answer grading by combining peer assessment with algorithmic scoring. ACM. https://doi.org/10.1145/2556325.2566238
Kumar, V., & Boulanger, D. (2020). Explainable automated essay scoring: Deep learning really has pedagogical value. Front. Educ. https://doi.org/10.3389/feduc.2020.572367
Kumar, Y., et al. (2019). Get IT Scored using AutoSAS- An automated system for scoring short answers. AAAI Press. https://doi.org/10.1609/aaai.v33i01.33019662
Lahitani, A. R., Permanasari, A. E., & Setiawan, N. A. (2016). Cosine similarity to determine similarity measure: Study case in online essay assessment (pp. 1–6). IEEE. https://doi.org/10.1109/CITSM.2016.7577578
Lamsiyah, S., Mahdaouy, A. E., Ouatik, S. E. A., & Espinasse, B., (2019). A supervised method for extractive single document summarization based on sentence embeddings and neural networks. Marrakech, Morocco, hal-02433565. https://doi.org/10.1007/978-3-030-36674-2_8
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
LaVoie, N., et al. (2019). Using latent semantic analysis to score short answer constructed responses: Automated scoring of the consequences test. Educational and Psychological Measurement. https://doi.org/10.1177/0013164419860575
Lee, C. V. D., & Bosch, A. V. D. (2017). Exploring Lexical and Syntactic Features for Language Variety Identification. https://aclanthology.org/W17-1224
Li, Y. & Yan, T., (2017). Word Embedding for Understanding Natural Language: A Survey. In: S. Srinivasan, ed. Guide to big data application. s.l.:Springer. https://doi.org/10.1007/978-3-319-53817-4_4
Li, X. et al., (2018). Coherence-based automated essay scoring using self-attention. s.l. (pp. 386–397). Springer. https://doi.org/10.1007/978-3-030-01716-3_32
Li, B., et al., (2020). On the sentence embeddings from pre-trained language models. arXiv:2011.05864v1.
Liang, G., et al. (2018). Automated essay scoring: A siamese bidirectional LSTM neural network architecture. Symmetry, 10(12), 682. https://doi.org/10.3390/sym10120682
Lilja, M., & Andersson, P. (2018). Automatic essay scoring of Swedish essays using neural networks. Uppsala University.
Link, S. M., (2015). Development and validation of an automated essay scoring engine to assess students’ development across program levels. Iowa State University. https://doi.org/10.31274/etd-180810-4115
Liu, O. L., et al. (2014). Automated scoring of constructed-response science items: Prospects and obstacles. Educational Measurement: Issues and Practice, 33(2), 19–28. https://doi.org/10.1111/emip.12028
Liu, Q., Kusner, M. J. & Blunsom, P. (2020). A survey on contextual embeddings. arXiv:2003.07278v2.
Lundström, D. (2017). Data-efficient transfer learning with pre-trained networks. Linköping University.
M. Syamala, D., & Mittal, H., (2013). Subjective Evaluation using LSA Technique. International Journal of Computers and Distributed Systems.
Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval. Cambridge University Press.
Martin, D. I., & Berry, M. W., (2007). Mathematical foundations behind Latent Semantic Analysis. In: T. K. Landauer, D. S. McNamara, S. D. & W. Kintsch, eds. Handbook of Latent Semantic Analysis. s.l.: Routledge Handbooks Online.
Masaki, U., & Masashi, O., (2020). Robust neural automated essay scoring using item response theory. s.l. Springer. https://doi.org/10.1007/978-3-030-52237-7_44
Mathias, S. & Bhattacharyya, P., (2018). ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores. Miyazaki, Japan, European Language Resources Association (ELRA). https://aclanthology.org/L18-1187
McNamara, D. S., et al. (2015). A hierarchical classification approach to automated essay scoring. Assessing Writing, 23, 35–59. https://doi.org/10.1016/j.asw.2014.09.002
Mesgar, M., & Strube, M. (2018). A neural local coherence model for text quality assessment (pp. 4328–4339). Brussels: Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1464
Mittal, H. & Devi, M. S., (2016). Computerized evaluation of subjective answers using hybrid technique. In: H. Saini, R. Sayal & S. Rawat, eds. Innovations in computer science and engineering. advances in intelligent systems and computing. Springer. https://doi.org/10.1007/978-981-10-0419-3_35
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. (2009). Preferred reporting items for systematic reviews and Meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151(4), 264–269. https://doi.org/10.1136/bmj.b2535
Mugasib, M. H. A., & Baraka, D. R. S. (2015). An ontology-based automated scoring system for short questions. Islamic University of Gaza.
Munir, Riza, L. S., & Mulyadi, A., (2016). An automatic scoring system for essay by using methods combination of term frequency and n-Gram. International Journal of Trend in Research and Development, 3(6). http://www.ijtrd.com/papers/IJTRD5412.pdf
Nadeem, F., Nguyen, H., Liu, Y., & Ostendorf, M. (2019). Automated essay scoring with discourse-aware neural models (pp. 484–493). Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-4450
Nam, D., Yasmin, J., & Zulkernine, F. (2020). Effects of pre-trained word embeddings on text-based deception detection (pp. 437–443). Calgary: IEEE. https://doi.org/10.1109/DASC-PICom-CBDCom-CyberSciTech49142.2020.00083
Ndukwe, I. G., Daniel, B. K., & Amadi, C. E., (2019). A machine learning grading system using chatbots. s.l. (pp. 365–368). Springer. https://doi.org/10.1007/978-3-030-23207-8_67
Oduntan, O. E., & Adeyanju, I. A. (2017). A comparative study of modified principal component analysis and generalized latent semantic analysis approach to automated marking of theoretical-based exams. IOSR Journal of Mobile Computing & Application, 4(2), 31–41. https://doi.org/10.9790/0050-04023141
Oduntan, O. E., Adeyanju, I. A., Olabiyisi, S. O., & Omidiora, E. O. (2015). Evaluation of N-gram text representations for automated essay-type grading systems. International Journal of Applied Information Systems, 9(4), 25–31. https://doi.org/10.5120/ijais15-451394
Oduntan, O. E., Olabiyisi, S. O., Adeyanju, I. A., & Omidiora, E. O. (2016). A modified principal component analysis approach to automated essay-type grading (pp. 94–98). IEEE. https://doi.org/10.1109/FTC.2016.7821595
Oduntan, O. E., Adeyanju, I., Falohun, A. S., & Obe, O. O. (2018). A comparative analysis of Euclidean distance and cosine similarity measure for automated essay type grading. Journal of Engineering and Applied Sciences, 13(11), 4198–4204. https://doi.org/10.3923/jeasci.2018.4198.4204
Palma, D., & Atkinson, J. (2018). Coherence-based automatic essay assessment. IEEE Intelligent Systems, 25, 26–36. https://doi.org/10.1109/MIS.2018.2877278
Pradhan, N., Gyanchandani, M. & Wadhvani, R., (2015). A Review on Text Similarity Technique used in IR and its Application. International Journal of Computer Applications (0975 –8887), 120(9). https://doi.org/10.5120/ijais15-451394
Pramukantoro, E. S., & Fauzi, M. A. (2016). Comparative analysis of string similarity and corpus-based similarity for automatic essay scoring system on e-learning gamification (pp. 149–155). IEEE. https://doi.org/10.1109/ICACSIS.2016.7872785
Prasetya, D. D., Wibawa, A. P. & Hirashima, T., (2018). The performance of text similarity algorithms. International Journal of Advances in Intelligent Informatics, 4(1), 63–69. https://doi.org/10.26555/ijain.v4i1.152
Pribadi, F. S. et al., (2017). Automatic Short Answer Scoring Using Words Overlapping Methods. s.l.. AIP Publishing LLC. https://doi.org/10.1063/1.4976906
Pribadi, F. S., Permanasari, A. E., & Adji, T. B. (2018). Short answer scoring system using automatic reference answer generation and geometric average normalized-longest common subsequence (GAN-LCS). Education and Information Technologies, 23, 2855–2866. https://doi.org/10.1007/s10639-018-9745-z
Qiu, X. et al. (2020). Pre-trained models for natural language processing: A survey. arXiv:2003.08271v3 [cs.CL].
Rababah, H., & Al-Taani, A. (2017). An automated scoring approach for Arabic short answers essay questions (pp. 697–702). IEEE. https://doi.org/10.1109/ICITECH.2017.8079930
Rahman, M. S. (2020). Development of text analytics for debriefing reflection essays, s.l. University Of Texas Arlington. http://hdl.handle.net/10106/29134
Rothe, S., Narayan, S., & Severyn, A. (2020). Leveraging Pre-trained Checkpoints for Sequence Generation Tasks. Transactions of the Association for Computational Linguistics, 8, 264–280. https://doi.org/10.1162/tacl_a_00313
Roy, S., Dandapat, S., Nagesh, A., & Narahari, Y. (2016). Wisdom of students: A consistent automatic short answer grading technique (pp. 178–187). NLP Association of India.
Saha, S. K., & Ch, D. R. (2019). Development of a practical system for computerized evaluation of descriptive answers of middle school level students. Interactive Learning Environments. https://doi.org/10.1080/10494820.2019.1651743
Saha, S., & Gupta, R. (2020). Adopting computer-assisted assessment in evaluation of handwritten answer books: An experimental study. Education and Information Technologies, 25, 4845–4860. https://doi.org/10.1007/s10639-020-10192-6
Sakhapara, A,. et al., (2019). Subjective answer grader system based on machine learning. Soft Computing and Signal Processing, 347–355.https://doi.org/10.1007/978-981-13-3393-4_36
scikit-learn, d., 2020. Scikit Learn. Retrieved June 23, 2020, from https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html
Seifried, E., (2016). Improving learning and teaching at universities: The potential of applying automatic essay scoring with latent semantic analysis. Heidelberg. https://doi.org/10.11588/heidok.00021837
Sendra, M., et al. (2016). Enhanced Latent Semantic Analysis by considering mistyped words in automated essay scoring. IEEE. https://doi.org/10.1109/IAC.2016.7905734
Shaker, A., & Hüllermeie, E. (2012). IBLStreams: A system for instance-based classification and regression on data streams. Evolving Systems, 3(4), 235–249. https://doi.org/10.1007/s12530-012-9059-0
Smolentzov, A. (2012). Automated essay scoring: scoring essays in Swedish, s.l.: Stockholms University.
Sokolova, M., Japkowicz, N. & Szpakowicz, S., (2006). Beyond Accuracy, F-score and ROC: a Family of Discriminant Measures for Performance Evaluation. Advances in Artificial Intelligence. AI 2006. Lecture Notes in Computer Science, 4304, 1015–1021. https://doi.org/10.1007/11941439_114
Srihari, S. et al., (2006). Automatic Scoring of Handwritten Essays using Latent Semantic Analysis. In: Document Analysis Systems VII. Center of Excellence for Document Analysis and Recognition (CEDAR) Department of Computer Science and Engineering University at Buffalo: Springer Berlin Heidelberg. https://doi.org/10.1007/11669487_7
Stefanovič, P., Kurasova, O., & Štrimaitis, R. (2019). The N-Grams based text similarity detection approach using self-organizing maps and similarity measure. Applied Science, 9(9), 1870. https://doi.org/10.3390/app9091870
Steimel, K., & Riordan, B. (2020). Towards instance-based content scoring with pre-trained transformer models. s.l., s.n.
Suleman, R. M., & Korkontzelo, I., (2021). Extending latent semantic analysis to manage its syntactic blindness. Expert Systems with Applications, 165(1). https://doi.org/10.1016/j.eswa.2020.114130
Suzen, N., Gorbana, A. N., Levesley, J., & Mirkes, E. M., (2020). Automatic short answer grading and feedback. s.l., Procedia Computer Science, 169, 726–743.https://doi.org/10.1016/j.procs.2020.02.171
Taghipour, K., & Ng, H. T. (2016). A Neural Approach to Automated Essay Scoring (pp. 1882–1891). Austin: Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1193
Tashu, T. M., & Horváth, T. (2020). Smart Score-Short Answer Scoring Made Easy Using Sem-LSH (pp. 145–149). IEEE: San Diego. https://doi.org/10.1109/ICSC.2020.00028
Tashu, T. M., Julius P., E. & Tomas, H., (2019). Intelligent On-line Exam Management and Evaluation System. s.l., Springer. https://doi.org/10.1007/978-3-030-22244-4_14
Uto, M. & Okano, M., (2020). Robust neural automated essay scoring using item response theory. s.l., Springer. https://doi.org/10.1007/978-3-030-52237-7_44
Vlachos, M., (2017). Similarity measures. In: C. Sammut & G. Webb, eds. Encyclopedia of machine learning and data mining. Springer. https://doi.org/10.1007/978-1-4899-7687-1_766
Vuk, B. & Dragan, B., (2015). Using part-of-speech tags as deep-syntax indicators in determining short-text semantic similarity. Computer Science and Information Systems, 12(1). https://doi.org/10.2298/CSIS131127082B
Wang, J., & Dong, Y. (2020). Measurement of text similarity: A survey. Information, 11(9), 421. https://doi.org/10.3390/info11090421
Wang, Z., Liu, J., & Dong, R. (2018). Intelligent Auto-grading System (pp. 430–435). Nanjing: IEEE. https://doi.org/10.1109/CCIS.2018.8691244
Wang, B., et al. (2019). Evaluating word embedding models: Methods and experimental results. arXiv:1901.09785v2 [cs.CL]. https://doi.org/10.1017/ATSIP.2019.12
Wedisa, M., et al. (2020). Automated essay type paper marking system. Sri Lanka (Online), Retrieved May 20, 2021, from http://www.erepo.lib.uwu.ac.lk/handle/123456789/5728
Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement Issues and Practices, 31(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x
Yang, R., et al. (2020). Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking (online). Association for Computational Linguistics.
Yannakoudakis, H., & Cummin, R. (2015). Evaluating the performance of automated text scoring systems (pp. 213–223). Association for Computational Linguistics. https://doi.org/10.3115/v1/W15-0625
Zhao, S., et al. (2017). A memory-augmented neural model for automated grading. ACM. https://doi.org/10.1145/3051457.3053982
Zhuang, F., et al. (2021). A Comprehensive Survey on Transfer Learning (pp. 43–76). IEEE. https://doi.org/10.1109/JPROC.2020.3004555
Zhu, X., Li, T., & Melo, G. D. (2018). Exploring semantic properties of sentence embeddings (pp. 632–637). Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-2100
Zupanc, K., & Bosnic, Z. (2020). Improvement of automated essay grading by grouping similar graders. Fundamenta Informaticae, 172(3), 239–259. https://doi.org/10.3233/FI-2020-1904
Author information
Authors and Affiliations
Contributions
V.Ramnarain-Seetohul: Conceptualization, Methodology, Investigation, Formal analysis, Writing—original draft, Writing—review & editing.
V.Bassoo: Conceptualization, Methodology, Writing—review & editing, Supervision.
Y. Rosunally: Conceptualization, Methodology, Writing—review & editing, Supervision.
Corresponding author
Ethics declarations
Conflicts of interest/Competing interests
The authors have no conflicts of interest to declare.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ramnarain-Seetohul, V., Bassoo, V. & Rosunally, Y. Similarity measures in automated essay scoring systems: A ten-year review. Educ Inf Technol 27, 5573–5604 (2022). https://doi.org/10.1007/s10639-021-10838-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-021-10838-z