Abstract
Coherence is a crucial aspect of evaluating text readability and can be assessed through two primary factors when evaluating an essay in a scoring scenario. The first factor is logical coherence, characterized by the appropriate use of discourse connectives and the establishment of logical relationships between sentences. The second factor is the appropriateness of punctuation, as inappropriate punctuation can lead to confused sentence structure. To address these concerns, we propose a coherence scoring model consisting of a regression model with two feature extractors: a local coherence discriminative model and a punctuation correction model. We employ gradient-boosting regression trees as the regression model and impose monotonicity constraints on the input features. The results show that our proposed model better generalizes unseen data. The model achieved third place in track 1 of NLPCC 2023 shared task 7. Additionally, we briefly introduce our solution for the remaining tracks, which achieves second place for track 2 and first place for both track 3 and track 4.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Jurafsky, D., Martin, J. H.: Speech and Language Processing, 3rd edn. (Draft of Jan 7, 2023) (2023)
Huang, L., Ye, Z., Qin, J., Lin, L., Liang, X.: GRADE: automatic graph-enhanced coherence metric for evaluating open-domain dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9230–9240 (2020)
Christensen, J., Soderland, S., Etzioni, O.: Towards coherent multi-document summarization. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1163–1173 (2013)
Miltsakaki, E., Kukich, K.: Evaluation of text coherence for electronic essay scoring systems. Nat. Lang. Eng. 10(1), 25–55 (2004)
Burstein, J., Tetreault, J., Andreyev, S.: Using entity-based features to model coherence in student essays. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 681–684 (2010)
Cahill, A., Evanini, K.: Natural language processing for writing and speaking. In: Handbook of Automated Scoring, pp. 69–92. Chapman and Hall/CRC, Boca Raton (2020)
Xu, P., et al.: A cross-domain transferable neural coherence model. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 678–687 (2019)
Zhang, H., et al.: PaddleSpeech: an easy-to-use all-in-one speech toolkit. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: System Demonstrations, pp. 114–123 (2022)
Halliday, M. A. K., Hasan, R.: Cohesion in English (No. 9). Routledge (1976)
Morris, J., Hirst, G.: Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Comput. Linguist. 17(1), 21–48 (1991)
Hearst, M.A.: Text Tiling: segmenting text into multi-paragraph subtopic passages. Comput. Linguist. 23(1), 33–64 (1997)
Foltz, P.W., Kintsch, W., Landauer, T.K.: The measurement of textual coherence with latent semantic analysis. Discourse Process. 25(2–3), 285–307 (1998)
Li, J., Li, R., Hovy, E.: Recursive deep models for discourse parsing. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2061–2069 (2014)
Mesgar, M., Strube, M.: A neural local coherence model for text quality assessment. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4328–4339 (2018)
Zhang, M., Hao, S., Xu, Y., Ke, D., Peng, H.: Automated essay scoring using incremental latent semantic analysis. J. Softw. 9(2), 429–436 (2014)
Fu, R., Wang, D., Wang, S., Hu, G., Liu, T.: Elegart sentence recognition for automated essay scoring. J. Chin. Inf. Process. 32(6), 10 (2018)
Song, W., Song, Z., Liu, L., Fu, R.: Hierarchical multi-task learning for organization evaluation of argumentative student essays. In: IJCAI, pp. 3875–3881 (2020)
Song, W., Zhang, K., Fu, R., Liu, L., Liu, T., Cheng, M.: Multi-stage pre-training for automated Chinese essay scoring. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6723–6733 (2020)
He, Y., Jiang, F., Chu, X., Li, P.: Automated Chinese essay scoring from multiple traits. In: Proceedings of the 29th International Conference on Computational Linguistics, pp. 3007–3016 (2022)
Cui, Y., Che, W., Liu, T., Qin, B., Wang, S., Hu, G.: Revisiting pre-trained models for Chinese natural language processing. Findings Assoc. Comput. Linguist.: EMNLP 2020, 657–668 (2020)
Federico, M., Cettolo, M., Bentivogli, L., Michael, P., Sebastian, S.: Overview of the IWSLT 2012 evaluation campaign. In: Proceedings of the International Workshop on Spoken Language Translation (IWSLT), pp. 12–33 (2012)
Iter, D., Guu, K., Lansing, L., Jurafsky, D.: Pretraining with contrastive sentence objectives improves discourse performance of language models. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4859–4870 (2020)
Long, W., Webber, B., Xiong, D.: TED-CDB: a large-scale Chinese discourse relation dataset on ted talks. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2793–2803 (2020)
Acknowledgements
This work is supported by NSFC (62206070), the Innovation Fund Project of the Engineering Research Center of Integration and Application of Digital Learning Technology, Ministry of Education (1221014, 1221052), and National Key R &D Program of China (2021YFF0901005).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, C., Zhang, H., Zhao, Y., Lai, Y. (2023). Improving the Generalization Ability in Essay Coherence Evaluation Through Monotonic Constraints. In: Liu, F., Duan, N., Xu, Q., Hong, Y. (eds) Natural Language Processing and Chinese Computing. NLPCC 2023. Lecture Notes in Computer Science(), vol 14304. Springer, Cham. https://doi.org/10.1007/978-3-031-44699-3_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-44699-3_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44698-6
Online ISBN: 978-3-031-44699-3
eBook Packages: Computer ScienceComputer Science (R0)
