Abstract
Essay generation falls under very rare and challenging cases of deep learning in which input data is way lower than the output data. It focuses on the generation of written texts in natural human language from some known semantic representation of topic information. Aim is to generate informative, diverse, and topic-consistent essays based on different topics. We implemented three different artificial intelligence models, i.e., topic average long short-term memory (TAV-LSTM), topic-attention LSTM (TAT-LSTM), multi-topic aware LSTM (MTA-LSTM) to find out the best suitable technology for the natural language generator. However, TAV and TAT LSTMs showed some valuable results, MTA-LSTM gave the most suitable outcomes. Experimental results verify that the MTA-LSTM model is able to generate topic-consistent text, diverse and essentially makes development as compared to strong baselines. After the implementation of the models, it was found that MTA-LSTM outperformed TAT-LSTM and TAV-LSTM in almost every metric. Overall MTA-LSTM outperformed TAT-LSTM and TAV-LSTM by 14.79 and 20.82% in human evaluation. Also, it performed better in BLEU score evaluation by 27.47% in TAV-LSTM and by 11.53% in TAT-LSTM.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
T.B. Hashimoto, H. Zhang, P. Liang, Unifying human and statistical evaluation for natural language generation. arXiv preprint arXiv:1904.02792 (2019)
J. Allen, Natural Language Understanding (Pearson, 1995)
T.N.T. Abd Rahim, Z. Abd Aziz, R.H. Ab Rauf, N. Shamsudin, Automated exam question generator using genetic algorithm, in 2017 IEEE Conference on e-Learning, e-Management and e-Services (IC3e), Nov 2017 (IEEE), pp. 12–17
https://www.kaggle.com/krsoninikhil/pual-graham-essays?select=paul_graham_essay.txt
I.F. Moawad, D.S. Fadl, M.M. Aref, Rich semantic representation based approach for text generation
C. Callison-Burch, P. Koehn, C. Monz, K. Peterson, M. Przybocki, O. Zaidan, Findings of the 2010 joint workshop on statistical machine translation and metrics for machine translation, in Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR, July 2010, pp. 17–53
E. Lloret, H. Saggion, M. Palomar, Experiments on summary-based opinion classification, in Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, June 2010, pp. 107–115
G. Paltoglou, M. Thelwall, A study of information retrieval weighting schemes for sentiment analysis, in Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, July 2010, pp. 1386–1395
D. Sayed, M. Aref, I. Fathy, Text generation model from rich semantic representations, in Egypt Society of Language Engineering ESOLEC, 68-67 (2011)
J. Valls, S. Ontañón, Natural language generation through case-based text modification, in International Conference on Case-Based Reasoning, Sept 2012 (Springer, Berlin, Heidelberg, 2012), pp. 443–457
R. Rajkumar, D. Espinosa, M. White, The OSU system for surface realization at generation challenges 2011, in Proceedings of the 13th European Workshop on Natural Language Generation, Sept 2011, pp. 236–238
E.M. De Novais, T.D. Tadeu, I. Paraboni, Improved text generation using n-gram statistics, in Ibero-American Conference on Artificial Intelligence, Nov 2010 (Springer, Berlin, Heidelberg, 2010), pp. 316–325
A. Wiratmo, C. Fatichah, Indonesian short essay scoring using transfer learning dependency tree LSTM
C. Retoré, Variable types for meaning assembly: a logical syntax for generic noun phrases introduced by most. Rech. linguistiques Vincennes 41, 83–102 (2012)
B. Qin, D. Tang, X. Geng, D. Ning, J. Liu, T. Liu, A planning based framework for essay generation. arXiv preprint arXiv:1512.05919 (2015)
E. Clark, A. Celikyilmaz, N.A. Smith, Sentence mover’s similarity: automatic evaluation for multi-sentence texts, in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, July 2019, pp. 2748–2760
C. Kiddon, L. Zettlemoyer, Y. Choi, Globally coherent text generation with neural checklist models, in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Nov 2016, pp. 329–339
S. Welleck, I. Kulikov, S. Roller, E. Dinan, K. Cho, J. Weston, Neural text generation with unlikelihood training. arXiv preprint arXiv:1908.04319 (2019)
X. Feng, M. Liu, J. Liu, B. Qin, Y. Sun, T. Liu, Topic-to-essay generation with neural networks, in IJCAI, July 2018, pp. 4078–4084
E. Wulczyn, C. Jacoby, Softmax RNN for short text classification
L.R. Medsker, L.C. Jain, Recurrent neural networks. Des. Appl. 5 (2001)
P. Rodriguez, J. Wiles, J.L. Elman, A recurrent neural network that learns to count. Connect. Sci. 11(1), 5–40 (1999)
P. Yang, L. Li, F. Luo, T. Liu, X. Sun, Enhancing topic-to-essay generation with external commonsense knowledge, in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, July 2019, pp. 2002–2012
A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, A.A. Bharath, Generative adversarial networks: An overview. IEEE Signal Process. Mag. 35(1), 53–65 (2018)
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial networks. Commun. ACM 63(11), 139–144 (2020)
M. Arjovsky, L. Bottou, Towards principled methods for training generative adversarial networks. arXiv preprint arXiv:1701.04862 (2017)
R. Dey, F.M. Salemt, Gate-variants of gated recurrent unit (GRU) neural networks, in 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), Aug 2017 (IEEE, 2017), pp. 1597–1600
J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014)
F. Lin, X. Ma, Y. Chen, J. Zhou, B. Liu, PC-SAN: pretraining-based contextual self-attention model for topic essay generation. KSII Trans. Internet Inf. Syst. (TIIS) 14(8), 3168–3186 (2020)
T. Zhang, V. Kishore, F. Wu, K.Q. Weinberger, Y. Artzi, Bertscore: evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)
I. Tenney, D. Das, E. Pavlick, BERT rediscovers the classical NLP pipeline. arXiv preprint arXiv:1905.05950 (2019)
J. Pennington, R. Socher, C.D. Manning, Glove: global vectors for word representation, in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Oct 2014, pp. 1532–1543
W. Zhao, M. Peyrard, F. Liu, Y. Gao, C.M. Meyer, S. Eger, Moverscore: text generation evaluating with contextualized embeddings and earth mover distance. arXiv preprint arXiv:1909.02622 (2019)
K.W. Church, Word2Vec. Nat. Lang. Eng. 23(1), 155–162 (2017)
X. Rong, word2vec parameter learning explained. arXiv preprint arXiv:1411.2738 (2014)
D. Herremans, C.H. Chuan, Modeling musical context with word2vec. arXiv preprint arXiv:1706.09088 (2017)
H.V. Nguyen, L. Bai, Cosine similarity metric learning for face verification, in Asian Conference on Computer Vision, 8 Nov 2010 (Springer, Berlin, Heidelberg, 2010), pp. 709–720
N. Dehak, R. Dehak, J.R. Glass, D.A. Reynolds, P. Kenny, Cosine similarity scoring without score normalization techniques, in Odyssey, June 2010, p. 15
F. Rahutomo, T. Kitasuka, M. Aritsugi, Semantic cosine similarity, in The 7th International Student Conference on Advanced Science and Technology ICAST, Oct 2012, vol. 4, no. 1
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need. Adv. Neural Inf. Process. Syst. 5998–6008 (2017)
S. Vashishth, S. Upadhyay, G.S. Tomar, M. Faruqui, Attention interpretability across NLP tasks. arXiv preprint arXiv:1909.11218 (2019)
K. Papineni, S. Roukos, T. Ward, W.J. Zhu, BLEU: a method for automatic evaluation of machine translation, in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, July 2002, pp. 311–318
X. He, L. Deng, Maximum expected bleu training of phrase and lexicon translation models, in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), July 2012, pp. 292–301
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Gaur, M., Arora, M., Prakash, V., Kumar, Y., Gupta, K., Nagrath, P. (2022). Analyzing Natural Language Essay Generator Models Using Long Short-Term Memory Neural Networks. In: Khanna, A., Gupta, D., Bhattacharyya, S., Hassanien, A.E., Anand, S., Jaiswal, A. (eds) International Conference on Innovative Computing and Communications. Advances in Intelligent Systems and Computing, vol 1394. Springer, Singapore. https://doi.org/10.1007/978-981-16-3071-2_21
Download citation
DOI: https://doi.org/10.1007/978-981-16-3071-2_21
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-16-3070-5
Online ISBN: 978-981-16-3071-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)