Skip to main content
SpringerLink
Log in

Natural Language Generation Using Sequential Models: A Survey

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Natural Language Generation (NLG) is one of the most critical yet challenging tasks in all Natural Language Processing applications. It is a process to automate text generation so that humans can understand its meaning. A handful of research articles published in the literature have described how NLG can produce understandable texts in various languages. The use of sequence-to-sequence modeling powered by deep learning techniques such as Long Term Short Term Memory, Recurrent Neural Networks, and Gated Recurrent Units has received much popularity as text generators. This survey provides a comprehensive overview of text generations and their related techniques, such as statistical, traditional, and neural network-based techniques. Generating text using the sequence-to-sequence model is not a simple task as it needs to handle continuous data, such as images, and discrete information, such as text. Therefore, in this study, we have identified some crucial areas for further research on text generation, such as incorporating a large text dataset, identifying and resolving grammatical errors, and generating extensive sentences or paragraphs. This work has also presented a detailed overview of the activation functions used in deep learning-based models and the evaluation metrics used for text generation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Abbreviations

AI:

Artificial intelligence

ML:

Machine learning

NLP:

Natural language processing

NLG:

Natural language generation

NLU:

Natural language understanding

RNN:

Recurrent neural network

LSTM:

Long short term memory

GRU:

Gated recurrent unit

CNN:

Convolution neural network

POS:

Part of speech

NER:

Named entity recognition

CBOW:

Common bag of word

GloVe:

Global vector

RST:

Rhetorical structure theory

REG:

Referring expression generation

BRNN:

Bi-direction recurrent neural network

ReLU:

Rectified linear activation function

ROUGE:

Recall Oriented Understudy for Gisting Evaluation

BLEU:

Bilingual evaluation understudy

TCN:

Temporal convolution nets

MCW:

Medical College of Wisconsin

LCS:

Longest common subsequence

References

  1. Dethlefs N, Schoene A, Cuayáhuitl H (2021) A divide-and-conquer approach to neural natural language generation from structured data. Neurocomputing 433:300–309. https://doi.org/10.1016/j.neucom.2020.12.083

    Article  Google Scholar 

  2. Cao J (2020) Generating natural language descriptions from tables. IEEE Access 8:46206–46216. https://doi.org/10.1109/ACCESS.2020.2979115

    Article  Google Scholar 

  3. Wolf T et al (2020) Transformers: state-of-the-art natural language processing, pp 38–45

  4. Ruder S (2019) Neural transfer learning for natural language processing

  5. Song M (2021) A study on the predictive analytics powered by the artificial intelligence in the movie industry. Int J Adv smart Converg 10(4):72–83

    Google Scholar 

  6. Weizenbaum J (1983) ELIZA—a computer program for the study of natural language communication between man and machine. Commun ACM 26(1):23–28. https://doi.org/10.1145/357980.357991

    Article  Google Scholar 

  7. Colby KM (1976) Artificial paranoia: a computer simulation of paranoid processes, vol 7, no 1

  8. Angeli G, Liang P, Klein D (2010) A simple domain-independent probabilistic approach to generation. In: EMNLP 2010—conference on empirical methods in natural language processing, proceedings of the conference, pp 502–512

  9. Meister C, Pimentel T, Wiher G, Cotterell R (2022) Typical decoding for natural language generation. 2022, [Online]. Available: http://arxiv.org/abs/2202.00666

  10. McShane M, Leon I (2022) Language generation for broad-coverage, explainable cognitive systems. Adv Cogn Syst X, pp 1–6 [Online]. Available: https://arxiv.org/abs/2201.10422v1

  11. Li Z (2022) Text language classification based on dynamic word vector and attention mechanism. In: 2021 international conference on big data analytics for cyber-physical system in smart city, pp 367–375

  12. Elahi GMM, Yang YH (2022) Online learnable keyframe extraction in videos and its application with semantic word vector in action recognition. Pattern Recognit. https://doi.org/10.1016/j.patcog.2021.108273

    Article  Google Scholar 

  13. Pennington J, Socher R, Manning C (2014) {G}lo{V}e: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing ({EMNLP}), pp 1532–1543. https://doi.org/10.3115/v1/D14-1162

  14. Dharma EM, Gaol FL, Warnars HLHS, Soewito B (2022) The accuracy comparison among Word2Vec, glove, and fasttext towards convolution neural network (CNN) text classification. J Theor Appl Inf Technol 100(2):349–359

    Google Scholar 

  15. Nandanwar AK, Choudhary J (2021) Semantic features with contextual knowledge-based web page categorization using the glove model and stacked bilstm. Symmetry (Basel). https://doi.org/10.3390/sym13101772

    Article  Google Scholar 

  16. Jagfeld G, Jenne S, Vu NT (2018) Sequence-to-sequence models for data-to-text natural language generation: word- vs. character-based processing and output diversity. In: INLG 2018—11th International Natural Language Generation Conference, Proceedings, pp 221–232. https://doi.org/10.18653/v1/w18-6529

  17. Gaur M, Arora M, Prakash V, Kumar Y, Gupta K, Nagrath P (2022) Analyzing natural language essay generator models using long short-term memory neural networks, pp 233–248

  18. Kannan S, Vathsala MK (2022) Mathematical model for application of natural language description in the creation of an animation. In: Emerging research in computing, information, communication and applications, pp 237–251

  19. Shi J, Yang Z, He J, Xu B, Lo D (2022) Can Identifier Splitting Improve Open-Vocabulary Language Model of Code?, no. 1, [Online]. Available: http://arxiv.org/abs/2201.01988

  20. Li M et al (2022) Automated data function extraction from textual requirements by leveraging semi-supervised CRF and language model. Inf Softw Technol 143:106770. https://doi.org/10.1016/j.infsof.2021.106770

    Article  Google Scholar 

  21. Liu Y, Wang L, Shi T, Li J (2021) Detection of spam reviews through a hierarchical attention architecture with N-gram CNN and Bi-LSTM. Inf Syst 103:101865. https://doi.org/10.1016/j.is.2021.101865

    Article  Google Scholar 

  22. Lin J, Sun G, Beydoun G, Li L (2022) Applying machine translation and language modelling strategies for the recommendation task of micro learning service. Educ Technol Soc 25(1):205–212

    Google Scholar 

  23. Reiter E, Dale R (1997) Building applied natural language generation systems. Nat Lang Eng 3(1):57–87. https://doi.org/10.1017/S1351324997001502

    Article  Google Scholar 

  24. Kunhi LM, Shetty J (2022) Generation of structured query language from natural language using recurrent neural networks. Invent Commun Comput Technol 63–73

  25. Zhang X, Lapata M (2014) Chinese poetry generation with recurrent neural networks. In: Proceedings of the 2014 conference on empirical methods in natural language processing ({EMNLP}), pp 670–680. https://doi.org/10.3115/v1/D14-1074

  26. Gatt A, Krahmer E (2018) Survey of the state of the art in natural language generation: core tasks, applications and evaluation. J Artif Intell Res 61(c):1–64. https://doi.org/10.1613/jair.5714

    Article  MathSciNet  MATH  Google Scholar 

  27. Palombella VJ, Rando OJ, Goldberg AL, Maniatis T (1994) The ubiquitin-proteasome pathway is required for processing the NF-kappa B1 precursor protein and the activation of NF-kappa B. Cell 78(5):773–785. https://doi.org/10.1016/s0092-8674(94)90482-0

    Article  Google Scholar 

  28. Mann WC, Thompson SA (1987) Rhetorical structure theory: description and construction of text structures. In: Kempen G (ed) Natural language generation: new results in artificial intelligence, psychology and linguistics. Springer, Dordrecht, pp 85–95

    Chapter  Google Scholar 

  29. Santhanam S (2020) Context based text-generation using LSTM networks. [Online]. Available: http://arxiv.org/abs/2005.00048

  30. Langkilde I (2000) Forest-based statistical sentence generation. [Online]. Available: https://aclanthology.org/A00-2023

  31. Yao T et al (2021) Compound figure separation of biomedical images with side loss. In: Deep generative models, and data augmentation, labelling, and imperfections: first workshop, DGM4MICCAI 2021, and first workshop, DALI 2021, held in conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, proceedings 1, pp 173–183

  32. Iglesias P, Sicilia M-A, García-Barriocanal E (2023) Detecting browser drive-by exploits in images using deep learning. Electronics. https://doi.org/10.3390/electronics12030473

    Article  Google Scholar 

  33. Zhao M et al (2021) VoxelEmbed: 3D instance segmentation and tracking with voxel embedding based deep learning. In: Machine learning in medical imaging, pp 437–446

  34. Roy S, Viswanatham VM (2016) Classifying spam emails using artificial intelligent techniques. Int J Eng Res Africa 22:152–161. https://doi.org/10.4028/www.scientific.net/JERA.22.152

    Article  Google Scholar 

  35. Roy S, Viswanatham VM, Krishna P (2016) Spam detection using hybrid model of rough set and decorate ensemble. Int J Comput Syst Eng 2:139. https://doi.org/10.1504/IJCSYSE.2016.079000

    Article  Google Scholar 

  36. Wei M, Zhang Y (2019) Natural answer generation with attention over instances. IEEE Access 7:61008–61017. https://doi.org/10.1109/ACCESS.2019.2904337

    Article  Google Scholar 

  37. Pawade D, Sakhapara A, Jain M, Jain N, Gada K (2018) Story scrambler—automatic text generation using word level RNN-LSTM. Int J Inf Technol Comput Sci 10(6):44–53. https://doi.org/10.5815/ijitcs.2018.06.05

    Article  Google Scholar 

  38. Shen S, Chen Y, Yang C, Liu Z, Sun M (2018) Zero-shot cross-lingual neural headline generation. IEEE/ACM Trans Audio Speech Lang Process 26(12):2319–2327. https://doi.org/10.1109/TASLP.2018.2842432

    Article  Google Scholar 

  39. Chen Y, Yang C, Liu Z, Sun M (2020) Reinforced zero-shot cross-lingual neural headline generation. IEEE/ACM Trans Audio Speech Lang Process 28(12):2572–2584. https://doi.org/10.1109/TASLP.2020.3009487

    Article  Google Scholar 

  40. Abujar S, Masum AKM, Chowdhury SMMH, Hasan M, Hossain SA (2019) Bengali text generation using bi-directional RNN. In: 2019 10th International conference on computing and communication networks technology, ICCCNT 2019, pp 1–5. https://doi.org/10.1109/ICCCNT45670.2019.8944784

  41. Bao J, Tang D, Duan N, Yan Z, Zhou M, Zhao T (2019) Text generation from tables. IEEE/ACM Trans Audio Speech Lang Process 27(2):311–320. https://doi.org/10.1109/TASLP.2018.2878381

    Article  Google Scholar 

  42. Wang HC, Hsiao WC, Chang SH (2020) Automatic paper writing based on a RNN and the TextRank algorithm. Appl Soft Comput J 97:106767. https://doi.org/10.1016/j.asoc.2020.106767

    Article  Google Scholar 

  43. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  44. Roy S, Kaul D, Roy R, Barna C, Mehta S, Misra A (2018) Prediction of customer satisfaction using Naive Bayes, multiclass classifier, K-star and IBK

  45. Ren Y, Hu W, Wang Z, Zhang X, Wang Y, Wang X (2021) A hybrid deep generative neural model for financial report generation. Knowl Based Syst 227:107093. https://doi.org/10.1016/j.knosys.2021.107093

    Article  Google Scholar 

  46. Hoogi A, Mishra A, Gimenez F, Dong J, Rubin D (2020) Mammography reports simulation, vol 24, no 9, pp 2711–2717

  47. Xiang L, Yang S, Liu Y, Li Q, Zhu C (2020) Novel linguistic steganography based on character-level text generation. Mathematics 8(9):1–18. https://doi.org/10.3390/math8091558

    Article  Google Scholar 

  48. Chakraborty S, Banik J, Addhya S, Chatterjee D (2020) Study of dependency on number of LSTM units for character based text generation models. In: 2020 International conference on computer science and engineering and applications, ICCSEA 2020. https://doi.org/10.1109/ICCSEA49143.2020.9132839

  49. Sanzidul IM, Sadia Sultana SM, Abujar S, Hossain SA (2019) Sequence-to-sequence Bangla sentence generation with LSTM recurrent neural networks. Procedia Comput Sci 152:51–58. https://doi.org/10.1016/j.procs.2019.05.026

    Article  Google Scholar 

  50. Liu T, Wang K, Sha L, Chang B, Sui Z (2018) Table-to-text generation by structure-aware seq2seq learning. In: 32nd AAAI conference on artificial intelligence, AAAI 2018, pp 4881–4888

  51. Sha L et al (2018) Order-planning neural text generation from structured data. In: 32nd AAAI conference on artificial intelligence, AAAI 2018, pp 5414–5421

  52. Fan A, Lewis M, Dauphin Y (2018) Hierarchical neural story generation. In: ACL 2018—56th annual meeting of the association for computational linguistics, proceedings conference (long papers), vol 1, pp 889–898. https://doi.org/10.18653/v1/p18-1082

  53. Li J, Monroe W, A Ritter, Galley M, Gao J, Jurafsky D (2016) Deep reinforcement learning for dialogue generation. IN: EMNLP 2016—conference on empirical methods in natural language processing proceedings, no 4, pp 1192–1202. https://doi.org/10.18653/v1/d16-1127

  54. Bourane S et al (2015) Gate control of mechanical itch by a subpopulation of spinal cord interneurons. Science 350(6260):550–554. https://doi.org/10.1126/science.aac8653

    Article  Google Scholar 

  55. Zheng Q, Yang M, Yang J, Zhang Q, Zhang X (2018) Improvement of generalization ability of deep CNN via implicit regularization in two-stage training process. IEEE Access 6:15844–15869. https://doi.org/10.1109/ACCESS.2018.2810849

    Article  Google Scholar 

  56. Zhu J, Li J, Zhu M, Qian L, Zhang M, Zhou G (2020) Modeling graph structure in transformer for better AMR-to-text generation. In: EMNLP-IJCNLP 2019—2019 conference on empirical methods natural language processing, 9th international joint conference natural language processing proceedings, vol 1, pp 5459–5468. https://doi.org/10.18653/v1/d19-1548

  57. Biswas R, Vasan A, Roy SS (2020) Dilated deep neural network for segmentation of retinal blood vessels in fundus images. Iran J Sci Technol Trans Electr Eng 44(1):505–518. https://doi.org/10.1007/s40998-019-00213-7

    Article  Google Scholar 

  58. Schmitt M, Sharifzadeh S, Tresp V, Schütze H (2020) An unsupervised joint system for text generation from knowledge graphs and semantic parsing. In EMNLP 2020—2020 conference on empirical methods natural language processing proceedings, pp 7117–7130. https://doi.org/10.18653/v1/2020.emnlp-main.577

  59. Qader R, Jneid K, Portet F, Labbé C (2018) Generation of company descriptions using concept-to-text and text-to-text deep models: dataset collection and systems evaluation. In: Proceedings of the 11th international conference on natural language generation, pp 254–263. https://doi.org/10.18653/v1/W18-6532

  60. Jin D, Jin Z, Hu Z, Vechtomova O, Mihalcea R (2022) Deep learning for text style transfer: a survey. Comput Linguist 48(1):155–205. https://doi.org/10.1162/COLI_a_00426

    Article  Google Scholar 

  61. Yermakov R, Ag B, Drago N, Ag B, Ziletti A, Ag B (2021) Biomedical data-to-text generation via fine-tuning transformers, pp 364–370

  62. Kim Y, Jang M, Allan J (2020) Explaining text matching on neural natural language inference. ACM Trans Inf Syst 38(4):1–23

    Article  Google Scholar 

  63. Wang M, Lu S, Zhu D, Lin J, Wang Z (2018) A high-speed and low-complexity architecture for softmax function in deep learning. In: 2018 IEEE Asia Pacific conference on circuits and systems (APCCAS), pp 223–226. https://doi.org/10.1109/APCCAS.2018.8605654

  64. Bouchard G (2007) Efficient bounds for the softmax function, applications to inference in hybrid models. Nips 1–9 [Online]. Available: http://eprints.pascal-network.org/archive/00003498/

  65. Yin X, Goudriaan J, Lantinga EA, Vos J, Spiertz HJ (2003) A flexible sigmoid function of determinate growth. Ann Bot 91(3):361–371. https://doi.org/10.1093/aob/mcg029

    Article  Google Scholar 

  66. Lin C-Y (2004) {ROUGE}: a package for automatic evaluation of summaries. In: Text summarization branches out, pp 74–81. Available: https://aclanthology.org/W04-1013

  67. Lin C-Y (2004) Looking for a few good metrics: ROUGE and its evaluation. In: NTCIR Work, pp 1–8

  68. Yadav D et al (2022) Qualitative analysis of text summarization techniques and its applications in health domain. Comput Intell Neurosci 2022:1–14. https://doi.org/10.1155/2022/3411881

    Article  Google Scholar 

  69. Yadav AK et al (2022) Extractive text summarization using deep learning approach. Int J Inf Technol. https://doi.org/10.1007/s41870-022-00863-7

    Article  Google Scholar 

  70. Sun Y et al (2022) Bidirectional difference locating and semantic consistency reasoning for change captioning. Int J Intell Syst. https://doi.org/10.1002/int.22821

    Article  Google Scholar 

  71. Papineni K, Roukos S, Ward T, Zhu WJ (2002) BLEU: a method for automatic evaluation of machine translation. https://doi.org/10.3115/1073083.1073135

  72. Singh C (2017) Alice in Wonderland Gutenberg. https://www.kaggle.com/datasets/chandan2495/alice-in-wonderland-gutenbergproject/metadata

  73. BG illustrated by A. Browne, Hansel and Gretel (1981). Julia MacRae Books, London, New York

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Vellore, 632014, India

    Abhishek Kumar Pandey & Sanjiban Sekhar Roy

Authors
  1. Abhishek Kumar Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sanjiban Sekhar Roy
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, AKP, and SSR; methodology, AKP; software, AKP; validation, AKP, and SSR; formal analysis, AKP; investigation, AKP; resources, AKP; data curation, AKP; writing—original draft preparation, AKP; writing—review and editing, AKP and SSR; visualization, AKP; supervision, SSR; All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Sanjiban Sekhar Roy.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pandey, A.K., Roy, S.S. Natural Language Generation Using Sequential Models: A Survey. Neural Process Lett 55, 7709–7742 (2023). https://doi.org/10.1007/s11063-023-11281-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11281-6

Keywords

Search

Navigation