Skip to main content

Evolution of Neural Text Generation: Comparative Analysis

  • 855 Accesses

Part of the Advances in Intelligent Systems and Computing book series (AISC,volume 1158)

Abstract

In the past few years, various advancements have been made in Language Models owing to the formulation of various new algorithms such as Generative Adversarial Networks (GANs), ELMo and Bidirectional Encoder Representations from Transformers (BERT). Text Generation, one of the most important language modeling problems has shown great promise recently due to the advancement of more efficient and competent context-dependent algorithms such as ElMo and BERT and GPT-2 compared to preceding context independent algorithms such as word2vec and GloVe. In this paper, we compare the various attempts to Text Generation showcasing the benefits of each in their own unique form.

Keywords

  • Word2vec
  • GloVe
  • ELMo
  • BERT
  • GANs
  • GPT-2

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-981-15-4409-5_71
  • Chapter length: 10 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   219.00
Price excludes VAT (USA)
  • ISBN: 978-981-15-4409-5
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   279.99
Price excludes VAT (USA)
Fig. 1

References

  1. T. Mikolov, I. Sutskever, K. Chen, G. Corrado, J. Dean, Distributed representations of words and phrases and their compositionality, in Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

    Google Scholar 

  2. J. Pennington, R. Socher, C.D. Manning, GloVe: global vectors for word representation, in Empirical Methods in Natural Language Processing, pp. 1532–1543 (2014)

    Google Scholar 

  3. M.E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, L. Zettlemoyer, Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018)

  4. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding (2018). arXiv preprint arXiv:1810.04805 (2019)

  5. A. Wang, K. Cho, BERT has a mouth, and it must speak: BERT as a Markov random field language model. arXiv preprint arXiv:1902.04094 (Apr 2019)

  6. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners (2019)

    Google Scholar 

  7. A. Wang, J. Hula, P. Xia, R. Pappagari, R. Thomas McCoy, R. Patel, N. Kim, I. Tenney, Y. Huang, K. Yu, S. Jin, B. Chen, B. Van Durme, E. Grave, E. Pavlick, S.R. Bowman, Can you tell me how to get past sesame street? Sentence-level pretraining beyond language modeling. arXiv preprint arXiv:1812.10860v5 (2019)

  8. K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, BLEU: a method for automatic evaluation of machine translation. in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), Philadelphia, July 2002, pp. 311–318

    Google Scholar 

  9. J. Howard, S. Ruder, Universal language model fine-tuning for text classification. arXiv preprint arXiv: 1801.06146v5, (2018)

    Google Scholar 

  10. M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, Ł. Kaiser, Universal transformer. arXiv preprint arXiv:1807.03819 (2019)

  11. R. McDermott, Trump-speeches, Github repository (2016). https://github.com/ryanmcdermott/trump-speeches/blob/master/speeches.txt

  12. Y. Wang, RNN_Sequence, Github Repository (2017). https://github.com/alanwang93/RNN_Sequence/blob/master/data/arxiv/arxiv_abstracts.txt

  13. Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, Y. Yu, Texygen: a benchmarking platform for text generation models. arXiv preprint arXiv:1802.0188v1 (2018)

  14. https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf. Accessed 2nd July 2019

  15. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, Q.V. Le, XLNet: generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237 (2019)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Lakshmi Kurup .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Singapore Pte Ltd.

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Kurup, L., Narvekar, M., Sarvaiya, R., Shah, A. (2021). Evolution of Neural Text Generation: Comparative Analysis. In: Bhatia, S.K., Tiwari, S., Ruidan, S., Trivedi, M.C., Mishra, K.K. (eds) Advances in Computer, Communication and Computational Sciences. Advances in Intelligent Systems and Computing, vol 1158. Springer, Singapore. https://doi.org/10.1007/978-981-15-4409-5_71

Download citation