Advertisement

LSTM Encoder-Decoder with Adversarial Network for Text Generation from Keyword

  • Dongju Park
  • Chang Wook Ahn
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 952)

Abstract

Natural Language Generation (NLG), one of the areas of Natural Language Processing (NLP), is a difficult task, but it is also important because it applies to our lives. So far, there have been various approaches to text generation, but in recent years, approaches using artificial neural networks have been used extensively. We propose a model for generating sentences from keywords using Generative Adversarial Network (GAN) composed of a generator and a discriminator among these artificial neural networks. Specifically, the generator uses the Long Short-Term Memory (LSTM) Encoder-Decoder structure, and the discriminator uses the bi-directional LSTM with self-attention. Also, the keyword for input to the encoder of the generator is input together with two words similar to oneself. This method contributes to the creation of sentences containing words that have similar meanings to the keyword. In addition, the number of unique sentences generated increases and diversity can be increased. We evaluate our model with BLEU Score and loss value. As a result, we can see that our model improves the performance compared to the baseline model without an adversarial network.

Keywords

Text generation Generative Adversarial Network Natural Language Processing 

Notes

Acknowledgements

This work was supported by Global University Project (GUP) grant funded by the GIST in 2018. Also, this work was supported by the NRF funded by MEST of Korea (No. 2015R1D1A1A02062017).

References

  1. 1.
    Serban, I.V., Sordoni, A., Bengio, Y., Courville, A.C., Pineau, J.: Building end-to-end dialogue systems using generative hierarchical neural network models. In: AAAI, pp. 3776–3784 (2016)Google Scholar
  2. 2.
    Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: International Conference on Machine Learning, pp. 2048–2057 (2015)Google Scholar
  3. 3.
    Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
  4. 4.
    Mikolov, T., Karafiát, M., Burget, L., Černocký, J., Khudanpur, S.: Recurrent neural network based language model. In: Eleventh Annual Conference of the International Speech Communication Association (2010)Google Scholar
  5. 5.
    Sundermeyer, M., Schlüter, R., Ney, H.: LSTM neural networks for language modeling. In: Thirteenth annual Conference of the International Speech Communication Association (2012)Google Scholar
  6. 6.
    Bengio, S., Vinyals, O., Jaitly, N., Shazeer, N.: Scheduled sampling for sequence prediction with recurrent neural networks. In: Advances in Neural Information Processing Systems, pp. 1171–1179 (2015)Google Scholar
  7. 7.
    Goodfellow, I., et al.: Generative adversarial nets. In: Advances in neural information processing systems, pp. 2672–2680 (2014)Google Scholar
  8. 8.
    Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015)
  9. 9.
    Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., Choo, J.: StarGAN: unified generative adversarial networks for multi-domain image-to-image translation. arXiv preprint arXiv:1711.09020 (2017)
  10. 10.
    Yu, L., Zhang, W., Wang, J., Yu, Y.: SeqGAN: sequence generative adversarial nets with policy gradient. In: AAAI, pp. 2852–2858 (2015)Google Scholar
  11. 11.
    Lin, K., Li, D., He, X., Zhang, Z., Sun, M.-T.: Adversarial ranking for language generation. In: Advances in Neural Information Processing Systems, pp. 3155–3165 (2017)Google Scholar
  12. 12.
    Guo, J., Lu, S., Cai, H., Zhang, W., Yu, Y., Wang, J.: Long text generation via adversarial training with leaked information. arXiv preprint arXiv:1709.08624 (2017)
  13. 13.
    Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)Google Scholar
  14. 14.
    Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
  15. 15.
    Lin, Z., et al.: A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130 (2017)
  16. 16.
    Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
  17. 17.
    Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992)zbMATHGoogle Scholar
  18. 18.
    Chen, X., et al.: Microsoft COCO captions: data collection and evaluation server. arXiv preprint arXiv:1504.00325 (2015)
  19. 19.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311–318. Association for Computational Linguistics (2002)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2018

Authors and Affiliations

  1. 1.School of Electrical Engineering and Computer ScienceGwangju Institute of Science and Technology (GIST)GwangjuRepublic of Korea

Personalised recommendations