Abstract
Probabilistic autoencoders are effective for text generation. However, they are unable to control the style of generated text, despite the training samples explicitly labeled with different styles. We present a Wasserstein autoencoder with a Gaussian mixture prior for style-aware sentence generation. Our model is trained on a multi-class dataset and generates sentences in the style of the desired class. It is also capable of interpolating multiple classes. Moreover, we can train our model on relatively small datasets. While a regular WAE or VAE cannot generate diverse sentences with few training samples, our approach generates diverse sentences and preserves the style of the desired classes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The source code is available at https://github.com/alwevks/GMM-WAE.
References
Bahuleyan, H., Mou, L., Vechtomova, O., Poupart, P.: Variational attention for sequence-to-sequence models. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 1672–1682. Association for Computational Linguistics (2018). https://www.aclweb.org/anthology/C18-1142
Bahuleyan, H., Mou, L., Zhou, H., Vechtomova, O.: Stochastic Wasserstein autoencoder for probabilistic sentence generation. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4068–4076 (2019). https://www.aclweb.org/anthology/N19-1411
Ben-Yosef, M., Weinshall, D.: Gaussian mixture generative adversarial networks for diverse datasets, and the unsupervised clustering of images. arXiv preprint arXiv:1808.10356 (2018). https://arxiv.org/abs/1808.10356
Fu, Z., Tan, X., Peng, N., Zhao, D., Yan, R.: Style transfer in text: exploration and evaluation. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pp. 663–670 (2018). https://ojs.aaai.org/index.php/AAAI/article/view/11330
Gao, X., et al.: Structuring latent spaces for stylized response generation. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 1814–1823 (2019). https://www.aclweb.org/anthology/D19-1190
Gaujac, B., Feige, I., Barber, D.: Gaussian mixture models with Wasserstein distance. arXiv preprint arXiv:1806.04465 (2018). https://arxiv.org/abs/1806.04465
Gu, X., Cho, K., Ha, J.W., Kim, S.: DialogWAE: multimodal response generation with conditional Wasserstein auto-encoder. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=BkgBvsC9FQ
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., Xing, E.P.: Toward controlled generation of text. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1587–1596. JMLR. org (2017)
John, V., Mou, L., Bahuleyan, H., Vechtomova, O.: Disentangled representation learning for non-parallel text style transfer. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 424–434 (2019). https://www.aclweb.org/anthology/P19-1041
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1746–1751 (2014). https://www.aclweb.org/anthology/D14-1181
Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference on Learning Representations (2014). https://openreview.net/forum?id=33X9fd2-9FyZd
Li, J., Galley, M., Brockett, C., Gao, J., Dolan, B.: A diversity-promoting objective function for neural conversation models. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110–119 (2016). https://www.aclweb.org/anthology/N16-1014/
Lin, J.: Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory 37(1), 145–151 (1991). https://ieeexplore.ieee.org/abstract/document/61115/
Mou, L., Vechtomova, O.: Stylized text generation: approaches and applications. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pp. 19–22 (2020). https://www.aclweb.org/anthology/2020.acl-tutorials.5
Shen, T., Lei, T., Barzilay, R., Jaakkola, T.: Style transfer from non-parallel text by cross-alignment. In: Advances in Neural Information Processing Systems, pp. 6830–6841 (2017). https://proceedings.neurips.cc/paper/2017/file/2d2c8394e31101a261abf1784302bf75-Paper.pdf
Shen, T., Ott, M., Auli, M., Ranzato, M.: Mixture models for diverse machine translation: tricks of the trade. In: Proceedings of the 36th International Conference on Machine Learning, pp. 5719–5728 (2019). http://proceedings.mlr.press/v97/shen19c.html
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014). https://papers.nips.cc/paper/2014/file/a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf
Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. In: International Conference on Learning Representations (2018). https://openreview.net/forum?id=HkL7n1-0b
Vechtomova, O., Bahuleyan, H., Ghabussi, A., John, V.: Generating lyrics with variational autoencoder and multi-modal artist embeddings. arXiv preprint arXiv:1812.08318 (2018). https://arxiv.org/abs/1812.08318
Wang, W., et al.: Topic-guided variational autoencoders for text generation. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 166–177 (2019). https://www.aclweb.org/anthology/N19-1015
Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1112–1122 (2018). http://aclweb.org/anthology/N18-1101
Xu, R., Ge, T., Wei, F.: Formality style transfer with hybrid textual annotations. arXiv preprint arXiv:1903.06353 (2019). https://arxiv.org/abs/1903.06353
Acknowledgments
The research was supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) under Grants RGPIN-2019-04897 and RGPIN-2020-04465. Lili Mou is supported by the Alberta Machine Intelligence Institute (Amii) Fellow Program and the Canada CIFAR AI (CCAI) Chair Program. This research was also enabled in part by the support of Compute Canada (www.computecanada.ca).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Ghabussi, A., Mou, L., Vechtomova, O. (2021). Wasserstein Autoencoders with Mixture of Gaussian Priors for Stylized Text Generation. In: EkÅ¡tein, K., Pártl, F., KonopÃk, M. (eds) Text, Speech, and Dialogue. TSD 2021. Lecture Notes in Computer Science(), vol 12848. Springer, Cham. https://doi.org/10.1007/978-3-030-83527-9_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-83527-9_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-83526-2
Online ISBN: 978-3-030-83527-9
eBook Packages: Computer ScienceComputer Science (R0)