Automatic Text Generation in Macedonian Using Recurrent Neural Networks

  • Ivona Milanova
  • Ksenija Sarvanoska
  • Viktor Srbinoski
  • Hristijan GjoreskiEmail author
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 1110)


Neural text generation is the process of a training neural network to generate a human understandable text (poem, story, article). Recurrent Neural Networks and Long-Short Term Memory are powerful sequence models that are suitable for this kind of task. In this paper, we have developed two types of language models, one generating news articles and the other generating poems in Macedonian language. We developed and tested several different model architectures, among which we also tried transfer-learning model, since text generation requires a lot of processing time. As evaluation metric we used ROUGE-N metric (Recall-Oriented Understudy for Gisting Evaluation), where the generated text was tested against a reference text written by an expert. The results showed that even though the generate text had flaws, it was human understandable, and it was consistent throughout the sentences. To the best of our knowledge this is a first attempt in automatic text generation (poems and articles) in Macedonian language using Deep Learning.


Text generation Storytelling Poems RNN Macedonian language NLP Transfer learning ROUGE-N 



We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.


  1. 1.
    Bailey, P.: Searching for storiness: story-generation from a reader’s perspective. In: Working Notes of the Narrative Intelligence Symposium (1999)Google Scholar
  2. 2.
    PÉrez, R.P.Ý., Sharples, M.: MEXICA: a computer model of a cognitive account of creative writing. J. Exp. Teor. Artif. Intell. 13, 119–139 (2001)CrossRefGoogle Scholar
  3. 3.
    Sutskever, I., Martens, J., Hinton, G.E.: Generating text with recurrent neural networks. In: Proceedings of the 28th International Conference on Machine Learning (ICML-2011), pp. 1017–1024 (2011)Google Scholar
  4. 4.
    Jain, P., Agrawal, P., Mishra, A., Sukhwani, M., Laha, A., Sankaranarayanan, K.: Story generation from sequence of independent short descriptions. In: Proceedings of Workshop on Machine Learning for Creativity, Halifax, Canada, August 2017 (SIGKDD 2017) (2017)Google Scholar
  5. 5.
    McIntyre, N., Lapata, M.: Learning to tell tales: a data-driven approach to story generation. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, Association for Computational Linguistics, vol. 1, pp. 217–225 (2009)Google Scholar
  6. 6.
    Li, B., Lee-Urban, S., Johnston, G., Riedl, M.: Story generation with crowdsourced plot graphs. In: AAAI (2013)Google Scholar
  7. 7.
    Swanson, R., Gordon, A.: Say anything: a massively collaborative open domain story writing companion. Interact. Storytelling 2008, 32–40 (2008)CrossRefGoogle Scholar
  8. 8.
    Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: CVPR (2015)Google Scholar
  9. 9.
    Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: ICML (2015)Google Scholar
  10. 10.
    Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T., Saenko, K.: Sequence to sequence-video to text. In: ICCV (2015)Google Scholar
  11. 11.
    Pan, Y., Mei, T., Yao, T., Li, H., Rui, Y.: Jointly modeling embedding and translation to bridge video and language. In: CVPR (2016)Google Scholar
  12. 12.
    Rush, A.M., Chopra, S., Weston, J.: A neural attention model for abstractive sentence summarization. In: EMNLP (2015)Google Scholar
  13. 13.
    Kim, G., Xing, E.P.: Reconstructing storyline graphs for image recommendation from web community photos. In: CVPR (2014)Google Scholar
  14. 14.
    Sigurdsson, G.A., Chen, X., Gupta, A.: Learning visual storylines with skipping recurrent neural networks. In: ECCV (2016)Google Scholar
  15. 15.
    Glorianna Jagfeld, S.J.: Sequence-to-sequence models for data-to-text natural language (2018)Google Scholar
  16. 16.
    Sutskever, I.: Generating text with recurrent neural networks. In: 28th International Conference on Machine Learning (ICML-2011), pp. 1017–1024 (2011)Google Scholar
  17. 17.
    Racin, K.: Beli mugri, pp. 3–33. Makedonska kniga, Skopje (1989)Google Scholar
  18. 18.
    Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Hahnloser, R.H., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., Seung, H.S.: Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405(6789), 947 (2000)CrossRefGoogle Scholar
  20. 20.
    Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv:1412.6980 [cs.LG], December 2014
  21. 21.
    Montavon, G., Orr, Geneviève B., Müller, K.-R. (eds.): Neural Networks: Tricks of the Trade. LNCS, vol. 7700. Springer, Heidelberg (2012). Scholar
  22. 22.
    Mikolov, T., et al.: Distributed representations of words and phrases and their compositionality. In: NIPS (2013)Google Scholar
  23. 23.
    Pradhan, S.: Exploring the depths of recurrent neural networks with stochastic residual learning (2016)Google Scholar
  24. 24.
    Lin, C.-Y.: ROUGE: a package for automatic evaluation of summaries. In: ACL Workshop: Text Summarization Braches Out 2004, p. 10 (2004)Google Scholar
  25. 25.
    Huang, Q., Gan, Z., Celikyilmaz, A., Wu, D., Wang, J., He, X.: Hierarchically structured reinforcement learning for topically coherent visual story generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 8465–8472, July 2019Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Ivona Milanova
    • 1
  • Ksenija Sarvanoska
    • 1
  • Viktor Srbinoski
    • 1
  • Hristijan Gjoreski
    • 1
    Email author
  1. 1.Faculty of Electrical Engineering and Information TechnologiesUniversity of Ss. Cyril and Methodius in SkopjeSkopjeNorth Macedonia

Personalised recommendations