Abstract
Natural language summaries of codes are important during software development and maintenance. Recently, deep learning-based models have achieved good performance on automatic code summarization, which encode token sequence or abstract syntax tree (AST) of code with neural networks. However, almost all of these models are trained using maximum likelihood estimation, which do not guarantee the quality of generated summaries. Moreover, existing models that benefit from multiple encoders lack a fined-grained selection between different encoders, and the encoders may be insufficiently optimized. To address these issues and generate better code summaries, we propose a novel code summarization framework based on adversarial training and ensemble learning. It includes two separately trained encoder-decoder models, one for source code sequence and the other for its AST. Here, an efficient approach to obtain AST node sequence is introduced. We train our models via adversarial training, where each model is guided by a well-designed discriminator that learns to evaluate its outputs. During inference, a module named mixture network is introduced to compute an adaptive combination weight of the models’ outputs. We evaluate our framework on a large Java corpus and compare it to several state-of-the-art models. Experimental results show that our approach outperforms the best baseline by 22.6% on BLEU-4, 5.7% on ROUGE-L and 7.6% on METEOR.
Similar content being viewed by others
References
Sridhara G, Hill E, Muppaneni D, Pollock LL, Vijay-Shanker K (2010) Towards automatically generating summary comments for Java methods. In: Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, ACM, pp 43–52
Haiduc S, Aponte J, Moreno L, Marcus A (2010) On the use of automated text summarization techniques for summarizing source code. In: Reverse Engineering (WCRE), 2010 17th Working Conference on, IEEE, pp 35–44
McBurney PW, McMillan C (2016) Automatic source code summarization of context for java methods. IEEE Trans Software Eng 42(2):103–119
Movshovitz-Attias D, Cohen WW (2013) Natural language models for predicting programming comments. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACL, pp 35–40
Moreno L, Aponte J, Sridhara G, Marcus A, Pollock LL, Vijay-Shanker K (2013) Automatic generation of natural language summaries for Java classes. In: IEEE 21st International Conference on Program Comprehension, pp 23–32
Haiduc S, Aponte J, Marcus A (2010) Supporting program comprehension with source code summarization. In: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering. Vol 2, ACM, pp 223–226
Iyer S, Konstas I, Cheung A, Zettlemoyer L (2016) Summarizing source code using a neural attention model. In: Proceedings of the 54th annual meeting of the association for computational linguistics, ACL
Allamanis M, Peng H, Sutton CA (2016) A convolutional attention network for extreme summarization of source code. In: International conference on machine learning, pp 2091–2100
Hu X, Li G, Xia X, Lo D, Jin Z (2018) Deep code comment generation. In: Proceedings of the 26th conference on program comprehension, ACM, pp 200–210
Wan Y, Zhao Z, Yang M, Xu G, Ying H, Wu J, Yu PS (2018) Improving automatic source code summarization via deep reinforcement learning. In: Proceedings of the 33rd ACM/IEEE international conference on automated software engineering, ACM, pp 397–407
LeClair A, Jiang S, McMillan C (2019) A neural model for generating natural language summaries of program subroutines. In: Proceedings of the 41st international conference on software engineering, IEEE/ACM, pp 795–806
Hu X, Li G, Xia X, Lo D, Jin Z (2020) Deep code comment generation with hybrid lexical and syntactical information. Empir Softw Eng 25(3):2179–2217
Liang Y, Zhu KQ (2018) Automatic generation of text descriptive comments for code blocks. In: Proceedings of the Thirty-Second AAAI Conference on Artificial intelligence, AAAI Press, pp 5229–5236
Fernandes P, Allamanis M, Brockschmidt M (2019) Structured neural summarization. In: 7th international conference on learning representations
Alon U, Brody S, Levy O, Yahav E (2019) code2seq: Generating sequences from structured representations of code. In: 7th international Conference On Learning Representations
Hu X, Li G, Xia X, Lo D, Lu S, Jin Z (2018) Summarizing source code with transferred api knowledge. In: Proceedings of the twenty-seventh international joint conference on artificial intelligence, pp 2269–2275
Wei B, Li G, Xia X, Fu Z, Jin Z (2019) Code generation as a dual task of code summarization. In: Advances in neural information processing systems, pp 6559–6569
Ye W, Xie R, Zhang J, Hu T, Wang X, Zhang S (2020) Leveraging code generation to improve code retrieval and summarization via dual learning. In: The Web Conference, pp 2309–2319
Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems. pp 3104–3112
Cho K, Merrienboer Bv, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. In: Proceedings of the 2014 conference on empirical methods in natural language processing, pp 1724–1734
Luong T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 conference on empirical methods in natural language processing, ACL, pp 1412–1421
Wu Y, Schuster M, Chen Z, Le QV, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, Klingner J, Shah A, Johnson M, Liu X, Kaiser L, Gouws S, Kato Y, Kudo T, Kazawa H, Stevens K, Kurian G, Patil N, Wang W, Young C, Smith J, Riesa J, Rudnick A, Vinyals O, Corrado G, Hughes M, Dean J (2016) Google's neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144
Hellendoorn VJ, Devanbu PT (2017) Are deep neural networks the best choice for modeling source code?. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering, ACM, pp 763–773
Ranzato MA, Chopra S, Auli M, Zaremba W (2016) Sequence level training with recurrent neural networks. In: 4th International conference on learning representations
Papineni K, Roukos S, Ward T, Zhu W-J (2002) BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, ACL, pp 311–318
Lin C-Y (2004) ROUGE: A package for automatic evaluation of summaries. In: Proceedings of the workshop on text summarization branches out
Callison-Burch C, Osborne M, Koehn P (2006) Re-evaluation the role of bleu in machine translation research. In: 11th Conference of the European chapter of the association for computational linguistics
Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. In: NIPS, pp 2672–2680
Chen B, Cherry C (2014) A systematic comparison of smoothing techniques for sentence-level BLEU. In: ACL, pp 362–367
Wong E, Liu T, Tan L (2015) CloCom: mining existing source code for automatic comment generation. In: Software analysis, evolution and reengineering (SANER), 2015 IEEE 22nd international Conference on. IEEE, pp 380–389
Wong E, Yang J, Tan L (2013) AutoComment: mining question and answer sites for automatic comment generation. In: Proceedings of the 28th IEEE/ACM international conference on automated software engineering, IEEE, pp 562–567
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780
Cho K, Merrienboer Bv, Bahdanau D, Bengio Y (2014) On the properties of neural machine translation: encoder-decoder approaches. In: Proceedings of SSST@EMNLP, ACL, pp 103–111
Mirza M, Osindero S (2014) Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784
Yu L, Zhang W, Wang J, Yu Y (2017) SeqGAN: sequence generative adversarial nets with policy gradient. In: AAAI, pp 2852–2858
Li J, Monroe W, Shi T, Jean S, Ritter A, Jurafsky D (2017) Adversarial learning for neural dialogue generation. In: EMNLP, pp 2157–2169
Dognin PL, Melnyk I, Mroueh Y, Ross J, Sercu T (2019) Adversarial semantic alignment for improved image captions. In: CVPR, pp 10463–10471
Chen C, Mu S, Xiao W, Ye Z, Wu L, Ju Q (2019) Improving image captioning with conditional generative adversarial nets. In: AAAI, pp 8142–8150
Yang Z, Chen W, Wang F, Xu B (2018) Improving neural machine translation with conditional sequence generative adversarial nets. In: NAACL-HLT, pp 1346–1355
Wu L, Xia Y, Tian F, Zhao L, Qin T, Lai J, Liu T-Y (2018) Adversarial neural machine translation. In: ACML, pp 534–549
Garmash E, Monz C (2016) Ensemble learning for multi-source neural machine translation. In: 26th International conference on computational linguistics: technical papers, pp 1409–1418
Kim Y (2014) Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882
Hu B, Lu Z, Li H, Chen Q (2015) Convolutional neural network architectures for matching natural language sentences. arXiv preprint arXiv:1503.03244
Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning, pp 807–814
Rennie SJ, Marcheret E, Mroueh Y, Ross J, Goel V (2017) Self-critical sequence training for image captioning. Reinforcement learning. Springer, Berlin, pp 1179–1195
Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8:229–256
Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 249–256
Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: 3rd International conference on learning representations
Srivastava N, Hinton GE, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15(1):1929–1958
Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing internal covariate shift. pp 448–456
Denkowski M, Lavie A (2014) Meteor universal: language specific translation evaluation for any target language. In: Proceedings of the EACL 2014 Workshop on Statistical Machine Translation. ACL
Konda VR, Tsitsiklis JN (1999) Actor-critic algorithms. In: Advances in neural information processing systems, pp 1008–1014
LeClair A, Haque S, Wu L, McMillan C (2020) Improved code summarization via a graph neural network. In: 28th International Conference on Program Comprehension, pp 184–195
Gui J, Sun Z, Wen Y, Tao D, Ye J (2020) A review on generative adversarial networks: algorithms, theory, and applications. arXiv preprint arXiv:2001.06937v1
Funding
This work is partially supported by the NSF of China under Grants No. 61772200 and 61702334, Shanghai Pujiang Talent Program under Grants No. 17PJ1401900, Shanghai Municipal Natural Science Foundation under Grants No. 17ZR1406900 and 17ZR1429700, Educational Research Fund of ECUST under Grant No. ZH1726108.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zhou, Z., Yu, H. & Fan, G. Adversarial training and ensemble learning for automatic code summarization. Neural Comput & Applic 33, 12571–12589 (2021). https://doi.org/10.1007/s00521-021-05907-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00521-021-05907-w