Wuhan University Journal of Natural Sciences

, Volume 23, Issue 6, pp 525–534 | Cite as

Predicting Effectiveness of Generate-and-Validate Patch Generation Systems Using Random Forest

  • Yong Xu
  • Bo Huang
  • Xiaoning Zou
  • Liying Kong


One way to improve practicability of automatic program repair (APR) techniques is to build prediction models which can predict whether an application of a APR technique on a bug is effective or not. Existing prediction models have some limitations. First, the prediction models are built with hand crafted features which usually fail to capture the semantic characteristics of program repair task. Second, the performance of the prediction models is only evaluated on Genprog, a genetic-programming based APR technique. This paper develops prediction models, i.e., random forest prediction models for SPR, another kind of generate- and-validate APR technique, which can distinguish ineffective repair instances from effective repair instances. Rather than handcrafted features, we use features automatically learned by deep belief network (DBN) to train the prediction models. The empirical results show that compared to the baseline models, that is, all effective models, our proposed models can at least improve the F1 by 9% and AUC(area under the receiver operating characteristics curve) by 19%. At the same time, the prediction model using learned features at least outperforms the one using hand-crafted features in terms of F1 by 11%

Key words

automatic program repair deep belief network effectiveness prediction repair instance patch generation random forest 

CLC number

TP 311.5 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    Goues C L, Nguyen T V, Forrest S, et al. Genprog: A generic method for automatic software repair[J]. Software Engineering, IEEE Transactions on, 2012,38(1):54–72.Google Scholar
  2. [2]
    Xu Y, Wu G Q, Yuan M T, et al. Automatic program repair based on version–space algebra[J]. Acta Electronica Sinica, 2007, 45(10): 2498–2505.Google Scholar
  3. [3]
    Nguyen H D T, Dawei Q, Roychoudhury A, et al. Semfix: Program repair via semantic analysis[C]//Proceedings of the 2013 International Conference on Software Engineering. Washington D C: IEEE, 2013: 772–781.Google Scholar
  4. [4]
    Goues C L, Dewey–Vogt M, Forrest S, et al. A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each [C]//Proceedings of Software Engineering (ICSE), 2012 34th International Conference on. Washington D C: IEEE, 2012: 3–13.Google Scholar
  5. [5]
    Fan L, Rinard M. An analysis of the search spaces for generate and validate patch generation systems[C]//Proceedings of Software Engineering (ICSE), 2016 38th International Conference on. Washington D C: IEEE, 2016: 702–713.Google Scholar
  6. [6]
    Le X B D, Le T D B, Lo D. Should fixing these failures be delegated to automated program repair?[C]//Proceedings of Software Reliability Engineering (ISSRE), 2015 IEEE 26th International Symposium on. Washington D C: IEEE, 2015: 427–437.Google Scholar
  7. [7]
    Fan L, Rinard M. Staged program repair with condition synthesis[C]//Proceedings of the 2015 10th Jointing on Foundations of Software Engineering, ESEC/FSE 2015. Washington D C: IEEE, 2015:166–178.Google Scholar
  8. [8]
    Bengio Y, Courville A, Vincent P. Rep–Resentation learning: A review and new perspectives[C]//IEEE Transactions on Pattern Analysis Machine Intelligence. Washington D C: IEEE, 2012: 1798–1828.Google Scholar
  9. [9]
    Wang S, Liu T Y, Tan L. Automatically learning semantic features for defect prediction [C]_//Proceedings of IEEE/ACM International Conference on Software Engineering. Washington D C: IEEE, 2016:297–308.Google Scholar
  10. [10]
    Dam H K, Tran T, Pham T, et al. Automatic feature learning for vulnerability prediction[EB/OL]. [2017–12–10]. http: //cn. arxiv. org/pdf/1708.02368.Google Scholar
  11. [11]
    White M, Tufano M, Vendome C, et al. Deep learning code fragments for code clone detection[C]//Proceedings of IEEE/ACM International Conference on Automated Software Engineering. Washington D C: IEEE, 2016: 87–98.Google Scholar
  12. [12]
    Hinton G E, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets [J]. Neural Computation, 2006, 18(7): 1527–1554.CrossRefGoogle Scholar
  13. [13]
    Wong W E, Debroy V. A Survey of Software Fault Localization [R]. Dallas:Department of Computer Science, University of Texas, 2009.Google Scholar
  14. [14]
    Bengio Y. Learning deep architectures for AI [J]. Foundations and Trends in Machine Learning, 2009, 2(1): 1–127.CrossRefGoogle Scholar
  15. [15]
    Breiman L. Random forests [J]. Machine Learning, 2001, 45(1): 5–32.CrossRefGoogle Scholar
  16. [16]
    Llvm–clang. Clang: A Clanguage family front–and for LIVM [DB/OL]. [2016–01–03]. Scholar
  17. [17]
    Peng H, Mou L L, Li G, et al. Building program vector representations for deep learning[C]//Knowledge Science, Engineering and Management. Berlin: Springer–Verlag, 2015: 547–553.Google Scholar
  18. [18]
    Sarikaya R, Hinton G E, Deoras A. Application of deep belief networks for natural language understanding [J]. IEEE/ ACM Transactions on Audio Speech Language Processing, 2014, 22(4): 778–784.CrossRefGoogle Scholar
  19. [19]
    Papa J P, Scheirer W, Cox D D. Fine–tuning deep belief networks using harmony search[J]. Applied Soft Computing, 2015, 46(C): 875–885.Google Scholar
  20. [20]
    Assiri F Y, Bieman J M. Fault localization for automated program repair: effectiveness, performance, repair correctness [J]. Software Quality Journal, 2016, 25(1): 1–29Google Scholar
  21. [21]
    Goues C L, Holtschulte N, Smith E K, et al. The manybugs and introclass benchmarks for automated repair of c programs [J]. IEEE Transactions on Software Engineering, 2015, 41(12): 1236–1256.CrossRefGoogle Scholar
  22. [22]
    Elish K O, Elish M O. Predicting defect–prone software modules using support vector machines [J]. Journal of Systems Software, 2008, 81(5): 649–660.CrossRefGoogle Scholar
  23. [23]
    Ciresan D, Meier U, Schmidhuber J. Multi–column deep neural networks for image classification[EB/OL]. [2017–10–20]. 2012: 3642–3649. https://arxiv.orp/abs/1202.2745.Google Scholar
  24. [24]
    An N L, Nguyen A T, Nguyen H A, et al. Combining deep learning with information retrieval to localize buggy files for bug reports[C]//Proceedings of IEEE/ACM International Conference on Automated Software Engineering. Washington D C: IEEE, 2016: 476–481.Google Scholar
  25. [25]
    Roux N L, Bengio Y. Representational power of restricted boltzmann machines and deep belief networks [J]. Neural Computation, 2008, 20(6): 1631–1649.CrossRefGoogle Scholar
  26. [26]
    Platt J C. Sequential minimal optimization: A fast algorithm for training support vector machines [J]. Advances in Kernel Methods–Support Vector Learning, 1998, 208(1): 212–223.Google Scholar
  27. [27]
    Hosmer Jr D W, Lemeshow S, Sturdivant R X. Applied Logistic Regression [M]. New York: John Wiley & Sons, 2013.CrossRefGoogle Scholar
  28. [28]
    Quinlan J R. Programs for Machine Learning [M]. San Francisco: Morgan Kaufmann Publishers Inc,1993.Google Scholar
  29. [29]
    Cleary J G, Trigg L E. K*:An instance–based learner using an entropic distance measure[C]//Proceedings of International Conference on Machine Learning. New York: ACM Press, 1995:108–114.Google Scholar

Copyright information

© Wuhan University and Springer-Verlag GmbH Germany, part of Springer Nature 2018

Authors and Affiliations

  1. 1.School of Mathematics and StatisticsZhaoqing UniversityZhaoqing, GuangdongChina
  2. 2.School of Electronic and Electrical EngineeringShanghai University of Engineering ScienceShanghaiChina

Personalised recommendations