Feature based problem hardness understanding for requirements engineering

  • Zhilei Ren
  • He Jiang
  • Jifeng Xuan
  • Shuwei Zhang
  • Zhongxuan Luo
Research Paper


Heuristics and metaheuristics have achieved great accomplishments in various fields, and the investigation of the relationship between these algorithms and the problem hardness has been a hot topic in the research field. Related research work has contributed much to the understanding of the underlying mechanisms of the algorithms for problem solving. However, most existing studies consider traditional combinatorial problems as their case studies. In this study, taking the Next Release Problem (NRP) from the requirements engineering as a case study, we investigate the relationship between software engineering problem instances and heuristics. We employ an evolutionary algorithm to evolve NRP instances, which are uniquely hard or easy for the target heuristic (Greedy Randomized Adaptive Search Procedure and Randomized Hill Climbing in this paper). Then, we use a feature-based method to estimate the hardness of the evolved instances, with respect to the target heuristic. Experimental results demonstrate that, evolutionary algorithm can be used to evolve NRP instances that are uniquely hard or easy to solve. Moreover, the features enable the estimation of the target heuristics’ performance.


problem hardness next release problem computational intelligence requirements engineering evolution algorithm 




启发式和元启发式算法在各种生产实践中, 已经取得了巨大的成功。 其中, 算法与问题难度间的分析一直是该领域中的研究热点。 然而, 已有的研究大多关注经典的组合优化问题。 本文中, 我们以需求工程中的下一版本问题为案例, 着重分析软件工程问题实例与启发式算法之间的关系。 具体的, 我们使用演化算法, 来针对目标算法演化困难与简单的问题实例。 接着, 我们使用基于问题特征的方法来估计所得实例的难度。 实验结果表明, 我们设计的算法能够生成具有显著困难区分度的问题实例。 并且, 我们提出的问题特征能够用于实现对目标算法性能的有效预测。


经验困难模型 下一版本问题 计算智能 需求工程 演化算法 原子模型 


  1. 1.
    Gong W, Cai Z, Liang D. Adaptive ranking mutation operator based differential evolution for constrained optimization. IEEE Trans Cybern, 2015, 45: 716–727CrossRefGoogle Scholar
  2. 2.
    Gong W, Cai Z. Differential evolution with ranking-based mutation operators. IEEE Trans Cybern, 2013, 43: 2066–2081CrossRefGoogle Scholar
  3. 3.
    Luo C, Cai S, Su K, et al. Clause states based configuration checking in local search for satisfiability. IEEE Trans Cybern, 2015, 45: 1014–1027CrossRefGoogle Scholar
  4. 4.
    Chen WN, Zhang J. Ant colony optimization for software project scheduling and staffing with an event-based scheduler. IEEE Trans Softw Eng, 2013, 39: 1–17CrossRefGoogle Scholar
  5. 5.
    Haeri S, Trajkovic L. Intelligent deflection routing in buffer-less networks. IEEE Trans Cybern, 2015, 45: 316–327CrossRefGoogle Scholar
  6. 6.
    Tang K, Yang P, Yao X. Negatively correlated cooperative search. arXiv:1504.04914Google Scholar
  7. 7.
    Jiang H, Xuan J, Ren Z. Approximate backbone based multilevel algorithm for next release problem. In: Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, Portland, 2010. 1333–1340CrossRefGoogle Scholar
  8. 8.
    Smith-Miles K A. Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput Surv, 2008, 41: 6CrossRefGoogle Scholar
  9. 9.
    Smith-Miles K A, van Hemert J I. Discovering the suitability of optimisation algorithms by learning from evolved instances. Ann Math Artif Intell, 2011, 61: 87–104MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Mersmann O, Bischl B, Trautmann H, et al. A novel feature-based approach to characterize algorithm performance for the traveling salesperson problem. Ann Math Artif Intell, 2013, 69: 151–182MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Smith-Miles K A. Towards insightful algorithm selection for optimisation using meta-learning concepts. In: Prcoeedings of International Joint Conference on Neural Networks, Hong Kong, 2008. 4118–4124Google Scholar
  12. 12.
    van Hemert J I. Evolving combinatorial problem instances that are difficult to solve. Evol Comput, 2006, 14: 433–462CrossRefGoogle Scholar
  13. 13.
    Vilalta R, Drissi Y. A perspective view and survey of meta-learning. Artif Intell Rev, 2002, 18: 77–95CrossRefGoogle Scholar
  14. 14.
    Oentaryo R J, Handoko S D, Lau H C. Algorithm selection via ranking. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence, Austin Texas, 2015. 1826–1832Google Scholar
  15. 15.
    van Hemert J I. Property analysis of symmetric travelling salesman problem instances acquired through evolution. In: Evolutionary Computation in Combinatorial Optimization. Berlin: Springer, 2005. 122–131CrossRefGoogle Scholar
  16. 16.
    Xuan J, Jiang H, Ren Z, et al. Solving the large scale next release problem with a backbone-based multilevel algorithm. IEEE Trans Softw Eng, 2012, 38: 1195–1212CrossRefGoogle Scholar
  17. 17.
    Nie L, Jiang H, Ren Z, et al. Query expansion based on crowd knowledge for code search. IEEE Trans Serv Comput, 2016, 9: 771–783CrossRefGoogle Scholar
  18. 18.
    Xuan J F, Martinez M, DeMarco F, et al. Nopol: automatic repair of conditional statement bugs in java programs. IEEE Trans Softw Eng, in press. doi: 10.1109/TSE.2016.2560811Google Scholar
  19. 19.
    Bagnall A J, Rayward-Smith V J, Whittley I M. The next release problem. Inf Softw Tech, 2001, 43: 883–890CrossRefGoogle Scholar
  20. 20.
    Greer D, Ruhe G. Software release planning: an evolutionary and iterative approach. Inf Softw Tech, 2004, 46: 243–253CrossRefGoogle Scholar
  21. 21.
    Ignatiev A, Janota M, Marques-Silva J. Towards efficient optimization in package management systems. In: Proceedings of the 36th International Conference on Software Engineering, Hyderabad, 2014. 745–755Google Scholar
  22. 22.
    Sun J, Zhang Q, Yao X. Meta-heuristic combining prior online and offline information for the quadratic assignment problem. IEEE Trans Cybern, 2014, 44: 429–444CrossRefGoogle Scholar
  23. 23.
    Ren Z, Jiang H, Xuan J, et al. New insights into diversification of hyper-heuristics. IEEE Trans Cybern, 2014, 44: 1747–1761CrossRefGoogle Scholar
  24. 24.
    Gao WF, Liu S Y, Huang L L. A novel artificial bee colony algorithm based on modified search equation and orthogonal learning. IEEE Trans Cybern, 2013, 43: 1011–1024CrossRefGoogle Scholar
  25. 25.
    Ren Z, Zhang A, Wen C, et al. A scatter learning particle swarm optimization algorithm for multimodal problems. IEEE Trans Cybern, 2014, 44: 1127–1140CrossRefGoogle Scholar
  26. 26.
    He J, Zhang J Y, Xuan J F, et al. A hybrid ACO algorithm for the next release problem. In: Proceedings of the 2nd International Conference on Software Engineering and Data Mining, Chengdu, 2010. 166–171Google Scholar
  27. 27.
    He J, Ren Z L, Li X C, et al. Transformed search based software engineering: a new paradigm of sbse. In: Search-Based Software Engineering. Berlin: Springer, 2015. 203–218Google Scholar
  28. 28.
    van Hemert J I. Evolving binary constraint satisfaction problem instances that are difficult to solve. In: Proceedings of Congress on Evolutionary Computation, Canberra, 2003. 1267–1273Google Scholar
  29. 29.
    Julstrom B A. Evolving heuristically difficult instances of combinatorial problems. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, Montreal, 2009. 279–286Google Scholar
  30. 30.
    Smith-Miles K A, Lopes L. Generalising algorithm performance in instance space: a timetabling case study. In: Proceedings of the 5th International Conference on Learning and Intelligent Optimization, Rome, 2011. 524–538CrossRefGoogle Scholar
  31. 31.
    Toledo-Suárez C D, Valenzuela-Rendón M, Terashima-Marín H, et al. On the relativity in the assessment of blind optimization algorithms and the problem-algorithm coevolution. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation. New York: ACM, 2007. 1436–1443CrossRefGoogle Scholar
  32. 32.
    Hall N G, Posner M E. Performance prediction and preselection for optimization and heuristic solution procedures. Oper Res, 2007, 55: 703–716CrossRefzbMATHGoogle Scholar
  33. 33.
    Leyton-Brown K, Nudelman E, Shoham Y. Empirical hardness models: methodology and a case study on combinatorial auctions. J ACM, 2009, 56: 22MathSciNetCrossRefzbMATHGoogle Scholar
  34. 34.
    Smith-Miles K, Wreford B, Lopes L, et al. Predicting metaheuristic performance on graph coloring problems using data mining. In: Hybrid Metaheuristics. Berlin: Springer, 2013. 417–432CrossRefGoogle Scholar
  35. 35.
    Nallaperuma S, Wagner M, Neumann F, et al. A feature-based comparison of local search and the christofides algorithm for the travelling salesperson problem. In: Proceedings of the 12th Workshop on Foundations of Genetic Algorithms XII. New York: ACM, 2013. 147–160CrossRefGoogle Scholar
  36. 36.
    Smith-Miles K, Baatar D, Wreford B, et al. Towards objective measures of algorithm performance across instance space. Comput Oper Res, 2014, 45: 12–24MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Ruhe G, Ngo-The A. Optimized resource allocation for software release planning. IEEE Trans Softw Eng, 2009, 35: 109–123CrossRefGoogle Scholar
  38. 38.
    Wen L, Dromey R, Kirk D. Software engineering and scale-free networks. IEEE Trans Syst Man Cybern, 2009, 39: 845–854CrossRefGoogle Scholar
  39. 39.
    Pisinger D. Core problems in knapsack algorithms. Oper Res, 1999, 47: 570–575MathSciNetCrossRefzbMATHGoogle Scholar
  40. 40.
    Pisinger D. Where are the hard knapsack problems? Comput Oper Res, 2005, 32: 2271–2284MathSciNetCrossRefzbMATHGoogle Scholar
  41. 41.
    Balas E, Zemel E. An algorithm for large zero-one knapsack problems. Oper Res, 1980, 28: 1130–1154MathSciNetCrossRefzbMATHGoogle Scholar
  42. 42.
    Cule E. Ridge: Ridge Regression With Automatic Selection of The Penalty Parameter, 2014. R package version 2.1-3. Scholar
  43. 43.
    Friedman J H. Multivariate adaptive regression splines1. Ann Stat, 1991, 19: 1–141CrossRefGoogle Scholar
  44. 44.
    Breiman L. Random forests. Mach Learn, 2001, 45: 5–32CrossRefzbMATHGoogle Scholar
  45. 45.
    Hutter F, Xu L, Hoos H H, et al. Algorithm runtime prediction: methods & evaluation. Artif Intell, 2014, 206: 79–111MathSciNetCrossRefzbMATHGoogle Scholar
  46. 46.
    Cule E, de Iorio M. Ridge regression in prediction problems: automatic choice of the ridge parameter. Genetic Epidemiol, 2013, 37: 704–714CrossRefGoogle Scholar
  47. 47.
    López-Ibánez M, Dubois-Lacoste J, Stützle T, et al. The Irace Package, Iterated Race for Automatic Algorithm Configuration. Technical Report Technical Report TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium, 2011Google Scholar
  48. 48.
    Kuhn M. Caret: Classification and Regression Training, 2012. R package version 5.15-044. src/contrib/Archive/caret/Google Scholar
  49. 49.
    Milborrow S. Earth: Multivariate Adaptive Regression Splines, 2015. R package version 4.4.2. https://cran.r-project. org/src/contrib/Archive/earth/Google Scholar
  50. 50.
    Liaw A, Wiener M. Classification and regression by randomforest. R News, 2002, 2: 18–22Google Scholar
  51. 51.
    Jiang H, Sun W, Ren Z, et al. Evolving hard and easy traveling salesman problem instances: a multi-objective approach. In: Simulated Evolution and Learning. Berlin: Springer, 2014. 216–227Google Scholar
  52. 52.
    Sun X, Gong D, Jin Y, et al. A new surrogate-assisted interactive genetic algorithm with weighted semisupervised learning. IEEE Trans Cybern, 2013, 43: 685–698CrossRefGoogle Scholar
  53. 53.
    Gong W, Zhou A, Cai Z. A multioperator search strategy based on cheap surrogate models for evolutionary optimization. IEEE Trans Evol Comput, 2015, 19: 746–758CrossRefGoogle Scholar
  54. 54.
    Nallaperuma S, Wagner M, Neumann F. Parameter prediction based on features of evolved instances for ant colony optimization and the traveling salesperson problem. In: Parallel Problem Solving from Nature–PPSN XIII. Berlin: Springer, 2014. 100–109Google Scholar
  55. 55.
    Tang K, Peng F, Chen G, et al. Population-based algorithm portfolios with automated constituent algorithms selection. Inf Sci, 2014, 279: 94–104CrossRefGoogle Scholar

Copyright information

© Science China Press and Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Zhilei Ren
    • 1
    • 2
  • He Jiang
    • 1
    • 2
  • Jifeng Xuan
    • 3
  • Shuwei Zhang
    • 1
    • 2
  • Zhongxuan Luo
    • 1
    • 2
  1. 1.Key Laboratory for Ubiquitous Network and Service Software of Liaoning ProvinceDalian University of TechnologyDalianChina
  2. 2.School of SoftwareDalian University of TechnologyDalianChina
  3. 3.State Key Laboratory of Software EngineeringWuhan UniversityWuhanChina

Personalised recommendations