Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

Feature based problem hardness understanding for requirements engineering


  • 100 Accesses

  • 2 Citations


Heuristics and metaheuristics have achieved great accomplishments in various fields, and the investigation of the relationship between these algorithms and the problem hardness has been a hot topic in the research field. Related research work has contributed much to the understanding of the underlying mechanisms of the algorithms for problem solving. However, most existing studies consider traditional combinatorial problems as their case studies. In this study, taking the Next Release Problem (NRP) from the requirements engineering as a case study, we investigate the relationship between software engineering problem instances and heuristics. We employ an evolutionary algorithm to evolve NRP instances, which are uniquely hard or easy for the target heuristic (Greedy Randomized Adaptive Search Procedure and Randomized Hill Climbing in this paper). Then, we use a feature-based method to estimate the hardness of the evolved instances, with respect to the target heuristic. Experimental results demonstrate that, evolutionary algorithm can be used to evolve NRP instances that are uniquely hard or easy to solve. Moreover, the features enable the estimation of the target heuristics’ performance.



启发式和元启发式算法在各种生产实践中, 已经取得了巨大的成功。 其中, 算法与问题难度间的分析一直是该领域中的研究热点。 然而, 已有的研究大多关注经典的组合优化问题。 本文中, 我们以需求工程中的下一版本问题为案例, 着重分析软件工程问题实例与启发式算法之间的关系。 具体的, 我们使用演化算法, 来针对目标算法演化困难与简单的问题实例。 接着, 我们使用基于问题特征的方法来估计所得实例的难度。 实验结果表明, 我们设计的算法能够生成具有显著困难区分度的问题实例。 并且, 我们提出的问题特征能够用于实现对目标算法性能的有效预测。

This is a preview of subscription content, log in to check access.


  1. 1

    Gong W, Cai Z, Liang D. Adaptive ranking mutation operator based differential evolution for constrained optimization. IEEE Trans Cybern, 2015, 45: 716–727

  2. 2

    Gong W, Cai Z. Differential evolution with ranking-based mutation operators. IEEE Trans Cybern, 2013, 43: 2066–2081

  3. 3

    Luo C, Cai S, Su K, et al. Clause states based configuration checking in local search for satisfiability. IEEE Trans Cybern, 2015, 45: 1014–1027

  4. 4

    Chen WN, Zhang J. Ant colony optimization for software project scheduling and staffing with an event-based scheduler. IEEE Trans Softw Eng, 2013, 39: 1–17

  5. 5

    Haeri S, Trajkovic L. Intelligent deflection routing in buffer-less networks. IEEE Trans Cybern, 2015, 45: 316–327

  6. 6

    Tang K, Yang P, Yao X. Negatively correlated cooperative search. arXiv:1504.04914

  7. 7

    Jiang H, Xuan J, Ren Z. Approximate backbone based multilevel algorithm for next release problem. In: Proceedings of the 12th Annual Conference on Genetic and Evolutionary Computation, Portland, 2010. 1333–1340

  8. 8

    Smith-Miles K A. Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput Surv, 2008, 41: 6

  9. 9

    Smith-Miles K A, van Hemert J I. Discovering the suitability of optimisation algorithms by learning from evolved instances. Ann Math Artif Intell, 2011, 61: 87–104

  10. 10

    Mersmann O, Bischl B, Trautmann H, et al. A novel feature-based approach to characterize algorithm performance for the traveling salesperson problem. Ann Math Artif Intell, 2013, 69: 151–182

  11. 11

    Smith-Miles K A. Towards insightful algorithm selection for optimisation using meta-learning concepts. In: Prcoeedings of International Joint Conference on Neural Networks, Hong Kong, 2008. 4118–4124

  12. 12

    van Hemert J I. Evolving combinatorial problem instances that are difficult to solve. Evol Comput, 2006, 14: 433–462

  13. 13

    Vilalta R, Drissi Y. A perspective view and survey of meta-learning. Artif Intell Rev, 2002, 18: 77–95

  14. 14

    Oentaryo R J, Handoko S D, Lau H C. Algorithm selection via ranking. In: Proceedings of the 29th AAAI Conference on Artificial Intelligence, Austin Texas, 2015. 1826–1832

  15. 15

    van Hemert J I. Property analysis of symmetric travelling salesman problem instances acquired through evolution. In: Evolutionary Computation in Combinatorial Optimization. Berlin: Springer, 2005. 122–131

  16. 16

    Xuan J, Jiang H, Ren Z, et al. Solving the large scale next release problem with a backbone-based multilevel algorithm. IEEE Trans Softw Eng, 2012, 38: 1195–1212

  17. 17

    Nie L, Jiang H, Ren Z, et al. Query expansion based on crowd knowledge for code search. IEEE Trans Serv Comput, 2016, 9: 771–783

  18. 18

    Xuan J F, Martinez M, DeMarco F, et al. Nopol: automatic repair of conditional statement bugs in java programs. IEEE Trans Softw Eng, in press. doi: 10.1109/TSE.2016.2560811

  19. 19

    Bagnall A J, Rayward-Smith V J, Whittley I M. The next release problem. Inf Softw Tech, 2001, 43: 883–890

  20. 20

    Greer D, Ruhe G. Software release planning: an evolutionary and iterative approach. Inf Softw Tech, 2004, 46: 243–253

  21. 21

    Ignatiev A, Janota M, Marques-Silva J. Towards efficient optimization in package management systems. In: Proceedings of the 36th International Conference on Software Engineering, Hyderabad, 2014. 745–755

  22. 22

    Sun J, Zhang Q, Yao X. Meta-heuristic combining prior online and offline information for the quadratic assignment problem. IEEE Trans Cybern, 2014, 44: 429–444

  23. 23

    Ren Z, Jiang H, Xuan J, et al. New insights into diversification of hyper-heuristics. IEEE Trans Cybern, 2014, 44: 1747–1761

  24. 24

    Gao WF, Liu S Y, Huang L L. A novel artificial bee colony algorithm based on modified search equation and orthogonal learning. IEEE Trans Cybern, 2013, 43: 1011–1024

  25. 25

    Ren Z, Zhang A, Wen C, et al. A scatter learning particle swarm optimization algorithm for multimodal problems. IEEE Trans Cybern, 2014, 44: 1127–1140

  26. 26

    He J, Zhang J Y, Xuan J F, et al. A hybrid ACO algorithm for the next release problem. In: Proceedings of the 2nd International Conference on Software Engineering and Data Mining, Chengdu, 2010. 166–171

  27. 27

    He J, Ren Z L, Li X C, et al. Transformed search based software engineering: a new paradigm of sbse. In: Search-Based Software Engineering. Berlin: Springer, 2015. 203–218

  28. 28

    van Hemert J I. Evolving binary constraint satisfaction problem instances that are difficult to solve. In: Proceedings of Congress on Evolutionary Computation, Canberra, 2003. 1267–1273

  29. 29

    Julstrom B A. Evolving heuristically difficult instances of combinatorial problems. In: Proceedings of the 11th Annual Conference on Genetic and Evolutionary Computation, Montreal, 2009. 279–286

  30. 30

    Smith-Miles K A, Lopes L. Generalising algorithm performance in instance space: a timetabling case study. In: Proceedings of the 5th International Conference on Learning and Intelligent Optimization, Rome, 2011. 524–538

  31. 31

    Toledo-Suárez C D, Valenzuela-Rendón M, Terashima-Marín H, et al. On the relativity in the assessment of blind optimization algorithms and the problem-algorithm coevolution. In: Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation. New York: ACM, 2007. 1436–1443

  32. 32

    Hall N G, Posner M E. Performance prediction and preselection for optimization and heuristic solution procedures. Oper Res, 2007, 55: 703–716

  33. 33

    Leyton-Brown K, Nudelman E, Shoham Y. Empirical hardness models: methodology and a case study on combinatorial auctions. J ACM, 2009, 56: 22

  34. 34

    Smith-Miles K, Wreford B, Lopes L, et al. Predicting metaheuristic performance on graph coloring problems using data mining. In: Hybrid Metaheuristics. Berlin: Springer, 2013. 417–432

  35. 35

    Nallaperuma S, Wagner M, Neumann F, et al. A feature-based comparison of local search and the christofides algorithm for the travelling salesperson problem. In: Proceedings of the 12th Workshop on Foundations of Genetic Algorithms XII. New York: ACM, 2013. 147–160

  36. 36

    Smith-Miles K, Baatar D, Wreford B, et al. Towards objective measures of algorithm performance across instance space. Comput Oper Res, 2014, 45: 12–24

  37. 37

    Ruhe G, Ngo-The A. Optimized resource allocation for software release planning. IEEE Trans Softw Eng, 2009, 35: 109–123

  38. 38

    Wen L, Dromey R, Kirk D. Software engineering and scale-free networks. IEEE Trans Syst Man Cybern, 2009, 39: 845–854

  39. 39

    Pisinger D. Core problems in knapsack algorithms. Oper Res, 1999, 47: 570–575

  40. 40

    Pisinger D. Where are the hard knapsack problems? Comput Oper Res, 2005, 32: 2271–2284

  41. 41

    Balas E, Zemel E. An algorithm for large zero-one knapsack problems. Oper Res, 1980, 28: 1130–1154

  42. 42

    Cule E. Ridge: Ridge Regression With Automatic Selection of The Penalty Parameter, 2014. R package version 2.1-3. https://cran.r-project.org/src/contrib/Archive/ridge/

  43. 43

    Friedman J H. Multivariate adaptive regression splines1. Ann Stat, 1991, 19: 1–141

  44. 44

    Breiman L. Random forests. Mach Learn, 2001, 45: 5–32

  45. 45

    Hutter F, Xu L, Hoos H H, et al. Algorithm runtime prediction: methods & evaluation. Artif Intell, 2014, 206: 79–111

  46. 46

    Cule E, de Iorio M. Ridge regression in prediction problems: automatic choice of the ridge parameter. Genetic Epidemiol, 2013, 37: 704–714

  47. 47

    López-Ibánez M, Dubois-Lacoste J, Stützle T, et al. The Irace Package, Iterated Race for Automatic Algorithm Configuration. Technical Report Technical Report TR/IRIDIA/2011-004, IRIDIA, Université Libre de Bruxelles, Belgium, 2011

  48. 48

    Kuhn M. Caret: Classification and Regression Training, 2012. R package version 5.15-044. https://cran.r-project.org/ src/contrib/Archive/caret/

  49. 49

    Milborrow S. Earth: Multivariate Adaptive Regression Splines, 2015. R package version 4.4.2. https://cran.r-project. org/src/contrib/Archive/earth/

  50. 50

    Liaw A, Wiener M. Classification and regression by randomforest. R News, 2002, 2: 18–22

  51. 51

    Jiang H, Sun W, Ren Z, et al. Evolving hard and easy traveling salesman problem instances: a multi-objective approach. In: Simulated Evolution and Learning. Berlin: Springer, 2014. 216–227

  52. 52

    Sun X, Gong D, Jin Y, et al. A new surrogate-assisted interactive genetic algorithm with weighted semisupervised learning. IEEE Trans Cybern, 2013, 43: 685–698

  53. 53

    Gong W, Zhou A, Cai Z. A multioperator search strategy based on cheap surrogate models for evolutionary optimization. IEEE Trans Evol Comput, 2015, 19: 746–758

  54. 54

    Nallaperuma S, Wagner M, Neumann F. Parameter prediction based on features of evolved instances for ant colony optimization and the traveling salesperson problem. In: Parallel Problem Solving from Nature–PPSN XIII. Berlin: Springer, 2014. 100–109

  55. 55

    Tang K, Peng F, Chen G, et al. Population-based algorithm portfolios with automated constituent algorithms selection. Inf Sci, 2014, 279: 94–104

Download references

Author information

Correspondence to He Jiang.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Ren, Z., Jiang, H., Xuan, J. et al. Feature based problem hardness understanding for requirements engineering. Sci. China Inf. Sci. 60, 032105 (2017). https://doi.org/10.1007/s11432-016-0089-7

Download citation


  • problem hardness
  • next release problem
  • computational intelligence
  • requirements engineering
  • evolution algorithm


  • 经验困难模型
  • 下一版本问题
  • 计算智能
  • 需求工程
  • 演化算法
  • 原子模型