Matrix Factorization Based Benchmark Set Analysis: A Case Study on HyFlex

  • Mustafa MısırEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10593)


The present paper offers an analysis strategy to examine benchmark sets of combinatorial search problems. Experimental analysis has been widely used to compare a set of algorithms on a group of instances from such problem domains. These studies mostly focus on the algorithms’ performance rather than the quality of the target benchmark set. In relation to that, the insights about the algorithms’ varying performance happen to be highly limited. The goal here is to introduce a benchmark set analysis strategy that can tell the quality of a benchmark set while allowing to retrieve some insights regarding the algorithms’ performance. A matrix factorization based strategy is utilized for this purpose. A Hyper-heuristic framework, i.e. HyFlex, involving 6 problem domains is accommodated as the testbed to perform the analysis on.


  1. 1.
    Koren, Y., Bell, R., Volinsky, C., et al.: Matrix factorization techniques for recommender systems. Computer 42(8), 30–37 (2009)CrossRefGoogle Scholar
  2. 2.
    Su, X., Khoshgoftaar, T.M.: A survey of collaborative filtering techniques. Adv. Artif. Intell. 2009, 4 (2009)CrossRefGoogle Scholar
  3. 3.
    Mısır, M., Sebag, M.: Alors: an algorithm recommender system. Artif. Intell. 244, 291–314 (2017)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Rice, J.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)CrossRefGoogle Scholar
  5. 5.
    Golub, G.H., Reinsch, C.: Singular value decomposition and least squares solutions. Numer. Math. 14(5), 403–420 (1970)MathSciNetCrossRefzbMATHGoogle Scholar
  6. 6.
    Ochoa, G., et al.: HyFlex: a benchmark framework for cross-domain heuristic search. In: Hao, J.-K., Middendorf, M. (eds.) EvoCOP 2012. LNCS, vol. 7245, pp. 136–147. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-29124-1_12 CrossRefGoogle Scholar
  7. 7.
    Burke, E.K., Hyde, M., Kendall, G., Ochoa, G., Ozcan, E., Woodward, J.R.: A classification of hyper-heuristic approaches. In: Gendreau, M., Potvin, J.Y. (eds.) Handbook of Metaheuristics. International Series in Operations Research & Management Science, vol. 146, pp. 449–468. Springer, Boston (2010)CrossRefGoogle Scholar
  8. 8.
    Burke, E.K., Gendreau, M., Hyde, M., Kendall, G., Ochoa, G., Özcan, E., Qu, R.: Hyper-heuristics: a survey of the state of the art. J. Oper. Res. Soc. 64(12), 1695–1724 (2013)CrossRefGoogle Scholar
  9. 9.
    Chen, S., Li, Z., Yang, B., Rudolph, G.: Quantum-inspired hyper-heuristics for energy-aware scheduling on heterogeneous computing systems. IEEE Trans. Parallel Distrib. Syst. 27(6), 1796–1810 (2016)CrossRefGoogle Scholar
  10. 10.
    Pillay, N.: A review of hyper-heuristics for educational timetabling. Ann. Oper. Res. 239(1), 3–38 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Terashima-Marin, H., Morán-Saavedra, A., Ross, P.: Forming hyper-heuristics with gas when solving 2D-regular cutting stock problems. In: IEEE Congress on Evolutionary Computation (CEC), vol. 2, pp. 1104–1110. IEEE (2005)Google Scholar
  12. 12.
    Sotelo-Figueroa, M.A., Soberanes, H.J.P., Carpio, J.M., Huacuja, H.J.F., Reyes, L.C., Alcaraz, J.A.S., Espinal, A.: Generating bin packing heuristic through grammatical evolution based on bee swarm optimization. In: Melin, P., Castillo, O., Kacprzyk, J. (eds.) Nature-Inspired Design of Hybrid Intelligent Systems. SCI, vol. 667, pp. 655–671. Springer, Cham (2017). doi: 10.1007/978-3-319-47054-2_43 CrossRefGoogle Scholar
  13. 13.
    Bader-El-Den, M., Poli, R.: Generating SAT local-search heuristics using a GP hyper-heuristic framework. In: Monmarché, N., Talbi, E.-G., Collet, P., Schoenauer, M., Lutton, E. (eds.) EA 2007. LNCS, vol. 4926, pp. 37–49. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-79305-2_4 CrossRefGoogle Scholar
  14. 14.
    Burke, E.K., Hyde, M.R., Kendall, G., Ochoa, G., Ozcan, E., Woodward, J.R.: Exploring hyper-heuristic methodologies with genetic programming. In: Mumford, C.L., Jain, L.C. (eds.) Computational Intelligence. Intelligent Systems Reference Library, vol. 1, pp. 177–201. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  15. 15.
    Burke, E.K., MacCarthy, B.L., Petrovic, S., Qu, R.: Knowledge discovery in a hyper-heuristic for course timetabling using case-based reasoning. In: Burke, E., De Causmaecker, P. (eds.) Practice and Theory of Automated Timetabling IV. LNCS, vol. 2740, pp. 276–287. Springer, Heidelberg (2003). doi: 10.1007/978-3-540-45157-0_18 CrossRefGoogle Scholar
  16. 16.
    Maashi, M., Kendall, G., Özcan, E.: Choice function based hyper-heuristics for multi-objective optimization. Appl. Soft Comput. 28, 312–326 (2015)CrossRefGoogle Scholar
  17. 17.
    Marín-Blázquez, J.G., Schulenburg, S.: A hyper-heuristic framework with XCS: learning to create novel problem-solving algorithms constructed from simpler algorithmic ingredients. In: Kovacs, T., Llorà, X., Takadama, K., Lanzi, P.L., Stolzmann, W., Wilson, S.W. (eds.) IWLCS 2003-2005. LNCS, vol. 4399, pp. 193–218. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-71231-2_14 CrossRefGoogle Scholar
  18. 18.
    Cowling, P., Kendall, G., Soubeiga, E.: A hyperheuristic approach to scheduling a sales summit. In: Burke, E., Erben, W. (eds.) PATAT 2000. LNCS, vol. 2079, pp. 176–190. Springer, Heidelberg (2001). doi: 10.1007/3-540-44629-X_11 CrossRefGoogle Scholar
  19. 19.
    Da Costa, L., Fialho, A., Schoenauer, M., Sebag, M.: Adaptive operator selection with dynamic multi-armed bandits. In: Proceedings of Genetic and Evolutionary Computation Conference (GECCO), PP. 913–920. Atlanta, Georgia, USA (2008)Google Scholar
  20. 20.
    Epitropakis, M.G., Caraffini, F., Neri, F., Burke, E.K.: A separability prototype for automatic memes with adaptive operator selection. In: IEEE Symposium on Foundations of Computational Intelligence (FOCI), PP. 70–77. IEEE (2014)Google Scholar
  21. 21.
    Kotthoff, L.: Algorithm selection for combinatorial search problems: a survey. AI Mag. 35(3), 48–60 (2014)CrossRefGoogle Scholar
  22. 22.
    Park, J., Mei, Y., Nguyen, S., Chen, G., Johnston, M., Zhang, M.: Genetic programming based hyper-heuristics for dynamic job shop scheduling: cooperative coevolutionary approaches. In: Heywood, M.I., McDermott, J., Castelli, M., Costa, E., Sim, K. (eds.) EuroGP 2016. LNCS, vol. 9594, pp. 115–132. Springer, Cham (2016). doi: 10.1007/978-3-319-30668-1_8 CrossRefGoogle Scholar
  23. 23.
    Sotelo-Figueroa, M., Soberanes, H., Carpio, J., Fraire Huacuja, H., Reyes, L., Soria Alcaraz, J.: Evolving bin packing heuristic using micro-differential evolution with indirect representation. In: Castillo, O., Melin, P., Kacprzyk, J. (eds.) Recent Advances on Hybrid Intelligent Systems, vol. 451, pp. 349–359. Studies in Computational Intelligence. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  24. 24.
    Cheeseman, P., Kanefsky, B., Taylor, W.M.: Where the really hard problems are. In: IJCAI, vol. 91, pp. 331–337 (1991)Google Scholar
  25. 25.
    Jones, T., Forrest, S., et al.: Fitness distance correlation as a measure of problem difficulty for genetic algorithms. ICGA 95, 184–192 (1995)Google Scholar
  26. 26.
    Ruan, Y., Kautz, H.A., Horvitz, E.: The backdoor key: a path to understanding problem hardness. In: AAAI, vol. 4, pp. 118–123 (2004)Google Scholar
  27. 27.
    Smith-Miles, K., Lopes, L.: Measuring instance difficulty for combinatorial optimization problems. Comput. Oper. Res. 39(5), 875–889 (2012)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Leyton-Brown, K., Hoos, H.H., Hutter, F., Xu, L.: Understanding the empirical hardness of NP-complete problems. Commun. ACM 57(5), 98–107 (2014)CrossRefGoogle Scholar
  29. 29.
    van Hemert, J.I.: Evolving combinatorial problem instances that are difficult to solve. Evol. Comput. 14(4), 433–462 (2006)CrossRefGoogle Scholar
  30. 30.
    Smith-Miles, K., van Hemert, J.I.: Discovering the suitability of optimisation algorithms by learning from evolved instances. Ann. Math. Artif. Intell. 61(2), 87 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Lopes, L., Smith-Miles, K.: Generating applicable synthetic instances for branch problems. Oper. Res. 61(3), 563–577 (2013)CrossRefzbMATHGoogle Scholar
  32. 32.
    Smith-Miles, K., Bowly, S.: Generating new test instances by evolving in instance space. Comput. Oper. Res. 63, 102–113 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Malitsky, Y., Merschformann, M., O’Sullivan, B., Tierney, K.: Structure-preserving instance generation. In: Festa, P., Sellmann, M., Vanschoren, J. (eds.) LION 2016. LNCS, vol. 10079, pp. 123–140. Springer, Cham (2016). doi: 10.1007/978-3-319-50349-3_9 CrossRefGoogle Scholar
  34. 34.
    Smith-Miles, K., Tan, T.T.: Measuring algorithm footprints in instance space. In: IEEE Congress on Evolutionary Computation (CEC), pp. 1–8. IEEE (2012)Google Scholar
  35. 35.
    Jolliffe, I.: Principal Component Analysis. Wiley Online Library, Hoboken (2002)zbMATHGoogle Scholar
  36. 36.
    Saul, L.K., Roweis, S.T.: Think globally, fit locally: unsupervised learning of low dimensional manifolds. J. Mach. Learn. Res. 4, 119–155 (2003)MathSciNetzbMATHGoogle Scholar
  37. 37.
    Kaufman, L., Rousseeuw, P.J.: Finding Groups in Data: an Introduction to Cluster Analysis, vol. 344. John Wiley & Sons, Hoboken (2009)zbMATHGoogle Scholar
  38. 38.
    Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987)CrossRefzbMATHGoogle Scholar
  39. 39.
    Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)CrossRefzbMATHGoogle Scholar
  40. 40.
    Mısır, M.: Intelligent hyper-heuristics: a tool for solving generic optimisation problems. Ph.D. thesis, Department of Computer Science, KU Leuven (2012)Google Scholar
  41. 41.
    Mısır, M., Handoko, S.D., Lau, H.C.: OSCAR: online selection of algorithm portfolios with case study on memetic algorithms. In: Dhaenens, C., Jourdan, L., Marmion, M.-E. (eds.) LION 2015. LNCS, vol. 8994, pp. 59–73. Springer, Cham (2015). doi: 10.1007/978-3-319-19084-6_6 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.College of Computer Science and TechnologyNanjing University of Aeronautics and AstronauticsNanjingChina

Personalised recommendations