First Investigations on Noisy Model-Based Multi-objective Optimization

  • Daniel Horn
  • Melanie Dagge
  • Xudong Sun
  • Bernd Bischl
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10173)


In many real-world applications concerning multi-objective optimization, the true objective functions are not observable. Instead, only noisy observations are available. In recent years, the interest in the effect of such noise in evolutionary multi-objective optimization (EMO) has increased and many specialized algorithms have been proposed. However, evolutionary algorithms are not suitable if the evaluation of the objectives is expensive and only a small budget is available. One popular solution is to use model-based multi-objective optimization (MBMO) techniques. In this paper, we present a first investigation on noisy MBMO. For this purpose we collect several noise handling strategies from the field of EMO and adapt them for MBMO algorithms. We compare the performance of those strategies in two benchmark situations: Firstly, we perform a purely artificial benchmark using homogeneous Gaussian noise. Secondly, we choose a setting from the field of machine learning, where the structure of the underlying noise is unknown.


Noisy optimization Machine learning Bayesian optimization Model-based optimization Multi-objective optimization 


  1. 1.
    Aizawa, A.N., Wah, B.W.: Scheduling of genetic algorithms in a noisy environment. Evol. Comput. 2(2), 97–122 (1994)CrossRefGoogle Scholar
  2. 2.
    Bischl, B., Mersmann, O., Trautmann, H., Weihs, C.: Resampling methods for meta-model validation with recommendations for evolutionary computation. Evol. Comput. 20(2), 249–275 (2012)CrossRefGoogle Scholar
  3. 3.
    Bischl, B., Lang, M., Kotthoff, L., Schiffner, J., Richter, J., Studerus, E., Casalicchio, G., Jones, Z.M.: mlr: machine learning in R. J. Mach. Learn. Res. 17(170), 1–5 (2016)MathSciNetMATHGoogle Scholar
  4. 4.
    Bischl, B., Lang, M., Mersmann, O., Rahnenführer, J., Weihs, C.: BatchJobs and BatchExperiments: abstraction mechanisms for using R in batch environments. J. Stat. Softw. 64(11), 1–25 (2015)CrossRefGoogle Scholar
  5. 5.
    Bischl, B., Richter, J., Bossek, J., Horn, D., Lang, M.: mlrMBO: A Toolbox for Model-Based Optimization of Expensive Black-Box Functions.
  6. 6.
    Breiderhoff, B., Bartz-Beielstein, T., Naujoks, B., Zaefferer, M., Fischbach, A., Flasch, O., Friese, M., Mersmann, O., Stork, J.: Simulation and optimization of cyclone dust separators. In: Proceedings of the 23rd Workshop on Computational Intelligence, pp. 177–195 (2013)Google Scholar
  7. 7.
    Dem\(\breve{\rm {s}}\)ar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)Google Scholar
  8. 8.
    Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Fieldsend, J.E., Everson, R.M.: The rolling tide evolutionary algorithm: a multiobjective optimizer for noisy optimization problems. IEEE Trans. Evol. Comput. 19(1), 103–117 (2015)CrossRefGoogle Scholar
  10. 10.
    Hernández-Lobato, D., Hernández-Lobato, J.M., Shah, A., Adams, R.P.: Predictive entropy search for multi-objective bayesian optimization. arXiv preprint arXiv:1511.05467 (2015)
  11. 11.
    Horn, D., Bischl, B.: Multi-objective parameter configuration of machine learning algorithms using model-based optimization. In: IEEE Symposium on Computational Intelligence in Multicriteria Decision-Making (2016, accepted)Google Scholar
  12. 12.
    Horn, D., Demircioğlu, A., Bischl, B., Glasmachers, T., Weihs, C.: A comparative study on large scale kernelized support vector machines. Adv. Data Anal. Classif., 1–17 (2016)Google Scholar
  13. 13.
    Horn, D., Wagner, T., Biermann, D., Weihs, C., Bischl, B.: Model-based multi-objective optimization: taxonomy, multi-point proposal, toolbox and benchmark. In: International Conference on Evolutionary Multi-criterion Optimization, pp. 64–78 (2015)Google Scholar
  14. 14.
    Jin, Y., Branke, J.: Evolutionary optimization in uncertain environments - a survey. IEEE Trans. Evol. Comput. 9(3), 303–317 (2005)CrossRefGoogle Scholar
  15. 15.
    Jones, D.R., Schonlau, M., Welch, W.J.: Efficient global optimization of expensive black-box functions. J. Global Optim. 13(4), 455–492 (1998)MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Kleijnen, J.P.C.: White noise assumptions revisited: regression metamodels and experimental designs in practice. In: Proceedings of the 38th Conference on Winter Simulation, pp. 107–117 (2006)Google Scholar
  17. 17.
    Knowles, J.: Parego: a hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Trans. Evol. Comput. 10(1), 50–66 (2006)CrossRefGoogle Scholar
  18. 18.
    Knowles, J., Corne, D., Reynolds, A.: Noisy multiobjective optimization on a budget of 250 evaluations. In: International Conference on Evolutionary Multi-criterion Optimization, pp. 36–50 (2009)Google Scholar
  19. 19.
    Koch, P., Wagner, T., Emmerich, M.T., Bck, T., Konen, W.: Efficient multi-criteria optimization on noisy machine learning problems. Appl. Soft Comput. 29, 357–370 (2015)CrossRefGoogle Scholar
  20. 20.
    Ponweiser, W., Wagner, T., Biermann, D., Vincze, M.: Multiobjective optimization on a limited amount of evaluations using model-assisted \(\cal{S}\)-metric selection. In: Proceedings of the 10th International Conference on Parallel Problem Solving from Nature (PPSN), pp. 784–794 (2008)Google Scholar
  21. 21.
    Schiffner, J., Bischl, B., Lang, M., Richter, J., Jones, Z.M., Probst, P., Pfisterer, F., Gallo, M., Kirchhoff, D., Kühn, T., Thomas, J., Kotthoff, L.: mlr tutorial. CoRR abs/1609.06146 (2016)Google Scholar
  22. 22.
    Schutze, O., Esquivel, X., Lara, A., Coello, C.A.C.: Using the averaged hausdorff distance as a performance measure in evolutionary multiobjective optimization. IEEE Trans. Evol. Comput. 16(4), 504–522 (2012)CrossRefGoogle Scholar
  23. 23.
    Syberfeldt, A., Ng, A., John, R.I., Moore, P.: Evolutionary optimisation of noisy multi-objective problems using confidence-based dynamic resampling. Eur. J. Oper. Res. 204(3), 533–544 (2010)MathSciNetCrossRefMATHGoogle Scholar
  24. 24.
    Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: networked science in machine learning. SIGKDD Explor. Newsl. 15(2), 49–60 (2014)CrossRefGoogle Scholar
  25. 25.
    Zhan, J., Ma, Y., Zhu, L.: Multiobjective simulation optimization using stochastic kriging. In: Proceedings of the 22nd International Conference on Industrial Engineering and Engineering Management 2015, pp. 81–91 (2016)Google Scholar
  26. 26.
    Zhang, Q., Zhou, A., Zhaoy, S., Suganthany, P.N., Liu, W., Tiwari, S.: Multiobjective optimization test instances for the cec 2009 special session and competition. University of Essex and Nanyang Technological University, Technical report (2008)Google Scholar
  27. 27.
    Zhou, A., Qu, B.Y., Li, H., Zhao, S.Z., Suganthan, P.N., Zhang, Q.: Multiobjective evolutionary algorithms: a survey of the state of the art. Swarm Evol. Comput. 1, 32–49 (2011)CrossRefGoogle Scholar
  28. 28.
    Zuluaga, M., Krause, A., Püschel, M.: e-pal: an active learning approach to the multi-objective optimization problem. J. Mach. Learn. Res. 17(104), 1–32 (2016)MathSciNetMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Daniel Horn
    • 1
  • Melanie Dagge
    • 2
  • Xudong Sun
    • 3
  • Bernd Bischl
    • 3
  1. 1.TU Dortmund University, Computational StatisticsDortmundGermany
  2. 2.TU Dortmund University, Statistics with Applications in the Field of Engineering SciencesDortmundGermany
  3. 3.LMU Munich, Computational StatisticsMunichGermany

Personalised recommendations