A Case Study on Multi-Criteria Optimization of an Event Detection Software under Limited Budgets

  • Martin Zaefferer
  • Thomas Bartz-Beielstein
  • Boris Naujoks
  • Tobias Wagner
  • Michael Emmerich
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7811)


Several methods were developed to solve cost-extensive multi-criteria optimization problems by reducing the number of function evaluations by means of surrogate optimization. In this study, we apply different multi-criteria surrogate optimization methods to improve (tune) an event-detection software for water-quality monitoring. For tuning two important parameters of this software, four state-of-the-art methods are compared: S-Metric-Selection Efficient Global Optimization (SMS-EGO), S-Metric-Expected Improvement for Efficient Global Optimization SExI-EGO, Euclidean Distance based Expected Improvement Euclid-EI (here referred to as MEI-SPOT due to its implementation in the Sequential Parameter Optimization Toolbox SPOT) and a multi-criteria approach based on SPO (MSPOT).

Analyzing the performance of the different methods provides insight into the working-mechanisms of cutting-edge multi-criteria solvers. As one of the approaches, namely MSPOT, does not consider the prediction variance of the surrogate model, it is of interest whether this can lead to premature convergence on the practical tuning problem. Furthermore, all four approaches will be compared to a simple SMS-EMOA to validate that the use of surrogate models is justified on this problem.


Receiver Operator Characteristic Receiver Operator Characteristic Curve False Positive Rate Pareto Front False Alarm Rate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bartz-Beielstein, T., Parsopoulos, K.E., Vrahatis, M.N.: Design and analysis of optimization algorithms using computational statistics. Applied Numerical Analysis and Computational Mathematics (ANACM) 1(2), 413–433 (2004)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Beume, N., Naujoks, B., Emmerich, M.: SMS-EMOA: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research 181(3), 1653–1669 (2007)zbMATHCrossRefGoogle Scholar
  3. 3.
    Emmerich, M.: Single- and Multi-objective Evolutionary Design Optimization: Assisted by Gaussian Random Field Metamodels. PhD thesis, Universität Dortmund, Germany (2005)Google Scholar
  4. 4.
    Emmerich, M., Deutz, A., Klinkenberg, J.: Hypervolume-based expected improvement: Monotonicity properties and exact computation. In: 2011 IEEE Congress on Evolutionary Computation (CEC), pp. 2147–2154. IEEE (2011)Google Scholar
  5. 5.
    Everson, R.M., Fieldsend, J.E.: Multi-class ROC analysis from a multi-objective optimisation perspective. Pattern Recognition Letters 27(8), 918–927 (2006)CrossRefGoogle Scholar
  6. 6.
    Everson, R.M., Fieldsend, J.E.: Multi-objective Optimisation for Receiver Operating Characteristic Analysis. In: Jin, Y. (ed.) Multi-Objective Machine Learning. SCI, vol. 16, pp. 533–556. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  7. 7.
    Fieldsend, J.E., Everson, R.M.: ROC Optimisation of Safety Related Systems. In: Hernández-Orallo, J., Ferri, C., Lachiche, N., Flach, P.A. (eds.) ROCAI, pp. 37–44 (2004)Google Scholar
  8. 8.
    Flach, P.A., Wu, S.: Repairing concavities in ROC curves. In: Proceedings of the 19th International Joint Conference on Artificial Intelligence, IJCAI 2005, pp. 702–707. Morgan Kaufmann Publishers Inc., San Francisco (2005)Google Scholar
  9. 9.
    Forrester, A., Sobester, A., Keane, A.: Engineering Design via Surrogate Modelling. Wiley (2008)Google Scholar
  10. 10.
    Hart, D.B., Klise, K.A., Vugrin, E.D., McKenna, S.A., Wilson, M.P.: Canary user’s manual and software upgrades. Technical Report EPA/600/R-08/040A, U.S. Environmental Protection Agency, Washington, DC (2009)Google Scholar
  11. 11.
    Jeong, S., Obayashi, S.: Efficient global optimization (EGO) for multi-objective problem and data mining. In: Corne, D., et al. (eds.) IEEE Congress on Evolutionary Computation, pp. 2138–2145. IEEE (2005)Google Scholar
  12. 12.
    Jin, Y.: A comprehensive survey of fitness approximation in evolutionary computation. Soft Computing 9(1), 3–12 (2005)CrossRefGoogle Scholar
  13. 13.
    Jones, D., Schonlau, M., Welch, W.: Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13, 455–492 (1998)MathSciNetzbMATHCrossRefGoogle Scholar
  14. 14.
    Keane, A.: Statistical improvement criteria for use in multiobjective design optimisation. AIAA Journal 44(4), 879–891 (2006)CrossRefGoogle Scholar
  15. 15.
    Knowles, J.: Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation 10(1), 50–66 (2006)CrossRefGoogle Scholar
  16. 16.
    Knowles, J.D., Nakayama, H.: Meta-Modeling in Multiobjective Optimization. In: Branke, J., Deb, K., Miettinen, K., Słowiński, R. (eds.) Multiobjective Optimization. LNCS, vol. 5252, pp. 245–284. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  17. 17.
    Kupinski, M.A., Anastasio, M.A.: Multiobjective genetic optimization of diagnostic classifiers with implications for generating receiver operating characteristic curves. IEEE Transactions on Medical Imaging 18, 675–685 (1999)CrossRefGoogle Scholar
  18. 18.
    Lophaven, S., Nielsen, H., Søndergaard, J.: DACE—A Matlab Kriging Toolbox. Technical Report IMM-REP-2002-12, Informatics and Mathematical Modelling, Technical University of Denmark, Copenhagen, Denmark (2002)Google Scholar
  19. 19.
    Murray, R., Haxton, T., McKenna, S.A., Hart, D.B., Klise, K., Koch, M., Vugrin, E.D., Martin, S., Wilson, M., Cruz, V., Cutler, L.: Water quality event detection systems for drinking water contamination warning systems—development, testing, and application of CANARY. Technical Report EPA/600/R-10/036, National Homeland Security Research Center (May 2010)Google Scholar
  20. 20.
    Ponweiser, W., Wagner, T., Biermann, D., Vincze, M.: Multiobjective Optimization on a Limited Budget of Evaluations Using Model-Assisted \(\mathcal{S}\)-Metric Selection. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN X. LNCS, vol. 5199, pp. 784–794. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  21. 21.
    Voutchkov, I., Keane, A.: Multiobjective optimization using surrogates. In: Adaptive Computing in Design and Manufacture ACDM, pp. 167–175 (2006)Google Scholar
  22. 22.
    Wagner, T., Emmerich, M., Deutz, A., Ponweiser, W.: On Expected-Improvement Criteria for Model-based Multi-objective Optimization. In: Schaefer, R., Cotta, C., Kołodziej, J., Rudolph, G. (eds.) PPSN XI. LNCS, vol. 6238, pp. 718–727. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  23. 23.
    Zaefferer, M.: Optimization and empirical analysis of an event detection software for water quality monitoring. Master’s thesis, Cologne University of Applied Sciences (May 2012)Google Scholar
  24. 24.
    Zaefferer, M., Bartz-Beielstein, T., Friese, M., Naujoks, B., Flasch, O.: Multi-criteria optimization for hard problems under limited budgets. In: Soule, T., et al. (eds.) GECCO 2012 Proceedings, Philadelphia, Pennsylvania, USA, pp. 1451–1452. ACM (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Martin Zaefferer
    • 1
  • Thomas Bartz-Beielstein
    • 1
  • Boris Naujoks
    • 1
  • Tobias Wagner
    • 2
  • Michael Emmerich
    • 3
  1. 1.Faculty for Computer and Engineering SciencesCologne University of Applied SciencesGummersbachGermany
  2. 2.Institute of Machining Technology (ISF)TU Dortmund UniversityDortmundGermany
  3. 3.Leiden Institute for Advanced Computer ScienceLeiden UniversityThe Netherlands

Personalised recommendations