Abstract
Several methods were developed to solve cost-extensive multi-criteria optimization problems by reducing the number of function evaluations by means of surrogate optimization. In this study, we apply different multi-criteria surrogate optimization methods to improve (tune) an event-detection software for water-quality monitoring. For tuning two important parameters of this software, four state-of-the-art methods are compared: S-Metric-Selection Efficient Global Optimization (SMS-EGO), S-Metric-Expected Improvement for Efficient Global Optimization SExI-EGO, Euclidean Distance based Expected Improvement Euclid-EI (here referred to as MEI-SPOT due to its implementation in the Sequential Parameter Optimization Toolbox SPOT) and a multi-criteria approach based on SPO (MSPOT).
Analyzing the performance of the different methods provides insight into the working-mechanisms of cutting-edge multi-criteria solvers. As one of the approaches, namely MSPOT, does not consider the prediction variance of the surrogate model, it is of interest whether this can lead to premature convergence on the practical tuning problem. Furthermore, all four approaches will be compared to a simple SMS-EMOA to validate that the use of surrogate models is justified on this problem.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Bartz-Beielstein, T., Parsopoulos, K.E., Vrahatis, M.N.: Design and analysis of optimization algorithms using computational statistics. Applied Numerical Analysis and Computational Mathematics (ANACM) 1(2), 413–433 (2004)
Beume, N., Naujoks, B., Emmerich, M.: SMS-EMOA: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research 181(3), 1653–1669 (2007)
Emmerich, M.: Single- and Multi-objective Evolutionary Design Optimization: Assisted by Gaussian Random Field Metamodels. PhD thesis, Universität Dortmund, Germany (2005)
Emmerich, M., Deutz, A., Klinkenberg, J.: Hypervolume-based expected improvement: Monotonicity properties and exact computation. In: 2011 IEEE Congress on Evolutionary Computation (CEC), pp. 2147–2154. IEEE (2011)
Everson, R.M., Fieldsend, J.E.: Multi-class ROC analysis from a multi-objective optimisation perspective. Pattern Recognition Letters 27(8), 918–927 (2006)
Everson, R.M., Fieldsend, J.E.: Multi-objective Optimisation for Receiver Operating Characteristic Analysis. In: Jin, Y. (ed.) Multi-Objective Machine Learning. SCI, vol. 16, pp. 533–556. Springer, Heidelberg (2006)
Fieldsend, J.E., Everson, R.M.: ROC Optimisation of Safety Related Systems. In: Hernández-Orallo, J., Ferri, C., Lachiche, N., Flach, P.A. (eds.) ROCAI, pp. 37–44 (2004)
Flach, P.A., Wu, S.: Repairing concavities in ROC curves. In: Proceedings of the 19th International Joint Conference on Artificial Intelligence, IJCAI 2005, pp. 702–707. Morgan Kaufmann Publishers Inc., San Francisco (2005)
Forrester, A., Sobester, A., Keane, A.: Engineering Design via Surrogate Modelling. Wiley (2008)
Hart, D.B., Klise, K.A., Vugrin, E.D., McKenna, S.A., Wilson, M.P.: Canary user’s manual and software upgrades. Technical Report EPA/600/R-08/040A, U.S. Environmental Protection Agency, Washington, DC (2009)
Jeong, S., Obayashi, S.: Efficient global optimization (EGO) for multi-objective problem and data mining. In: Corne, D., et al. (eds.) IEEE Congress on Evolutionary Computation, pp. 2138–2145. IEEE (2005)
Jin, Y.: A comprehensive survey of fitness approximation in evolutionary computation. Soft Computing 9(1), 3–12 (2005)
Jones, D., Schonlau, M., Welch, W.: Efficient global optimization of expensive black-box functions. Journal of Global Optimization 13, 455–492 (1998)
Keane, A.: Statistical improvement criteria for use in multiobjective design optimisation. AIAA Journal 44(4), 879–891 (2006)
Knowles, J.: Parego: A hybrid algorithm with on-line landscape approximation for expensive multiobjective optimization problems. IEEE Transactions on Evolutionary Computation 10(1), 50–66 (2006)
Knowles, J.D., Nakayama, H.: Meta-Modeling in Multiobjective Optimization. In: Branke, J., Deb, K., Miettinen, K., Słowiński, R. (eds.) Multiobjective Optimization. LNCS, vol. 5252, pp. 245–284. Springer, Heidelberg (2008)
Kupinski, M.A., Anastasio, M.A.: Multiobjective genetic optimization of diagnostic classifiers with implications for generating receiver operating characteristic curves. IEEE Transactions on Medical Imaging 18, 675–685 (1999)
Lophaven, S., Nielsen, H., Søndergaard, J.: DACE—A Matlab Kriging Toolbox. Technical Report IMM-REP-2002-12, Informatics and Mathematical Modelling, Technical University of Denmark, Copenhagen, Denmark (2002)
Murray, R., Haxton, T., McKenna, S.A., Hart, D.B., Klise, K., Koch, M., Vugrin, E.D., Martin, S., Wilson, M., Cruz, V., Cutler, L.: Water quality event detection systems for drinking water contamination warning systems—development, testing, and application of CANARY. Technical Report EPA/600/R-10/036, National Homeland Security Research Center (May 2010)
Ponweiser, W., Wagner, T., Biermann, D., Vincze, M.: Multiobjective Optimization on a Limited Budget of Evaluations Using Model-Assisted \(\mathcal{S}\)-Metric Selection. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (eds.) PPSN X. LNCS, vol. 5199, pp. 784–794. Springer, Heidelberg (2008)
Voutchkov, I., Keane, A.: Multiobjective optimization using surrogates. In: Adaptive Computing in Design and Manufacture ACDM, pp. 167–175 (2006)
Wagner, T., Emmerich, M., Deutz, A., Ponweiser, W.: On Expected-Improvement Criteria for Model-based Multi-objective Optimization. In: Schaefer, R., Cotta, C., Kołodziej, J., Rudolph, G. (eds.) PPSN XI. LNCS, vol. 6238, pp. 718–727. Springer, Heidelberg (2010)
Zaefferer, M.: Optimization and empirical analysis of an event detection software for water quality monitoring. Master’s thesis, Cologne University of Applied Sciences (May 2012)
Zaefferer, M., Bartz-Beielstein, T., Friese, M., Naujoks, B., Flasch, O.: Multi-criteria optimization for hard problems under limited budgets. In: Soule, T., et al. (eds.) GECCO 2012 Proceedings, Philadelphia, Pennsylvania, USA, pp. 1451–1452. ACM (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zaefferer, M., Bartz-Beielstein, T., Naujoks, B., Wagner, T., Emmerich, M. (2013). A Case Study on Multi-Criteria Optimization of an Event Detection Software under Limited Budgets. In: Purshouse, R.C., Fleming, P.J., Fonseca, C.M., Greco, S., Shaw, J. (eds) Evolutionary Multi-Criterion Optimization. EMO 2013. Lecture Notes in Computer Science, vol 7811. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37140-0_56
Download citation
DOI: https://doi.org/10.1007/978-3-642-37140-0_56
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-37139-4
Online ISBN: 978-3-642-37140-0
eBook Packages: Computer ScienceComputer Science (R0)