Structural and Multidisciplinary Optimization

, Volume 37, Issue 3, pp 279–294 | Cite as

Ensemble of metamodels with optimized weight factors

  • E. AcarEmail author
  • M. Rais-Rohani
Research Paper


Approximate mathematical models (metamodels) are often used as surrogates for more computationally intensive simulations. The common practice is to construct multiple metamodels based on a common training data set, evaluate their accuracy, and then to use only a single model perceived as the best while discarding the rest. This practice has some shortcomings as it does not take full advantage of the resources devoted to constructing different metamodels, and it is based on the assumption that changes in the training data set will not jeopardize the accuracy of the selected model. It is possible to overcome these drawbacks and to improve the prediction accuracy of the surrogate model if the separate stand-alone metamodels are combined to form an ensemble. Motivated by previous research on committee of neural networks and ensemble of surrogate models, a technique for developing a more accurate ensemble of multiple metamodels is presented in this paper. Here, the selection of weight factors in the general weighted-sum formulation of an ensemble is treated as an optimization problem with the desired solution being one that minimizes a selected error metric. The proposed technique is evaluated by considering one industrial and four benchmark problems. The effect of different metrics for estimating the prediction error at either the training data set or a few validation points is also explored. The results show that the optimized ensemble provides more accurate predictions than the stand-alone metamodels and for most problems even surpassing the previously reported ensemble approaches.


Metamodeling Surrogate modeling Optimization Ensemble 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Armstrong M, Champigny N (1989) A study on Kriging small blocks. CIM Bull 82(923):128–133Google Scholar
  2. Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, New York, pp 364–369Google Scholar
  3. Buckland ST, Burnham KP, Augustin NH (1997) Model selection: an integral part of inference. Biometrics 53:603–618zbMATHCrossRefGoogle Scholar
  4. Buhmann MD (2003) Radial basis functions: theory and implementations. Cambridge University Press, New YorkzbMATHGoogle Scholar
  5. Clarke SM, Griebsch JH, Simpson TW (2005) Analysis of support vector regression for approximation of complex engineering analyses. ASME J Mech Des 127(11):1077–1087CrossRefGoogle Scholar
  6. Daberkow DD, Marvis DN (2002) An investigation of metamodeling techniques for complex systems design. In: Proceedings of the 9th AIAA/ISSMO symposium on multidisciplinary analysis and optimization, Atlanta, GA, SeptemberGoogle Scholar
  7. Dyn N, Levin D, Rippa S (1986) Numerical procedures for surface fitting of scattered data by radial basis functions. SIAM J Sci Stat Comput 7(2):639–659zbMATHCrossRefMathSciNetGoogle Scholar
  8. Evensen G (2003) The ensemble Kalman filter: theoretical formulation and practical implementation. Ocean Dyn 53:343–367CrossRefGoogle Scholar
  9. Fang H, Rais-Rohani M, Liu Z, Horstemeyer MF (2005) A comparative study of metamodeling methods for multi-objective crashworthiness optimization. Comput Struct 83:2121–2136CrossRefGoogle Scholar
  10. Friedman JH (1991) Multivariate adaptive regression splines. Ann Stat 19(1):1–141zbMATHCrossRefGoogle Scholar
  11. Giunta A, Watson LT (1998) A comparison of approximation modeling techniques: polynomial versus interpolating models. In: Proceedings of the 7th AIAA/USAF/NASA/ISSMO symposium on multidisciplinary analysis and optimization, St. Louis, MO, 2–4 September 1998, vol. 1, pp. 392–404.Google Scholar
  12. Goel T, Haftka RT, Shyy W, Queipo NV (2007) Ensemble of surrogates. Struct Multidisc Optim 33(3):199–216CrossRefGoogle Scholar
  13. Gunn SR (1997) Support vector machines for classification and regression. Technical report. Image Speech and Intelligent Systems Research Group, University of Southampton, SouthhamptonGoogle Scholar
  14. Hardy RL (1971) Multiquadratic equations of topography and other irregular surfaces. J Geophys Res 76:1905–1915CrossRefGoogle Scholar
  15. Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodeling techniques under multiple modeling criteria. Struct Multidisc Optim 23:1–13CrossRefGoogle Scholar
  16. Lophaven SN, Nielsen HB, Søndergaard J (2002) DACE—A MATLAB Kriging toolbox. Informatics and mathematical modeling. Technical University of Denmark, LyngbyGoogle Scholar
  17. MacKay DJC (1998) Introduction to Gaussian processes. In: Bishop CM (ed) Neural networks and machine learning, vol 168 of NATO ASI Series, 1998. Springer, Berlin, pp 133–165Google Scholar
  18. Madigan D, Raftery AE (1994) Model selection and accounting for model uncertainty in graphical models using Occam’s window. J Am Stat Assoc 89(428):1535–1546zbMATHCrossRefGoogle Scholar
  19. Martin JD, Simpson TW (2005) Use of Kriging models to approximate deterministic computer models. AIAA J 43(4):853–863CrossRefGoogle Scholar
  20. Mullur AA, Messac A (2004) Extended radial basis functions: more flexible and effective metamodeling. In: Proceedings of the 10th AIAA/ISSMO symposium on multidisciplinary analysis and optimization, Albany, NY, Aug–Sept, 2004Google Scholar
  21. Myers RH, Montgomery DC (2002) Response surface methodology: process and product optimization using designed experiments. Wiley, New YorkzbMATHGoogle Scholar
  22. Papila N, Shyy W, Griffin LW, Dorney DJ (2001) Shape optimization of supersonic turbines using response surface and neural network methods. J Propuls Power 18:509–518CrossRefGoogle Scholar
  23. Perrone MP, Cooper LN (1993) When networks disagree: ensemble methods for hybrid neural networks. In: Mammone RJ (ed) Artificial neural networks for speech and vision. Chapman and Hall, London, pp 126–142Google Scholar
  24. Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Tucker PK (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41:1–28CrossRefGoogle Scholar
  25. Rais-Rohani M, Solanki K, Eamon C (2006) Reliability-based optimization of lightweight automotive structures for crashworthiness. In: Proceedings of the 11th AIAA/ISSMO multidisciplinary analysis and optimization conference, September 2006, AIAA Paper 2006–7004, Portsmouth, VAGoogle Scholar
  26. Rasmussen CE, Williams CKI (2006) Gaussian processes for machine learning. MIT, CambridgeGoogle Scholar
  27. Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4(4):409–435zbMATHCrossRefMathSciNetGoogle Scholar
  28. Sanchez E, Pintos S, Queipo N (2006) Toward an optimal ensemble of kernel based approximations with engineering applications. In: IEEE world congress on computational intelligence, Vancouver, Canada, July 16–21, 2006Google Scholar
  29. Simpson TW, Peplinski JD, Koch PN, Allen JK (2001a) Metamodels for computer based engineering design: survey and recommendations. Eng Comput 17:129–150zbMATHCrossRefGoogle Scholar
  30. Simpson TW, Mauery TM, Korte JJ, Mistree F (2001b) Kriging models for global approximation in simulation-based multidisciplinary design optimization. AIAA J 39(12):2233–2241CrossRefGoogle Scholar
  31. Smith M (1993) Neural networks for statistical modeling. Von Nostrand Reinhold, New YorkzbMATHGoogle Scholar
  32. Stander N, Roux W, Giger M, Redhe M, Fedorova N, Haarhoff J (2004) A comparison of metamodeling techniques for crashworthiness optimization. In: Proceedings of the 10th AIAA/ISSMO multidisciplinary analysis and optimization conference, Albany, NY, Aug 2004Google Scholar
  33. Vapnik V, Golowich S, Smola A (1997) Support vector method for function approximation, regression estimation, and signal processing. In: Mozer M, Jordan M, Petsche T (eds) Advances in neural information processing systems. MIT, Cambridge, pp 281–287Google Scholar
  34. Venkataraman S, Haftka RT (2004) Structural optimization complexity: what has Moore’s law done for us? Struct Multidisc Optim 28:375–387CrossRefGoogle Scholar
  35. Wang GG, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. ASME J Mech Des 129(4):370–380CrossRefMathSciNetGoogle Scholar
  36. Wang G, Dong Z, Aitchison P (2001) Adaptive response surface method—a global optimization scheme for computation-intensive design problems. J Eng Optim 33(6):707–734CrossRefGoogle Scholar
  37. Wang JM, Fleet DJ, Hertzmann A (2005) Gaussian process dynamical moels. In: Proceedings of the 18th advanced neural information processing systems conference, December 2005, Vancouver, Canada, pp. 1441–1448Google Scholar
  38. Wang L, Beeson D, Wiggs G, Rayasam M (2006) A comparison of metamodeling methods using practical industry requirements. In: Proceedings of the 47th AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics, and materials conference, Newport, RI, May 2006Google Scholar
  39. Wujek BA, Renaud JE (1998a) New adaptive move-limit management strategy for approximate optimization, Part 1. AIAA J 36(10):1911–1921CrossRefGoogle Scholar
  40. Wujek BA, Renaud JE (1998b) New adaptive move-limit management strategy for approximate optimization, Part 2. AIAA J 36(10):1922–1934CrossRefGoogle Scholar
  41. Zerpa L, Queipo NV, Pintos S, Salager J (2005) An optimization methodology of alkaline–surfactant–polymer flooding processes using field scale numerical simulation and multiple surrogates. J Pet Sci Eng 47:197–208CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2008

Authors and Affiliations

  1. 1.Center for Advanced Vehicular SystemsMississippi State UniversityMississippi StateUSA
  2. 2.Department of Aerospace EngineeringMississippi State UniversityMississippi StateUSA

Personalised recommendations