Skip to main content
SpringerLink
Log in

A comparative study of metamodeling methods considering sample quality merits

  • Research Paper
  • Published:
Structural and Multidisciplinary Optimization Aims and scope Submit manuscript

Abstract

This research focuses on the study of the relationships between sample data characteristics and metamodel performance considering different types of metamodeling methods. In this work, four types of metamodeling methods, including multivariate polynomial method, radial basis function method, kriging method and Bayesian neural network method, three sample quality merits, including sample size, uniformity and noise, and four performance evaluation measures considering accuracy, confidence, robustness and efficiency, are considered. Different from other comparative studies, quantitative measures, instead of qualitative ones, are used in this research to evaluate the characteristics of the sample data. In addition, the Bayesian neural network method, which is rarely used in metamodeling and has never been considered in comparative studies, is selected in this research as a metamodeling method and compared with other metamodeling methods. A simple guideline is also developed for selecting candidate metamodeling methods based on sample quality merits and performance requirements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  • Bishop CM (1995) Neural networks for pattern recognition. Clarendon, Oxford

    Google Scholar 

  • Chen S, Cowan CFN, Grant PM (1991) Orthogonal least squares learning algorithm for radial basis function networks. IEEE Trans Neural Netw 2(2):302–309

    Article  Google Scholar 

  • Chen VCP, Tsui KL, Barton RR, Meckesheimer M (2006) A review on design, modeling and applications of computer experiments. IIE Trans 38(4):273–291

    Article  Google Scholar 

  • Dyn N, Levin D, Rippa S (1986) Numerical procedures for surface fitting of scattered data by radial basis function. SIAM J Sci Statist Comput 7(1):639–659

    MATH  MathSciNet  Google Scholar 

  • Fang KT, Lin DKJ, Winker P, Zhang Y (2000) Uniform design: theory and applications. Technometrics 42(1):237–248

    Article  MATH  MathSciNet  Google Scholar 

  • Fang H, Rais-Rohani M, Liu Z, Horstemeyer MF (2005) A comparative study of metamodeling methods for multiobjective crashworthiness optimization. Comput Struct 83(25–26):2121–2136

    Article  Google Scholar 

  • Fang KT, Li R, Sudjianto A (2006) Design and modeling for computer experiments. Chapman & Hall/CRC, London

    MATH  Google Scholar 

  • Feller W (1968) An introduction to probability theory and its applications, 3rd edn. Wiley, Hoboken

    MATH  Google Scholar 

  • Forrester AIJ, Keane AJ (2009) Recent advances in surrogate-based optimization. Prog Aerosp Sci 45(1–3):50–79

    Article  Google Scholar 

  • Forsberg J, Nilsson L (2005) On polynomial response surfaces and kriging for use in structural optimization of crashworthiness. Struct Multidisc Optim 29(3):232–243

    Article  Google Scholar 

  • Giunta A, Watson LT, Koehler J (1998) A comparison of approximation modeling techniques: polynomial versus interpolating models. In: Proceedings of the 7th AIAA/USAF/NASA/ISSMO symposium on multidisciplinary analysis & optimization, No. AIAA-98–4758, St. Louis, MO, 1998

  • Goel T, Haftka RT, Shyy W, Watson LT (2008) Pitfalls of using a single criterion for selecting experimental designs. Int J Numer Methods Eng 75(2):127–155

    Article  MATH  Google Scholar 

  • Gu L (2001) A comparison of polynomial based regression models in vehicle safety analysis. In: Proceedings of the ASME 2001 design engineering technical conferences and computer and information in engineering conference, Pittsburgh, PA, 2001

  • Hedayat A, Sloane NJA, Stufken J (1999) Orthogonal arrays: theory and applications. Springer, Heidelberg

    MATH  Google Scholar 

  • Hickernell FJ (1998) A generalized discrepancy and quadrature error bound. Math Comput 67(221):299–322

    Article  MATH  MathSciNet  Google Scholar 

  • Hock W, Schittkowski K (1981) Test examples for nonlinear programming codes. Springer, Heidelberg

    MATH  Google Scholar 

  • Hua LK, Wang Y (1981) Applications of number theory to numerical analysis. Springer, Berlin

    MATH  Google Scholar 

  • Ian TN (2004) Netlab: algorithms for pattern recognition. Springer, Heidelberg

    Google Scholar 

  • Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodelling techniques under multiple modeling criteria. Struct Multidisc Optim 23:1–13

    Article  Google Scholar 

  • Jin R, Chen W, Sudjianto A (2002) On sequential sampling for global metamodeling in engineering design. In: Proceedings of the ASME 2002 design engineering technical conferences and computer and information in engineering conference, No. DETC2002/DAC-34092, Montreal, Canada, 2002

  • Kim BS, Lee YB, Choi DH (2009) Comparison study on the accuracy of metamodeling technique for non-convex functions. J Mech Sci Technol 23(4):1175–1181

    Article  Google Scholar 

  • Kleijnen JPC (1987) Statistical tools for simulation practitioners. Marcel Dekker, New York

    MATH  Google Scholar 

  • Koch PN, Simpson TW, Allen JK, Mistree F (1999) Statistical approximations for multidisciplinary design optimization: the problem of size. J Aircr 36(1):275–286

    Article  Google Scholar 

  • Koehler JR, Owen A (1996) Computer experiments. In: Ghosh S, Rao CR (eds) Handbook of statistics. Elsevier Science, New York, pp 261–308

    Google Scholar 

  • Lophaven SN, Nielsen HB, Søndergaard J (2002) DACE—a Matlab kriging toolbox—version 2.0. Technical Report IMMREP-2002-12, Informatics and Mathematical Modelling, Technical University of Denmark, Kgs. Lyngby, Denmark

  • MacKay DJC (1991) Bayesian methods for adaptive models. Dissertation, California Institute of Technology

  • Matheron G (1963) Principles of geostatistics. Econ Geol 58(1):1246–1266

    Article  Google Scholar 

  • McDonald DB, Grantham WJ, Tabor WL, Murphy MJ (2007) Global and local optimization using radial basis function response surface models. Appl Math Model 31(10):2095–2110

    Article  MATH  Google Scholar 

  • Mckay MD, Beckman RJ, Conover WJ (1979) A comparison of three methods for selecting values of input variables from a computer code. Technometrics 21(2):239–245

    Article  MATH  MathSciNet  Google Scholar 

  • Montès P (1994) Smoothing noisy data by kriging with nugget effects. In: Laurent PJ, Le Méhauté A, Schumaker LL (eds) Wavelets, images and surface fitting. A.K. Peters, Wellesley, pp 371–378

  • Mullur AA, Messac A (2006) Metamodeling using extended radial basis functions: a comparative approach. Eng Comput 21(3):203–217

    Article  Google Scholar 

  • Myers RH, Montgomery DC (1995) Response surface methodology: process and product optimization using designed experiments. Wiley, New York

    MATH  Google Scholar 

  • Paiva RM, Carvalho ARD, Crawford C, Suleman A (2010) Comparison of surrogate models in a multidisciplinary optimization framework for wing design. AIAA J 48(5):995–1006

    Article  Google Scholar 

  • Papila N, Shyy W, Fitz-Coy N, Haftka RT (1999) Assessment of neural net and polynomial-based techniques for aerodynamic applications. In: Proceedings of the 17th applied aerodynamics conference, No. AIAA 99-3167, Norfolk, VA, 1999

  • Sacks J, Schiller SB, Welch WJ (1989a) Designs for computer experiments. Technometrics 31(1):41–47

    Article  MathSciNet  Google Scholar 

  • Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989b) Design and analysis of computer experiments. Stat Sci 4(1):409–435

    Article  MATH  MathSciNet  Google Scholar 

  • Simpson TW, Mauery TM, Korte JJ, Mistree F (1998) Comparison of response surface and kriging models for multidisciplinary design optimization. In: Proceedings of the 7th AIAA/USAF/NASA/ISSMO symposium on multidisciplinary analysis & optimization, No. AIAA-98-4755, St. Louis, MO, 1998

  • Simpson TW, Lin DKJ, Chen W (2001a) Sampling strategies for computer experiments: design and analysis. Int J Reliab Appl 2(3):209–240

    Google Scholar 

  • Simpson TW, Peplinski J, Koch PN, Allen JK (2001b) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17(2):129–150

    Article  MATH  Google Scholar 

  • Simpson TW, Toropov V, Balabanov V, Viana FAC (2008) Design and analysis of computer experiments in multidisciplinary design optimization: a review of how far we have come—or not. In: Proceedings of the 12th AIAA/ISSMO multidisciplinary analysis and optimization conference, No. AIAA 2008–5802, Victoria, Canada, 2008

  • Stander N, Roux W, Giger M, Redhe M, Fedorova N, Haarhoff J (2004) A comparison of meta-modeling techniques for crashworthiness optimization. In: Proceedings of the 10th AIAA/ISSMO multidisciplinary analysis and optimization conference, No. AIAA-2004-4489, Albany, NY, 2004

  • Varadarajan S, Chen W, Pelka C (2000) The robust concept exploration method with enhanced model approximation capabilities. Eng Optim 32(3):309–334

    Article  Google Scholar 

  • Wang GG, Shan S (2007) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129(1):370–380

    Article  Google Scholar 

  • Wang L, Beeson D, Wiggs G, Rayasam M (2006) A comparison meta-modeling methods using practical industry requirements. In: Proceedings of the 47th AIAA/ASME/ASCE/AHS/ASC structures, structural dynamics, and materials, No. AIAA 2006-1811, Newport, Rhode Island, 2006

  • Xiong F, Xiong Y, Chen W, Yang S (2009) Optimizing Latin hypercube design for sequential sampling of computer experiments. Eng Optim 41(8):793–810

    Article  Google Scholar 

  • Yang RJ, Gu L, Liaw L, Gearhart C, Tho CH, Liu X, Wang BP (2000) Approximations for safety optimization of large systems. In: Proceeding of the 2000 ASME design engineering technical conferences and computers and information in engineering conference, No. DETC-00/DAC-14245, Baltimore, MD, 2000

  • Zhu P, Zhang Y, Chen GL (2009) Metamodel-based lightweight design of an automotive front-body structure using robust optimization. Proc Instn Mech Eng Part D—J Automobile Eng 223(D9):1133–1147

    Article  Google Scholar 

Download references

Acknowledgments

This research is supported by Natural Science and Engineering Research Council (NSERC) of Canada. The use of the Western Canada Research Grid computing services is also acknowledged.

Author information

Authors and Affiliations

  1. Department of Mechanical and Manufacturing Engineering, University of Calgary, Calgary, Alberta, Canada, T2N 1N4

    Dong Zhao & Deyi Xue

Authors
  1. Dong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Deyi Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Deyi Xue.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhao, D., Xue, D. A comparative study of metamodeling methods considering sample quality merits. Struct Multidisc Optim 42, 923–938 (2010). https://doi.org/10.1007/s00158-010-0529-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00158-010-0529-3

Keywords

Search

Navigation