Abstract
Test and evaluation is a process that is used to determine if a product/system satisfies its performance specifications across its entire operating regime. The operating regime is typically defined using factors such as types of terrains/sea-states/altitudes, weather conditions, operating speeds, etc., and involves multiple performance metrics. With each test being expensive to conduct and with multiple factors and performance metrics under consideration, design of a test and evaluation schedule is far from trivial. Design of experiments (DOE) still continues to be the most prevalent approach to derive the test plans, although there is significant opportunity to improve this practice through optimization. In this paper, we introduce a surrogate-assisted optimization approach to uncover the performance envelope with a small number of tests. The approach relies on principles of decomposition to deal with multiple performance metrics and employs bi-directional search along each reference vector to identify the best and worst performance simultaneously. To limit the number of tests, the search is guided by multiple surrogate models. At every iteration the approach delivers a test plan involving at most \(K_T\) tests, and the information acquired is used to generate future test plans. In order to evaluate the performance of the proposed approach, a set of scalable test functions with various Pareto front characteristics and objective space bias are introduced. The performance of the approach is quantitatively assessed and compared with two popular DOE strategies, namely Latin Hypercube Sampling (LHS) and Full Factorial Design (FFD). Further, we also demonstrate its practical use on a simulated catapult system.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bhattacharjee, K.S., Singh, H.K., Ray, T.: Multi-objective optimization with multiple spatially distributed surrogates. J. Mech. Des. 138(9), 091401 (2016)
Bhattacharjee, K.S., Singh, H.K., Ray, T.: Multiple surrogate-assisted many-objective optimization for computationally expensive engineering design. J. Mech. Des. 140(5), 051403 (2018)
Chugh, T., Jin, Y., Meittinen, K., Hakanen, J., Sindhya, K.: A surrogate-assisted reference vector guided evolutionary algorithm for computationally expensive many-objective optimization. IEEE Trans. Evol. Comput. 22(1), 129–142 (2018)
Das, I., Dennis, J.E.: Normal-boundary intersection: a new method for generating the Pareto surface in nonlinear multicriteria optimization problems. SIAM J. Optim. 8(3), 631–657 (1998)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002)
Deb, K., Thiele, L., Laumanns, M., Zitzler, E.: Scalable test problems for evolutionary multiobjective optimization. In: Proceedings of the International Conference on Evolutionary Multiobjective Optimization, pp. 105–145 (2005)
Ishibuchi, H., Yu, S., Hiroyuki, M., Yusuke, N.: Performance of decomposition-based many-objective algorithms strongly depends on Pareto front shapes. IEEE Trans. Evol. Comput. 21(2), 169–190 (2017)
Jin, Y.: A comprehensive survey of fitness approximation in evolutionary computation. Soft Comput. - Fusion Found. Methodol. Appl. 9(1), 3–12 (2005)
Kelly, M.: Simple catapult simulation. https://au.mathworks.com/matlabcentral/fileexchange/56469-simple-catapult-simulation?focused=6163843&tab=function
Lillard, V.B.: Science of test: improving the efficiency and effectiveness of DoD test and evaluation (2014). http://fs.fish.govt.nz/Page.aspx?pk=7&sc=SUR
Pan, L., He, C., Tian, Y., Wang, H., Zhang, X., Jin, Y.: A classification based surrogate-assisted evolutionary algorithm for expensive many-objective optimization. IEEE Trans. Evol. Comput. (2018). https://ieeexplore.ieee.org/document/8281523
Storn, R., Price, K.: Differential evolution - a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim. 11(4), 341–359 (1997). https://doi.org/10.1023/A:1008202821328
Stuber, M.D.: Evaluation of process systems operating envelopes. Ph.D. thesis, Massachusetts Institute of Technology (2013)
Trivedi, A., Srinivasan, D., Sanyal, K., Ghosh, A.: A survey of multi-objective evolutionary algorithms based on decomposition. IEEE Trans. Evol. Comput. 21(3), 440–462 (2017)
Acknowledgments
The authors would like to acknowledge Defence Related Research (DRR) grant from the University of New South Wales (UNSW), Canberra, Australia.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Ray, T., Habib, A., Singh, H.K., Ryan, M. (2018). Uncovering Performance Envelopes Through Optimum Design of Tests. In: Mitrovic, T., Xue, B., Li, X. (eds) AI 2018: Advances in Artificial Intelligence. AI 2018. Lecture Notes in Computer Science(), vol 11320. Springer, Cham. https://doi.org/10.1007/978-3-030-03991-2_41
Download citation
DOI: https://doi.org/10.1007/978-3-030-03991-2_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-03990-5
Online ISBN: 978-3-030-03991-2
eBook Packages: Computer ScienceComputer Science (R0)