Uncovering Performance Envelopes Through Optimum Design of Tests
Test and evaluation is a process that is used to determine if a product/system satisfies its performance specifications across its entire operating regime. The operating regime is typically defined using factors such as types of terrains/sea-states/altitudes, weather conditions, operating speeds, etc., and involves multiple performance metrics. With each test being expensive to conduct and with multiple factors and performance metrics under consideration, design of a test and evaluation schedule is far from trivial. Design of experiments (DOE) still continues to be the most prevalent approach to derive the test plans, although there is significant opportunity to improve this practice through optimization. In this paper, we introduce a surrogate-assisted optimization approach to uncover the performance envelope with a small number of tests. The approach relies on principles of decomposition to deal with multiple performance metrics and employs bi-directional search along each reference vector to identify the best and worst performance simultaneously. To limit the number of tests, the search is guided by multiple surrogate models. At every iteration the approach delivers a test plan involving at most \(K_T\) tests, and the information acquired is used to generate future test plans. In order to evaluate the performance of the proposed approach, a set of scalable test functions with various Pareto front characteristics and objective space bias are introduced. The performance of the approach is quantitatively assessed and compared with two popular DOE strategies, namely Latin Hypercube Sampling (LHS) and Full Factorial Design (FFD). Further, we also demonstrate its practical use on a simulated catapult system.
KeywordsDesign of tests Performance envelope Multi-objective optimization
The authors would like to acknowledge Defence Related Research (DRR) grant from the University of New South Wales (UNSW), Canberra, Australia.
- 6.Deb, K., Thiele, L., Laumanns, M., Zitzler, E.: Scalable test problems for evolutionary multiobjective optimization. In: Proceedings of the International Conference on Evolutionary Multiobjective Optimization, pp. 105–145 (2005)Google Scholar
- 9.Kelly, M.: Simple catapult simulation. https://au.mathworks.com/matlabcentral/fileexchange/56469-simple-catapult-simulation?focused=6163843&tab=function
- 10.Lillard, V.B.: Science of test: improving the efficiency and effectiveness of DoD test and evaluation (2014). http://fs.fish.govt.nz/Page.aspx?pk=7&sc=SUR
- 11.Pan, L., He, C., Tian, Y., Wang, H., Zhang, X., Jin, Y.: A classification based surrogate-assisted evolutionary algorithm for expensive many-objective optimization. IEEE Trans. Evol. Comput. (2018). https://ieeexplore.ieee.org/document/8281523
- 13.Stuber, M.D.: Evaluation of process systems operating envelopes. Ph.D. thesis, Massachusetts Institute of Technology (2013)Google Scholar
- 14.Trivedi, A., Srinivasan, D., Sanyal, K., Ghosh, A.: A survey of multi-objective evolutionary algorithms based on decomposition. IEEE Trans. Evol. Comput. 21(3), 440–462 (2017)Google Scholar