Feature-based classifiers for design optimization

Abstract

We present a design optimization method for systems with high-dimensional parameter spaces using inductive decision trees. The essential idea is to map designs into a relatively low-dimensional feature space, and to derive a classifier to search for high-performing design alternatives within this space. Unlike learning classifier systems that were pioneered by Holland and Goldberg, classifiers defined by inductive decision trees were not originally developed for design optimization. In this paper, we explore modifications to such classifiers to make them more effective in the optimization problem. We expand the notions of feature space, generalize the tree construction heuristic beyond the original information-theoretic definitions, increase the reliance on domain expertise, and facilitate the transfer of design knowledge between related systems. There is a relatively small but rapidly growing body of work in the use of inductive trees for engineering design; the method presented herein is complementary to this research effort.

Appendix

1.1 Derivation details for the generalized heuristic

The difference between the minimum entropy and expected utility heuristics for tree construction can be reconciled by examining the incremental value of information. The basic idea is to use Bayesian analysis to compare the value of information obtained in the cases for finite and infinite supervised data. We begin by defining n _ij and P _ij =  n _ij /N _i as the number and observed proportion of class j designs in leaf i, where N _i =  ∑ ^M _j=1 n _ij is the total number of data in leaf i. If standard Bayesian conjugate distributions are used, then the posterior distribution of the actual proportions π _ij of class j designs in leaf i would be Dirichlet with joint density (DeGroot 1970)

$$\phi_{i} {\left({{\left\{{\pi_{{ij}}}\right\}}}\right)} = \Gamma {\left({N_{i}}\right)}{\prod\limits_{j = 1}^M}\frac{{\pi^{{n_{{ij}} - 1}}_{{ij}}}}{{\Gamma {\left({n_{{ij}}}\right)}}}$$

(A1)

The mean value of π _ij is approximately equal to the observed proportion P _ij and the standard deviation of π _ij , which is a measure of the uncertainty in the actual proportion, is inversely proportional to the square root of the number of data N _i . The expected utility U _i for leaf i given the N _i data is

$$E{\left[ {u_{i}} | c,N_{i}\;\hbox{data} \right]} = U_{i} {\left[ {{\left\{{P_{{ij}}}\right\}},c} \right]}$$

(A2)

where the functional form of U _i is given in (14) in terms of the classifier c.

Next, we consider the effect of additional data on classifier accuracy. What is needed here is an explicit decision rule for redefining the classifier c as data is collected. Equations 15 and 16 are two examples of such explicit rules in which the observed proportions P _ij are recomputed and the classifiers c _max and c ^* are redefined using the new values for P _ij . The topology and the binary relations defining the tree T and its leaves are fixed at this stage; improvements to the tree are considered later. Given a decision rule for redefining c, it can be shown that, in the limit of infinite data, the expected utility for leaf j would be

$$E{\left[ {u_{i}} | c,\infty\;\hbox{data} \right]} = {\int\limits_0^1} \cdots {\int\limits_0^1}U_{i} {\left[ {{\left\{{\pi_{{ij}}}\right\}},c} \right]}\phi_{i} {\left({{\left\{{\pi_{{ij}}}\right\}}}\right)}{\rm d}\pi_{{i1}} \cdots {\rm d}\pi_{{iM}} $$

(A3)

It follows from the definitions that as the number of data N _i in leaf i becomes large, the preceding expected utilities must agree:

$$E{\left[u_{i} | c,\infty\;\hbox{data} \right]} - E{\left[ {u_{i}} | c,N_{i}\;\hbox{data}\right]} \to 0 \quad \hbox{as}\;N_{i} \to \infty $$

(A4)

In the special case where the maximum likelihood classifier c _max in (15) is used, it can be shown, by a slight generalization of the arguments presented in (Buntine 1990), that the difference in the preceding expected utilities has the following asymptotic result:

$$E{\left[ u_{i} | c_{{\max}} ,\infty\;\hbox{data} \right]} - E{\left[ {u_{i}} | c_{{\max}}, N_{i}\;\hbox{data} \right]} \sim \exp {\left({- \lambda_{i} N_{i}}\right)}$$

(A5)

This shows that the difference in expected utilities decays exponentially with the number of data N _i . The exponential rate λ _i is

$$\frac{{\lambda_{i}}}{{\log 2}} = - I_{i} {\left({{\left\{{P_{{ij}}}\right\}}}\right)} + 1$$

(A6)

If the product of the differences in expected utilities is examined, then the following asymptotic result is obtained

$${\prod\limits_{i = 1}^L}{\left({E{\left[ {u_{i}} |c_{{\max}}, \infty\;\hbox{data} \right]} - E{\left[ {u_{i}} | c_{{\max}}, N_{i}\;\hbox{data} \right]}}\right)} \sim \exp {\left({- \lambda N}\right)}$$

(A7)

where N =  ∑ ^L _i=1 N _i is the total number of supervised data for all L leaves and the exponential rate is

$$\frac{\lambda}{{\log 2}} = - I + 1$$

(A8)

The preceding equations provide useful relationships between the expected utility of the maximum likelihood classifier and the information entropy.

In the next stage of the analysis, changes in the tree T are considered. Following the information-theoretic viewpoint, the accuracy of the tree is reflected by how fully it reflects the information in the supervised data. With this viewpoint, the difference of utilities in (A7) is a measure of the information in the N data that is not incorporated into the tree. To increase the accuracy of the tree, it is necessary to make the rate λ as large as possible. As shown in (A8), this is equivalent to minimizing the information entropy. Hence, the information entropy heuristic can be tied to a Bayesian analysis of the utility of information.

With these results, we are now able to obtain a tree construction heuristic based on the utility function in Sect. 4.1 for the generation of design alternatives. Starting from the Bayesian analysis results in Eqs. A1–A8, the main task is to determine if an exponential rate λ _i exists when the maximum likelihood classifier c _max in (A5) is replaced by the classifier c ^* for generating design alternatives. With some approximation, it can be shown that the exponential rate remains valid for c ^*; the result is

$$\frac{{\lambda_{i}}}{{\log 2}} \approx - I_{i} {\left({{\left\{{{P}^{\prime}_{{ij}}}\right\}}}\right)} + \hbox{constant}$$

(A9)

where P ^′ _ij are given by

$${P}^{\prime}_{{ij}} = \left\{{\begin{array}{*{20}l} {{u_{0} P_{{i1}} {\left[ {1 + {\left({u_{0} - 1}\right)}P_{{i1}}} \right]}^{{- 1}}}}& {{\hbox{if}\;j = 1}} \\ {{P_{{ij}} {\left[ {1 + {\left({u_{0} - 1}\right)}P_{{j1}}} \right]}^{{- 1}}}}& {{\hbox{otherwise}}}\\ \end{array}} \right.$$

(A10)

with u ₀ defined after (16). The revised proportion P ^′ _i1 can be interpreted as the proportion of class 1 design at leaf i if the number of class 1 designs is weighted by the utility-related value u ₀.