SPMoE: a novel subspace-projected mixture of experts model for multi-target regression problems

Abstract

In this paper, we focus on modeling multi-target regression problems with high-dimensional feature spaces and a small number of instances that are common in many real-life problems of predictive modeling. With the aim of designing an accurate prediction tool, we present a novel mixture of experts (MoE) model called subspace-projected MoE (SPMoE). Training the experts of the SPMoE is done using a boosting-like manner by a combination of ideas from subspace projection method and the negative correlation learning algorithm (NCL). Instead of using whole original input space for training the experts, we develop a new cluster-based subspace projection method to obtain projected subspaces focused on the difficult instances at each step of the boosting approach for training the diverse experts. The experts of the SPMoE are trained on the obtained subspaces using a new NCL algorithm called sequential NCL. The SPMoE is compared with the other ensemble models using three real cases of high-dimensional multi-target regression problems; the electrical discharge machining, energy efficiency and an important problem in the field of operations strategy called the practice–performance problem. The experimental results show that the prediction accuracy of the SPMoE is significantly better than the other ensemble and single models and can be considered to be a promising alternative for modeling the high-dimensional multi-target regression problems.

Appendix

• The Friedman test ranks the models so the best performing model gets the rank of 1, the second best rank 2, and so on. Let $r_i^j$ be the rank of the $j$th of kth models on the $i$th of $N$ observations (treatments). The Friedman test compares the average ranks of models, $R_j =\frac{1}{N}\mathop \sum \nolimits _{i=1}^N r_i^j$. Under the null hypothesis, which states that all the models are equivalent and so their ranks $R_{j}$ should be equal, the Friedman statistic:

  $$\begin{aligned} \chi _F^2 =\frac{12N}{k(k+1)}\left[ {\mathop \sum \limits _{j=1}^k jR_j^2 -\frac{k(k+1)^2}{4}} \right] \end{aligned}$$

  (16)

  $\chi _F^2$ is distributed according to $\chi ^2$ with $k - 1$ df.

• Bonferroni–Dunn is a post hoc test that can be used after the Friedman test when it rejects the null hypothesis. This method assumes that the performance of two models is significantly different if the corresponding average ranks differ by at least the critical difference:

  $$\begin{aligned} \mathrm{CD}=\frac{q_\alpha }{\sqrt{\frac{k( {k+1})}{6N}} } \end{aligned}$$

  (17)

  ${q}_{\alpha }$ value is the critical value ${Q}{'}$ for a multiple non-parametrical comparison with a control

• Holm’s test: it is a multiple comparison procedure that can work with a control model and be compared with the remaining methods. The test statistics for comparing the ith and jth method using this procedure is:

  $$\begin{aligned} z=\frac{R_i -R_j }{\sqrt{\frac{k( {k+1})}{6N_{ds} }} }. \end{aligned}$$

  (18)

  The $z$ value is used to find the corresponding probability from the table of normal distribution, which is then compared with an appropriate level of confidence $\alpha$. In the Bonferroni–Dunn comparison, this $\alpha$ value is always $\alpha /(k-1)$, but Holm’s test adjusts the value for $\alpha$ to compensate for multiple comparisons. Holm’s test is a step-up procedure that sequentially tests the hypothesis ordered by their significance. We will denote the ordered $p$values by p$_{1}$, p$_{2}$ $\ldots$ so that $p_1 \le p_2 \le \cdots \le p_{k-1}$. Holm’s test compares each $p_{i}$ with $\alpha /(k-i)$, starting from the most significant $p$ value. If $p_{1}$ is below$\alpha /(k-1)$, the corresponding hypothesis is rejected and we are allowed to compare $p_{2}$ with $\alpha /(k-2)$. If the second hypothesis is rejected, the test proceeds with the third, and so on. As soon as a certain null hypothesis cannot be rejected, all the remaining hypotheses are retained as well.

• Hochberg’s procedure: it is a step-up procedure that works in the opposite direction from Holm’s method, comparing the largest $p$ value with $\alpha$, the next largest with $\alpha /2$ and so forth until it encounters a hypothesis that it can reject. All hypotheses with smaller $p$ values are then rejected as well.