An unsupervised learning approach for multilayer perceptron networks

Abstract

Multilayer perceptron networks have been designed to solve supervised learning problems in which there is a set of known labeled training feature vectors. The resulting model allows us to infer adequate labels for unknown input vectors. Traditionally, the optimal model is the one that minimizes the error between the known labels and those inferred labels via such a model. The training process results in those weights that achieve the most adequate labels. Training implies a search process which is usually determined by the descent gradient of the error. In this work, we propose to replace the known labels by a set of such labels induced by a validity index. The validity index represents a measure of the adequateness of the model relative only to intrinsic structures and relationships of the set of feature vectors and not to previously known labels. Since, in general, there is no guarantee of the differentiability of such an index, we resort to heuristic optimization techniques. Our proposal results in an unsupervised learning approach for multilayer perceptron networks that allows us to infer the best model relative to labels derived from such a validity index which uncovers the hidden relationships of an unlabeled dataset.

Appendices

Appendix A: Experimental process for setting heuristic parameters

An important element in the performance of any heuristic is the way in which its parameters are set. Many attempts to solve this problem have been reported (Lobo et al. 2007; Grefenstette 1986; Storn and Price 1997; Talbi 2009). Most of them have important dependencies associated to the properties of the problem that disable the generalization of the suggested optimal values. We conducted an experimental process to determine the most suitable parameters setting relative to the characteristics of our problem. A parameters setting can be defined as m-tuple of the form $\tau =[p_1,p_2,\ldots ,p_{m}]$ that includes those parameters $p_i$ that are necessary to execute H (e.g., population size, crossover and mutation probabilities). Given a heuristic H, the experimental process is summarized as follows:

• 1. The domain for each parameter $p_i$ is previously defined.

• 2. A random set of possible instances of $\tau$ is systematically generated (about 300 different instances). Such a set is denoted as ${\mathcal {I}}$.

• 3. From ${\mathcal {I}}$, an instance $\tau _i$ is randomly chosen.

• 4. A set of 32 clustering problems (from a reservoir described in “Appendix A”) is solved via Algorithm 1 using H with the parameters $\tau _i$. Based on the solutions obtained using $\tau _i$, an average value is calculated and taken as a performance value $F_i$.

• 5. Step 3 is repeated until $i=300$.

• 6. The instance that achieves the best value of $F_i$ is selected as the most suitable setting.

The above process was executed with EGA using the indices SD, DB, DD. We reformulated every index as an objective function that requires to be minimized and whose range is the interval [0, 1]. When EGA is running, we got snapshots of the value of the index to be optimized every 50 generations. This allows us to obtain the value of performance F throughout the adaptive process. In Fig. 7 is shown the value of F obtained by the top 5 instances of $\tau$ throughout 500 iterations.

The best instance of $\tau$ is that with the fastest convergence and the best value of F. The values of such an instance were shown in Table 1.

Appendix B: Generating clustering problems

The relative performance of a clustering method must refer to its ability when compared to other methods to solve the same set of problems. A generalization of the performance will be possible as long as these problems represent a random sample of all clustering problems in a wide a numerical space. A systematic process has been followed to generate numerical datasets in this space. Each dataset contains elements grouped into k clusters which are generated via a set of parametric functions as follows:

Let k, $\aleph _i$ and ${\mathbb {F}}$ the number of clusters, the size of the ith cluster and a set of generator functions, respectively. A cluster will be a set of d-dimensional vectors generated as follows:

• 1. From ${\mathbb {F}}$, a subset of functions $f_i:{\mathbb {R}}\rightarrow {\mathbb {R}}$ are randomly chosen ($i=1,2,\ldots ,d$).

• 2. A vector of the form $\mathbf {v}=[f_1(x_1),f_2(x_2),\ldots ,f_{d}(x_d)]$ is generated. The values of $x_i$ are drawn randomly from the domain of $f_i$.

• 3. Step 2 is repeated until $\aleph _i$ vectors have been obtained.

This process is executed until k clusters have been obtained. Set ${\mathbb {F}}$ includes the functions reported in Pohlheim (2012) and Molga and Smutnicki (2005). The reservoir generated includes clustering problems with $k = 2,3,\ldots ,20$. The above process was implemented in Java language and the resulting data were stored in a relational database (MySQL).