Granular Computing Techniques for Classification and Semantic Characterization of Structured Data

Abstract

We propose a system able to synthesize automatically a classification model and a set of interpretable decision rules defined over a set of symbols, corresponding to frequent substructures of the input dataset. Given a preprocessing procedure which maps every input element into a fully labeled graph, the system solves the classification problem in the graph domain. The extracted rules are then able to characterize semantically the classes of the problem at hand. The structured data that we consider in this paper are images coming from classification datasets: they represent an effective proving ground for studying the ability of the system to extract interpretable classification rules. For this particular input domain, the preprocessing procedure is based on a flexible segmentation algorithm whose behavior is defined by a set of parameters. The core inference engine uses a parametric graph edit dissimilarity measure. A genetic algorithm is in charge of selecting suitable values for the parameters, in order to synthesize a classification model based on interpretable rules which maximize the generalization capability of the model. Decision rules are defined over a set of information granules in the graph domain, identified by a frequent substructures miner. We compare the system with two other state-of-the-art graph classifiers, evidencing both its main strengths and limits.

Appendices

Appendix 1: The Image Segmentation Procedure

In this appendix we describe an implementation of the water-shed image segmentation procedure.

In the beginning, the image is scanned with a sampling step of z pixels; for each position a vector representing the signature of the pixel is extracted. The feature vector $\mathbf s$, called pixel signature, is composed by the following sections:

$$\begin{aligned} \begin{array}{l} \mathbf s = [\mathbf s _0 ;\mathbf s _1 ;\mathbf s _2 ],\\ {\mathrm{where:}} \\ \mathbf s _0 = \texttt {CrCb} = [\texttt {Cr}, \texttt {Cb}], \\ \mathbf s _1 = \texttt {Brg}, \\ \mathbf s _2 = \texttt {Wlet} = [W_{HL} ,W_{LH} ,W_{HH} ]. \end{array} \end{aligned}$$

The components CrCb (which are organized in a real-valued vector of $\mathbb {R}^2$) and Brg contains chromatic and brightness information about the pixel. Cr and Cb are the two chrominance values in the YCrCb color space, while Brg is the brightness Y value from the same color space. The wavelet vector Wlet, which contains the components $W_{HL}$, $W_{LH}$, and $W_{HH}$, carries information about the texture characterizing the neighborhood of the pixel. In order to calculate these values, a 2D version of the well-known Daubechies wavelet transformation [3] is applied to the Brg channel of the image, using a $B\times B$ window (where B is usually chosen to be 4 or 8) around the pixel of interest. The squares of the values contained in the HL, LH, and HH sections of the resulting matrix (see Fig. 17) are averaged, yielding, respectively, the $W_{HL}$, $W_{LH}$, and $W_{HH}$ coefficients [42].

Fig. 17

Extraction of the wavelet components of a pixel p

As a first segmentation step, a single channel image is obtained calculating the following value for each pixel at position (i, j):

$$\begin{aligned} L(i,j) = \frac{1}{8}\sum \limits _{h = - 1}^1 {\sum \limits _{k = - 1}^1 {d\left( \mathbf s (i,j),\mathbf s (i + h,j + k) \right) } }, \end{aligned}$$

(27)

where $\mathbf s (i,j)$ is the signature of the pixel in the i-th row and j-th column. Signatures of pixels are compared using the following weighted Manhattan distance:

$$\begin{aligned} d(\mathbf {s}(i,j), \mathbf {s}(h,k)) = \sum _{l=0}^{2} \nu _l (\mathbf {s}_l(i,j) - \mathbf {s}_l(h,k)) \end{aligned}$$

(28)

which depends on 3 weight parameters $\nu _0$, $\nu _1$ and $\nu _2$, defined as follows:

$$\begin{aligned} \begin{array}{l} \nu _0 = \left( {1 - \omega _{\text {Brg}} } \right) \cdot \left( {1 - \omega _{\text {bin}} } \right) , \\ \nu _1 = \omega _{\text {Brg}} \cdot \left( {1 - \omega _{\text {bin}} } \right) , \\ \nu _2 = \omega _{\text {bin}}, \end{array} \end{aligned}$$

where $\omega _{\text {Brg}}\in \left[ 0,1\right]$ and $\omega _{\text {bin}} \in \left[ 0,1\right]$ are two parameters of the segmentation procedure.

Note that the entry L(i, j) represents a measure of how much the pixel at position (i, j) is similar (on average) to its closest neighbors, according to the definition of pixel signature and the dissimilarity measure defined in the signature domain. After the computation of the L channel, an affine normalization into the range $\left[ 0,1\right]$ is performed. Successively, the following steps are executed in order to decompose the image into connected regions (or segments). Note that we are always referring to path-connection, meaning that between two points of a region must exist at least one path joining them, formed exclusively by points (pixels in our case) belonging to the region itself.

• 1. A discrete version of the L map is obtained by a two-value (1 bit) quantization with the threshold given by the parameter $\tau _{\text {bin}}$.

• 2. After the step 1 the image is subdivided in two generally non-connected parts: the “stable” one, corresponding to $L(i,j)<\tau _{\text {bin}}$, and the “unstable” one, corresponding to $L(i, j)\ge \tau _{\text {bin}}$. Each one of the two parts is then decomposed into its connected components. Each obtained component is called a region and marked as “stable” if originating from the stable part, “unstable” otherwise.

• 3. In order to reduce the total area of unstable regions, their pixels are absorbed by their adjacent stable regions. The pixel is absorbed if its distance from the centroid of the nearest adjacent stable region is less than an absorption threshold $\tau _{\text {abs}}$. The remaining parts of the unstable regions are now marked as stable regions, so that after this step there are no more unstable regions. As it can be observed there is no guarantee that the newly obtained stable regions are connected, but this is not an issue at this stage.

• 4. In order to merge very similar stable regions, the BSAS clustering algorithm [26, 40] is used. The clustering threshold used in this step is the segmentation parameter $\tau _{\text {fus}}$. The distance between two regions is calculated as the distance between their centroids. In general the regions obtained after the steps 3 and 4 are non-connected, so the routine of decomposition into connected components is applied again to each region. The regions obtained at the end of this stage are certainly connected.

In both steps 3 and 4, the following weights are used in the signature dissimilarity measure (see Eq. (28)):

$$\begin{aligned} \begin{array}{l} \nu _0 = \left( {1 - \omega _{\text {Brg}} } \right) \cdot \left( {1 - \omega _{\text {W}} } \right) , \\ \nu _1 = \omega _{\text {Brg}} \cdot \left( {1 - \omega _{\text {W}} } \right) , \\ \nu _2 = \omega _{\text {W}}, \end{array} \end{aligned}$$

where $\omega _{\text {W}} \in [0,1]$ is an additional weighting parameters which replaces $\omega _{\text {bin}}$, introduced to enhance the flexibility of the algorithm.

Thus the set of the segmentation parameters $\mathcal {H}$ is the following: $\mathcal {H} = \{ \tau _{\text {bin}}, \tau _{\text {abs}}, \tau _{\text {fus}}; \omega _{\text {Brg}}, \omega _{\text {bin}}, \omega _{\text {W}} \}$.

Appendix 2: Features Extraction from Image Segments

In this appendix, we describe how to compute the 12 features used for describing a segment (node label feature space) and the 3 features used for representing the mutual spatial displacement in the image between a pair of segments (edge label feature space).

With reference to notation described in Eq. (1), let V be the generic segment in the image. The features Xc and Yc are, respectively, the horizontal and vertical position of the center of mass $\mathbf C$ of the segment. The components CrCb and Wlet are the average values over V of the corresponding quantities for the pixel signature used in the segmentation process. The additional color feature, Sat, is calculated from CrCb and Brg and it represents the saturation value from the well-known HSB color space.

The Area, Symm (symmetry), Rnd (roundness), and Cmp (compactness) values carry information about the size and shape of the segment; they are designed to be invariant to scaling and rotation. The “area” feature is given by: $\texttt {Area}=P/P_{tot}$, where P is the number of pixels of the segment V and $P_{tot}$ is the total number of pixels in the image.

Let $\mathbf p =(p_i, p_j)$ be a generic pixel of V described by a $\mathbb {R}^2$ vector, being $p_i, p_j$ the components that describe its coordinates in the image, while $\mathbf C =(\texttt {Xc}, \texttt {Yc})$ is the center of mass of the segment. Moreover, let $\mathbf p '$ be the position of the symmetric pixel of $\mathbf p$ with respect to $\mathbf C$. A measure of the symmetry degree of V with respect to $\mathbf C$ is computed by the following expression:

$$\begin{aligned} \texttt {Symm} = \frac{{\sum _V {\mathbf{s }\left( \mathbf{p } \right) } }}{P},\ {\mathrm{where}}\ \mathbf{s }\left( \mathbf{p } \right) = {\left\{ \begin{array}{ll} 1 &{} \mathrm {if}\ \mathbf{p }' \in V, \\ 0 &{} {\mathrm{otherwise.}} \end{array}\right. } \end{aligned}$$

(29)

Now, let $\mathbf r$ be the generalized radius of V, computed as follows:

$$\begin{aligned} \mathbf r = \mathop {\max }\limits _S \left\{ {d_E \left( {\mathbf{p },\mathbf{C }} \right) } \right\} , \end{aligned}$$

(30)

where $d_E(\mathbf p , \mathbf C )$ is the Euclidean distance between a given pixel p of V and the center of mass C. The equivalent circle ${\varGamma }$ is defined as the circle centered in C and such that its area is equal to the previously defined area feature of V. As it can be easily seen, the radius $\mathbf r '$ of ${\varGamma }$ is: $\mathbf{r }' = \sqrt{{\texttt {Area}} / {\pi }}$. The roundness of V can now be defined as follows: $\texttt {Rnd}=\mathbf r '/\mathbf r$.

Finally, let us introduce the function $\mathbf t (\mathbf p )$ defined as follows:

$$\begin{aligned} \mathbf{t }(\mathbf{p }) = {\left\{ \begin{array}{ll} 0 &{} \mathrm {if}\ \mathbf{p } \in {\varGamma }, \\ 1 &{} {\mathrm{otherwise.}} \end{array}\right. } \end{aligned}$$

(31)

We define the compactness of V as:

$$\begin{aligned} \texttt {Cmp} = 1 - \frac{{\sum _V {\mathbf{t }(\mathbf{p })} }}{P}. \end{aligned}$$

(32)

Figure 18 helps to understand the procedure for computing the shape features of a sample region.

Fig. 18

Retrieval of the symmetric pixels, used for computing the Symm features (a), and of the equivalent radius r’, used for evaluating the roundness Rnd and the compactness Cmp of a segment (b)

When a segment has a fairly linear shape (elongated shape), it is useful to characterize its direction and orientation. Since the previous three features do not hold this information, we introduce in the following four additional measurements. In order to calculate these values, a Principal Component Analysis (PCA) is evaluated from the $P\times 2$ pixel matrix $\mathbf {M}$:

$$\begin{aligned} \mathbf {M} = \left[ {\begin{array}{c} {\mathbf{p }_0 } \\ \vdots \\ {\mathbf{p }_{P - 1} } \\ \end{array}} \right] . \end{aligned}$$

(33)

As a result of the PCA analysis, we get the $2\times 2$ unitary rotation matrix R holding the principal direction of the pixel cluster, as well as the standard deviations of the two components in the rotated space. Let us define a local reference system with the origin coincident with C and axes given by the two eigenvectors $\mathbf x '$ and $\mathbf y '$ in R (where $\mathbf x '$ is the principal direction); we force the orientation of $\mathbf x '$ such that it lies in the upper half-plane. Let $x'$ and $y'$ be the coordinates in the reference system $<\mathbf C , \mathbf x ',\mathbf y '>$ and $\sigma _{x'}$, $\sigma _{y'}$ the respective standard deviations. The angle between $\mathbf x '$ and the x axis of the main reference system is the direction of V. The feature Dir is a complex number defined as:

$$\begin{aligned} \texttt {Dir} = S_{DN}\cdot \exp \{j2\pi D_{N}\}, \end{aligned}$$

(34)

where its magnitude $S_{DN}$ represents the significance (i.e., reliability) of the Dir value, and it is defined as follows:

$$\begin{aligned} S_{DN} = \left( {1 - \frac{{\sigma _{x'} }}{{\sigma _{y'} }}} \right) ^2 . \end{aligned}$$

(35)

The $S_{DN}$ value is a measure of eccentricity with suitable normalization properties (it ranges from 0 to 1), and it worked well for our application. The phase, $D_N$, is the normalized direction and is defined as follows:

$$\begin{aligned} D_N = \frac{\theta }{\pi } \end{aligned}$$

(36)

where $\theta$ is the angle between $\mathbf x$ and $\mathbf x ^{'}$ (according to a counterclockwise rotation). The value of $D_N$ ranges from $0^ \circ$ to $180^ \circ$, and it is meaningful only in the case of an elongated-shape segment, i.e., when $S_{DN}$ assumes values close to 1. Dir represents a unique feature, which is invariant with respect to a $180^ \circ$ rotation (orientation flipping).

In order to introduce an orientation discerning feature, we define a new angular quantity, which is related to the mass distribution of the pixels in V. First, we perform the following affine normalization:

$$\begin{aligned} \hat{x'} = \frac{{x' - \min \left\{ x' \right\} }}{{\max \left\{ x' \right\} - \min \left\{ x' \right\} }}, \end{aligned}$$

(37)

in order to ensure that every pixel coordinate ranges between 0 and 1. Let us now introduce the values:

$$\begin{aligned} m_\mathbf{x } = \frac{1}{P}\sum \limits _V^{} \hat{x'} ,\quad {\mathrm{and}}\quad md_\mathbf{x } = m_\mathbf{x } - 0.5. \end{aligned}$$

(38)

As it can be easily seen, the absolute value of $md_\mathbf{x }$ is a measure of the displacement of the normalized center of mass of the segment in the $\mathbf x '$ direction, with respect to the geometric center point 0.5. On the other hand, the sign of $md_\mathbf{x }$ is positive when the vector $\mathbf x$ “points toward the side where most of the mass accumulates” and negative in the opposite case. Let us define $\mathbf x ^{''}$ as:

$$\begin{aligned} \mathbf x ^{''} = \text {sign}(md_\mathbf{x }) \mathbf x ^{'}, \end{aligned}$$

(39)

where $\text {sign}(\cdot )$ is the common sign function. Let $\theta ^{'}$ be the angle among $\mathbf x$ and $\mathbf x ^{''}$ (according to a counterclockwise rotation), which allows to introduce the following quantity (the normalized orientation):

$$\begin{aligned} O_N = \frac{\theta ^{'}}{2\pi }. \end{aligned}$$

(40)

We can now introduce the following complex number:

$$\begin{aligned} \texttt {Or} = S_{ON} \cdot \exp \{j2\pi O_N\} \end{aligned}$$

(41)

where

$$\begin{aligned} S_{ON} = g(|md_\mathbf{x }|) \cdot S_{DN}, \end{aligned}$$

(42)

is the magnitude of the complex number Or and represents the reliability of the phase value $O_N$. $g(\cdot )$ is a nonlinear monotonic function that performs a dynamic correction of the displacement of the normalized center of mass with respect to the geometric center (see Eq, (38)), in order to make it closer to human perception. In particular, we use a function $g(\cdot )$ which performs a soft-clipping, since a region whose $md_\mathbf{x }$ value is above a certain threshold must be completely characterized by an orientation (supposing that the value $S_{DN}$ is sufficiently high).

Figure  19 shows the Dir and Or magnitude of two image segments characterized by a pronounced direction and orientation.

Fig. 19

The values Dir and Or of an elongated and asymmetric image segment

It is worth to highlight the following remarks:

• 1. The value $O_{N}$ ranges from $-180^\circ$ to $+180^\circ$; it always represents the orientation that “points toward the side where most of the mass accumulates”. So, its definition is coherent for all the possible segments.

• 2. The value $S_{ON}$ is significantly greater than zero only when the segment both has an elongated shape and an asymmetric mass distribution with respect to the geometric center $\mathbf G$, while the value $S_{DN}$ requires only that the segment is characterized by an elongated shape.

For what concerns the 3 features which describe the spatial displacement between two segments, $\delta \texttt {Xc}$ and $\delta \texttt {Yc}$ are the values of the differences, respectively, between the coordinates of the centers of mass of the two segments. The value $\delta \texttt {B}$ is the minimum Euclidean distance between two pixels belonging to the boundaries of the two segments (boundary distance).