Model-based classification with dissimilarities: a maximum likelihood approach

Abstract

Most of classification problems concern applications with objects lying in an Euclidean space, but, in some situations, only dissimilarities between objects are known. We are concerned with supervised classification analysis from an observed dissimilarity table, which task is classifying new unobserved or implicit objects (only known through their dissimilarity measures with previously classified ones forming the training data set) into predefined classes. This work concentrates on developing model-based classifiers for dissimilarities which take into account the measurement error w.r.t. Euclidean distance. Basically, it is assumed that the unobserved objects are unknown parameters to estimate in an Euclidean space, and the observed dissimilarity table is a random perturbation of their Euclidean distances of gaussian type. Allowing the distribution of these perturbations to vary across pairs of classes in the population leads to more flexible classification methods than usual algorithms. Model parameters are estimated from the training data set via the maximum likelihood (ML) method, and allocation is done by assigning a new implicit object to the group in the population and positioning in the Euclidean space maximizing the conditional group likelihood with the estimated parameters. This point of view can be expected to be useful in classifying dissimilarity tables that are no longer Euclidean due to measurement error or instabilities of various types. Two possible structures are postulated for the error, resulting in two different model-based classifiers. First results on real or simulated data sets show interesting behavior of the two proposed algorithms, ant the respective effects of the dissimilarity type and of the data intrinsic dimension are investigated. For these latter two aspects, one of the constructed classifiers appears to be very promising. Interestingly, the data intrinsic dimension seems to have a much less adverse effect on our classifiers than initially feared, at least for small to moderate dimensions.

Appendix

1.1 The computation of an initial approximation ${\widehat{U}}_{k,0}$ to minimize ${\widehat{H}_{k}(U)}$

Since we intend to compute $U = {\widehat{U}}_{k},$ the point at which the function ${\widehat{H}_{k}(U)}$ defined by (44) reaches its minimum value, intuitively this minimum should satisfy, as closely as possible, the approximate equalities:

$$ {{\|{U-{{\widehat{X}}_{i}}}\|}} \approx d_{i},\quad \hbox{for}\; i = 1,\ldots,n, $$

which implies, by squaring and expanding the squared Euclidean norms of the differences:

$$ 2 < {{\widehat{X}}_{i}}, U > - {\|{U}\|}^{2} \approx {\|{{{\widehat{X}}_{i}}}\|}^{2} - d_{i}^{2} (i = 1,\ldots,n), $$

or

$$ 2 {\widehat{x}_{i1}} u_1 + \cdots + 2 {\widehat{x}_{ip}} u_p - u_{p+1} \approx {\|{{{\widehat{X}}_{i}}}\|}^{2} - d_{i}^{2} (i = 1,\ldots,n), $$

(49)

where

$$ U = ({{u}_{{1}},\ldots,{u}_{{p}}})^{\rm T}, u_{p+1} = {\|{U}\|}^{2}, $$

and

$$ {{\widehat{X}}_{i}}=({\widehat{x}_{i1}}, \ldots, {\widehat{x}_{ip}})\;(i = 1,\ldots,n). $$

Now, (49) can be regarded as an approximate linear system of n equations in the p + 1 scalar unknowns ${{u}_{{1}},\ldots,{u}_{{p+1}}}.$ The unknown u _p+1 introduces a nonlinear constraint: u _p+1 = u ²₁  + ⋯ + u ² _p . This constraint can be eliminated by suppressing an arbitrarily chosen equation from (49) while subtracting it from all the remaining n − 1 ones, therefore obtaining an approximate linear system of n − 1 equations in the p unknowns u ₁,...,u _p , which one then solves by Least Squares. Instead we chose the simpler strategy to directly solve (49) by Least Squares and merely discard the estimation obtained for u _p+1.

In any event, whatever the strategy chosen between the two outlined above to approximately solve (49), it is important to note that the matrix of the solved Least Squares system is entirely determined by the estimated learning configuration ${\widehat{{\mathbf{X}}}}.$ Thus, its QR factorization can be computed once for all after the Learning Phase for use throughout the Prediction Phase to quickly solve (49) for each new explicit observation d to classify.

However, since n ≫ p, there are obvious far more economical ways to approximately solve (49). For instance (see [22, 26]), one could retain just p + 1 equations among the n in (49) and solve a (hopefully) Cramer system for ${{u}_{{1}},\ldots,{u}_{{p+1}}}$ (or for u ₁,...,u _p by first suppressing one of the p + 1 equations and subtracting it from the p others). The reason we chose not to follow these cheap paths is twofold:

• 1. One would have to devise a cheap selecting criterion for the p + 1 equations among the initial n in (49). Now, the two simplest choices are that of the first p + 1 ones or a random choice, but this may result in undesirable effects for the numerical stability of the resulting system in certain cases and/or the convergence of the numerical minimization algorithm starting from the obtained initial approximation of ${\widehat{U}}_{k}.$ On the other hand, more sophisticated selecting criteria may imply a significant overhead in computation with no appreciable gain over the use of all the equations. Whereas, since n ≫ p, using all the equations almost certainly guarantees that we least squarely solve a linear system of full rank p.

• 2. By using all the equations in (49) for computing an initial approximation ${\widehat{U}}_{k,0}$ to ${\widehat{U}}_{k},$ we increase our chances of obtaining a good one in order to guarantee fast convergence in the numerical nonlinear minimizer used to minimize the function ${\widehat{H}_{k}(U)}$ defined by (44).

However, pushing this latter argument further, one may legitimately argue that, considering the structure of the function ${\widehat{H}_{k}(U)},$ weighted least squares should be the right strategy in solving our suggested linear systems approximately to obtain ${\widehat{U}}_{k,0}.$ This is right, but would entail a serious increase in computational cost, since the weights would then vary according to each implicit object U and each possible group label in {1,...,G}.

1.2 Contribution and Originality

Traditional classification problems concern applications with objects lying in an Euclidean space, but, in certain situations, only some type of pairwise dissimilarity measure between objects is available. Today, practical applications are numerous (e.g., [3, 8, 11, 28–31, 40, 41]). Moreover, using dissimilarity measures can be of much interest to analyze proximity between curves or objects in high dimensional spaces, or, more generally, between objects of complicated intrinsic structure.

None of the existing algorithms for dissimilarity data classification is based on standard principles of statistical inference. Moreover, apart from [37], they do not take into account measurement error in the dissimilarities. Therefore, they are more suited to classify dissimilarity tables that result from exactly computed pairwise distances between objects in a data set that were originally given through attributes in an Euclidean space. They are seldom recommended for coarser or noisy dissimilarity types, whereas real world dissimilarity data quite often fall in that category.

In the work presented here, we develop a new approach based on the purely statistical viewpoint of MDS [39]. Although interesting alternatives exist, MDS remains today the leading mathematical methodology for handling dissimilarity data. We are concerned with classification analysis from a table of such observed data from an otherwise non observable population of objects. The main goal is to assign a new object to one of a priori groups in the population using as only information its dissimilarities with previously classified ones which thus form the Training Data Set. Basically, our approach to solving this problem assumes a probability model in which the observed dissimilarities are Euclidean distances perturbed with random gaussian errors. Actually, two possible probability models are investigated, differing by the structure they postulate for the perturbation of the Euclidean distances which led to the observed dissimilarities. In each of these models, the unobserved objects are regarded as unknown parameters lying in an Euclidean space. Estimating these parameters in the statistical sense is thus equivalent to positioning the unobserved objects in the Euclidean space given their respective pairwise dissimilarities, which is the traditional MDS main concern. Such a model-based approach then has the advantage to simultaneously estimate objects positioning and group labeling. Since it is unknown, the dimension p of the Euclidean space shall serve as a tuning parameter to be estimated from the data by cross validation and the “within one standard error of the minimum error towards model parsimony” rule.

Each of the two probabilistic models so postulated for dissimilarity data allows us to derive a general purpose classifier for such data. The two constructed classifiers are nicknamed M1.BC and M2.BC respectively. The latter (M2.BC) exhibits high flexibility to adapt to the dissimilarity type at hand in the data. Its classification performance on some classical data sets appears comparable to that of some of the already available best classifiers on dissimilarity data.