Possibilistic classifiers for numerical data

Abstract

Naive Bayesian Classifiers, which rely on independence hypotheses, together with a normality assumption to estimate densities for numerical data, are known for their simplicity and their effectiveness. However, estimating densities, even under the normality assumption, may be problematic in case of poor data. In such a situation, possibility distributions may provide a more faithful representation of these data. Naive Possibilistic Classifiers (NPC), based on possibility theory, have been recently proposed as a counterpart of Bayesian classifiers to deal with classification tasks. There are only few works that treat possibilistic classification and most of existing NPC deal only with categorical attributes. This work focuses on the estimation of possibility distributions for continuous data. In this paper we investigate two kinds of possibilistic classifiers. The first one is derived from classical or flexible Bayesian classifiers by applying a probability–possibility transformation to Gaussian distributions, which introduces some further tolerance in the description of classes. The second one is based on a direct interpretation of data in possibilistic formats that exploit an idea of proximity between data values in different ways, which provides a less constrained representation of them. We show that possibilistic classifiers have a better capability to detect new instances for which the classification is ambiguous than Bayesian classifiers, where probabilities may be poorly estimated and illusorily precise. Moreover, we propose, in this case, an hybrid possibilistic classification approach based on a nearest-neighbour heuristics to improve the accuracy of the proposed possibilistic classifiers when the available information is insufficient to choose between classes. Possibilistic classifiers are compared with classical or flexible Bayesian classifiers on a collection of benchmarks databases. The experiments reported show the interest of possibilistic classifiers. In particular, flexible possibilistic classifiers perform well for data agreeing with the normality assumption, while proximity-based possibilistic classifiers outperform others in the other cases. The hybrid possibilistic classification exhibits a good ability for improving accuracy.

Appendix: Naive Bayesian Classifiers

Naive Bayesian Classifiers (NBC) are based on Bayes rule. They assume the independence of the input variables. Despite their simplicity, NBC can often outperform more sophisticated classification methods (Langley et al. 1992). A NBC can be seen as a Bayesian network in which predictive attributes are assumed to be conditionally independent given the class attribute.

Given a vector X = {x ₁, x ₂, …, x _n } to be classified, a NBC computes the posterior probability P(c _j |X) for each class c _j in a set of possible classes C = (c ₁, c ₂, …, c _m ) and labels the case X with the class c _j that achieves the highest posterior probability, that is:

$$ c^{\ast}=\hbox{arg} \max_{c_{j}} P(c_{j}|X). $$

(21)

Using the Bayes rule:

$$ P(c_j|x_1,x_2,\ldots,x_n)=\frac{P(x_1,x_2,\ldots,x_n|c_j)*P(c_j)}{P(x_1,x_2,\ldots,x_n)} $$

(22)

The denominator P(x ₁, x ₂, , x _n ) is a normalizing factor that can be ignored when determining the maximum posterior probability of a class, as it does not depend on the class. The key term in Eq. (2) is P(x ₁, x ₂, , x _n |c _j ) which is estimated from training data. Since Naive Bayes assumes that conditional probabilities of attributes are statistically independent we can decompose the likelihood into a product of terms:

$$ P(x_1,x_2,\ldots,x_n|c_j)=\prod_{i=1}^{n}p(x_i|c_j) $$

(23)

Even under the independence assumption, the NBC have shown good performance for datasets containing dependent attributes. Domingos and Pazzani (2002) explain that attribute dependency does not strongly affect the classification accuracy. They also relate good performance of NBC to the zero-one loss function which considers that a classifier is successful when the maximum probability is assigned to the correct class (even if estimated probability is inaccurate). The work in Zhang (2004) gives a deeper explanation about the reasons for which the efficiency of NBC is not affected by attribute dependency. The author shows that, even if attributes are strongly dependent (if we look at each pairs of attributes), the global dependencies among all attributes could be insignificant because dependencies may cancel each other out and so they do not affect classification.

The most well-known Bayesian classification approach uses an estimation based on a multinomial distribution over the discretized variables and leads to so-called multinomial classifiers. Such a classifier, which handles only discrete attributes (continuous attributes must be discretized), assumes that all attributes follow a multinomial probability distribution. A variety of multinomial classifiers have been proposed for handling an arbitrary number of independent attributes. Let us mention especially (Langley et al. 1992; Langley and Sage 1994; Grossman and Dominigos 2004), semi-naive Bayesian classifiers (Kononenko 1991; Denton and Perrizo 2004), tree-augmented naive Bayesian classifiers (Friedman et al. 1997), k-dependence Bayesian classifiers (Sahami 1996) and Bayesian Network-augmented naive Bayesian classifiers (Cheng and Greiner 1999).

A second family of NBC is suitable for continuous attribute values. They directly estimate the true density of attributes using parametric density. A supplementary common assumption made by the NBC in that case is that within each class the values of numeric attributes are normally distributed around the mean, and they model each attribute through a single Gaussian. Then, the NBC represent such a distribution in terms of its mean and standard deviation and compute the probability of an observed value from such estimates. This probability is calculated as follows:

$$ p(x_{i}|c_{j})=g(x_{i}, \mu_{j}, \sigma_{j})=\frac{1}{\sqrt{2\Uppi}\sigma_{j}}\hbox{e}^{-\frac{(x_{i}- \mu_{j})^{2}}{2\sigma_{j}^2}} $$

(24)

The Gaussian classifiers (Geiger and Heckerman 1994; John and Langley 1995) are known for their simplicity and have a smaller complexity, compared with other non-parametric approximations. Although the normality assumption may be a valuable approximation for many benchmarks, it is not always the best estimation. Moreover, if the normality assumption is violated, classification results of NBC may deteriorate.

Other approaches using a non-parametric estimation are those breaking with the strong parametric assumption. The main approaches are based on the mixture model (Figueiredo and Leit ao 1999; McLachlan and Peel 2000) and the Gaussian mixture models (Bishop 1999; McLachlan and Peel 2000). Other approaches use kernel densities (John and Langley 1995; Pérez et al. 2009), leading to so-called Flexible Classifiers. This name is due to the ability of such classifier to represent densities with more than one mode in contrast with simple Gaussian classifiers. Flexible classifiers represent densities of different shapes with high accuracy; however, it results into a considerable increase in complexity.

John and Langley (1995) have proposed a Flexible Naive Bayesian Classifier (FNBC) that abandons the normality assumption and instead uses nonparametric kernel density estimation for each conditional distribution. The FNBC has the same properties as those introduced for the NBC; the only difference is instead of estimating the density for each continuous attribute x by a single Gaussian g(x,  μ _j , σ _j ), this density is estimated using an averaged large set of Gaussian kernels. To compute continuous attribute density for a specific class j, FNBC calculates n Gaussians, where each of them stores each attribute value encountered during training for this class and then takes the average of the n Gaussians to estimate p(x _i |c _j ). More formally, probability distribution is estimated as follows:

$$ p(x_{i}|c_{j})=\frac{1}{N_{j}}\sum _{k=1}^{N_{j}} g(x_{i}, \mu_{ik}, \sigma_{j}) $$

(25)

where k ranges over the training set of attribute x _i in class c _j , N _j is the number of instances belonging to the class c _j . The mean μ _ik is equal to the real value of attribute i of the instance k belonging to the class j, e.g. μ _ik  = x _ik . For each class j, FNBC estimates this standard deviation by

$$ \sigma_j=\frac{1}{\sqrt{N_j}} $$

(26)

The authors also prove kernel estimation consistency using (26) (see John and Langley 1995, for details). It has been shown that the kernel density estimation used in the FNBC and applied on several datasets enables this classifier to perform well in datasets where the parametric assumption is violated with little cost for datasets where it holds.

Pérez et al. (2009) have recently proposed a new approach for Flexible Bayesian classifiers based on kernel density estimation that extends the FNBC proposed by John and Langley (1995) to handle dependent attributes and abandons the independence assumption. In this work, three classifiers: tree-augmented naive Bays, a k-dependence Bayesian classifier and a complete graph are adapted to the support kernel Bayesian network paradigm.