Boosting k-NN for Categorization of Natural Scenes

Abstract

The k-nearest neighbors (k-NN) classification rule has proven extremely successful in countless many computer vision applications. For example, image categorization often relies on uniform voting among the nearest prototypes in the space of descriptors. In spite of its good generalization properties and its natural extension to multi-class problems, the classic k-NN rule suffers from high variance when dealing with sparse prototype datasets in high dimensions. A few techniques have been proposed in order to improve k-NN classification, which rely on either deforming the nearest neighborhood relationship by learning a distance function or modifying the input space by means of subspace selection. From the computational standpoint, many methods have been proposed for speeding up nearest neighbor retrieval, both for multidimensional vector spaces and nonvector spaces induced by computationally expensive distance measures.

In this paper, we propose a novel boosting approach for generalizing the k-NN rule, by providing a new k-NN boosting algorithm, called UNN (Universal Nearest Neighbors), for the induction of leveraged k-NN. We emphasize that UNN is a formal boosting algorithm in the original boosting terminology. Our approach consists in redefining the voting rule as a strong classifier that linearly combines predictions from the k closest prototypes. Therefore, the k nearest neighbors examples act as weak classifiers and their weights, called leveraging coefficients, are learned by UNN so as to minimize a surrogate risk, which upper bounds the empirical misclassification rate over training data. These leveraging coefficients allows us to distinguish the most relevant prototypes for a given class. Indeed, UNN does not affect the k-nearest neighborhood relationship, but rather acts on top of k-NN search.

We carried out experiments comparing UNN to k-NN, support vector machines (SVM) and AdaBoost on categorization of natural scenes, using state-of-the art image descriptors (Gist and Bag-of-Features) on real images from Oliva and Torralba (Int. J. Comput. Vis. 42(3):145–175, 2001), Fei-Fei and Perona (IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 524–531, 2005), and Xiao et al. (IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3485–3492, 2010). Results display the ability of UNN to compete with or beat the other contenders, while achieving comparatively small training and testing times.

Appendix

Generic UNN algorithm

The general version of UNN is shown in Algorithm 2. This algorithm induces the leveraged k-NN rule (9) for the broad class of surrogate losses meeting conditions of (Bartlett et al. 2006), thus generalizing Algorithm 1. Namely, we constrain ψ to meet the following conditions: (i) im(ψ)=ℝ₊, (ii) ∇ _ψ (0)<0 (∇ _ψ  is the conventional derivative of ψ loss function), and (iii) ψ is strictly convex and differentiable. (i) and (ii) imply that ψ is classification-calibrated: its local minimization is roughly tied up to that of the empirical risk (Bartlett et al. 2006). (iii) implies convenient algorithmic properties for the minimization of the surrogate risk (Nock and Nielsen 2009b). Three common examples have been shown in (6)–(5).

Algorithm 2

Algorithm Universal Nearest Neighbors UNN()

Fig. 11

Two examples where UNN corrects misclassifications of k-NN. The query image is shown in the leftmost column. The 11-nearest prototype images are shown on the right: the first row refers to UNN with 20 % of retained prototypes (θ=0.2), whereas the second column refers to classic k-NN classification over all prototypes (θ=1). Neighbors in the same category as the query image are surrounded by black boxes. Votes given by each prototype for the true category (coast) are shown below each image (such values correspond to α _ic y _jc in (9), where c is the ground-truth category)

Fig. 12

Examples of image prototypes with their leveraging coefficients for category 1 (tall buildings)-versus-all

The main bottleneck of UNN is step [I.1], as (29) is non-linear, but it always has a solution, finite under mild assumptions (Nock and Nielsen 2009b): in our case, δ _j  is guaranteed to be finite when there is no total matching or mismatching of example j’s memberships with its reciprocal neighbors’, for the class at hand. The second column of Table 5 contains the solutions to (29) for surrogate losses mentioned in Sect. 2.2. Those solutions are always exact for the exponential loss (ψ ^exp) and squared loss (ψ ^squ); for the logistic loss (ψ ^log) it is exact when the weights in the reciprocal neighborhood of j are the same, otherwise it is approximated. Since starting weights are all the same, exactness can be guaranteed during a large number of inner rounds depending on which order is used to choice the examples. Table 5 helps to formalize the finiteness condition on δ _j mentioned above: when either sum of weights in (28) is zero, the solutions in the first and third line of Table 5 are not finite. A simple strategy to cope with numerical problems arising from such situations is that proposed by Schapire and Singer (1999). (See Sect. 2.4.) Table 5 also shows how the weight update rule (30) specializes for the mentioned losses.

Table 5 Three common loss functions and the corresponding solutions δ _j of (29) and w _i of (30). (Vector $\boldsymbol{r}^{(c)}_{j}$ designates column j of R ^(c) and ∥.∥₁ is the L ₁ norm.) The rightmost column says whether it is (A)lways the solution, or whether it is when the weights of reciprocal neighbors of j are the (S)ame

Table 6 Number of images for each of the first 100 categories of the SUN database

Proofsketch of Theorem 3

We plug in the weight notation the iteration t and class c, so that $w_{ti}^{(c)}$ denotes the weight of example x _i prior to iteration t for class c in UNN (inside the “for c” loop of Algorithm 2, letting w ₀ denote the initial value of w). To save space in some computations below, we also denote for short:

(25)

ψ is ω strongly smooth is equivalent to $\tilde{\psi}$ being strongly convex with parameter ω ⁻¹ (Kakade et al. 2009), that is,

(26)

is convex. Here, we have made use of the following notations: $\tilde{\psi}(x) \stackrel{\mathrm{.}}{=} \psi^{\star}(-x)$, where $\psi^{\star}(x) \stackrel{\mathrm{.}}{=} x\nabla_{\psi}^{-1}(x) - \psi(\nabla^{-1}_{\psi}(x))$ is the Legendre conjugate of ψ. Since a convex function h satisfies h(w′)≥h(w)+∇ _h (w)(w′−w), applying inequality (32) taking as h the function in (32) yields, ∀t=1,2,…,T, ∀i=1,2,…,m, ∀c=1,2,…,C:

(27)

where we recall that D _ψ denotes the Bregman divergence with generator ψ (21). On the other hand, Cauchy-Schwartz inequality yields:

(28)

The equality in (34) holds because $\sum_{i: j \sim_{k} i} {\mathrm{r}^{(c)}_{ij}w^{(c)}_{(t+1)i}} = 0$, which is exactly (29). We obtain:

(29)

(30)

(31)

Here, (35) follows from (33), (36) follows from (34), and (37) follows from (19). Adding (37) for c=1,2,…,C and t=1,2,…,T, and then dividing by C, we obtain:

(32)

We now work on the big parenthesis which depends solely upon the examples. We have:

(33)

(34)

(35)

Here, (39) holds because of the Arithmetic-Geometric-Harmonic inequality, and (40) is Young’s inequality⁷ with p=q=2. Plugging (41) into (38), we obtain:

(36)

Now, UNN meets the following property (Piro et al. 2012, A.2), which can easily be shown to hold with our class encoding as well:

(37)

Adding (43) for t=0,2,…,T−1 and c=1,2,…,C, we obtain:

(38)

Plugging (42) into (44), we obtain:

(39)

But the following inequality holds between the average surrogate risk and the empirical risk of the leveraged k-NN rule $\boldsymbol{h}^{\ell}_{T}$, because of (i):

(40)

so that, putting altogether (45) and (46) and using the fact that ψ(0)>0 because of (i)–(ii), we have after T rounds of boosting for each class: i.e.:

(41)

There remains to compute the minimal value of T for which the right hand side of (47) becomes no greater than some user-fixed τ∈[0,1] to obtain the bound in (22).