Metric learning-guided k nearest neighbor multilabel classifier

Abstract

Multilabel classification deals with the problem where each instance belongs to multiple labels simultaneously. The algorithm based on large margin loss with k nearest neighbor constraints (LM-kNN) is one of the most prominent multilabel classification algorithms. However, due to the use of square hinge loss, LM-kNN needs to iteratively solve a constrained quadratic programming at a high computational cost. To address this issue, we propose a novel metric learning-guided k nearest neighbor approach (MLG-kNN) for multilabel classification. Specifically, we first transform the original instance into the label space by least square regression. Then, we learn a metric matrix in the label space, which makes the predictions of an instance in the learned metric space close to its true class values while far away from others. Since our MLG-kNN can be formulated as an unconstrained strictly (geodesically) convex optimization problem and yield a closed-form solution, the computational complexity is reduced. An analysis of generalization error bound indicates that our MLG-kNN converges to the optimal solutions. Experimental results verify that the proposed approach is more effective than the existing ones for multilabel classification across many benchmark datasets.

Appendix

1.1 Proofs of the main theorems

Before proving, we first introduce several related definitions and lemmas.

Definition 1

[20, 21] Given a metric space $({\mathcal {X}}, d)$, let m be the smallest value such that every ball in ${\mathcal {X}}$ can be covered by $2^m$ balls of half the radius. The doubling dimension of ${\mathcal {X}}$ is defined as $ddim({\mathcal {X}})=m$.

Lemma 1

[29] Given a metric space $({\mathcal {X}}, d )$, an $\varepsilon$-cover for ${\mathcal {X}}$ is a set ${\mathcal {B}}\subseteq {\mathcal {X}}$ such that every element of ${\mathcal {X}}$ has a counterpart in ${\mathcal {B}}$ that is at a distance at most $\varepsilon$ (with respect to d). Denote the size of the smallest $\varepsilon$-cover by $N(\varepsilon , {\mathcal {X}}, d)$, then we have

$$\begin{aligned} N(\varepsilon , {\mathcal {X}}, d)\le \left( \frac{2diam({\mathcal {X}})}{\varepsilon }\right) ^{ddim({\mathcal {X}})} \end{aligned}$$

(20)

where $diam({\mathcal {X}})=\mathop {\sup }\limits _{\varvec{x},\varvec{x}^{'}\in {\mathcal {X}}}d(\varvec{x},\varvec{x}^{'})$.

1.2 Proof of Theorem 1

Proof

From the linearity of expectation and the definition of $L(\varvec{y}(q),h^{{\mathbb {S}}}_q(\varvec{x}))$, we can rewrite:

$$\begin{aligned} \begin{aligned}&\mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ \sum _{q=1}^cL(\varvec{y}(q),h^{{\mathbb {S}}}_q(\varvec{x}))\right] \\&=\sum _{q=1}^c\mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ {\mathbb {P}}[\varvec{y}(q)\ne h^{{\mathbb {S}}}_q(\varvec{x})]\right]. \end{aligned} \end{aligned}$$

(21)

Let’s start with ${\mathbb {P}}[\varvec{y}(q)\ne h^{{\mathbb {S}}}_q(\varvec{x})]$. For any two examples $(\varvec{x}, \varvec{y})$ and $(\varvec{x}^{'},\varvec{y}^{'})$ drawn from the distribution ${\mathcal {D}}$, we have

$$\begin{aligned} \begin{aligned}&{\mathbb {P}}[\varvec{y}(q)\ne \varvec{y}^{'}(q)|\varvec{x},\varvec{x}^{'}]\\&=\eta _q(\varvec{x})(1-\eta _q(\varvec{x}^{'}))+(1-\eta _q(\varvec{x}))\eta _q(\varvec{x}^{'})\\&=\eta _q(\varvec{x})(1-\eta _q(\varvec{x})+\eta _q(\varvec{x})-\eta _q(\varvec{x}^{'}))\\&\quad+(1-\eta _q(\varvec{x}))(\eta _q(\varvec{x})-\eta _q(\varvec{x})+\eta _q(\varvec{x}^{'}))\\&=2\eta _q(\varvec{x})(1-\eta _q(\varvec{x}))+(2\eta _q(\varvec{x})-1)(\eta _q(\varvec{x})-\eta _q(\varvec{x}^{'}))\\&\quad\le 2\eta _q(\varvec{x})(1-\eta _q(\varvec{x}))+(\eta _q(\varvec{x})-\eta _q(\varvec{x}^{'})). \end{aligned} \end{aligned}$$

(22)

Using $\eta ^q$ is l-Lipschitz, we have $\eta _q(\varvec{x})-\eta _q(\varvec{x}^{'})\le ld_{pro}(\varvec{x},\varvec{x}^{'})$. Due to $\mathbf{M }_{\alpha }$ is a symmetric positive definite matrix, we can factorize $\mathbf{M }_{\alpha }$ as $\mathbf{M }_{\alpha }=\mathbf{P }\mathbf{P }^{\top }$ [30]. Then, we have

$$\begin{aligned} \begin{aligned} d_{pro}(\varvec{x},\varvec{x}^{'})&=\Vert \mathbf{P }^{\top }\mathbf{W }^{\top }\varvec{x}-\mathbf{P }^{\top }\mathbf{W }^{\top }\varvec{x}^{'}\Vert _2\\&\quad\le \Vert \mathbf{W }\Vert _F\Vert \mathbf{P }\Vert _F\Vert \varvec{x}-\varvec{x}^{'}\Vert _2\\&=|\mathbf{W }\Vert _F(\mathrm {tr}(\mathbf{P }\mathbf{P }^{\top }))^{1/2}\Vert \varvec{x}-\varvec{x}^{'}\Vert _2\\&=\Vert \mathbf{W }\Vert _F(\mathrm {tr}(\mathbf{M }_{\alpha }))^{1/2}\Vert \varvec{x}-\varvec{x}^{'}\Vert _2. \end{aligned} \end{aligned}$$

(23)

Actually, the regularization term in MLG-kNN ensures that $\mathrm {tr}(\mathbf{M }_{\alpha })$ is bounded.

Let $(\varvec{x}^{'},\varvec{y}^{'})$ be the nearest neighbor of $(\varvec{x},\varvec{y})$ in ${\mathbb {S}}$ with respect to $d_{pro}$. Then, $\mathop {{\mathbb {E}}}\nolimits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ {\mathbb {P}}[\varvec{y}(q)\ne h^{{\mathbb {S}}}_q(\varvec{x})]\right]$ is equivalent to $\mathop {{\mathbb {E}}}\nolimits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ {\mathbb {P}}[\varvec{y}(q)\ne \varvec{y}^{'}(q)]\right]$. By combining the preceding with (22), we have

$$\begin{aligned} \begin{aligned}&\mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ {\mathbb {P}}[\varvec{y}(q)\ne h^{{\mathbb {S}}}_q(\varvec{x})]\right] \\&\quad\le \mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ 2\eta _q(\varvec{x})(1-\eta _q(\varvec{x}))\right] \\&\quad+l\Vert \mathbf{W }\Vert _F(\mathrm {tr}(\mathbf{M }_{\alpha }))^{1/2} \mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}} \left[ d(\varvec{x}, \varvec{x}^{'})\right]. \end{aligned} \end{aligned}$$

(24)

The loss of the Bayes optimal classifier for q-th label is

$$\begin{aligned} \begin{aligned} L(\varvec{y}(q),h^*_q(\varvec{x}))&=\mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ min\{\eta _q(\varvec{x}), 1-\eta _q(\varvec{x})\}\right] \\&\quad\ge \mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ \eta _q(\varvec{x})(1-\eta _q(\varvec{x}))\right]. \end{aligned} \end{aligned}$$

(25)

Combining (24) and (25) concludes our proof. $\square$

Lemma 2

[14] Let $({\mathcal {X}}, d)$ be a metric space with $diam({\mathcal {X}})=1$ and $ddim({\mathcal {X}})=m<\infty$. For any $\varvec{x}$, $\varvec{x}^{'}\in {\mathcal {X}}$, we have

$$\begin{aligned} \begin{aligned} \mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ d(\varvec{x}, \varvec{x}^{'})\right] \le \frac{3}{n^{1/(m+1)}}. \end{aligned} \end{aligned}$$

(26)

1.3 Proof of Theorem 2

Proof

Combining Theorem 1 with Lemma  2 concludes our proof. $\square$

1.4 Proof of Theorem 3

Proof

We use the proof of Theorem 19.5 in [22] to complete the proof of Theorem 3.

Let’s first consider $\mathop {{\mathbb {E}}}\nolimits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ L(\varvec{y}(q),h^{{\mathbb {S}}}_q(\varvec{x}))\right]$ for the q-th label. For the training set ${\mathbb {S}}=\{(\varvec{x}_1,\varvec{y}_1),...,(\varvec{x}_n,\varvec{y}_n)\}$ and any $\varvec{x}\in {\mathcal {X}}$, let $\pi _1(\varvec{x}), \ldots , \pi _n(\varvec{x})$ be a reordering of $\{1, \ldots , n\}$ according to their distance to $\varvec{x}$, $d_{\mathbf{M }}(\varvec{x},\varvec{x}_i)$. That is, for all $i<n$, $d_{pro}(\varvec{x}, \varvec{x}_{\pi _i(\varvec{x})})\le d_{pro}(\varvec{x}, \varvec{x}_{\pi _{i+1}(\varvec{x})})$. Let${\mathcal {B}}=\{B_1,\ldots , B_N\}$ is the $\varepsilon$-cover for ${\mathcal {X}}$ of cardinality N. From Eq. (19.3) in [22], we have

$$\begin{aligned} \begin{aligned}&\mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ L(\varvec{y}(q),h^{{\mathbb {S}}}_q(\varvec{x}))\right] \\&\quad\le \mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n}\left[ \sum _{j:|B_i\bigcap {\mathbb {S}}|<k}{\mathbb {P}}[B_i]\right] \\&\qquad+\max _i\mathop {{\mathbb {P}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}[\varvec{y}(q)\ne h^{{\mathbb {S}}}_q(\varvec{x})| \forall i~\in [k], d_{pro}(\varvec{x}, \varvec{x}_{\pi _i(\varvec{x})})\\&\quad\le \varepsilon \Vert \mathbf{W }\Vert _F(\mathrm {tr}(\mathbf{M }_{\alpha }))^{1/2}]. \end{aligned} \end{aligned}$$

(27)

According to [22, Lemma 19.6], the first term of right side of (27) is bounded by $\frac{2Nk}{en}$. The second term of right side of (27) is bounded by $\left(1+\sqrt{\tfrac{8}{k}}\right)L(\varvec{y}(q),h^*_q(\varvec{x}))+3l\varepsilon \Vert \mathbf{W }\Vert _F(\mathrm {tr}(\mathbf{M }_{\alpha }))^{1/2}$. Finally, setting $\varepsilon =2n^{-\tfrac{1}{m+1}}$ and combining Lemma 1, we have

$$\begin{aligned} \begin{aligned}&\mathop {{\mathbb {E}}}\limits _{{\mathbb {S}}\sim {\mathcal {D}}^n,(\varvec{x},\varvec{y})\sim {\mathcal {D}}}\left[ L(\varvec{y}(q),h^{{\mathbb {S}}}_q(\varvec{x}))\right] \\&\quad\le \left( 1+\sqrt{\frac{8}{k}}\right) L(\varvec{y}(q),h^*_q(\varvec{x})) +\frac{(6l\Vert \mathbf{W }\Vert _F(\mathrm {tr}(\mathbf{M }_{\alpha }))^{1/2}+k)}{n^{1/(m+1)}}. \end{aligned} \end{aligned}$$

(28)

Applying Eq.(28) for each label and taking the sum, we complete the proof. $\square$