Ranking-preserved generative label enhancement

Abstract

Label distribution learning (LDL) is effective for addressing label ambiguity. In LDL, ground-truth label distributions are hardly available due to the high annotation cost, whereas it is relatively easy to obtain examples with logical labels. Hence, label enhancement (LE) is proposed to automatically transform logical labels into label distributions. Most existing LE methods employ discriminative approaches. However, discriminative approaches specialize in obtaining better predictive performance under supervised learning, and their capability is limited in LE that lacks supervisory information. Therefore, we propose a generative LE model, and infer label distributions by the variational Bayes capable of preserving the label ranking within the logical label vector. Our method consists of a generation process and an inference process. In the generation process, we treat label distributions as latent variables, and assume that label distributions generate logical labels and feature values of the instance itself and logical labels of the neighbors of this instance. In the inference process, we design a function, which mines the label correlation and preserves the label ranking within the logical label vector, to parameterize the variational posterior. Finally, we conduct extensive experiments to validate our proposal.

Appendix

1.1 Why choose Kumaraswamy distribution

Here we discuss in more detail the rationale for modeling the variational posterior as Kumaraswamy distribution. We briefly show several probability distributions with a support of (0, 1) as follow:

$$\begin{aligned} \begin{aligned} \text {Beta}(x\vert \alpha ,\beta )&= \frac{x^{\alpha -1}(1-x)^{\beta -1}}{{\text {Beta}}(\alpha ,\beta )}\\ \text {Arcsine}(x)&={\frac{1}{\pi {\sqrt{x(1-x)}}}} \\ \text {PERT}(x)&= {\displaystyle {\frac{(x-a)^{\alpha -1}(c-x)^{\beta -1}}{\textrm{B} (\alpha ,\beta )(c-a)^{\alpha +\beta -1}}}}\\ \text {Logit-normal}(x)&={\frac{1}{\sigma {\sqrt{2\pi }}}}\,e^{{-{\frac{({\text {logit}}(x)-\mu )^{2}}{2\sigma ^{2}}}}}{\frac{1}{x(1-x)}}\\ \text {Kumaraswamy}(x\vert a,b)&= abx^{{a-1}}(1-x^{a})^{{b-1}} \\ \end{aligned} \end{aligned}$$

(35)

In addition to the probability distributions listed in Eq. (35), there are also some probability distribution with complicated form, such as truncated normal distribution, U-quadratic distribution, triangular distribution and trapezoidal distribution. Using these probability distributions to model variational posteriors may result in some problems. For example, the shape of “Arcsine” and “Uniform” cannot be controlled by the learnable function; “PERT” is difficult to apply the reparameterization trick; “Logit-normal” and truncated normal distribution do not have closed-form KL divergence w.r.t. the prior distribution (Beta distribution); Triangular distribution and trapezoidal distribution have segmented density functions, which are not conducive to gradient-based optimization; U-quadratic distribution is difficult to express the case that mean value is around 0.5. Among all these probability distributions, only the Kumaraswamy distribution is the most suitable for modeling the variational posterior of the variable with Beta prior because it not only allows easy implementation of the reparameterization trick but also has closed-form KL divergence w.r.t. the Beta prior.

1.2 Proof of Theorem 1

Suppose that ${{\varvec{y}}}$ is any logical label vector, ${\textbf{P}}$ is the symmetrically normalized Laplacian of any label co-occurrence graph, ${{\varvec{v}}}= {\varphi }({{\varvec{y}}},\alpha )$. Then we have ${{\varvec{v}}}\simeq {{\varvec{y}}}$ if $\alpha \in (0, \alpha ^\star _{{\varvec{y}}})$, where $\alpha _{{\varvec{y}}}^\star =\inf ({\mathbb {A}} \cup \{1\})$,

$$\begin{aligned} {\mathbb {A}} = \bigcup _{\begin{array}{c} {i\in \{i\mid {y}_i=1\}}\\ {j\in \{j\mid {y}_j=0\}} \end{array}} \Big \{ \frac{1}{2}\le \alpha <1\mid {\varphi }_i({{\varvec{y}}}, \alpha ) = {\varphi }_j({{\varvec{y}}},\alpha ) \Big \}. \end{aligned}$$

Proof

Let $[{{\varvec{u}}}]^+\triangleq \inf \{ {u}_i\mid {y}_i=1 \}$ and $[{{\varvec{u}}}]^-\triangleq \sup \{{u}_i\mid {y}_i=0\}$, where ${{\varvec{u}}}$ is a ${m}$-dimensional vector.

1) Proving that ${{\varvec{v}}}\simeq {{\varvec{y}}}$ holds for any $\alpha \in (0, 2^{-1})$. We consider the source of ${{\varvec{v}}}={\varphi }({{\varvec{y}}}, \alpha )$, i.e., ${{\varvec{v}}}^{(t+1)} = \alpha {\textbf{P}}{{\varvec{v}}}^{(t)} + (1 - \alpha ) {{\varvec{y}}}$. Suppose that ${{\varvec{v}}}^{(t)}\in [0,1]^{m}$, we have ${\textbf{P}}{{\varvec{v}}}^{(t)} \in [0, 1]^{m}$, and ${{\varvec{v}}}^{(t+1)}=\alpha {\textbf{P}}{{\varvec{v}}}^{(t)} + (1-\alpha ){{\varvec{y}}}\in [0,1]^{m}$. Since ${{\varvec{v}}}^{(0)}={{\varvec{y}}}$, we have ${{\varvec{v}}}^{(t)}\in [0,1]^{m}$ holds for any $t\ge 0$. Since ${{\varvec{v}}}^{(t)}\in [0,1]^{m}$ implies $[\alpha {\textbf{P}}{{\varvec{v}}}^{(t)}]^+=0$ and $[\alpha {\textbf{P}}{{\varvec{v}}}^{(t)}]^-=\alpha$, we can obtain $[{{\varvec{v}}}^{(t+1)}]^+=1-\alpha$ and $[{{\varvec{v}}}^{(t+1)}]^-=\alpha$; since $0<\alpha <0.5$, we have $[{{\varvec{v}}}^{(t+1)}]^+>[{{\varvec{v}}}^{(t+1)}]^-$. Therefore, $[{{\varvec{v}}}^{(t)}]^+>[{{\varvec{v}}}^{(t)}]^-$ holds for any $t\ge 0$, i.e., ${{\varvec{v}}}\simeq {{\varvec{y}}}$.

2) Proving that ${{\varvec{v}}}\simeq {{\varvec{y}}}$ holds for any $\alpha \in [2^{-1},\alpha _{{\varvec{y}}}^\star )$. Say there exists $\alpha _0\in [2^{-1}, \alpha _{{\varvec{y}}}^\star )$ such that $v_i < v_j$, where $i\in \{i\mid {y}_i=1\}$ and $j\in \{j\mid {y}_j=0\}$, then there must exist $\alpha ^\prime \in [2^{-1}, \alpha _0)$ such that $v_i=v_j$ (since $v_i > v_j$ always holds for $0<\alpha <2^{-1}$, and ${{\varvec{v}}}={\varphi }({{\varvec{y}}},\alpha )$ is continuous), which contradicts the fact that $\alpha ^\star _{{\varvec{y}}}$ is the minimum of the roots (between $2^{-1}$ and 1) of all equations $v_i=v_j$ (where $(i,j)\in \{(i,j)\mid {y}_i=1\wedge {y}_j=0\}$). Therefore, ${{\varvec{v}}}\simeq {{\varvec{y}}}$ holds for any $\alpha \in (2^{-1},\alpha _{{\varvec{y}}}^\star )$.

Therefore, ${{\varvec{v}}}\simeq {{\varvec{y}}}$ holds for any $\alpha \in (0, \alpha _{{\varvec{y}}}^\star )$. $\square$

1.3 Example of calculating $\alpha _{{\varvec{y}}}^\star$ in Theorem 1

Suppose that ${{\varvec{y}}}=[1,0,1]^\top$, Laplacian matrix ${\textbf{P}}$ of the label co-occurrence graph can be factorized as:

$$\begin{aligned} \begin{aligned}&{\textbf{P}}= \begin{bmatrix} 2^{-1} &{} 3^{-1} &{} 6^{-1} \\ 3^{-1} &{} 2^{-1} &{} 6^{-1} \\ 6^{-1} &{} 6^{-1} &{} 2\times 3^{-1} \end{bmatrix} = {\textbf{U}} \begin{bmatrix} 1 &{} 0 &{} 0 \\ 0 &{} 0.17 &{} 0 \\ 0 &{} 0 &{} 0.5 \end{bmatrix} {\textbf{U}}^\top ,\\&\text {where }{\textbf{U}}= [{{\varvec{u}}}_1,{{\varvec{u}}}_2,{{\varvec{u}}}_3]= \begin{bmatrix} -0.58 &{} -0.71 &{} -0.41 \\ -0.58 &{} 0.71 &{} -0.41 \\ -0.58 &{} 0 &{} 0.82 \end{bmatrix}. \end{aligned} \end{aligned}$$

Then, we have:

$$\begin{aligned} \begin{aligned} \frac{1}{1-\alpha } {{\varvec{v}}}&=\frac{ {{\varvec{u}}}_1{{\varvec{u}}}_1^\top {{\varvec{y}}}}{1-\alpha }+\frac{{{\varvec{u}}}_2{{\varvec{u}}}_2^\top {{\varvec{y}}}}{1-0.17\alpha } +\frac{{{\varvec{u}}}_3{{\varvec{u}}}_3^\top {{\varvec{y}}}}{1-0.5\alpha } \\&= \begin{bmatrix} {0.67}(1-\alpha )^{-1} + {0.5}{(1-0.17\alpha )^{-1}} {-0.17}{(1-0.5\alpha )^{-1}}\\ {0.67}(1-\alpha )^{-1} {-0.5}{(1-0.17\alpha )^{-1}} {-0.17}{(1-0.5\alpha )^{-1}} \\ {0.67}(1-\alpha )^{-1} + {0}{(1-0.17\alpha )^{-1}} + {0.33}{(1-0.5\alpha )^{-1}} \end{bmatrix}. \end{aligned} \end{aligned}$$

Finally, solving $v_1=v_2$ or $v_3 = v_2$ both yield $\alpha \in \varnothing$ (when $0.5<\alpha <1$), then we have ${\mathbb {A}} = \varnothing$, $\alpha _{{\varvec{y}}}^\star =\min ({\mathbb {A}}\cup \{1\})=1$.