Visually weighted neighbor voting for image tag relevance learning

Abstract

The presence of non-relevant tags in image folksonomies hampers the effective organization and retrieval of user-contributed images. In this paper, we propose to learn the relevance of user-supplied tags by means of visually weighted neighbor voting, a variant of the popular baseline neighbor voting algorithm proposed by Li et al. (IEEE Trans Multimedia 11(7):1310–1322, 2009). To gain insight into the effectiveness of baseline and visually weighted neighbor voting, we qualitatively analyze the difference in tag relevance when using a different number of neighbors, for both tags relevant and tags not relevant to the content of a seed image. Our qualitative analysis shows that tag relevance values computed by means of visually weighted neighbor voting are more stable and representative than tag relevance values computed by means of baseline neighbor voting. This is quantitatively confirmed through extensive experimentation with MIRFLICKR-25000, studying the variation of tag relevance values as a function of the number of neighbors used (for both tags relevant and tags not relevant with respect to the content of a seed image), as well as the influence of tag relevance learning on the effectiveness of image tag refinement, tag-based image retrieval, and image tag recommendation.

Appendix

This appendix details the derivation of the difference in accuracy of visual search over random sampling. To that end, given a seed image I, we make a distinction between a tag w ₁ relevant to the content of I and a tag w ₂ not relevant to the content of I.

Difference in search accuracy for w ₁   We make use of $V_{I,w_1}(k)$ to represent the number of images relevant to w ₁ in the set of k visual neighbors of I. We assume that the value of $V_{I,w_1}(k)$ is (1) upper-bounded by the number of images relevant to w ₁ when making use of perfectly working visual search and (2) lower-bounded by the number of images relevant to w ₁ when making use of random sampling. This is conceptually illustrated by Fig. 6.

Fig. 6

The number of images relevant to w ₁ in the set of k visual neighbors of I

When visual search works perfectly, $V_{I,w_1}(k)$ increases linearly from zero to $|R_{w_1}|$ for k varying from zero to $|R_{w_1}|$. Indeed, all images in the set of visual neighbors belong to $|R_{w_1}|$. For $k>|R_{w_1}|$ , $V_{I,w_1}(k)=|R_{w_1}|$ because Φ only contains $|R_{w_1}|$ images related to w ₁. This is denoted in Fig. 6 by “ideal”. When making use of random sampling, we assume that $V_{I,w_1}(k)$ increases linearly and that all images of $R_{w_1}$ can only be found in the set of visual neighbors when this set is identical to Φ (this is, when k is equal to |Φ|). This is denoted in Fig. 6 by “random”. In practice, we also assume that $V_{I,w_1}(k)$ increases linearly until the value of $V_{I,w_1}(k)$ is equal to $|R_{w_1}|$. This is denoted in Fig. 6 by “real”. When visual search is effective, the dashed line will be close to “ideal”. Otherwise, when visual search is not effective, the dashed line will be close to “random”. In Fig. 6, k′ represents the minimal value of k for which all images of $R_{w_1}$ can be found in the set of visual neighbors of I.

In general, given a tag w ₁, the accuracy of visual search $A_{I,w_{1},k}$ can be written as $V_{I,w_1}(k)/k$. Given the above observations made for $V_{I,w_1}(k)$, $A_{I,w_{1},k}$ can also be expressed as follows:

$$ \label{eq:15} A_{I,w_{1},k} = \left\{ \begin{array}{ll} \dfrac{|R_{w_1}|}{k'}, & {k \leq k'}\\\\[-9pt] \dfrac{|R_{w_1}|}{k}, & {k > k'}.\end{array} \right. $$

(15)

The difference in accuracy of visual search over random sampling for w ₁ can then be expressed as follows:

$$ \label{eq:16} \epsilon_{I,w_{1},k} = \left\{ \begin{array}{ll} \dfrac{|R_{w_1}|}{k'}-P(R_{w_1}), & {k \leq k'}\\\\[-9pt] \dfrac{|R_{w_1}|}{k}-P(R_{w_1}), & {k > k'}.\end{array} \right. $$

(16)

Difference in search accuracy for w ₂   We make use of $V_{I,w_2}(k)$ to represent the number of images relevant to w ₂ in the set of k visual neighbors of I. Further, we assume that the value of $V_{I,w_2}(k)$ is (1) lower-bounded by the number of images relevant to w ₂ when visual search works perfectly and (2) upper-bounded by the number of images relevant to w ₂ when making use of random sampling. This is conceptually illustrated by Fig. 7.

Fig. 7

The number of images relevant to w ₂ in the set of k visual neighbors of I

When visual search works perfectly (in this case, when visual search finds all images relevant to I in Φ), then the images in $R_{w_2}$ should not be among the visual neighbors of I when k ≤ |R _I |, where R _I represents the set of images relevant to I. Here, we assume that images are relevant to each other when they have semantic concepts in common (for the sake of simplicity, we also assume that images relevant to I are not relevant to w ₂). However, for k > |R _I |, the set of visual neighbors of I will start to contain images belonging to $R_{w_2}$. This is denoted in Fig. 7 by “ideal”. When making use of random sampling, we assume that the number of images of $R_{w_2}$ in the set of visual neighbors increases linearly when k varies from zero to |Φ|. This is denoted in Fig. 7 by “random”. In practice, we are able to find a k′ for which we can start to see images of $R_{w_2}$ in the set of visual neighbors. This is denoted in Fig. 7 by means of “real”. In practice, we also assume that the number of images of $R_{w_2}$ in the set of visual neighbors increases linearly. The accuracy of visual search for w ₂, $A_{I,w_2,k}$, is calculated by dividing $V_{I,w_2}(k)$ by k:

$$ \label{eq:17} A_{I,w_2,k} = \left\{ \begin{array}{ll} 0, & {k \leq k'}\\\\[-9pt] \dfrac{|R_{w_2}|-|R_{w_2}| \cdot \frac{k'}{k}}{|\Phi|-k'}, & {k > k'}.\end{array} \right. $$

(17)

The difference in accuracy of visual search over random sampling for w ₂ can then be expressed as follows:

$$ \label{eq:18} \epsilon_{I,w_2,k} = \left\{ \begin{array}{ll} -P(R_{w_2}), & {k \leq k'}\\\\[-9pt] \dfrac{|R_{w_2}|-|R_{w_2}| \cdot \frac{k'}{k}}{|\Phi|-k'}-P(R_{w_2}), & {k > k'}.\end{array} \right. $$

(18)