Learning with Noisy Correspondence

Abstract

This paper studies a new learning paradigm for noisy labels, i.e., noisy correspondence (NC). Unlike the well-studied noisy labels that consider the errors in the category annotation of a sample, the NC refers to the errors in the alignment relationship of two data points. Although such false positive pairs are common especially in the data harvested from the Internet, which however are neglected by most existing works. By taking cross-modal retrieval as a showcase, we propose a method called learning with noisy correspondence (LNC). In brief, the LNC first roughly obtains the clean and noisy subsets from the original data and then rectifies the false positive pairs by using a novel adaptive prediction function. Finally, the LNC adopts a novel triplet loss with soft margins to endow cross-modal retrieval the robustness to the NC. To verify the effectiveness of the proposed LNC, we conduct experiments on six benchmark datasets in image-text and video-text retrieval tasks. Besides the effectiveness of the LNC, the experimental results show the necessity of the explicit solution to the NC faced by not only the standard model training paradigm but also the pre-training and fine-tuning paradigms.

Appendices

Appendix A: Exactly Matched Element (EME)

To quantify the noise in the correspondence, here we introduce the exactly matched element (EME) score which evaluates the similarity of image and text pairs based on how many elements they share in common. Formally,

$$\begin{aligned} \begin{aligned} \textrm{EME} = \frac{1}{2N_I}\sum _{e_i} E(e_i, T) + \frac{1}{2N_T}\sum _{e_t} E(e_t, I) \end{aligned} \end{aligned}$$

(A1)

where $e_i$ and $e_t$ are meaningful elements extracted from the image I and the text T respectively, $N_I$ and $N_T$ are the number of elements in I and T respectively, the function $E(e_i, T)$ is an indicator function that outputs 1 if the element $e_i$ is accurately described in T, and 0 otherwise. Similarly, $E(e_t, I)$ is an indicator function that outputs 1 if the element $e_t$ is depicted in I, and 0 otherwise. EME could be considered as the correspondence label of cross modal pairs. To obtain EME score, we have two main approaches. Firstly, we can compute the EME score with human annotations to ensure accuracy. Alternatively, we can leverage advanced techniques such as semantic segmentation or object detection on images to extract all visual elements, and employ text segmentation methods on text to extract all textual elements. Subsequently, we can calculate EME by utilizing a visual-language model as the indicator (i.e., the function $E(e_i, T)$ and $E(e_t, I)$), such as CLIP. Such an indicator helps identify whether the extracted elements from one modality are described in the other modality.

The EME degree ranges from 0 to 1, where higher values indicate higher similarity between image and text pairs. For example, if an image and a text are completely unrelated, their EME degree will be 0. If they are partially related, their EME degree will be between 0 and 1. If they are fully related, their EME degree will be 1. A toy example is illustrated in Fig. 10 to show how to calculate the EME score of an image-text pair. Note that, the calculation of EME is contingent upon the quantity of shared content across various modalities. Thus EME score primarily estimates the level of cross-modal semantic completeness and lacks the capability to assess other intricate relationships, such as contradictions or complements.

Appendix B: Algorithm

Here we provide the detail algorithm of LNC in Algorithm 1.

Appendix C: Additional Experiments

1.1 C.1 Evaluation on the MS-COCO 5K Testing Set

Here we provide the additional comparison results on the 5K testing set of MS-COCO. As shown in Table 10, LNC achieves SOTA results in the non-noise case. In the cases with noisy correspondence, LNC remarkably outperforms all the baselines. Specifically, in the noisy setting, LNC improves R@1 by 3.9%, 2.7%, 3.7%, and 3.1% in text and image retrieval compared to the best baseline SGR-C.

Table 10 Image-text retrieval on MS-COCO 5K

1.2 C.2 Case Study

In this section, we show some qualitative results of LNC. The example image-text retrieval results are shown in Fig. 11 and Fig. 12. As shown in Fig. 11 (1)–(4) and Fig. 11 (1)–(5), LNC could successfully retrieve the corresponding samples with given queries. Moreover, we provide some failure cases from LNC in Fig. 11 (5)–(6) and Fig. 12 (6). Interestingly, the retrieved image from LNC is fit to the query compared to the ground truth in Fig. 12 (6).

Fig. 11

Some retrieved images by the LNC from CC152K. The left images are the ground truth while the right are the retrieved ones. The successfully retrieved images and the failure cases are highlighted by green dashed boxes and red dashed boxes, respectively (Color figure online)

Fig. 12

Some retrieved captions by the LNC from CC152K. The top sentence is the ground truth while the rest are the retrieved top 3 captions. The successfully retrieved images and the failure cases are highlighted by and , respectively

Fig. 13

a The visualization of loss distribution and GMM fitting results from LNC. b The visualization of the rectified correspondence from LNC

Table 11 Ablation study on co-divide module by using MS-COCO

1.3 C.3 Co-divide and Co-rectify from LNC

In this section, we conduct analysis experiments to further study the influence of co-divide and co-rectify modules. First, we provide the visualization on co-divide and co-rectify from LNC in Fig. 13. As one could observe, the noisy and clean pairs are well divided and rectified by our method.

Besides, to evaluate the impact of our confidence estimation, we performed an ablation study by setting $w_i$ to 1 for clean samples and 0 for noisy samples, based on the ground truth labels. We denote this method as LNC ($w^c_i=1, w^n_i=0$). The results are shown in Table 11. Interestingly, our LNC achieved better results than LNC ($w^c_i=1, w^n_i=0$), despite the latter having access to the ground truth labels. This indicates that our confidence estimation can effectively capture the uncertainty of data correspondence, including fully-matched, partially-matched, and unmatched image-text pairs, and thus improve the cross-modal matching performance.

Appendix D: Implementation Detail

1.1 D.1 Image-text Retrieval

Here we detail how LNC adopts SGR for cross-modal retrieval.

Specifically, for images, the visual features of K local regions are extracted by the Faster R-CNN (Ren et al., 2015). Then we obtain the local embeddings $\{{\textbf{v}}_1,\ldots ,{\textbf{v}}_K\}$ by embedding the above visual features by a fully connected layer f. The global embeddings are obtained via the self-attention mechanism (Vaswani et al., 2017). Moreover specifically, we aggregate all the local embeddings to obtain global embedding $\hat{{\textbf{v}}}$ by treating the average local embeddings as query. For captions, the caption is spited into L words and are further represented by the 300-dimensional features with word embedding technique. Then the 1024-dimensional local embeddings $\{{\textbf{t}}_1,\ldots ,{\textbf{t}}_L\}$ are obtained by a Bi-GRU (Schuster & Paliwal, 1997) g(T). The global embeddings $\hat{{\textbf{t}}}$ of captions are computed similar to the image.

With the extracted visual and textual embeddings, we compute the similarity vector for given pairs. In detail, the similarity vector is computed by:

$$\begin{aligned} s({\textbf{v}}_1, {\textbf{v}}_2; {\textbf{W}}) = \frac{{\textbf{W}}\vert {\textbf{v}}_1-{\textbf{v}}_2 \vert ^2}{{\textbf{W}}\Vert {\textbf{v}}_1-{\textbf{v}}_2\Vert ^2} \end{aligned}$$

(D2)

where ${\textbf{W}}$ denotes a learnable matrix. Then we compute the similarity of global visual and textual embeddings as:

$$\begin{aligned} {\textbf{s}}^g = {\textbf{s}}(\hat{{\textbf{v}}}, \hat{{\textbf{t}}}; {\textbf{W}}_g) \end{aligned}$$

(D3)

and the similarity of local visual and textual embeddings:

$$\begin{aligned} \begin{aligned} {\textbf{s}}^l_j&= {\textbf{s}}({\textbf{a}}^v_j, {\textbf{t}}_j; {\textbf{W}}_l)\\ {\textbf{a}}^v_j&= \sum _{i=1}^K\alpha _{ij}{\textbf{v}}_i \end{aligned} \end{aligned}$$

(D4)

where ${\textbf{a}}^v_j$ denotes aggregated embeddings, $\alpha _{ij}$ denotes the attention coefficient:

$$\begin{aligned} \alpha _{ij} = \frac{exp(\lambda {\hat{c}}_{ij})}{\sum _{j=1}^Kexp(\lambda {\hat{c}}_{ij})} \end{aligned}$$

(D5)

where ${\hat{c}}_{ij}$ denotes the cosine similarity between the i-th image region and j-th word in a given image-text pair.

Once the similarity vectors ${\mathcal {N}} = \{{\textbf{s}}_1^l, {\textbf{s}}_2^l, \cdots , {\textbf{s}}_K^l\}$ are obtained, we treat them as the similarity graph nodes and compute the graph edges as:

$$\begin{aligned} e({\textbf{s}}_p, {\textbf{s}}_q; {\textbf{W}}_{in}, {\textbf{W}}_{out}) = \frac{exp(({\textbf{W}}_{in}{\textbf{s}}_p)({\textbf{W}}_{out}{\textbf{s}}_q))}{\sum _q exp(({\textbf{W}}_{in}{\textbf{s}}_p)({\textbf{W}}_{out}{\textbf{s}}_q))} \end{aligned}$$

(D6)

where ${\textbf{W}}_{in}$ and ${\textbf{W}}_{out}$ are the learnable matrixes to transform the incoming and outgoing similarity. Finally, we aggregate all the similarities by updating the similarity of nodes and edges by

$$\begin{aligned} \begin{aligned} \hat{{\textbf{s}}}^n_p&= \sum _q e({\textbf{s}}_p^n, {\textbf{s}}_q^n; {\textbf{W}}_{in}^n, {\textbf{W}}_{out}^n) \cdot {\textbf{s}}_q^n\\ {\textbf{s}}_q^{n+1}&= ReLU({\textbf{W}}_r^n\hat{{\textbf{s}}}^n_p) \end{aligned} \end{aligned}$$

(D7)

where ${\textbf{W}}_{in}^n$, ${\textbf{W}}_{out}^n$ and ${\textbf{W}}_{r}^n$ are learnable matrixes, ${\textbf{s}}_p^0$ and ${\textbf{s}}_q^0$ are the initial nodes from ${\mathcal {N}}$ at step $n = 0$. Specifically, it iteratively updates the similarity for N steps, and treats the global node as the reasoned similarity. Finally, we use a fully connected layer to compute the final similarity as S(I, T) in LNC.

Table 12 Experiment parameters

Here we provide the used parameters for training LNC in Table 12 including the number of epochs for warmup, the number of epochs for training, the number of learning rate update intervals (LR Update) and batch size.

1.2 D.2 Video-Text Retrieval

In the video-text retrieval experiment, we take the model proposed by Miech et al. (2019) as an example and extend it to be robust again noisy correspondence. Specifically, with the given video clip and caption $({\textbf{v}}, {\textbf{t}})$, we adopt the class of non-linear embedding functions to obtain the visual and textual features, i.e.,

$$\begin{aligned} \begin{aligned}&f({\textbf{v}})=\left( W_{1}^{v} {\textbf{v}}+b_{1}^{v}\right) \circ \sigma \left( W_{2}^{v}\left( W_{1}^{v} {\textbf{v}}+b_{1}^{v}\right) +b_{2}^{v}\right) \\&g({\textbf{t}})=\left( W_{1}^{t} {\textbf{t}}+b_{1}^{c}\right) \circ \sigma \left( W_{2}^{c}\left( W_{1}^{c} {\textbf{t}}+b_{1}^{c}\right) +b_{2}^{c}\right) \end{aligned} \end{aligned}$$

(D8)

where $W_1^v$, $W_1^t$, $W_2^v$, and $W_2^t$ are the learnable weight, $b_1^v$, $b_1^t$, $b_2^v$, and $b_2^t$ are the learnable bias vectors, $\sigma$ is an element-wise sigmoid activation and $\circ$ is the element-wise multiplication. In all experiments, we embed the clip and caption into 4096-dimensional space.

For all video-text experiments, we adopt the Adam optimizer with a learning rate of 0.0001 and set the batch size to 256. For the pre-training on HowTo100M data, we follow the default settings in Miech et al. (2019). The number of warmup epochs is fixed to 3 for all video datasets.