Semantic Contrastive Bootstrapping for Single-Positive Multi-label Recognition

Abstract

Learning multi-label image recognition with incomplete annotation is gaining popularity due to its superior performance and significant labor savings when compared to training with fully labeled datasets. Existing literature mainly focuses on label completion and co-occurrence learning while facing difficulties with the most common single-positive label manner. To tackle this problem, we present a semantic contrastive bootstrapping (Scob) approach to gradually recover the cross-object relationships by introducing class activation as semantic guidance. With this learning guidance, we then propose a recurrent semantic masked transformer to extract iconic object-level representations and delve into the contrastive learning problems on multi-label classification tasks. We further propose a bootstrapping framework in an Expectation-Maximization fashion that iteratively optimizes the network parameters and refines semantic guidance to alleviate possible disturbance caused by wrong semantic guidance. Extensive experimental results demonstrate that the proposed joint learning framework surpasses the state-of-the-art models by a large margin on four public multi-label image recognition benchmarks. Codes can be found at https://github.com/iCVTEAM/Scob.

Appendix

Fig. 10

Visualization of semantic masks on MS-COCO. In each group, the left and right images denote the input image and the masked class activation maps in our proposed Scob approach

Table 9 The AP results of 10 small-scale classes in MS-COCO dataset

1.1 Appendix A: Visualizations on Class-specific Semantic Masks

Beyond the ablations on semantic masks in the main manuscript, here we exhibit more semantic masks generated on the MS-COCO dataset in Fig. 10. In each group, the left and right images denote the input image and the masked class activation maps in our proposed Scob approach. As in the first two columns, it can be found that our proposed approach focuses on the semantic objects e.g., airplane and bus, while filtering the background information. In the second to the fourth groups of Fig. 10, two major challenges occur to distinguish these objects: (1) the semantic objects are relatively small compared to the image size; (2) these objects show high dependencies on the other objects or namely co-occurrences. As can be seen from this generated semantic guidance, our proposed Scob has the ability to distinguish the specific object from its related objects and then forms the representative features.

Fig. 11

The network architecture of the proposed Recurrent Semantic Masked Transformer. We leverage the masked multi-head attention mechanisms with iterative semantic guidance generated by CAMs

1.2 Appendix B: Performances on Recognizing Small Objects

Besides Fig. 10 showing some CAM results of large and small-scale objects. We additionally list the AP results of 10 small-scale classes in the MS-COCO dataset to show our approach is robust to them in Table 9.

1.3 Appendix C: Details on Recurrent Semantic Masked Transformer

The proposed recurrent Semantic Masked Transformer (SMT) mainly consists of positional encoding, semantic mask, and multiple multi-head attention units as in previous work (Vaswani et al., 2017). Different from the implementation of these works, we rely on the feature maps of ResNet backbones as image feature patches (illustrated as image patches for a better view). Inspired by the masked coding manner (Devlin et al., 2019) in the field of natural language processing, here we adopt the class activation generated from the last optimization as the guidance for semantic masked attention. We present the detailed network architectures in Fig. 11.

Semantic masked encoding Denote $\textbf{F}=\Phi (\textbf{x};\theta _b) \in \mathbb R^{HW \times K}$ as the backbone feature extracted from ResNet stages with parameter $\theta _b \in \Theta$. $\textbf{F}$ is split into $H \times W$ patches $\left\{ \textbf{F}_{i,j} \mid i \in \{1,2,\dots ,H\}, j \in \{1,2,\dots ,W\} \right\}$ of which each patch has channel size K, as the input of SMT. As mentioned in Section 3.2, the recurrent SMT applies a learnable positional encoding $\Delta (\cdot ):\mathbb N^{W \times H} \mapsto \mathbb R^1$ and semantic mask to $\{ \textbf{F}_{i,j} \}$:

$$\begin{aligned} \mathcal M(\textbf{F}_{i,j})=(\textbf{F}_{i,j}+\Delta (i,j))\cdot (1-\textbf{M}^c), \end{aligned}$$

(14)

where the symbols are described as aforementioned. The implementation of $\Delta$ follows the previous Transformer architecture, i.e., DETR (Carion et al., 2020). Let $\Delta _h:\mathbb N^W \mapsto \mathbb R^1$ and $\Delta _v:\mathbb N^H \mapsto \mathbb R^1$ be the horizontal and vertical encoding. Then $\Delta$ is the concatenation of them:

$$\begin{aligned} \Delta (i,j) = \Delta _h(i) \oplus \Delta _v(j), \end{aligned}$$

(15)

where $\oplus$ denotes the feature concatenation operation. Following Devlin et al. (2019), positional encoding and masks are applied to the query and key of multi-head attention, providing global position information and constraining the extracted features $\textbf{H}_{i,j}^c$ showing high response to only a specific semantic class related to the masks.

Multi-scale SMTs As the different network stages during training are sensitive to semantic objects of different scales, hence incorporating multi-scale and multi-level features can be beneficial for final feature representations, which is also one of the challenging problems in multi-label classification tasks. In our implementation, we propose to use two SMTs connecting to Stage 3 and Stage 4 of ResNet-50, and combine their outputs to extract image features at multiple scales. In this manner, objects of small scales are more easily to be presented in earlier network stages without a significant loss in resolutions.

Algorithm 2:

Insert activated node v to $\mathbb G$ and adjust priority tree

Algorithm 3:

Pop top nodes $\mathcal N'$ from $\mathbb G^i$ for negative sampling

1.4 Appendix E: Algorithms of Instance Priority Trees

Instance Priority Tree (IPT) $\mathbb {G}=\{\mathcal {E}, \mathcal {N}\}$ is a heap implemented with a complete binary tree structure, maintaining high confident instances to construct negative samples for contrastive learning. Each element $u \in \mathcal N$ is a binary tuple $(\widetilde{H}_u, s_u)$, where $\widetilde{H}_u$ is a semantic masked feature and $s_u$ is the corresponding activation confidence score. Each edge $e \in \mathcal {E}$ indicates an affiliation relationship (u, v) where the parent node u is with higher confidence than the leaf ones v.

In our implementations, we adopt an array of length n, $[u_0,u_1,u_2,\dots ,u_n]$, to store the nodes $\mathcal N$ and describe the edges $\mathcal E$ by their indices in the array, i.e., $u_i$ is the parent node of $u_{2i}$ and $u_{2i+1}$. The success of IPT relies on three typical operations: “Insert activated nodes”, “Adjust priority tree”, and “Pop top nodes”. In our implementation, the tree adjustment operation needs to be conducted after every Insert or Pop operation for maintaining the tree structures. In other words, the “Insert activated nodes” and “Adjust priority tree” functions are operated together. Here we elaborate the detailed algorithms in Algorithm 2 and Algorithm 3.

In addition, we maintain the size of instance priority tree $\mathbb {G}$ in a proper range to reduce computation costs. In this manner, instances with lower confidence features would not be added to our tree storage, indicating the efficiency of our model.