Keywords

1 Introduction

Weakly Supervised Object Detection (WSOD) [3, 7, 10, 17, 18, 20, 21, 23, 32, 33, 35, 42,43,44] aims to detect objects only using image-level annotations for supervision. Despite remarkable progress, existing approaches still have difficulties in accurately identifying tight boxes of target objects with only image-level annotations, thus their performance is inferior to the fully supervised counterparts [6, 13, 22, 25, 28,29,30].

Fig. 1.
figure 1

Comparison of MIL-based approaches and our target. MIL-based approaches tend to assign high confidence to discriminative parts (blue boxes) of target objects. Our target is to alleviate such cases and lift the confidence of the tight ones (yellow boxes). Best viewed in color. (Color figure online)

Full size image

To localize objects with weak supervision information, one popular solution is to apply Multiple Instance Learning (MIL) for mining high-confidence region proposals [34, 47] with positive image-level annotations. However, MIL usually discovers the most discriminative part of the target object (e.g. the head of a cat) rather than the entire object region, as shown in Fig. 1. This inability of providing the complete object severely limits its effectiveness for WSOD. To address this issue, Li et al. [21] exploited the contrastive relationship between a selected region and its mask-out image for proposal selection. Nevertheless, the mask-out strategy fails for multi-instance cases. The selector is easily confused by remained instances with high responses, even though the correct object has been masked out.

Recently, some weakly supervised semantic segmentation approaches [19, 36, 38, 40] have demonstrated promising performance. Utilizing the inferred segmentation confidence maps, Diba et al. [10] presented a cascaded approach that leverages segmentation knowledge to filter noisy proposals and achieves competitive detection results. However, we argue that their solution is sub-optimal and insufficient as it only considers the segmentation confidence inside the proposal boxes, thus is unable to filter high-response fragments of object parts, as the magenta boxes shown in Fig. 2 (b).

In this work, we propose a principled and more effective approach, compared with [10], to mine tight object boxes by exploiting segmentation confidence maps in a creative way, aiming for addressing the challenging WSOD problems. Our approach is motivated by the following observations, as illustrated by two examples in Fig. 2 (a). We use blue and yellow to encode two kinds of boxes, which partially and tightly cover objects respectively. Based on the semantic segmentation confidence maps obtained in a weakly supervised manner, many pixels surrounding the blue boxes have high predicted segmentation confidence, while very few high-confidence pixels are included in the surrounding context for the yellow ones of higher tightness. We find that a desirable tight object box generally needs to satisfy two properties based on segmentation context:

  • Purity: most pixels inside the box should have high confidence scores, which guarantees that the box is located around the target object;

  • Completeness: very few pixels are with high confidence scores in the surrounding context of the target box.

Fig. 2.
figure 2

(a) Motivation of the proposed TS2C: fewer high response pixels on the segmentation confidence map are included by enlarging higher-quality boxes of object candidates (the yellow one) compared with partial bounding boxes (the blue one). (b) Comparison of the rank 1 proposal using the strategy proposed by [10] (magenta boxes) and ours (yellow boxes). Best viewed in color. (Color figure online)

Full size image

Based on these properties, we devise a simple yet effective approach, named Tight box mining with Surrounding Segmentation Context (TS2C), to efficiently select object candidates of high quality from thousands of candidates. Specifically, the proposed TS2C examines two kinds of regions for evaluating the tightness of bounding boxes: (1) the region included in the box and (2) the region surrounding the box. It computes objectness scores of the two regions by averaging the corresponding pixel confidence values on the segmentation maps. Tight boxes are expected to be with high and low objectness values of the two kinds of regions simultaneously. Thus, the difference of two objectness scores is then taken as the quality metric on the final tightness for ranking object candidates. Figure 2 (b) shows the top 1 object candidate inferred by the proposed TS2C. We can see that our approach is more effective for mining tight object boxes than [10]. Moreover, our proposed TS2C is generic and can be easily integrated into any WSOD framework by introducing a parallel semantic segmentation branch for class-specific confidence map prediction. Benefiting from our TS2C, we achieve 48.0% and 44.4% mAP scores on the challenging Pascal VOC 2007 and VOC 2012 benchmarks, which are the new state-of-the-arts in the WSOD community.

2 Related Work

Multiple Instance Learning (MIL) provides a suitable way for formulating and solving WSOD. In specific, if an image is annotated with a specific class, at least one proposal instance from the image is positive for this class; and no proposal instance is positive for unlabeled classes. Previous works on applying MIL to WSOD can be roughly categorized into two-step [7, 17, 21, 35] and end-to-end [3, 10, 18, 20, 32, 33] based approaches.

Two-Step Approaches. First extract proposal representation leveraging hand-crafted features or pre-trained CNN models and employ MIL to select the best object candidate for learning the object detector. For instance, Wang et al. [35] presented a latent semantic clustering approach to select the most discriminative cluster for each category. Cibis et al. [7] learned a multi-fold MIL detector by re-labeling proposals and re-training the object classifier iteratively. Li et al. [21] first trained a multi-label classification network on entire images and then selected class-specific proposal candidates using a mask-out strategy, followed by MIL for learning a Fast R-CNN detector. Recently, Jie et al. [17] took a similar strategy as Li et al. [21] and proposed a more robust self-taught approach to learn a detector by harvesting more accurate supportive proposals in an online manner. However, splitting the WSOD into two steps results in a non-convex optimization problem, making such approaches trapped in local optima.

End-to-End Approaches. Combine CNNs and MIL into a unified framework for addressing WSOD. Oquab et al. [27] and Wei et al. [39] adopted a similar strategy to learn a multi-label classification network with max-pooling MIL. The learned classification model was then applied to coarse object localization [27]. Bilen et al. [3] proposed a novel Weakly Supervised Deep Detection Network (WSDDN) including two key streams, one for classification and the other for object localization. The outputs of these two streams are then combined for better rating the objectness of proposals. Based on WSDDN, Kantorov et al. [18] proposed to learn a context-aware CNN with contrast-based contextual modeling. Both [18] and our approach employ proposal context to identify high-quality proposals. However, [18] exploits inside/outside context features of each bounding box for learning to classification, in contrast, we leverage objectness scores obtained by segmentation confidence maps to pick out tight candidates. Recently, Tang et al. [32] also employed WSDDN as the basic network and augmented it with several Online Instance Classifier Refinement (OICR) branches, which is the state-of-the-art on the challenging WSOD task. In this work, we employ both WSDDN and OICR to develop our framework where the proposed TS2C is leveraged to further improve performance. Both [10] and our approach utilizes object segmentation knowledge to benefit WSOD. However, Diba et al. [10] only considered the confidence of pixels included in the bounding box for rating the proposal objectness, which is not as effective as ours.

Beyond the above mentioned related works, some fully-supervised object detection approaches [5, 12, 22, 46] also exploit contextual information of bounding boxes for benefiting object detection. Both Chen et al. [5] and Li et al. [22] leveraged information of enlarged contextual proposals to enhance the accuracy of the classifier. Zhu et al. [46] proposed to use a pool of segments obtained in the bottom-up manner to obtain better detection boxes. Our TS2C is totally different from these works in terms of both motivation and methodology. In particular, our motivation is to employ surrounding segmentation context to suppress these false positive objects parts. In addition, our approach can be easily embedded into any WSOD framework to make a further performance improvement.

Fig. 3.
figure 3

Overview of the proposed TS2C for weakly supervised object detection. Several convolutional layers are leveraged to extract the intermediate features of an input image. The entire feature maps are firstly fed into a Classification branch to produce object localization maps corresponding to image-level labels. We then employ the localization maps to generate the segmentation masks, which serve as supervision to learn the Segmentation branch. Based on the segmentation confidence maps, we utilize TS2C to evaluate the objectness scores of proposals according to their purity and completeness, which collaborates with the OICR [32] for training the Detection branch.

Full size image

3 The Proposed Approach

We show the overall architecture of the proposed approach in Fig. 3. It consists of three key branches, i.e. image classification, semantic segmentation and object detection. In particular, the Classification branch is employed to generate class-specific localization maps. Following the previous weakly supervised semantic segmentation approaches [37], we leverage the inferred localization maps to produce pseudo segmentation masks of training images, which are then used as supervision to train the Segmentation branch. The segmentation confidence maps from the Segmentation branch are then employed to evaluate objectness scores of the proposals according to the proposed TS2C, which finally collaborates with the Detection branch for learning an improved object detector. The overall framework is trained by minimizing the following composite loss functions from the three branches using stochastic gradient descent:

$$\begin{aligned} L = L_{cls} + L_{seg} + L_{det}. \end{aligned}$$
(1)

We will introduce each branch below and then elaborate on details of TS2C.

3.1 Classification for Object Localization

Inspired by [10, 24, 45], the fully convolutional network along with the Global Average Pooling (GAP) operation is able to generate class-specific activation maps, which can provide coarse object localization prior. We conduct experiments on Pascal VOC benchmarks, in which each training image is annotated with one or several labels. We thus treat the classification task as a separate binary classification problem for each class. Following [27], the loss function \(L_{cls}\) is thus defined as a sum of C binary logistic regression losses.

3.2 Weakly Supervised Semantic Segmentation

The Classification branch can produce localization cues for foreground objects. We assign the pixels with values on the class-specific confidence map larger than a pre-defined normalized threshold (i.e. \(\ge \)0.78) with the corresponding class label. Beyond the object regions, background localization cues are also needed for training the segmentation branch. Motivated by [19, 36, 38, 40], we leverage the saliency detection technology [41] to produce the saliency map for each training image. Based on the generated saliency map, we choose the pixels with low normalized saliency values (i.e. \(\le \)0.06) as background. However, both the class-specific confidence map and the saliency map are not accurate enough to guarantee a high-quality segmentation mask. To alleviate the negative effect caused by falsely assigned pixels, we ignore the ambiguous pixels during training the Segmentation branch, including (1) pixels that are not assigned semantic labels, (2) foreground pixels of different categories that are in conflict, and (3) low-saliency pixels that fall in the foreground pixels. With the produced pseudo segmentation mask, we train the Segmentation branch with pixel-wise cross-entropy loss \(L_{seg}\), which is widely adopted by fully-supervised schemes [4, 26].

3.3 Learning Object Detection with TS2C

For each training or test image, Selective Search [34] is employed to generate object proposals and Spatial Pyramid Pooling (SPP) [15] is leveraged to generate constant size feature maps for different proposals. Our TS2C aims to select high-quality object candidates from thousands of candidates to improve the effectiveness of training, which can be easily implanted into any WSOD framework. We choose the state-of-the-art Online Instance Classifier Refinement (OICR) [32] as the backbone of the Detection branch, which collaborates with the proposed TS2C for learning a better object detector. In the following, we will first make a brief introduction of OICR, and then explain how to leverage our TS2C to benefit the learning process of WSOD.

OICR. As shown in Fig. 3, the OICR mainly includes two modules, i.e. multiple instance classification and instance refinement. In particular, the multiple instance classification module is inspired from [3], which includes two branches to extract parallel data streams from the input features pooled by SPP, as shown in Fig. 4 (a). The upper stream conducts softmax operation on each individual proposal for classification. The bottom stream estimates a probability distribution over all candidate proposals using softmax, which indicates the contribution of each proposal to classifier decision for each class. Therefore, these two streams provide classification-based and localization-based features for each proposal. Both inferred scores are then fused with element-wise product operation and finally aggregated into image-level prediction by sum-pooling over all proposals. With the supervision of image-level annotations, the multiple instance classification module can be learned with binary logistic regression losses as detailed in Sect. 3.1.

Fig. 4.
figure 4

Details of (a) multiple instance classification module and (b) instance refinement module in TS2C.

Full size image

By leveraging multiple instance classification module as a basic classifier for obtaining initial classification scores for each proposal, progressive refinement is then conducted via the instance refinement module, as detailed in Fig. 4 (b). In particular, the instance refinement module first selects the top-scoring proposal of each image-level label. Those proposals with high spatial overlap scores over the top-scoring one are then labeled correspondingly. The idea behind such a module is that the top-scoring proposal may only contain part of a target object and its adjacent proposals may cover more object regions. Benefiting from both two modules embedded in the OICR, each proposal is assigned with a pseudo class label, which is then employed as supervision for learning detection with the softmax cross-entropy loss [13, 14, 29]. To address the initialization issue (i.e. the classifier cannot well recognize proposals with randomly initialized parameters at the beginning of training), OICR adopts a weighted loss by assigning different weights to different proposals during different training iterations. Thus, the \(L_{det}\) is composed of binary logistic regression losses for image-level classification and softmax cross-entropy loss for proposal-level classification. Please refer to [32] for more details.

Problems. However, such progressive refinement operation of OICR highly relies on the quality of initial object candidates from the multiple instance classification module. This means without reasonable object candidates received from the multiple instance classification module for initialization, the following progressive refinement strategy of OICR cannot find the correct proposals with high IoU scores over ground-truth bounding boxes. This brings a critical risk: if the multiple instance classification module fails to produce reasonable object candidates then the OICR cannot recall the missed object with any hope. We propose to reduce such a risk by designing an objectness rating approach from a totally new perspective. In particular, we detail our proposed TS2C that rates the proposals’ objectness from the segmentation view in the following.

Fig. 5.
figure 5

Motivation of the conditional average strategy: only a small number of pixels belong to objects in the surrounding regions. To promote the objectness score of surrounding context, we only employ pixels with large confidence values (highlighted by red color) for conducting average calculation. Best viewed in color. (Color figure online)

Full size image

TS\(^{2}\)C for Learning Detection. As shown in Fig. 3, TS2C uses the segmentation confidence maps from the Segmentation branch to rate the proposal objectness. We consider \(x_i (i = 1 \cdots n)\) as one proposal from a given training image annotated by class c. Let \(H_c\) denote the confidence map of category c predicted by the semantic Segmentation branch. For \(x_i\), we calculate objectness scores of both the region inside the box \(P_I\) and the surrounding context \(P_S\) between \(x_i\) and the corresponding enlarged one. Let \(avg(H_c, x_i)\) denote the operation of computing \(P_I\), which takes all pixel values included in \(x_i\) into account. \(P_I\) of a large value can guarantee that \(x_i\) is around the target object. To obtain a robust surrounding objectness score \(P_S\), we adopt a conditional average strategy \(\hat{avg}(H_c, x_i)\). As shown in Fig. 5, many surrounding regions of negative candidates include a large number of un-related (i.e. background) pixels, which are with low confidence scores. Therefore, the resulted objectness score will be small if we average all the pixel values for computing \(P_S\), in a similar way as for \(P_I\). However, we expect the value of \(P_S\) to be large, so that negative candidates of such cases can be suppressed by \(P_I-P_S\). To this end, we first rank the pixels in the surrounding region according to their confidence scores and the conditional average strategy only employs the first 50% pixels for calculating the objectness score. Then, the objectness score \(O(x_i)\) of the proposed TS2C is finally calculated as

$$\begin{aligned} \begin{aligned} O(x_i) = P_I - P_S = avg(H_c, x_i) - \hat{avg}(H_c, x_i). \end{aligned} \end{aligned}$$

We rank all the object candidates according to \(O(x_i)\) and build a candidate pool by selecting the top two hundred proposals, collaborating with the OICR for learning a better detector. As shown in Fig. 3, \(\oplus \) means the OICR will only select object candidates from the pool produced by TS2C for the following training process.

During the testing stage, we ignore the Classification and Segmentation branches, and leverage the classification outputs from the instance refinement module to obtain the final detection results.

4 Experiments

4.1 Datasets and Evaluation Metrics

Datasets. We conduct experiments on Pascal VOC 2007 and 2012 datasets [11], which are the two most widely used benchmarks for weakly supervised object detection. For VOC 2007, we train the model on the trainval set (5,011 images) and evaluate on the test set (4,096 images). We also make extensive ablation analysis on VOC 2007 to verify the effectiveness of some settings. For VOC 2012, we train the model on the trainval set (11,540 images) and evaluate on test set (10,991 images) by submitting the testing result to the evaluation server.

Metrics. Following [10, 17, 32], we adopt two metrics for evaluation, i.e. mean average precision (mAP) and correct localization (CorLoc) [9], for evaluation on test and trainval sets, respectively. Both two metrics employ the same threshold of bounding box overlaps with ground-truth boxes, i.e. IoU \(>=\)0.5.

4.2 Implementation Details

We use the object proposals generated by Selective Search [34], and adopt the VGG16 network [31] pre-trained on ImageNet [8] as the backbone of the proposed framework. We employ the Deeplab-CRF-LargeFOV [4] model to initialize the corresponding layers in the segmentation branch. For the newly added layers, the parameters are randomly initialized with a Gaussian distribution \(\mathcal {N}(\mu , \delta )(\mu =0, \delta =0.01)\). We take a mini-batch size of 2 images and set the learning rates of the first 40 K and the following 30 K iterations as 0.001 and 0.0001 respectively. During training, we take five image scales \(\{480, 576, 688, 864, 1200\}\) for data augmentation. For TS2C, we adopt an enlarged ratio of 1.2 to obtain the surrounding context, which is further employed for evaluating completeness of object candidates. Our experiments use the OICR [32] code, which is implemented based on the publicly available Caffe [16] deep learning framework. All of our experiments are run on NVIDIA TITAN X PASCAL GPUs.

Table 1. Comparison of detection average precision (AP) (%) on PASCAL VOC.
Full size table
Table 2. Comparison of detection AP (%) by training FRCNN detectors.
Full size table
Fig. 6.
figure 6

Examples of our object detection results on VOC 2007 test set. Ground-truth annotations, predictions of OICR and ours are indicated by red, green and blue bounding boxes respectively. Best viewed in color. (Color figure online)

Full size image
Table 3. Comparison of correct localization (CorLoc) (%) on PASCAL VOC.
Full size table

4.3 Comparison with Other State-of-the-arts

We compare our approach with both two-step [7, 17, 21, 35] and end-to-end [3, 10, 18, 20, 32, 33] approaches. Top-3 results are indicated by , and colors. Table 1 shows the comparison in terms of AP on the VOC 2007. It can be observed that the proposed TS2C is effective and outperforms all the other approaches. In particular, we adopt OICR proposed by Tang et al. [32] as the detection backbone in the proposed framework. Our approach outperforms OICR by 3.1%. The gains are mainly from using both purity and completeness metrics for filtering noisy object candidates. We also show the comparison between our approach and other state-of-the-arts on PASCAL VOC 2012 in terms of AP. Our resultFootnote 1 outperforms the baseline (i.e. Tang et al.[32]) and the state-of-the-art approach (i.e. Jie et al.[17]) by 2.1% and 1.7%, respectively.

Following [32], we also train a FRCNN [13] detector using top-scoring proposals produced by TS2C as pseudo ground-truth bounding boxes. As shown in Table 2, the performance can be further enhanced to 48.0% and 44.4%Footnote 2 on VOC 2007 and 2012, respectively. Our results from a single model are much better than those of [32] obtained by models (e.g. VGG16 and VGG-M) fusion. In addition, we conduct additional experiments using CorLoc as the evaluation metric. Table 3 shows the comparison on the VOC 2007 and 2012. Our approach achieves 61.0% and 64.4% in terms of CorLoc score, which are competitive compared with the state-of-the-arts. We visualize some successful detection results (blue boxes) on VOC 2007, as shown in Fig. 6. Results from OICR (green boxes) and ground truth (red boxes) are employed for comparison. It can be seen that our approach effectively reduces false positives including partial objects.

4.4 Ablation Experiments

We conduct extensive ablation analyses of the proposed TS2C, including the influence of the enlarged scale for obtaining surrounding context and the proposed tightness criteria (i.e. purity and completeness). All experiments are based on VOC 2007 benchmark.

Table 4. Ablation study on PASCAL VOC 2007.
Full size table

Purity and Completeness. One of our main contributions is the proposed criteria of purity and completeness for measuring the tightness of object candidates based on the semantic segmentation confidence maps. To validate the effectiveness of our approach (i.e. \(P_I-P_S\)), we test the other popular setting where only the purity (e.g. \(P_I\)) is taken into account. Specifically, we firstly leverage the two metrics to rank object candidates for annotated class(es). For example, if the image is annotated with two labels, we will produce two rankings according to segmentation confidence maps of the two classes, which are then employed for evaluating recall scores. As shown in Fig. 7, we vary the top number of object candidates based on the rankings from two metrics. Since our evaluation method only takes one object candidate for each annotated category in the top-1 case, the upper bound of the recall is 57.9% due to the existence of multi-instance images. Despite the apparent simplicity, the recall scores of our proposed \(P_I-P_S\) significantly outperform those of \(P_I\) under different settings according to the top number, which demonstrates that the completeness metric is effective for reducing noisy object candidates. More visualizations of rank 1 boxes produced by \(P_I-P_S\) and \(P_I\) are shown in Fig. 8. We can observe that our approach can successfully discover the tight ones from thousands of candidates. To further validate the effectiveness of the proposed TS2C, we also conduct experiments using purity i.e. \(P_I\) for ranking object candidates as adopted in [10] for proposal selection, which results in 42.2% in mAP. By simultaneously taking purity and completeness into account, i.e. \(P_I-P_S\), the result surpasses the baseline by 2.1% as shown in Table 4.

Fig. 7.
figure 7

Comparison of recall scores (%) between the proposed TS2C (\(P_I-P_S\)) and the purity strategy (\(P_I\)).

Full size image

Influence of Enlarged Scale. To evaluate the completeness of object candidates, we need to enlarge the original box with a specific ratio. As shown in Table 4, we examine four ratios (i.e. from 1.1 to 1.4) for obtaining the surrounding context of object candidates, which are then employed to calculate objectness scores with the proposed TS2C. We can observe that all the models trained with the proposed TS2C can outperform the baseline by more than 1.4%. In particular, the best result is achieved by adopting the ratio of 1.2. By continually enlarging the ratio, the performance will be decreased. The reason may be that some training images include multiple instances with the same semantics, and the completeness score of each object candidate will be influenced by adjacent instances in the case of using larger ratios.

Influence of Conditional Averaging Strategy. As shown in Table 4, we also examine the threshold of conditional average strategy. The best result is achieved by employ the first 50% largest pixels to calculate the objectness score of surrounding region.

Fig. 8.
figure 8

Rank 1 object candidates inferred by the proposed TS2C (yellow boxes) and the strategy only using purity metric for ranking (magenta boxes). Some failure cases are given in the last row. Best viewed in color. (Color figure online)

Full size image

Discussion. Some failure cases are shown in the last row of Fig. 8. These samples share some similar characteristics: low-quality segmentation predictions or many semantically identical instances are linked together. For instance (the middle image of the last row), the semantic segmentation branch makes a false prediction for the object under the bird, leading to incorrect inference of our approach. It is believed that such a case can be well addressed with the development of weakly supervised semantic segmentation techniques. For other failure samples, although the segmentation branch can provide high quality confidence maps, the overlap between objects results in false prediction of our TS2C. In this case, we may need to develop effective instance-level semantic segmentation approaches in a weakly supervised manner.

However, the limitation of our TS2C to deal with overlapping objects with the same semantics does not affect its good performance on WSOD. We do not employ the top-1 proposal according to the objectness score as the object candidate, but build a candidate pool by selecting the top two hundred proposals. In this case, these tight boxes may still be recalled even without the largest tightness score. The effectiveness of our TS2C can be well proved by the performance gains on VOC 2007 and 2012 compared with [32].

5 Conclusion and Future Work

In this work, we proposed a simple approach, i.e. TS2C, for mining tight boxes by exploiting surrounding segmentation context. The TS2C is effective for suppressing low quality object candidates and promoting high quality ones tightly covering the target object. Based on the segmentation confidence map, TS2C introduces two simple criteria, i.e. purity and completeness, to evaluate objectness scores of object candidates. Despite apparent simplicity, the proposed TS2C can effectively filter thousands of noisy candidates and be easily embedded into any end-to-end weakly supervised framework for performance improvement. In the future, we plan to design more effective metrics for mining tight boxes by further boosting our current approach.