Drawing the Same Bounding Box Twice? Coping Noisy Annotations in Object Detection with Repeated Labels

Abstract

The reliability of supervised machine learning systems depends on the accuracy and availability of ground truth labels. However, the process of human annotation, being prone to error, introduces the potential for noisy labels, which can impede the practicality of these systems. While training with noisy labels is a significant consideration, the reliability of test data is also crucial to ascertain the dependability of the results. A common approach to addressing this issue is repeated labeling, where multiple annotators label the same example, and their labels are combined to provide a better estimate of the true label. In this paper, we propose a novel localization algorithm that adapts well-established ground truth estimation methods for object detection and instance segmentation tasks. The key innovation of our method lies in its ability to transform combined localization and classification tasks into classification-only problems, thus enabling the application of techniques such as Expectation-Maximization (EM) or Majority Voting (MJV). Although our main focus is the aggregation of unique ground truth for test data, our algorithm also shows superior performance during training on the TexBiG dataset, surpassing both noisy label training and label aggregation using Weighted Boxes Fusion (WBF). Our experiments indicate that the benefits of repeated labels emerge under specific dataset and annotation configurations. The key factors appear to be (1) dataset complexity, the (2) annotator consistency, and (3) the given annotation budget constraints.

Appendices

Appendix 1

This section presents our adaptation of the weighted box fusion (WBF) technique, tailored specifically for instance segmentation as a weighted mask fusion (WMF).

In their study, [18] propose a method for combining annotations from multiple annotators using a weighted box fusion [32] approach. In this method, bounding boxes are matched greedily only with boxes of the same class, and no annotations are discarded. The WBF algorithm fuses boxes that exceed a specified overlap threshold, resulting in new boxes that represent the weighted average of the original boxes. The approach also allows for inclusion of box confidence scores and prior weights for each annotator.

To extend the WBF method for instance segmentation, we introduce an option to fuse segmentation masks, which involves four steps: (1) calculating the weighted area and weighted center points from the different masks, (2) compute the average center point and average area from the selected masks, (3) determining the closest center point of the original masks to the weighted center point and selecting this mask, and (4) dilating or eroding the chosen mask until the area is close to the averaged area. The resulting mask is used as the aggregated segmentation mask and is also used as the averaging operation during the aggregation for LAEM and MJV with uniform weight.

Moreover, we integrate the WBF approach with LAEM, yielding WBF+EM. This integration involves assessing annotator confidence using LAEM, and subsequently incorporating it into the WBF method to produce weighted average areas instead of simply averaged areas. While the differences between LAEM and WBF might seem subtle, WBF+EM offers a more thorough approach to annotator fusion. This modification is relatively minor, and its impact is modest, as corroborated by our experiments delineated in Appendix 2.

Appendix 2

Table 5. Cross-Validation of ground truth inference combinations between training and test data, for the DetectoRS with a ResNet-50 backbone on the TexBiG dataset. Showing the mAP@[.5 : .95] for instance masks and bounding boxes. Union is represented by $\cup$, intersection by $\cap$ and averaging by $\mu$. RL denotes training conducted on un-aggregated noisy labels. The two rows on the bottom show how the training methods perform on average.

In this experiment, we carried out a comparative analysis of different ground truth inference methods. To do this, we separated the annotations for training and testing, and created various combinations of train-test datasets using the available ground truth estimation methods. Afterward, a model was trained on these combinations. The results from this experiment reveal how aggregation methods can impact the performance of the trained models and show how these outcomes can vary based on the specific combination of training and testing aggregation used.

Tables 5 and 6 present the application of various ground truth estimation methods on repeated labels. In the TexBiG dataset, each method is employed to aggregate the labels of both training and test data, and all possible train-test combinations are learned and tested to perform a cross comparison of the different ground truth inference methods, as shown in Table 5. The hyperparameter for the area combination is denoted as $\cup$ for union, $\mu$ for averaging and $\cap$ for intersection. Additionally, the plain repeated labels, without any aggregation, are compared with the different aggregated test data. Our findings reveal that on a high-agreement dataset, weighted boxes fusion does not perform well. This could be attributed to the inclusion of most annotations by WBF, whereas in cases with high agreement, it is more desirable to exclude non-conforming instances. Majority voting and localization-aware expectation maximization perform similarly; however, LAEM provides a more elegant solution for addressing edge cases. Calculating the annotator confidence, as performed in LAEM, is highly advantageous. However, in rare cases, spammer annotators could potentially circumvent annotation confidence by annotating large portions of simple examples correctly but failing at hard cases. Such cases would result in a high confidence level for the spammer, potentially outvoting the correct annotators on challenging and crucial cases.

Table 6. Comparing results with the private Kaggle leaderboard [22] for the VinDr-CXR dataset using the double headed R-CNN at $mAP_{40}$. Union is represented by $\cup$, intersection by $\cap$ and averaging by $\mu$. RL denotes training conducted on un-aggregated noisy labels.

The main performance differences between MJV and LAEM arise due to the application of the three different combination operations – union, averaging, and intersection. Combining areas by taking their union results in larger areas, making it easier for a classifier to identify the respective regions. Analysis of the mean results of the training methods reveals that both MJV+$\cup$ and LAEM+$\cup$ exhibit the highest performance across various test configurations. On the contrary, methods parameterized with intersection $\cap$ yield the lowest mean results. Training with repeated labels without any aggregation yields results similar to training with aggregated labels. However, while it may be generally feasible to train with noisy labels, the performance is slightly dampened. Since the test data aggregation method is LAEM-$\mu$ as described in Sect. 3.2, the best performing training method LAEM-$\cup$ is chosen as the aggregation method for the training data in the experiments shown in Sect. 4.2 and 4.3.

For the VinDr-CXR dataset, a smaller, similar experiment is performed as shown in Table 6. As the Kaggle leaderboard already provides an aggregated ground truth and labels are unavailable, only the training data are aggregated. Our findings indicate that training with plain repeated labels leads to higher results. Given the low agreement of the dataset, training with repeated labels may be seen as a form of “label augmentation.” Interestingly, the methods used to aggregate the test data, such as WBF, do not outperform the other methods. However, ground truth estimation methods are not designed to boost performance but rather to provide a suitable estimation for the targeted outcome. Based on these results, the following experiments on VinDr-CXR will be run with the repeated labels for training.

Fig. 3.

Qualitative results on three test images from the VinDr-CXR. Left: the original image with the repeated labels indicated by the different line types. Right: the four smaller images from top left to bottom right are, MJV+$\cap$, LAEM+$\mu$, LAEM+$\cup$ and WBF.

Appendix 3

This section shows three more comparisons between different ground truth aggregation methods, exemplary on the VinDr-CXR dataset [23]. All of them follow the same structure. Left: the original image with the repeated labels indicated by the different line types. Right: the four smaller images from top left to bottom right are, MJV+$\cap$, LAEM+$\mu$, LAEM+$\cup$ and WBF (Fig. 3).