On-line object detection: a robotics challenge

Abstract

Object detection is a fundamental ability for robots interacting within an environment. While stunningly effective, state-of-the-art deep learning methods require huge amounts of labeled images and hours of training which does not favour such scenarios. This work presents a novel pipeline resulting from integrating (Maiettini et al. in 2017 IEEE-RAS 17th international conference on humanoid robotics (Humanoids), 2017) and (Maiettini et al. in 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), 2018), which naturally trains a robot to detect novel objects in few seconds. Moreover, we report on an extended empirical evaluation of the learning method, justifying that the proposed hybrid architecture is key in leveraging powerful deep representations while maintaining fast training time of large scale Kernel methods. We validate our approach on the Pascal VOC benchmark (Everingham et al. in Int J Comput Vis 88(2): 303–338, 2010), and on a challenging robotic scenario (iCubWorld Transformations (Pasquale et al. in Rob Auton Syst 112:260–281, 2019). We address real world use-cases and show how to tune the method for different speed/accuracy trades-off. Lastly, we discuss limitations and directions for future development.

Appendices

Complete Minibootstrap procedure

This section reports the complete pseudo-code (Alg. 2) for the Minibootstrap procedure described in Sect. 3.3.

The iCubWorld Transformations Dataset

For validating the proposed pipeline in a robotic scenario, we considered the iCubWorld Transformations Dataset (iCWT). This dataset is part of a robotic project, called iCubWorld⁸ which main goal is to benchmark the development of the visual recognition capabilities of the iCub Humanoid Robot (Metta et al. 2010). Datasets from the iCubWorld project are collections of images recording the visual experience of iCub while observing objects in its typical environment, a laboratory or an office. iCWT is the last released dataset and the largest one of the project. We refer to Pasquale et al. (2019) for details about the acquisition setup.

1.1 Dataset description

iCWT contains images for 200 objects instances belonging to 20 different categories (10 instances for each category). Each object instance is acquired in two separate days, in a way that isolates, for each day, different viewpoints transformations: planar 2D rotation (2D ROT), generic rotation (3D ROT), translation with changing background (BKG) and scale (SCALE) and, finally, a sequence that contains all transformations (MIX).

While the dataset has been acquired as benchmark for object recognition, lately we also provided object detection annotations in Imagenet-like format. Moreover we manually annotated a subset of images, that could be used to validate object detection methods trained with automatically collected data, as we did in Maiettini et al. (2017). Specifically, for this work we released a new and larger set of manually annotated images.⁹

Fig. 6

Randomly sampled examples of detections on the PASCAL VOC 2007 test set, obtained using the proposed learning pipeline. CNN backbone for Faster R-CNN for feature extraction is Resnet101, train data is the set of images voc07++12 and the configuration used is FALKON + Minibootstrap$10 \times 2000$ (1 min and 40 s of Train Time and 70.4% of mAP)

For the experiments of this work we consider as train set, for the objects of the considered task, a subset of the union set of 2D ROT, 3D ROT, BKG and SCALE, while as test set we used a subset of 150 images from the first day of acquisition of the MIX sequence for each object, manually annotated adopting the labelImg tool.¹⁰ We fixed an annotating policy such that an object must be annotated if at least a 50-25% of its total shape is visible (i.e. not cut out from the image or occluded).

For defining the FEATURE-TASK presented in 4.3, we consider all the 10 instances of the categories: ’cellphone’, ’mouse’, ’perfume’, ’remote’, ’soapdispenser’, ’sunglasses’, ’glass’, ’hairbrush’, ’ovenglove’, ’squeezer’, while we define the TARGET-TASKs presented in Sect. 4.3 and in Sect. 6 by considering the remaining 10 categories by choosing the instances as following:

• 1 Object Task: obtained by averaging results from considering ’sodabottle2’, ’mug1’, ’sprayer6’, ’hairclip2’

• 10 Objects Task: ’sodabottle2’, ’mug1’, ’pencilcase5’, ’ringbinder4’, ’wallet6’, ’flower7’, ’book6’, ’bodylotion8’, ’hairclip2’, ’sprayer6’

• 20 Objects Task: 10 Objects Task + ’sodabottle3’, ’mug3’, ’pencilcase3’, ’ringbinder5’, ’wallet7’, ’flower5’, ’book4’, ’bodylotion2’, ’hairclip8’, ’sprayer8’

• 30 Objects Task: 20 Objects Task + ’sodabottle4’, ’mug4’ , ’pencilcase6’, ’ringbinder6’, ’wallet10’, ’flower2’, ’book9’, ’bodylotion5’, ’hairclip6’, ’sprayer9’

• 40 Objects Task: 30 Objects Task + ’sodabottle5’, ’mug9’, ’pencilcase1’, ’ringbinder7’, ’wallet2’, ’flower9’, ’book1’, ’bodylotion4’, ’hairclip9’, ’sprayer2’

Examples of detected images

We report in Figs. 6 and 8 examples of detections predicted by the FALKON + Minibootstrap on random sampled images from respectively the test set of Pascal VOC (Everingham et al. 2010) and of iCWT (Pasquale et al. 2019).

Stopping Criterion for Faster R-CNN Fine-tuning

Fig. 7

We show the validation accuracy trend with respect to the number of epochs for the Pascal VOC dataset (blue line) and we highlight (red star) the number of epochs chosen to train the Faster R-CNN baseline, reported in Table 1)

Fig. 8

Randomly sampled examples of detections on iCWT, obtained using the proposed learning pipeline. CNN backbone for Faster R-CNN for feature extraction is Resnet50, FEATURE-TASK and TARGET-TASK are respectively the 100 and 30 objects tasks described in Sect. 4.3 and the configuration used is FALKON + Minibootstrap$10 \times 2000$ (40 s of Train Time and 71.2% of mAP)

Fig. 9

We show the validation accuracy trend with respect to the number of epochs for the iCWT dataset (blue line) and we highlight (red star) the number of epochs chosen to train the Faster R-CNN baselines, reported in Table 2 and Fig. 3

Fig. 10

We show the validation accuracy trend for different configurations of epochs of the 4-Steps alternating training procedure (Ren et al. 2015) for the iCWT dataset (blue line). Each tick of the horizontal axis represents a different configuration. The numbers of epochs used for the RPN and for the Detection Network are reported, respectively after the labels RPN and DN. We highlight (red star) the configuration chosen to train the Faster R-CNN baseline, reported in Table 3

In this section we report on the cross validation carried out to study the convergence of Faster R-CNN and to choose when to stop the learning for the tasks of this work.

In Fig. 7 we report the validation accuracy trend on the Pascal VOC dataset, when learning the last layers of Faster R-CNN for increasing number of epochs. To this aim, within the Pascal VOC, we split the available images considering the union of the validation sets of Pascal 2007 and 2012 as validation set and the union of the training sets of Pascal 2007 and 2012 as training set.

Similarly, in Fig. 9 we report the validation accuracy trend with respect to the number of epochs for the iCWT dataset. In this case, we considered as train set the same $\sim$8k images used for the TARGET-TASK in Sect. 4, while we selected a different set of 4.5k images as validation set, considering the remaining images in the 2D ROT, 3D ROT, SCALE and TRANSL transformations.

Finally, in Fig. 10 we show the validation accuracy trend of the full train of Faster R-CNN (i.e. the optimization of the convolutional layers, RPN, feature extractor and output layers on the TARGET-TASK). Specifically, in this case, since we used the 4-Steps alternating training procedure as in Ren et al. (2015), we report the mAP trend, considering different numbers of epochs, when learning the RPN and the Detection Network. Therefore, the two numbers reported for each tick of the horizontal axis, represent respectively the number of epochs (i) for learning the RPN during steps 1 and 3 of the procedure and (ii) for learning the Detection Network during steps 2 and 4 of the procedure. We consider the same training and validation splitting as in Fig. 9.

Note that, we used these results for our stopping criterion, that consists in choosing the model at the epoch achieving the highest mAP on the validation set (we stopped when no mAP gain was observed in the three plots). We highlighted in red in the three plots, the configurations chosen to train the baselines on the tasks at hand, in Tables 1, 2 and 3 and in Fig. 3.