DeepProposals: Hunting Objects and Actions by Cascading Deep Convolutional Layers

In this paper, a new method for generating object and action proposals in images and videos is proposed. It builds on activations of different convolutional layers of a pretrained CNN, combining the localization accuracy of the early layers with the high informative-ness (and hence recall) of the later layers. To this end, we build an inverse cascade that, going backward from the later to the earlier convolutional layers of the CNN, selects the most promising locations and refines them in a coarse-to-fine manner. The method is efficient, because i) it re-uses the same features extracted for detection, ii) it aggregates features using integral images, and iii) it avoids a dense evaluation of the proposals thanks to the use of the inverse coarse-to-fine cascade. The method is also accurate. We show that our DeepProposals outperform most of the previously proposed object proposal and action proposal approaches and, when plugged into a CNN-based object detector, produce state-of-the-art detection performance.


Introduction
In recent years, the paradigm of generating a reduced set of window candidates to be evaluated with a powerful classifier has become very popular in object detection.Indeed, most of the recent state-of-the-art detection methods (Cinbis et al, 2013;Ren et al, 2015;He et al, 2015;Wang et al, 2013) are based on such propos-als.Furthermore, generating a limited number of proposals also helps weakly supervised object detection, which consists of learning to localize objects without any bounding box annotations (Deselaers et al, 2010;Song et al, 2014).
In fact, detection methods based on proposals can be seen as a two-stage cascade: first, the selection of a reduced set of promising and class-independent hypotheses, the proposals, and second, a class-specific classification of each hypothesis.Similarly to sliding window, this pipeline casts the detection problem to a classification problem.However, in contrast to sliding window, more powerful and time consuming detectors can be employed as the number of candidate windows is reduced.
Methods proposed in the literature for the generation of window candidates are based on two very different approaches.The first approach uses bottom-up cues like image segmentation (Arbelaez et al, 2014;Van de Sande et al, 2011), object edges and contours (Zitnick and Dollár, 2014) for window generation.The second approach is based on top-down cues which learn to separate correct object hypotheses from other possible window locations (Alexe et al, 2010;Cheng et al, 2014).So far, the latter strategy seems to have inferior performance.In this paper we show that, with the proper features, accurate and fast top-down window proposals can be generated.
We consider for this task the convolutional neural network (CNN) "feature maps" extracted from the intermediate layers of Alexnet-like (Krizhevsky et al, 2012) networks.In the first part of this work we present a performance analysis of different CNN layers for generating proposals.More specifically, similarly to BING (Cheng et al, 2014), we select a reduced set of window sizes and aspect ratios and slide them on each possi-Fig.1: DeepProposals pipeline.Our method uses the activation layers of a deep convolutional neural network in a coarse-to-fine inverse cascading to obtain proposals for object detection.Starting from a dense proposal sampling in the last convolutional layer (layer 5) we gradually filter out irrelevant boxes until reaching the initial layers of the net.In the last stage we use contours extracted from layer 2, to refine the proposals.Finally the generated boxes can be used within an object detection pipeline.ble location of the feature map generated by a certain CNN layer.The relevance (or objectness) of the windows is learned using a linear classifier.As the proposal generation procedure should be fast, we base the feature aggregation for each candidate window on average pooling, which can be computed in constant time using integral images (Viola and Jones, 2004).
From this analysis we see that there is not a single best layer for candidate windows generation.Instead we notice that deeper layers, having a more semantic representation, perform very well in recalling the objects with a reduced set of hypotheses.Unfortunately, as noticed also for other tasks (Hariharan et al, 2014), they provide a poor localization of the object due to their coarseness.In contrast, earlier layers are better in accurately localizing the object of interest, but their recall is reduced as they do not represent strong object cues.Thus, we conclude that, for a good window candidate generation, we should leverage multiple layers of the CNN.However, even with the very fast integral images for the feature extraction, evaluating all window locations at all feature layers is too expensive.Instead we propose a method based on a cascade starting from the last convolutional layer (layer 5) and going down with subsequent refinements until the initial layers of the net.As the flow of the cascade is inverse to the flow of the feature computation we call this approach an inverse cascade.Also, as we start from a coarse spatial window resolution, and throughout the layers we select and spatially refine the window hypotheses un-til we obtain a reduced and spatially well localized set of hypotheses, it is a coarse-to-fine inverse cascade.An overview of our approach, which we coined DeepProposals, is illustrated in Fig. 1.
In addition, we go beyond object proposals and extend the DeepProposals framework (first proposed in Ghodrati et al (2015)) to video, generating action proposals.To this end, we first apply the coarse-to-fine inverse cascade on each frame of a video.Then, we group the proposals into tubes, by imposing time continuity constraints, based on the assumption that the object of interest has a limited speed.We show that such proposals can provide excellent results for action localization.
We evaluate the performance of the DeepProposals in terms of recall vs. number of proposals as well as in terms of recall vs. object overlap.We show that in both evaluations the method is better than the current state of the art, and computationally very efficient.However, the biggest gains are achieved when using the method as part of a CNN-based detector like one proposed in Girshick (2015).In this case the approach does not need to compute any feature, because it reuses the same features already computed by the CNN network for detection.Thus, we can execute the full detection pipeline at a very low computational cost.
This paper is an extension of our earlier work (Ghodrati et al, 2015).In this version we have included some additional related work and have compared it against some recent approaches.In addition, we have extended DeepProposals for generating actions propos-als in videos and have compared it with other state-ofthe-art methods on two action datasets.
In the next section, we describe the related work.Next, in section 3, we analyze the quality of different CNN layers for window proposal generation.Section 4 describes our inverse coarse-to-fine cascade and in section 5 its extension to action proposals.In section 6 we compare quantitatively and qualitatively our method with the state-of-the-art, for both object and action proposal generation.Section 7 concludes the paper.

Related work
Object proposal methods Object proposal generators aim at obtaining an accurate object localization with few object window hypotheses.These proposals can help object detection in two ways: searching objects in fewer locations to reduce the detector running time and/or using more sophisticated and expensive models to achieve better performance.
A first set of approaches measures the objectness of densely sampled windows (i.e.how likely is it for an image window to represent an object) (Alexe et al, 2010;Cheng et al, 2014;Zitnick and Dollár, 2014).Alexe et al (2010) propose a measure based on image saliency and other cues like color and edges to discriminate object windows from background.BING (Cheng et al, 2014) is a very fast proposal generator, obtained by training a classifier on edge features, but it suffers from low localization accuracy.Moreover, Zhao et al (2014) has shown that the BING classifier has minimal impact on locating objects and without looking at the actual image a similar performance can be obtained.Edgeboxes (Zitnick and Dollár, 2014) uses structural edges of Dollár and Zitnick (2013), a state-of-the-art contour detector, to compute proposal scores in a sliding window fashion without any parameter learning.For a better localization it uses a final window refinement step.Like these methods, our approach densely samples hypotheses in a sliding window fashion.However, in contrast to them, we use a hierarchy of high-to-low level features extracted from a deep CNN which has proven to be effective for object detection (Girshick et al, 2014;Wang et al, 2013).
An alternative approach to sliding-window methods are the segmentation-based algorithms.This approach extracts from the image multiple levels of bottom-up segmentation and then merges the generated segments in order to generate object proposals (Arbelaez et al, 2014;Carreira and Sminchisescu, 2012;Manen et al, 2013;Van de Sande et al, 2011).The first and most widely used segmentation-based algorithm is selective search (Van de Sande et al, 2011).It hierarchically aggregates multiple segmentations in a bottom-up greedy manner without involving any learning procedure, but based on low level cues, such as color and texture.Multiscale Combinatorial Grouping (MCG) (Arbelaez et al, 2014) extracts multiscale segmentations and merges them by using the edge strength in order to generate object hypotheses.Carreira and Sminchisescu (2012) propose to segment the object of interest based on graph-cut.It produces segments from randomly generated seeds.As in selective search, each segment represents a proposal bounding box.Randomized Prim's (Manen et al, 2013) uses the same segmentation strategy as selective search.However, instead of merging the segments in a greedy manner it learns the probabilities for merging, and uses those to speed up the procedure.Geodesic object proposals (Krähenbühl and Koltun, 2014) are based on classifiers that place seeds for a geodesic distance transform on an over-segmented image.
Recently, following the great success of CNN in different computer vision tasks, CNN-based methods have been used to either generate proposals or directly regress the coordinates of the object bounding box.MultiBox (Erhan et al, 2014) proposes a network which directly regresses the coordinates of all object bounding boxes (without a sliding fashion approach) and assigns a confidence score for each of them in the image.However, MultiBox is not translation invariant and it does not share features between the proposal and detection networks i.e. it dedicates a network just for generating proposals.
DeepMask (Pinheiro et al, 2015) learns segmentation proposals by training a network to predict a classagnostic mask for each image patch and an associated score.Same as MultiBox, they do not share features between the proposal generation and detection.Moreover, they need segmentation annotations to train their network.OverFeat (Sermanet et al, 2013) is a method which does proposal generation and detection in onestage.In OverFeat, region-wise features are extracted from a sliding window and are used to simultaneously determine the location and category of the objects.In contrast to it, our goal is to predict class-agnostic proposals which can be used in a second stage for classspecific detections.
Probably, the most similar to our work is the concurrent work of Region Proposal Network (RPN) proposed in Ren et al (2015).RPN is a convolutional network that simultaneously predicts object bounds and objectness scores at each position.To generate region proposals, they slide a small network over the convolutional feature map.At each sliding-window location, they define k reference boxes (anchors) at different scales and aspect ratios and predict multiple region proposals parameterized relative to the anchors.Similarly to us, RPN builds on the convolutional features of the detection network.However, we leverage low-level features in early layers of the network to improve the localization quality of proposals.In addition, in contrast to them, our method can build on any pre-trained network without the need to re-train it explicitly for the proposal generation task.
Action proposal methods Action proposals are 3D boxes or temporal tubes extracted from videos that can be used for action localization, i.e. predicting the action label in a video and spatially localizing it.Also in this case, the main advantage of using proposals is to reduce the computational cost of the task and therefore make the method faster or allow for the use of more powerful classification approaches.The action proposal methods proposed in the literature to date mainly extend ideas originally developed for 2D object proposals in static images to 3D space.Jain et al ( 2014) is an extension of selective search (Van de Sande et al, 2011) to video.It extract super-voxels instead of super-pixels from a video and by hierarchical grouping it produces spatiotemporal tubes.
Bergh et al ( 2013) is an action proposal method inspired by the objectness method (Alexe et al, 2010), while a spatio-temporal variant of randomized Prime (Manen et al, 2013) is proposed in Oneata et al (2014).Since most of those methods are based on a super-pixel segmentation approach as a pre-processing step, they are computationally very expensive.To avoid such computationally demanding pre-processing, van Gemert et al (2015) proposed Action Localization Proposals (APT) which use the same features used in detection to generate action proposals.
Several action localization methods use 2D proposals in each frame but without generating intermediate action proposals at video-level.Typically, they leverage 2D object proposals that are generated separately for each frame in order to find the most probable path of bounding boxes across time for each action class separately (Gkioxari and Malik, 2015;Tran et al, 2014;Weinzaepfel et al, 2015;Yu and Yuan, 2015).Our method is similar to these works in spirit.However, these methods use class-specific detectors for action localization while we propose a class-agnostic method to generate a reduced set of action proposals.The idea of using classagnostic proposals allows us to filter out many negative tubes with a reduced computational time which enables the use of more powerful classifiers in the final stage.

CNN layers for proposals generation
In this section we analyze the quality of the different layers of a CNN as features for window proposal generation.The window proposals ideally should cover all objects in an image.To evaluate the baselines in this section, we use the PASCAL VOC 2007 dataset (Everingham et al, 2010).

Basic Approach
Sliding window Computing all possible boxes in a feature map of size N × N is in the order of O(N 4 ) and therefore computationally unfeasible.Hence, similarly to Cheng et al (2014), we select a set of window sizes that best cover the training data in terms of size and aspect ratio and use them in a sliding window fashion over the selected CNN layer.This approach is much faster than evaluating all possible windows and avoids to select windows with sizes or aspect ratios different from the training data and therefore probably false positives.
For the selection of the window sizes, we start with a pool of windows W all in different sizes and aspect ratios W all : {ω|ω ∈ Z 2 , Z = [1..20]}.It is important to select a set of window sizes that gives high recall and at the same time produces well localized proposals.To this end, for each window size, we compute its recall with different Intersection over Union (IoU) thresholds and greedily pick one window size at a time that maximizes α recall(IoU > α) over all the objects in the training set.Using this procedure, 50 window sizes are selected for the sliding window procedure.In Fig. 2 (middle) we show the maximum recall that can be obtained with the selected window sizes, which is an upper bound of the achievable recall of our method.
Multiple scales Even though it is possible to cover all possible objects using a sliding window on a single scale of feature map, it is inefficient since by using a single scale the stride is fixed and defined by the feature map resolution.For an efficient sliding window, the window stride should be proportional to the window size.Therefore, in all the experiments we evaluate our set of windows on multiple scales.For each scale, we resize the image such that min(w, h) = s where s ∈ {227, 300, 400, 600}.Note that the first scale is the network original input size.
Pooling As the approach should be very fast we represent a window by the average pooling of the convolutional features that are inside the window.As averaging is a linear operation, after computing the integral image, the features of any proposal window can be extracted in a constant time.Let f (x, y) be the specific channel of the feature map from a certain CNN layer and F (x, y) its integral image.Then, average pooling avg of a box defined by the top left corner a = (a x , a y ) and the bottom right corner b = (b x , b y ) is obtained as: Thus, after computing the integral image, the average pooling of any box is obtained in a constant time that corresponds to summing 4 integral values and dividing by the area of the box.
Pyramid One of the main cues used to detect general objects is the object boundaries.Using an approach based on average pooling can dilute the importance of the object boundaries because it discards any geometrical information among features.Therefore, to introduce more geometry to the description of a window we consider a spatial pyramid representation (Lazebnik et al, 2006).It consists of dividing the proposal window into a number of same size sub-windows (e.g. 2 × 2), and for each one build a different representation.
Bias on size and aspect ratio Objects tend to appear at specific sizes and aspect ratios.Therefore we add in the feature representation 3 additional dimensions (w, h, w × h) where w and h are the width and height of a window.This can be considered as an explicit kernel which lets the SVM learn which object sizes can be covered in a specific scale.For the final descriptor, we normalize the pooled features and size-related features separately with l 2 norm.
Classifier We train a linear classifier for each scale separately.For a specific scale, we randomly select at most 10 regions per object that overlap the annotation bounding boxes more than 70%, as positive training data and 50 regions per image that overlap less than 30% with ground-truth objects as negative data.In all experiments we use a linear SVM (Fan et al, 2008) because of its simplicity and fast training.We did not test non-linear classifiers since they would be too slow for our approach.
Non-maximum suppression The ranked window proposals in each scale are finally reduced through a nonmaximum suppression step.A window is removed if its IoU with a higher scored window is more than a threshold α, which defines the trade-off between recall and accurate localization.So, this threshold is directly related to the IoU criteria that is used for evaluation (see sec 3.2).By tuning α, it is possible to maximize recall at arbitrary IoU of β.Particularly, in this work we define two variants of our DeepProposals, namely DeepPro-posals50 and DeepProposals70 for maximizing recall at IoU of β = 0.5 and β = 0.7 respectively by fixing α to β + 0.05 (like Zitnick and Dollár ( 2014)).In addition, to aggregate boxes from different scales, we use another non-maximum suppression, fixing α = β.

Evaluation for object proposals
To evaluate the quality of our proposals we use the PASCAL VOC 2007 dataset (Everingham et al, 2010).PASCAL VOC 2007 includes 9,963 images with 20 object categories.4,952 images are used for testing, while the remaining ones are used for training.We use two different evaluation metrics; the first is Detection Rate (or Recall) vs. Number of proposals.This measure indicates how many objects can be recalled for a certain number of proposals.We use Intersection over Union (IoU) as evaluation criterion for measuring the quality of an object proposal ω.IoU is defined as | ω∩b ω∪b | where b is the ground truth object bounding box.Initially, an object was considered correctly recalled if at least one generated window had an IoU of 0.5 with it, the same overlap used for evaluating the detection performance of a method.Unfortunately this measure is too lose because a detector, for working properly, needs also good alignment with the object (Hosang et al, 2015).Thus we evaluate our method for an overlap of 0.7 as well.We also evaluate recall vs. overlap for a fixed number of proposals.As shown in Hosang et al (2015), the average recall obtained from this curve seems highly correlated with the performance of an object detector built on top of these proposals.
In this section, we investigate the effect of different parameters of our method, namely the different convolutional layers, the number of used window sizes and different levels of spatial pyramid pooling.
Layers We evaluate each convolutional layer (from 1 to 5) of Alexnet (Krizhevsky et al, 2012) using the sliding window settings explained above.For sake of simplicity, we do not add spatial pyramids on top of pooled features in this set of experiments.As shown in Fig. 2 (left) the top convolutional layers of the CNN perform better than the bottom ones.Also their computational cost is lower as their representation is coarser.Note this simple approach already performs on par or even better than the best proposal generator approaches from the literature.For instance, our approach at layer 3 for 100 proposals achieves a recall of 52%, whereas selective search (Van de Sande et al, 2011) obtains only   40%.This makes sense because the CNN features are specific for object classification and therefore can easily localize the object of interest.
However, this is only one side of the coin.If we compare the performance of the CNN layers for high overlap (see Fig. 2 (middle)), we see that segmentation-based methods are much better (Van de Sande et al, 2011; Arbelaez et al, 2014).For instance the recall of selective search for 1000 proposals at 0.8 overlap is around 55% whereas ours at layer 3 is only 38%.This is due to the coarseness of the CNN feature maps that do not allow a precise bounding box alignment to the object.In contrast, lower levels of the net have a much finer resolution that can help to align better, but their encoding is not powerful enough to properly localize objects.In Fig. 2 (middle) we also show the maximum recall for different overlap that a certain layer can attain with our selected sliding windows.In this case, the first layers of the net can recall many more objects with high overlap.
This shows that a problem of the higher layers of the CNN is the lack of a good spatial resolution.
In this sense we could try to change the structure of the net in a way that the top layers still have high spatial resolution.However, this would be computationally expensive and, more importantly, it would not allow to reuse the same features used for detection.Instead, in the next section we propose an efficient way to leverage the expressiveness of the top layers of the net together with the better spatial resolution of the bottom layers.

Number of Window Sizes
In Fig. 2 (right) we present the effect of a varying number of window sizes in the sliding window procedure for proposal generation.The windows are selected based on the greedy algorithm explained in Sec 3.1.As the number of used window sizes increases, we obtain a better recall at a price of a higher cost.In the following experiments we will fix the number of windows to 50 because that is a good trade-off between speed and top performance.The values in the figure refer to layer 5, however, similar behavior has been observed for the other layers as well.

Spatial Pyramid
We evaluate the effect of using a spatial pyramid pooling in Fig. 3 (left).As expected, adding geometry improves the quality of the proposals.Moving from a pure average pooling representation (sp level=0) to a 2 × 2 pyramid (sp level=1) gives a gain that varies between 2 and 4 percent in terms of recall, depending on the number of proposals.Moving from the 2 × 2 pyramid to the 4 × 4 (sp level=2) gives a slightly lower gain.At 4 × 4 the gain does not saturate yet.However, as we aim at a fast approach, we also need to consider the computational cost, which is linear in the number of spatial bins used.Thus, the representation of a window with a 2 × 2 spatial pyramid is 5 times slower than a flat representation and the 4 × 4 pyramid is 21 times slower.For this reason, in our final representation we limit the use of the spatial pyramid to 2 × 2 spatial pyramid.

Coarse-to-fine Inverse Cascade
Even if the features used for our window proposals come without any additional computational cost (because they are needed for the detector), still a dense evaluation in a sliding window fashion over the different layers would be too expensive.Instead here we leverage the structure of the CNN layers to obtain a method that combines in an efficient way the high recall of the top convolutional layers of a CNN, with the fine localization provided at the bottom layers of the net.In Table 1 we summarize the characteristics of each CNN layer.
We start the search with the top convolutional layers of the net, that have features well adapted to recognize high-level concepts like objects and actions, but are coarse, and then move to the bottom layers, that use simpler features but have a much finer spatial representation of the image (see Fig. 1).As we go from a coarse to a fine representation of the image and we follow a flow that is exactly the opposite of how those features are computed we call this approach coarse-to-fine inverse cascade.We found that a cascade with 3 layers is an optimal trade-off between complexity of the method and gain obtained from the cascading strategy.
Stage 1: Dense Sliding Window on Layer 5 The first stage of the cascade uses layer 5.As the feature representation is coarse, we can afford a dense sliding window approach with 50 different window sizes collected as explained in Sec.3.1.Even though a pyramid representation could further boost the performance, we do not use spatial binning at this stage to not increase the computational cost.We linearly re-scale the window scores to [0, 1] such that the lowest and highest scores are mapped to 0 and 1 respectively.Afterwards we select the best N 1 = 4000 windows obtained from a non-maximum suppression algorithm with threshold β + 0.05 in order to propagate them to the next stage.
Stage 2: Re-scoring Selected Windows on Layer 3 In this stage, as we use a reduced set of windows, we can afford to spend more computation time per window.Therefore we add more geometry in the representation by encoding each window with a pyramid representation composed of two levels: 1×1 and 2×2.The proposal scores from this layer are again mapped to [0, 1].The final score for each proposal is obtained by multiplying the scores of both stages.Afterwards we apply a nonmaximum suppression with overlap threshold β + 0.05 and select the 3000 best candidates.At the end of this stage, we aggregate the boxes from different scales using non-maximum suppression with threshold β and select the N desired = 1000 best for refinement.

Stage 3: Local Refinement on Layer 2
The main objective of this final stage is to refine the localization obtained from the previous stage of the cascade.For this stage the best candidate is layer 2 because it has a higher resolution than upper layers and contains lowlevel information which is suitable for the refinement task.Specifically, we refine the N desired windows received from the previous stage using the procedure explained in Zitnick and Dollár (2014).To this end, we train a structured random forest (Dollár and Zitnick, 2013) on the second layer of the convolutional features to estimate contours similarly to DeepContour (Xinggang et al, 2015).After computing the edgemap, a greedy iterative search tries to maximize the score of a proposal over different locations and aspect ratios using the scoring function used in Zitnick and Dollár (2014).It is worth mentioning that since our contour detector is based on the CNN-features, we again do not need to extract any extra features for this step.

Evaluation for object proposals
We discuss the performance of the inverse cascade stage by stage in terms of both computational cost and performance.A summary of the computational cost of each stage is given in Table 2.The entire cascade has a computational cost of 0.75 on a 8-core CPU of 3.50GHz, which is the composition of 0.3 , 0.25 and 0.2 for the first, second and third stage respectively.Note the first stage is very fast because even if we use a dense sliding window approach, with the integral image and without any pyramid level the cost of evaluating each window is very low.
As shown in Fig. 3 (middle and right), the second stage is complementary to the first and employed with a 2 × 2 pyramid improves the recall of the cascade by 5%.However, this boost is valid only up to an overlap of 0.75.After this point the contribution of the second stage is negligible.This is due to the coarse resolution of layer 5 and 3 that do not allow a precise overlap of the candidate windows with the ground truth object bounding boxes.We found that, for our task, layer 3 and 4 have a very similar performance (Recall@1000 is 79% in both cases) and adding the latter in the pipeline did not help in improving performance (Recall@1000 is still 79%).
As shown in Hosang et al (2015), for a good detection performance, not only the recall is important, but also a good alignment of the candidates is needed.At stage 3 we improve the alignment without performing any further selection of windows; instead we refine the proposals generated by the previous stages by aligning them to the edges of the object.In our experiments for contour detection we observed that the first layer of the CNN did not provide as good performance as layer 2 (0.61 vs. 0.72 AP on BSDS dataset (Arbelaez et al, 2011)), so we choose the second layer of the network for this task.Fig. 3 (middle) shows that this indeed improves the recall for high IoU values (above 0.7).

Proposals in Videos
Given a video sequence of length T , the goal is to generate a set of action proposals (tubes).Each proposal P = {R 1 , ..., R t , ..., R T } corresponds to a path from the box R 1 in the first frame to the box R T in the last frame, and it spatially localizes the action.
In case of object proposals, applying the inverse cascade to an image already gives the desired output.When the goal is to find proposals in videos, we need a) to capture the motion information that a video naturally provides us with, and b) to satisfy time continuity constraints over time.
One advantage of DeepProposals is that it can be setup on top of any fine-to-coarse convolutional network regardless of its input/output (and possibly its architecture).To benefit from both appearance and motion cues in a given video, we use two networks for the task of action proposal generation.The first network takes as input the RGB frames of a video, and is based on an Alexnet-like architecture, fine-tuned on VOC2007 for the task of object detection.The second network takes as input the optical flow of each frame extracted from the video.We use the motion-CNN network of Gkioxari and Malik (2015) trained on UCF101 (split1) (Soomro et al, 2012).The architecture of this network is identical to the first network.
To generate a set of proposals in each frame, in the first and second stage of DeepProposals, we use an early fusion strategy, concatenating the feature maps generated by both networks and treating them as a single set of feature maps.For the last stage, since it is an alignment process, we only use the feature map of the appearance network.
So far the output is a set of proposals in each frame.In order to make the proposals temporally coherent, we follow the procedure of Gkioxari and Malik (2015) and link the proposals of each single frame over time into tubes.We define the linking scoring function between every two consecutive boxes R t and R t+1 as follows: where C(.) is the confidence score for a box and O(.) is the intersection over union value if the overlap of the two boxes is more than 0.5, otherwise it is −Inf .Intuitively, the scoring function gives a high score if the two boxes R t and R t+1 overlap significantly and if each of them most likely contains an object of an action.
Finally, we are interested in finding the optimal path over all frames.To this end, we first compute the overall score for each path P by ). Computing score for all the possible paths can be done efficiently using the Viterbi algorithm.The optimal path  P opt is then the one with highest score.After finding the best path, all boxes in P opt are removed and we solve the optimization again in order to find second best path.This procedure is repeated until finding the last feasible path (those paths whose scores are higher than −Inf ).We consider each of these paths as an action proposal.

Evaluation for action proposals
We evaluate the performance of the inverse cascade for action proposals on the UCF-Sports (Rodriguez et al, 2008) dataset.We train our models on a training set that contains a total of 6, 604 frames.Additionally, we have 2, 976 frames in the test set, spread over 47 videos.Like van Gemert et al ( 2015), we measure the overlap between an action proposal and the ground-truth video tubes using the average intersection-over-union score of 2D boxes for all frames where there is either a ground-truth box or proposal box.Formally: where P and G are action proposal and ground-truth action tube respectively.F is the set of frames where either P or G is not empty.G t is empty if there is no action in the frame t of the ground-truth tube.In this case, P t ∩ G t is set to 0 for that frame.
Considering each frame as an image and applying DeepProposals on each frame individually, the framebased recall of objects/actors for an IoU of 0.7 is 78% for 10 windows and 96% for 100 windows.One possible explanation for such a promising frame-based recall is that an action mainly contains an actor performing it and hunting that actor in each frame is relatively easier than hunting general objects in the task of object proposal generation.However, this does not take into account any temporal continuity.Constructing tubes from these static windows, which results in our action proposals, is our final goal.
The extension of the inverse cascade for actions introduces an additional parameter which is the number of windows that we select in each frame.In figure 4 (left) we show the recall of the action proposals while varying the number of windows we select per frame.As expected, selecting more windows in each frame leads to a higher recall of the action proposals.However, it also leads to an increasing computational cost, since the computational complexity of the Viterbi algorithm is proportional to the square of the number of windows per frame.For example, the Viterbi algorithm for a video of length 240 frames takes 1.3 and 12.1 seconds for N = 100 and N = 300 respectively.From now on, during all the following experiments we select N = 100 windows per frame to have a good balance between performance and time complexity.
Figure 4 (middle) shows the action proposals recall for different number of proposals.As it is shown even for a very small number of proposals, DeepProposals obtains very good performance as already observed also for object proposals.Figure 4 (right) shows the recall of our method for 20 action proposals (tubes) per video over different IoU values.Our method works very well in the regime of [0.3..0.6].Notice that the definition of action proposals recall is different than object proposals recall and the performance in IoU=0.5 is already quite promising.

Experiments
In this section we compare the quality of the proposed DeepProposals with state-of-the-art object (section 6.1) and action (section 6.2) proposals.

Object Proposals
To evaluate our proposals, like previous works on object proposal generation, we focus on the well-known PASCAL VOC 2007 dataset.PASCAL VOC 2007 (Everingham et al, 2010) includes 9, 963 images divided in 20 object categories.4, 952 images are used for testing, while the remaining ones are used for training.We compare the quality of our DeepProposals in terms of recall and localization accuracy with multiple state-ofthe-art methods.Detection results and run-time are reported for PASCAL VOC 2007(Everingham et al, 2010), integrating DeepProposals in the Fast-RCNN framework (Girshick, 2015).Then, we evaluate the generalization performance of DeepProposals on unseen categories and finally some qualitative comparisons are presented.Comparison with state-of-the-art We compare our DeepProposals against well-known, state-of-the-art object proposal methods.Fig. 5 and Fig. 7 show the recall with a varying number of object proposals or IoU threshold respectively.From Fig. 5, we can see that, even with a small number of windows, DeepProposals can achieve higher recall for any IoU threshold.Methods like BING (Cheng et al, 2014) and objectness Alexe et al (2010) are providing high recall only at IoU = 0.5 because they are tuned for IoU of 0.5.
In Table 3 we summarize the quality of the proposals generated by the most promising methods.Achieving 75% recall with IoU of 0.7 would be possible with 540 proposals of DeepProposals, 800 of Edge boxes, 922 of RPN-ZF, 1400 of selective search proposals and 3000 of Randomized Prim's proposals (Manen et al, 2013).
Figure 7 left and middle show the curves related to recall over IoU with 100 and 1000 proposals.Again, DeepProposals obtain good results.The hand crafted segmentation based methods like selective search and MCG have good recall rate at higher IoU values.Instead DeepProposals perform better in the range of IoU = [0.6,0.8] which is desirable in practice and playing an important role in the object detectors performance (Hosang et al, 2015).
Figure 7 (right) shows average recall(AR) versus number of proposals for different methods.For a specific number of proposals, AR measures the proposal quality across IoU of [0.5, 1].Hosang et al (2015) show that AR correlates well with detection performance.Using this criteria, DeepProposals are on par or better than other methods with 700 or fewer boxes but with more boxes, selective search and Edgeboxes perform better.

Run-time
The run-time tests for our proposed method and the others are also available in Table 3.Since our approach is using the CNN features which are used by state-of-the-art object detectors like RCNN (Girshick et al, 2014) and SppNet (He et al, 2015), it does not need any extra cues and features and we can consider just the running time of our algorithm without CNN extraction time1 .DeepProposals takes 0.75 second on CPU and 0.4 second to generate object proposals on a GeForce GTX 750 Ti GPU, which is slightly slower than Edgeboxes.The fastest method is RPN-ZF, a convolutional network based on Zeiler and Fergus (2014) network architecture, tuned for generating object proposals.Note that for RPN-ZF, the running-time on GPU is reported while the others are reported on CPU.The remaining methods are segmentation based and take considerably more time.
Object Detection Perfomrance In the previous experiments we evaluate our proposal generator with different metrics and show that it is among the best methods in all of them.However, we believe that the best way to evaluate the usefulness of the generated proposals is a direct evaluation of the detector performance.Indeed, recently it has become clear (see Hosang et al (2015)) that an object proposal method with high recall at 0.5 IoU does not automatically lead to a good detector.
Some state-of-the-art detectors at the moment are: RCNN (Girshick et al, 2014), SppNet (He et al, 2015), fast-RCNN (Girshick, 2015).All are based on CNN features and use object proposals for localizing the object of interest.The first uses the window proposals to crop the corresponding regions of the image, compute the CNN features and obtain a classification score for each region.This approach is slow and takes around 10 sec on a high-end GPU and more than 50 sec on the GPU used for our experiments (GeForce GTX 750 Ti).Spp-Net and fast-RCNN instead compute the CNN features  AUC N@25% N@50% N@75% Recall Time BING (Cheng et al, 2014) .19292 --29% .2sObjectness (Alexe et al, 2010) .2627 --39% 3s Rand.Prim's (Manen et al, 2013) .3042 349 3023 71% 1s Selective Search (Van de Sande et al, 2011) .3428 199 1434 79% 10s Edge boxes 70 (Zitnick and Dollár, 2014) .42 12 108 800 84% .3sMCG (Arbelaez et al, 2014) .42 9 81 1363 78% 30s RPN-ZF (Ren et al, 2015) .Table 3: Our method compared to other methods for IoU threshold of 0.7.AUC is the area under recall vs. IoU curve for 1000 proposals.N@25%, N@50%, N@75% are the number of proposals needed to achieve a recall of 25%, 50% and 75% respectively.For reporting Recall, at most 2000 boxes are used.The run-times for the other methods are obtained from Hosang et al (2015).* In contrast to the other methods, for RPM-ZF the run-time is evaluated on GPU.
only once, on the entire image.Then, the proposals are used to select the sub-regions of the feature maps from where to pull the features.This allows this approach to be much faster.With these approaches then, we can also reuse the CNN features needed for the generation of the proposal so that the complete detection pipeline can be executed without any pre-computed component roughly in 1 second on our GPU.
Concurrently to our method also Faster-RCNN was recently introduced (Ren et al, 2015).It uses a Region Proposal Network (RPN) for generating proposals that shares full-image convolutional features with the detection network.
We compare the detection performance of our Deep-Proposals70 with selective search and RPN proposals.For RPN proposals the detector is trained as in the orig-inal paper (Ren et al, 2015) with an alternating procedure, where detector and localization sub-network update the shared parameters alternatively.Our method and selective search are instead evaluated using a detector fine-tuned with the corresponding proposals, but without any alternating procedure, i.e. the boxes remain the same for the entire training.The training is conducted using faster-RCNN code on PASCAL VOC 2007 with 2000 proposals per image.In Fig. 6 we report the detector mean average precision on the PAS-CAL VOC 2007 test data for different number of used proposals.
The difference of selective search with CNN-based approaches is quite significant and it appears mostly in a regime with low number of proposals.For instance, when using 50 proposals selective search obtains a mean average precision (mAP) of 28.1, while RPN and our proposals obtains a mAP already superior to 50.We believe that this different behavior is due to the fact that our method and RPN are supervised to select good object candidates, whereas selective search is not.
Comparing our proposals with RPN, we observe a similar trend.DeepProposals produces superior results with a reduced amount of proposals (< 100), while RPN performs better in the range of between 100 and 700 proposals.With more than 700 proposals both methods perform again similarly and better than selective search.Finally, with 2000 proposals per image, selective search, RPN and DeepProposals reach to the detection performance of 59.3, 59.4 and 59.8 respectively.
Thus, from these results we can see that RPN and our approach perform very similar.The main difference between the two approaches lies in the way they are trained.Our approach assumes an already pre-trained network, and learns to localize the object of interest by leveraging the convolutional activations generated for detection.RPN instead needs to be trained together with the detector with an alternating approach.In this sense, our approach is more flexible because it can be applied to any CNN based detector without modifying its training procedure.

Generalization to Unseen Categories
We evaluate the generalization capability of our approach on the Microsoft COCO dataset (Lin et al, 2014).The evaluation of the approach has been done by learning either from the 20 classes from VOC07 or COCO or from 1, 5, 20, 40, or 80 categories randomly sampled from COCO.As shown in figure 8, when the DeepProposals are trained using only 5 classes, the recall at 0.5 IoU with 1000 proposals is slightly reduced (56%).With more classes, either using VOC07 or COCO, recall remains stable around 59% -60%.This shows that the method can generalize well over all classes.We believe this is due to the simplicity of the classifier (average pooling on CNN features) that avoids over-fitting to specific classes.Note that in this case our recall is slightly lower than the Selective Search with 1000 proposals (63%).This is probably due to the presence of very small objects in the COCO dataset, that are missed by our method as it was not tuned for this setting.These results on COCO demonstrate that our proposed method is capable to generalize learnt objectness beyond the training categories.

Qualitative Results
Figure 9 shows some qualitative results of DeepProposals and another state of the art method, Edge boxes.In general, when the image contains high-level concepts cluttered with many edges (e.g. Figure 9 rows 1 and 3, first column) our method gives better results.However, for small objects with clear boundaries edge boxes performs better since it is completely based on contours and can easily detect smaller objects.

Action Proposals
We evaluate our action proposals on two different datasets namely UCF-Sports (Rodriguez et al, 2008) and UCF101 (Soomro et al, 2012).UCF-Sports contains 10 action categories and consists of 150 video samples, extracted from sport broadcasts.The actions in this dataset are temporally trimmed.For this dataset we use the train and test split proposed in Lan et al (2011).UCF101 is collected from YouTube and has 101 action categories where 24 of the annotated classes (corresponding to 3, 204 videos) are used in literature for action localization.In this dataset, for evaluation we report the  2015).For UCF101, like ours, we report the APT performance for split3.
average recall of 3 splits.Finally, for both datasets, we select the first top 100 boxes in each frame and find the N best paths over time for each video.
Comparison with the state-of-the-art In Table 4 we compare our proposal generation method against state-of-the-art methods in the presented datasets.As shown, our method is competitive or improves over all other methods with fewer proposals.In the UCF-Sports dataset, DeepProposals have higher recall compared to the recently published APT proposal generator (van Gemert et al, 2015) with almost 70x fewer proposals.
Notice that the method proposed by Brox and Malik (2010) is designed for motion segmentation and we use it here to emphasize the difficulty of generating good video proposals.In the UCF101 dataset we see the same trend, we outperform APT while using 67x fewer proposals.
Run-time Computationally, given the optical flow images, our method needs 1.2 seconds per-frame to generate object proposals and on average 1.3 seconds for linking all the windows.Most of the other methods are generally order of magnitude more expensive mainly because of performing super-pixel segmentation and grouping.

Qualitative Results
In figure 10 we provide examples of our action proposals extracted from some videos of UCF-sports dataset.For each video we show 5 cross sections of a tube.These sections are equally distributed in the video.

Conclusion
DeepProposalsis a new method to produce fast proposals for object detection and action localization.In this paper, we have presented how DeepProposals produces proposals at low computational cost through the use of an efficient coarse to fine cascade on multiple layers of a detection network, reusing the features already computed for detection.We have accurately evaluated the method in the most recent benchmarks and against previous approaches and we have shown that in most of the cases it is comparable or better than state-of-the-art approaches in terms of both accuracy and computation.
The source code of DeepProposals is available online2 .
Fig. 2: (Left) Recall versus number of proposals for IoU=0.7.(Middle) recall versus overlap for 1000 proposals for different layers.(Right) Recall versus number of proposals at IoU=0.7 on layer 5 for different number of window sizes.All are reported on the PASCAL VOC 2007 test set.

Fig. 3 :
Fig. 3: (Left) Recall versus number of proposals in IoU=0.7 for different spatial pyramid levels (Middle) Recall versus IoU for 1000 proposals for different stages of the cascade.(Right) Recall versus number of proposals in IoU=0.7 for the different stages of the cascade.All are reported on the PASCAL VOC 2007 test set.

Fig. 4 :
Fig. 4: (left) Action proposals recall at IoU of 0.5 varying the number of windows per frame.(middle) Recall versus number of proposals in IoU=0.5 for the different number of action proposals.(right) Action proposal recall at different IoU values for 20 action proposals.All results are reported on the UCF-sports test set.

Fig. 5 :
Fig. 5: Recall versus number of proposals on the PASCAL VOC 2007 test set for (left) IoU threshold 0.5 and (right)IoU threshold 0.7.

Fig. 8 :
Fig.8: Generalization of DeepProposals: we train models with different number of categories and evaluate it on whole eval-set of COCO dataset.We set IoU threshold to 0.5 and number of proposals to 1000.

Fig. 9 :
Fig.9: Qualitative examples of our object proposals (1st and 3rd column) versus Edge boxes proposals (2nd and 4th column).For first three rows our method performs better while in the last row Edge boxes is better.An object is correctly localized if its IoU with the ground-truth bounding box is more than 0.7.We use 1000 proposals for each method.Blue boxes are the closest proposal to each ground truth bounding box.Red and green boxes are ground-truth boxes where green indicates a localized object while red indicates a missed object.

Table 1 :
Characteristics and performance of the CNN layers.Feature map size is reported for an image of size 600 × 860.Recall(#1000,β) is the recall of 1000 proposals for the overlap threshold β.Max(β) is the maximum recall for the overlap threshold β using our selected window sizes set.

Table 2 :
Characteristics of the stages of our inverse cascade (NMS: non maximum suppression).

Table 4 :
Our action proposals generator compared to other methods at IoU threshold of 0.5.The number of proposals is averaged over all test videos.All the reported numbers on UCF-sports except ours are obtained from van Gemert et al (