Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Detection

In data association based MOT, the tracking performance is heavily affected by the detection results. We implement our detector based on Faster R-CNN [14]. In our implementation, the CNN model is fine-tuned from the VGG-16 on ImageNet. The additional training data includes ETHZ pedestrian dataset [4], Caltech pedestrian dataset [2] and the self-collected surveillance dataset (365653 boxes in 47556 frames). We adopt the multi-scale training strategy by randomly sampling a pyramid scale for each time. However, we only use a single scale and a single model during test. Moreover, we also use skip pooling [1] and multi-region [5] strategies to combine features at different scales and levels.

In considering the definition of MOTA in MOT16 [12], the sum of false negatives (FN) and false positives (FP) poses a large impact on the value of MOTA. In Table 1, we show that our detection optimization strategies lead to the significant decrease in the sum of FP and FNFootnote 1.

Table 1. Detection performance evaluation (on MOT16 train set)
Full size table

2 Appearance Feature

The distance between appearance features is used for computing the affinity value in data association. The affinity value based on the ideal appearance feature should be large for persons of the same identity, and be small for persons of different identities. In our implementation, we extract the appearance feature using a network which is similar to GoogLeNet [15]. The input size of our network is \(96 \times 96\), and the kernel size of pool5 layer is \(3 \times 3\) instead of \(7 \times 7\). The output layer is a fully connected layer which outputs the 128 dimensional feature. In the tracking phase, patches are first cropped according to the detection responses, and then resized to \(96 \times 96\) for feature extraction. The cosine distance is used for measuring the appearance affinity.

For training, we collect a dataset which contains nearly 119 K patches from 19835 identities. Such a dataset consists of multiple person re-id datasets, including PRW [18], Market-1501 [18], VIPeR [13] and CUHK03 [8]. We use the softmax and triplet loss jointly during training. The softmax loss guarantees the discriminative ability of the appearance feature, while the triplet loss ensures the cosine distance of the appearance features of the same identity to be small.

3 Online Tracker

We implement a simple online tracker, which uses Kalman filter [6] for motion prediction and Kuhn-Munkres algorithm [7] for data association. The overall tracking procedure is described in Algorithm 1.

figure a

In the following, we introduce the affinity matrix construction, data association method, threshold value setting and tracking quality metric.

Affinity Matrix Construction. To construct an affinity matrix for the Kuhn-Munkres algorithm, we calculate the affinity between tracklets and detections. We combine motion, shape and appearance affinity as the final affinity. Specifically, the appearance affinity is calculated based on the appearance feature described in Sect. 2. Details of the affinity calculation are given below:

$$\begin{aligned} \textit{aff}_{app}(trk_{i},det_{j})= & {} cosine(feat_{trk_{i}},feat_{det_{j}}) \end{aligned}$$
(1)
$$\begin{aligned} \textit{aff}_{mot}(trk_{i},det_{j})= & {} e^{-w_1 * ((\frac{X_{trk_{i}}-X_{det_{j}}}{W_{det_{j}}})^2+(\frac{Y_{trk_{i}}-Y_{det_{j}}}{H_{det_{j}}})^2)} \end{aligned}$$
(2)
$$\begin{aligned} \textit{aff}_{shp}(trk_{i},det_{j})= & {} e^{-w_2 * (\frac{|H_{trk_{i}}-H_{det_{j}}|}{H_{trk_{i}}+H_{det_{j}}} + \frac{|W_{trk_{i}}-W_{det_{j}}|}{W_{trk_{i}}+W_{det_{j}}})} \end{aligned}$$
(3)
$$\begin{aligned} \textit{affinity}(trk_{i},det_{j})= & {} \textit{aff}_{app}(trk_{i},det_{j}) * \textit{aff}_{mot}(trk_{i},det_{j}) * \textit{aff}_{shp}(trk_{i},det_{j}) \end{aligned}$$
(4)

\(\textit{aff}_{app}\), \(\textit{aff}_{mot}\) and \(\textit{aff}_{shp}\) indicate appearance, motion and shape affinity between the detection and tracklet, respectively. We combine these affinities with weights \(w_1\) and \(w_2\) as the final affinity.

Data Association. The tracklets and new detections are associated using the Kuhn-Munkres algorithm. Since the Kuhn-Munkres algorithm attempts to yield the global optimal result, it may fail when some detections are missing. To this end, we use a two-stage matching strategy, which divides \(T^{t-1}\) into high tracking quality set \(T^{t-1}_{high}\) and low quality set \(T^{t-1}_{low}\). The matching is first performed between \(T^{t-1}_{high}\) and D, and then performed between \((T^{t-1}_{high} - T^{t-1}_{success}) \cup T^{t-1}_{low}\) and \(D-D_{success}\).

Threshold Value Setting. On line 2 of Algorithm 1, we introduce \(\tau _{t}\) to divide \(T^{t-1}\) into high and low tracking quality set. The strategy is intuitive: we mark a tracklet with high flag whose tracking quality is higher than \(\tau _{t}\), other tracklets will be mark as low. On line 4, we use \(\tau _{a}\) to mark the association to be success or fail based on the affinity value. On line 7, we use \(\tau _{m}\) as a threshold to drop a tracklet which is lost for more than \(\tau _{m}\) frames.

Tracking Quality Metric. Tracking quality is designed to measure whether a object is tracking well or not. We use following formula to define tracking quality:

$$\begin{aligned} Quality(tracklet_i) = \frac{\sum _{k \in couples(tracklet_i)}^{}{\textit{affinity}_k}}{length(tracklet_i)}(1 - e^{-w_3 * \sqrt{length(tracklet_i)}}) \end{aligned}$$
(5)

where \(couples(tracklet_i)\), with the form \(\{trk_x,det_y\}\), is a set that contains every success association couple in history.

4 Offline Tracker

Our offine tracker an improved version of H\(^2\)T [16] while based on K-Dense Neighbors [11]. It is more robust and efficient than H\(^2\)T in handling the complex tracking scenarios. The overall procedure of the tracker is described in Algorithm 2.

figure b

Footnote 2

We make the following improvements over H\(^2\)T [16].

Appearance Representation. To construct the affinity matrix for the dense neighbors (DN) search, we need to calculate three affinities, i.e, appearance, motion, and smoothness affinity. Among these three affinities, the appearance affinity is the most important one and we use the CNN based feature described in Sect. 2, instead of the hand-crafted feature in [16].

Big Target. A scenario that H\(^2\)T [16] does not work well is the mixture of small and big targets. The reason is that the motion and smoothness affinities are unreliable for the big targets. Such unreliability is caused by the unsteady detection responses of the big targets. We introduce two thresholds, \(\tau _{s}\) and \(\tau _{r}\), regarding the object scale to deal with this challenge, i.e, \(\tau _{s}\) for preventing associating detection responses from very different scale, and \(\tau _{r}\) for determining whether to reduce the weights of motion and smoothness affinity. Specifically, if the ratio of the detection response scale and the target scale is less than \(\tau _{s}\), such a detection response will not be associated with the target. If the ratio of the detection response height and the image height is greater than \(\tau _{r}\), such a detection response will not be associated with the target. Both \(\tau _{s}\) and \(\tau _{r}\) are set as 0.5.

Algorithm Efficiency. H\(^2\)T is slow in handling the long tracking sequence where there exist plenty of targets. Among the steps in the algorithm, the step of DN search is the most time-consuming. To be more specific, the larger an affinity matrix, the longer time it will take to perform the DN search. Thus, we abandon the high-order information [16] when constructing the affinity matrix, which significantly reduces the matrix dimensions and improves the algorithm efficiency.

5 Evaluation

Our online and offline tracker are not learning based algorithm. We only tuning detection score threshold on train set and apply it to its similar scene from test set. For evaluation and submission, 0.1 is set for MOT16-03 and MOT16-04 due to high precision of detection result (03 and 04 are both surveillance scene, which is quite easy while our detector have been trained by self-collected surveillance dataset), and 0.3 is set for other sequences.

For both onlineFootnote 3 and offline tracker, we compare our detector with the official detector, and compare our feature with default CNN feature. The comparison results on MOT16 [12] train set are listed in Tables 2 and 3, respectively. Note that our detector leads to much better results in MT, ML, FP and FN, and our feature helps reduce both IDS and FM.

Table 2. Online tracker result on the train set
Full size table
Table 3. Offline tracker result on the train set
Full size table
Table 4. Comparison to the state-of-the-art methods on MOT16 rank list
Full size table

6 ECCV 2016 Challenge Results

Our ECCV 2016 Challenge results are listed in Table 4. Obviously, both our online and offline trackers outperform the state-of-the-art approaches by a large margin. Note that our offline tracker achieves the best performance in FN. However, its performance in FP is moderate, due to the interpolation module.

7 Conclusion

In this submission, we take many efforts to get high performance detection and deep learning based appearance feature. We show that they lead to the state-of-the-art multiple object tracking results, even with very simple online tracker. One observation is that with high performance detection and appearance feature, the state-of-the-art offline tracker does not have expected advantages over the much simpler online one. This observation is not reported in many current MOT papers, which often use detections that are not good enough. We make our detections and deep learning based re-ID features on MOT2016 publicly available, and hope that they can help more sophisticated trackers to get better performance.