Skip to main content
SpringerLink
Log in

Rank-1 Tensor Approximation for High-Order Association in Multi-target Tracking

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

High-order motion information is important in multi-target tracking (MTT) especially when dealing with large inter-target ambiguities. Such high-order information can be naturally modeled as a multi-dimensional assignment (MDA) problem, whose global solution is however intractable in general. In this paper, we propose a novel framework to the problem by reshaping MTT as a rank-1 tensor approximation problem (R1TA). We first show that MDA and R1TA share the same objective function and similar constraints. This discovery opens a door to use high-order tensor analysis for MTT and suggests the exploration of R1TA. In particular, we develop a tensor power iteration algorithm to effectively capture high-order motion information as well as appearance variation. The proposed algorithm is evaluated on a diverse set of datasets including aerial video sequences containing ariel borne dense highway scenes, top-view pedestrian trajectories, multiple similar objects, normal view pedestrians and vehicles. The effectiveness of the proposed algorithm is clearly demonstrated in these experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. The local tracking denotes the two-frame association. The terms association, tracking and matching are used interchangeably depending on context.

  2. The two metrics are not fully dependent on each other and the sum of \(P_c\) and \(P_w\) is not exactly 1, as shown in the experiments. This is attributed to the noisy false positives. In addition, the missing associations are not counted here.

  3. Finally, short trajectories with less than 5 instances are removed. The remaining trajectories are smoothed with spline fitting.

References

  • Andriluka, M., Roth, S., & Schiele, B. (2008). People-tracking-by-detection and people-detection-by-tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Bae, S.-H., & Yoon, K.-J. (2014). Robust online multi-object tracking based on tracklet confidence and online discriminative appearance learning. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Ban, Y., Ba, S., Alameda-Pineda, X., & Horaud, R. (2016). Tracking multiple persons based on a variational bayesian model. In Proceedings of European conference on computer vision.

  • Bar-Shalom, Y., & Fortmann, T. (1988). Tracking and data association. London: Academic Press.

    MATH  Google Scholar 

  • Benfold, B., & Reid, I. (2011). Stable multi-target tracking in real-time surveillance video. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Berclaz, J., Fleuret, F., Turetken, E., & Fua, P. (2011). Multiple object tracking using k-shortest paths optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(9), 1806–1819.

    Article  Google Scholar 

  • Bernardin, K., & Stiefelhagen, R. (2008). Evaluating multiple object tracking performance: The clear mot metrics. EURASIP Journal on Image and Video Processing, 2008, 246309.

    Article  Google Scholar 

  • Black, J., Ellis, T., & Rosin, P. (2002). Multi view image surveillance and tracking. In Workshop motion and video computing: Proceedings .

  • Blackman, S., & Popoli, R. (1999). Design and analysis of modern tracking systems. Norwood, MA: Artech House.

    MATH  Google Scholar 

  • Breitenstein, M. D., Reichlin, F., Leibe, B., Koller-Meier, E., & Van Gool, L. (2011). Online multi-person tracking-by-detection from a single, uncalibrated camera. IEEE Transactions on Pattern Analysis and Machine Interlligence, 33(9), 1820–1833.

    Article  Google Scholar 

  • Butt, A., & Collins, R. T. (2012). Multiple target tracking using frame triplets. In Proceedings of Asian conference computer vision.

  • Butt, A., & Collins, R. T. (2013). Multi-target tracking by lagrangian relaxation to min-cost network flow. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Chari, V., Julien, S. L., Laptev, I., & Sivic, J. (2015). On pairwise costs for network flow multi-object tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Collins, R. T. (2012). Multitarget data association with higher-order motion models. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Cox, I. (1993). A review of statistical data association techniques for motion correspondence. International Journal Computer Vision, 10(1), 53–66.

    Article  Google Scholar 

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Deb, S., Yeddanapudi, M., Pittipati, K., & Bar-Shalom, Y. (1997). A generalized S-D assignment algorithm for multi-sensor multi-target state estimation. IEEE Transactions Aerospace and Electronic Systems, 33(2), 523–538.

    Article  Google Scholar 

  • Dehghan, A., Modiri, S., & Shah, M. (2015). GMMCP tracker: Globally optimal generalized maximum multi clique problem for multiple object tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • De Lathauwer, L., De Moor, B., & Vandewalle, J. (2000). On the best rank-1 and rank-(\(R_1\), \(R_2\),., \(R_N\)) approximation of higher-order tensors. SIAM Journal on Matrix Analysis and Applications, 21(4), 1324–1342.

    Article  MathSciNet  MATH  Google Scholar 

  • Dicle, C., Sznaier, M., & Camps, O. (2013). The way they move: Tracking multiple targets with similar appearance. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Duchenne, O., Bach, F., Kweon, I., & Ponce, J. (2011). A tensor-based algorithm for high-order graph matching. IEEE Transactions on Pattern Analysis and Machine Interlligence, 33(12), 2383–2395.

    Article  Google Scholar 

  • Felzenszwalb, P., Girshick, R., McAllester, D., & Ramanan, D. (2010). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Interlligence, 32(9), 1627–1645.

    Article  Google Scholar 

  • Fortmann, T. E., Bar-Shalom, Y., & Scheffe, M. (1980). Multi-target tracking using joint probabilistic data association. In Proceedings of the IEEE conference on decision and control (Vo. 19, pp. 807–812).

  • Ge, W., Collins, R. T., & Ruback, R. (2012). Vision-based analysis of small groups in pedestrian crowds. IEEE Transactions on Pattern Analysis and Machine Interlligence, 34(5), 1003–1016.

    Article  Google Scholar 

  • Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving?. In Proceedings of the IEEE conference on Computer vision and pattern recognition: The KITTI vision benchmark suite.

  • Huang, C., Wu, B., & Nevatia, R. (2008). Robust object tracking by hierarchical association of detection responses. In Proceedings of European conference on computer vision.

  • Jiang, H., Fels, S., & Little, J. (2007). A linear programming approach for multiple object tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Kumar, K. C. A., & Vleeschouwer, C. (2013). Discriminative label propagation for multi-object tracking with sporadic appearance features. In Proceedings of IEEE conference on computer vision.

  • Le, N., Heili, A., & Odobez, J. (2016). Long-term time-sensitive costs for crf-based tracking by detection. In Proceedings of European conference on computer vision.

  • Leal-Taixé, L., Canton-Ferrer, C., & Schindler, K. (2016). Learning by tracking: Siamese CNN for robust target association. In Proceedings of IEEE conference on computer vision and pattern recognition workshops.

  • Leal-Taixé, L., Milan, A., Reid, I., Roth, S., & Schindler, K. (2015). MOTChallenge 2015: Towards a benchmark for multi-target tracking. arXiv:1504.01942 [cs].

  • Lenz, P., Geiger, A., & Urtasun, R. (2015). FollowMe: Efficient online min-cost flow tracking with bounded memory and computation. In Proceedings of IEEE international conference on computer vision.

  • Li, Y., Huang, C., & Nevatia, R. (2009). Learning to associate: Hybridboosted multi-target tracker for crowded scene. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Luo, W., Xing, J., Zhang, X., Zhao, X., & Kim, T.-K.(2014). Multiple object tracking: A review. arXiv:1409.7618.

  • Milan, A., Rezatofighi, S., Dick, A., Reid, I., & Schindler, K. (2017). Online multi-target tracking using recurrent neural networks. In Proceedings of AAAI.

  • Milan, A., Roth, S., & Schindler, K. (2014). Continuous energy minimization for multitarget tracking. IEEE Transactions on Pattern Analysis and Machine Interlligence, 36(1), 58–72.

    Article  Google Scholar 

  • Milan, A., Schindler, K., & Roth, S. (2016). Multi-target tracking by discrete-continuous energy minimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(10), 2054–2068.

    Article  Google Scholar 

  • Oh, S., Russell, S., & Sastry, S. (2009). Markov chain monte carlo data association for multi-target tracking. IEEE Transactions on Automatic Control, 54(3), 481–497.

    Article  MathSciNet  MATH  Google Scholar 

  • Okuma, K., Taleghani, A., Freitas, O. D., Little, J. J., & Lowe, D. G. (2004). A boosted particle filter: Multitarget detection and tracking. In Proceedings of European conference on computer vision.

  • Pirsiavash, H., Ramanan, D., & Fowlkes, C. (2011). Globally-optimal greedy algorithms for tracking a variable number of objects. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Poore, A. (1994). Multidimensional assignment formulation of data association problems arising from multitarget and multisensor tracking. Computational Optimization and Applications, 3(1), 27–57.

    Article  MathSciNet  MATH  Google Scholar 

  • Possegger, H., Mauthner, T., Roth, P. M., & Bischof, H. (2014). Occlusion geodesics for online multi-object tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Regalia, P., & Kofidis, E. (2000). The higher-order power method revisited: Convergence proofs and effective initialization. In Proceedings of IEEE international conference on acoustics speech and signal processing.

  • Reid, D. (1979). An algorithm for tracking multiple targets. IEEE Transactions on Automatic Control, 24(6), 843–854.

    Article  Google Scholar 

  • Ricardo, S., Fabio, P., & Andrea, C. (2016). Online multi-target tracking with strong and weak detections. In Computer vision: European conference.

  • Schulter, S., Vernaza, P., Choi, W., & Chandraker, M. (2017). Deep network flow for multi-object tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Shafique, K., Lee, M., & Haering, N. (2008). A rank constrained continuous formulation of multi-frame multi-target tracking problem. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Shafique, K., & Shah, M. (2005). A noniterative greedy algorithm for multiframe point correspondence. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(1), 51–65.

    Article  Google Scholar 

  • Shi, X., Ling, H., Blash, E., & Hu, W. (2012). Context-driven moving vehicle detection in wide area motion imagery. In Proceedings of IEEE International conference on pattern recognition.

  • Shi, X., Ling, H., Xing, J., & Hu, W. (2013). Multiple target tracking by rank-1 tensor approximation. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Sinkhorn, R. (1964). A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics, 35, 876–879.

    Article  MathSciNet  MATH  Google Scholar 

  • Son, J., Baek, M., Cho, M., & Han, B. Y. (2017). Multi-object tracking with quadruplet convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition.

  • Tang, S., Andres, B., Andriluka, M., & Schiele, B. (2015). Subgraph decomposition for multitarget tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Tang, S., Andres, B., Andriluka, M., & Schiele, B. (2016). Multi-person tracking by multicut and deep matching. In Computer vision: European conference.

  • The CLIF dataset. (2006). www.sdms.afrl.af.mil/index.php?collectionHrB=clifHrB.

  • Wang, B., Wang, L., Shuai, B., Zhen, Z., Liu, T., Chan, K. C., et al. (2016). Joint learning of convolutional neural networks and temporally constrained metrics for tracklet association. In Proceedings of IEEE conference on computer vision and pattern recognition workshops.

  • Wang, S., & Fowlkes, C. (2016). Learning optimal parameters for multi-target tracking with contextual interactions. In International journal of computer vision.

  • Wen, L., Li, W., Yan, J., Lei, Z., Yi, D., & Li, S. Z. (2014). Multiple target tracking based on undirected hierarchical relation hypergraph. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Yang, B., & Nevatia, R. (2012a). Multi-target tracking by online learning of non-linear motion patterns and robust appearance models. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Yang, B., & Nevatia, R. (2012b). An online learned crf model for multi-target tracking. In Proceedings of IEEE conference on computer vision and pattern recognition.

  • Yu, Q., & Medioni, G. (2009). Multiple target tracking by spatiotemporal monte carlo markov chain data association. IEEE Transactions on Automatic Control, 31(12), 2196–2210.

    Google Scholar 

  • Zamir, A., Dehghan, A., & Shah, M. (2012). GMCP-tracker: Global multi-object tracking using generalized minimum clique graphs. In Proceedings of European conference on computer vision.

  • Zhang, L., Li, Y., & Nevatia, R. (2008). Global data association for multi-object tracking using network flows. In Proceedings of IEEE conference on computer vision and pattern recognition.

Download references

Acknowledgements

We would like to express our sincere appreciation to Professor Steve Maybank for his valuable suggestion and careful revision on the wordings and grammar in the paper. This work is supported by Beijing Natural Science Foundation (Grant No. L172051), the Natural Science Foundation of China (Grant Nos. 61502492, 61751212, 61721004), the NSFC-general technology collaborative Fund for basic research (Grant No. U1636218), the Key Research Program of Frontier Sciences, CAS, Grant No. QYZDJ-SSW-JSC040, and the CAS External cooperation key project. H. Ling was supported in part by US NSF (Grant Nos. 1814745, 1407156, and 1350521).

Author information

Authors and Affiliations

  1. National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Xinchu Shi & Junliang Xing

  2. CAS Center for Excellence in Brain Science and Intelligence Technology, National Laboratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Weiming Hu

  3. University of Chinese Academy of Sciences, Beijing, China

    Weiming Hu

  4. Department of Computer and Information Sciences, Temple University, Philadelphia, USA

    Haibin Ling, Yu Pang & Peng Chu

Authors
  1. Xinchu Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haibin Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu Pang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weiming Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Peng Chu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Junliang Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weiming Hu.

Additional information

Communicated by Stefan Roth.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, X., Ling, H., Pang, Y. et al. Rank-1 Tensor Approximation for High-Order Association in Multi-target Tracking. Int J Comput Vis 127, 1063–1083 (2019). https://doi.org/10.1007/s11263-018-01147-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-018-01147-z

Keywords

Search

Navigation