AutoTrajectory: Label-Free Trajectory Extraction and Prediction from Videos Using Dynamic Points

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12358)


Current methods for trajectory prediction operate in supervised manners, and therefore require vast quantities of corresponding ground truth data for training. In this paper, we present a novel, label-free algorithm, AutoTrajectory, for trajectory extraction and prediction to use raw videos directly. To better capture the moving objects in videos, we introduce dynamic points. We use them to model dynamic motions by using a forward-backward extractor to keep temporal consistency and using image reconstruction to keep spatial consistency in an unsupervised manner. Then we aggregate dynamic points to instance points, which stand for moving objects such as pedestrians in videos. Finally, we extract trajectories by matching instance points for prediction training. To the best of our knowledge, our method is the first to achieve unsupervised learning of trajectory extraction and prediction. We evaluate the performance on well-known trajectory datasets and show that our method is effective for real-world videos and can use raw videos to further improve the performance of existing models.

Supplementary material

504454_1_En_38_MOESM1_ESM.pdf (7.5 mb)
Supplementary material 1 (pdf 7653 KB)


  1. 1.
    Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Li, F.F., Savarese, S.: Social LSTM: Human trajectory prediction in crowded spaces. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961–971 (2016)Google Scholar
  2. 2.
    Başar, T.: A new approach to linear filtering and prediction problems (2001)Google Scholar
  3. 3.
    Bergmann, P., Meinhardt, T., Leal-Taixe, L.: Tracking without bells and whistles. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 941–951 (2019)Google Scholar
  4. 4.
    Bertinetto, L., Valmadre, J., Henriques, J.F., Vedaldi, A., Torr, P.H.S.: Fully-convolutional siamese networks for object tracking. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 850–865. Springer, Cham (2016). Scholar
  5. 5.
    Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B.: Simple online and realtime tracking. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464–3468. IEEE (2016)Google Scholar
  6. 6.
    Cao, C., et al.: Look and think twice: Capturing top-down visual attention with feedback convolutional neural networks. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 2956–2964 (2015)Google Scholar
  7. 7.
    Chai, Y., Sapp, B., Bansal, M., Anguelov, D.: Multipath: multiple probabilistic anchor trajectory hypotheses for behavior prediction. ArXiv abs/1910.05449 (2019)Google Scholar
  8. 8.
    Chandra, R., Bhattacharya, U., Roncal, C., Bera, A., Manocha, D.: Robusttp: End-to-end trajectory prediction for heterogeneous road-agents in dense traffic with noisy sensor inputs. In: CSCS 2019 (2019)Google Scholar
  9. 9.
    Chandra, R., et al.: Forecasting trajectory and behavior of road-agents using spectral clustering in graph-LSTMS. ArXiv abs/1912.01118 (2019)Google Scholar
  10. 10.
    Chang, M.F., et al.: Argoverse: 3D tracking and forecasting with rich maps. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8740–8749 (2019)Google Scholar
  11. 11.
    Crawford, E., Pineau, J.: Exploiting spatial invariance for scalable unsupervised object tracking. ArXiv abs/1911.09033 (2019)Google Scholar
  12. 12.
    Eslami, S.M.A., et al.: Attend, infer, repeat: Fast scene understanding with generative models. In: NIPS (2016)Google Scholar
  13. 13.
    Fang, K., Xiang, Y., Li, X., Savarese, S.: Recurrent autoregressive networks for online multi-object tracking. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 466–475. IEEE (2018)Google Scholar
  14. 14.
    Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Bigun, J., Gustavsson, T. (eds.) SCIA 2003. LNCS, vol. 2749, pp. 363–370. Springer, Heidelberg (2003). Scholar
  15. 15.
    Feichtenhofer, C., Pinz, A., Zisserman, A.: Detect to track and track to detect. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3038–3046 (2017)Google Scholar
  16. 16.
    Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: the kitti dataset. Int. J. Rob. Res. 32, 1231–1237 (2013)CrossRefGoogle Scholar
  17. 17.
    Gupta, A., Johnson, J.E., Fei-Fei, L., Savarese, S., Alahi, A.: Social GAN: Socially acceptable trajectories with generative adversarial networks. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2255–2264 (2018)Google Scholar
  18. 18.
    Hall, M.A.: Correlation-based feature selection for machine learning (2003)Google Scholar
  19. 19.
    He, Z., Li, J., Liu, D., He, H., Barber, D.: Tracking by animation: unsupervised learning of multi-object attentive trackers. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1318–1327 (2018)Google Scholar
  20. 20.
    Jakab, T., Gupta, A., Bilen, H., Vedaldi, A.: Conditional image generation for learning the structure of visual objects. ArXiv abs/1806.07823 (2018)Google Scholar
  21. 21.
    Kim, Y., Nam, S., Cho, I.S., Kim, S.J.: Unsupervised keypoint learning for guiding class-conditional video prediction. ArXiv abs/1910.02027 (2019)Google Scholar
  22. 22.
    Kosiorek, A.R., Kim, H., Posner, I., Teh, Y.W.: Sequential attend, infer, repeat: generative modelling of moving objects. In: NeurIPS (2018)Google Scholar
  23. 23.
    Leal-Taixé, L., Fenzi, M., Kuznetsova, A., Rosenhahn, B., Savarese, S.: Learning an image-based motion context for multiple people tracking. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3542–3549 (2014)Google Scholar
  24. 24.
    Lee, N., et al.: Desire: Distant future prediction in dynamic scenes with interacting agents. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2165–2174 (2017)Google Scholar
  25. 25.
    Lefevre, S., Laugier, C., Guzman, J.I.: Exploiting map information for driver intention estimation at road intersections. In: 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 583–588 (2011)Google Scholar
  26. 26.
    Lerner, A., Chrysanthou, Y., Lischinski, D.: Crowds by example. Comput. Graph Forum 26, 655–664 (2007)CrossRefGoogle Scholar
  27. 27.
    Luo, Z., Peng, B., Huang, D.A., Alahi, A., Fei-Fei, L.: Unsupervised learning of long-term motion dynamics for videos. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7101–7110 (2017)Google Scholar
  28. 28.
    Ma, Y., Manocha, D., Wang, W.: Autorvo: Local navigation with dynamic constraints in dense heterogeneous traffic. arXiv preprint arXiv:1804.02915 (2018)
  29. 29.
    Ma, Y., Zhu, X., Zhang, S., Yang, R., Wang, W., Manocha, D.: Trafficpredict: trajectory prediction for heterogeneous traffic-agents. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 6120–6127 (2019)Google Scholar
  30. 30.
    Minderer, M., Sun, C., Villegas, R., Cole, F., Murphy, K., Lee, H.: Unsupervised learning of object structure and dynamics from videos. ArXiv abs/1906.07889 (2019)Google Scholar
  31. 31.
    Palaz, D.: Towards end-to-end speech recognition (2016)Google Scholar
  32. 32.
    Pan, J., Sun, H., cheng Xu, K., Jiang, Y., Xiao, X., Hu, J., Miao, J.: Lane attention: Predicting vehicles’ moving trajectories by learning their attention over lanes. ArXiv abs/1909.13377 (2019)Google Scholar
  33. 33.
    Park, S., Kim, B., Kang, C.M., Chung, C.C., Choi, J.W.: Sequence-to-sequence prediction of vehicle trajectory via LSTM encoder-decoder architecture. In: 2018 IEEE Intelligent Vehicles Symposium (IV), pp. 1672–1678 (2018)Google Scholar
  34. 34.
    Pellegrini, S., Ess, A., Van Gool, L.: Improving data association by joint modeling of pedestrian trajectories and groupings. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6311, pp. 452–465. Springer, Heidelberg (2010). Scholar
  35. 35.
    Pellegrini, S., Ess, A., Schindler, K., Gool, L.V.: You’ll never walk alone: modeling social behavior for multi-target tracking. In: 2009 IEEE 12th ICCV, pp. 261–268 (2009)Google Scholar
  36. 36.
    Piekniewski, F., Laurent, P.A., Petre, C., Richert, M., Fisher, D., Hylton, T.: Unsupervised learning from continuous video in a scalable predictive recurrent network. ArXiv abs/1607.06854 (2016)Google Scholar
  37. 37.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  38. 38.
    Rhinehart, N., McAllister, R., Kitani, K.M., Levine, S.: Precog: prediction conditioned on goals in visual multi-agent settings. ArXiv abs/1905.01296 (2019)Google Scholar
  39. 39.
    Sadeghian, A., Kosaraju, V., Sadeghian, A., Hirose, N., Savarese, S.: Sophie: An attentive GAN for predicting paths compliant to social and physical constraints. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1349–1358 (2018)Google Scholar
  40. 40.
    Sharma, S., Ansari, J.A., Murthy, J.K., Krishna, K.M.: Beyond pixels: Leveraging geometry and shape cues for online multi-object tracking. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3508–3515. IEEE (2018)Google Scholar
  41. 41.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)Google Scholar
  42. 42.
    Tang, S., Andriluka, M., Andres, B., Schiele, B.: Multiple people tracking by lifted multicut and person re-identification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3539–3548 (2017)Google Scholar
  43. 43.
    Tang, Y., Salakhutdinov, R.: Multiple futures prediction. In: NeurIPS (2019)Google Scholar
  44. 44.
    Wang, M., Shi, D., Guan, N., Zhang, T., Wang, L., Li, R.: Unsupervised pedestrian trajectory prediction with graph neural networks. In: 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pp. 832–839 (2019)Google Scholar
  45. 45.
    Wang, N., Song, Y., Ma, C., Zhou, W., Liu, W., Li, H.: Unsupervised deep tracking. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1308–1317 (2019)Google Scholar
  46. 46.
    Xu, J., Cao, Y., Zhang, Z., Hu, H.: Spatial-temporal relation networks for multi-object tracking. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3988–3998 (2019)Google Scholar
  47. 47.
    Zhang, S., et al.: Tracking persons-of-interest via unsupervised representation adaptation. Int. J. Comput. Vis. 128, 120–96 (2017)Google Scholar
  48. 48.
    Zhao, T., et al.: Multi-agent tensor fusion for contextual trajectory prediction. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12118–12126 (2019)Google Scholar
  49. 49.
    Zhu, J., Yang, H., Liu, N., Kim, M., Zhang, W., Yang, M.H.: Online multi-object tracking with dual matching attention networks. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 366–382 (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Hong Kong Baptist UniversityHong KongChina
  2. 2.Chinese University of Hong KongHong KongChina
  3. 3.InceptioFremontUSA
  4. 4.University of Maryland at College ParkCollege ParkUSA

Personalised recommendations