Skip to main content
SpringerLink
Log in

Adaptive trajectory prediction without catastrophic forgetting

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Pedestrian trajectory prediction is a necessary component of autonomous driving technology. However, current methods face two troubles when utilized to the actual world, one is the distribution difference between training and testing environments, and the other is catastrophic forgetting. These two issues will lead to an inevitable drop in the overall performance of the model in real-world scenarios. To tackle these two issues, we propose a framework that consists of modules for domain adaptation and continual learning. Specifically, a pedestrian interplay modeling method based totally on pedestrian social habits is proposed. Moreover, we add a domain adaptation module to analyze the data distribution difference between the source domain and the target domain, so as to alleviate the domain difference problem. Finally, a continual learning module is introduced to retain the information which is learned to limit the change of model parameters to deal with the catastrophic forgetting. We design trajectory prediction experiments that conform to real-world activities, and the experimental results verify the superiority of our proposed model. To the best of our knowledge, we are the first work that attempts to apply domain adaptation and continual learning methods to remedy real-world trajectory prediction problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Luo Y, Cai P, Bera A, Hsu D, Lee WS, Manocha D (2018) Porca: Modeling and planning for autonomous driving among many pedestrians. IEEE Robot Autom Lett 3(4):3418–3425

    Article  Google Scholar 

  2. Raksincharoensak P, Hasegawa T, Nagai M (2016) Motion planning and control of autonomous driving intelligence system based on risk potential optimization framework. Int J Automot Eng 7(AVEC14):53–60

    Article  Google Scholar 

  3. Bisagno N, Zhang B, Conci N (2018) Group lstm: group trajectory prediction in crowded scenarios. In: Proceedings of the European Conference on Computer Vision (ECCV) workshops, pp 0–0

  4. Alahi A, Goel K, Ramanathan V, Robicquet A, Fei–Fei L, Savarese S (2016) Social lstm: human trajectory prediction in crowded spaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 961–971

  5. Hu Y, Chen S, Zhang Y, Gu X (2020) Collaborative motion prediction via neural motion message passing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6319–6328

  6. Zhu Y, Ren D, Xu Y, Qian D, Fan M, Li X, Xia H (2021) Simultaneous past and current social interaction-aware trajectory prediction for multiple intelligent agents in dynamic scenes. ACM Trans Intell Syst Technol (TIST) 13(1):1–16

    Google Scholar 

  7. Liang J, Jiang L, Niebles JC, Hauptmann AG, Fei–Fei L (2019) Peeking into the future: predicting future person activities and locations in videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 5725–5734

  8. Xu Y, Yang J, Du S (2020) Cf-lstm: cascaded feature-based long short-term networks for predicting pedestrian trajectory. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, pp 12541–12548

  9. Zhang P, Ouyang W, Zhang P, Xue J, Zheng N (2019) Sr-lstm: state refinement for lstm towards pedestrian trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 12085–12094

  10. Chen G, Li J, Lu J, Zhou J (2021) Human trajectory prediction via counterfactual analysis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 9824–9833

  11. Chen G, Li J, Zhou N, Ren L, Lu J (2021) Personalized trajectory prediction via distribution discrimination. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp 15580–15589

  12. Cheng H, Liao W, Tang X, Yang MY, Sester M, Rosenhahn B (2021) Exploring dynamic context for multi-path trajectory prediction. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp 12795–12801. IEEE

  13. Salzmann T, Ivanovic B, Chakravarty P, Pavone M (2020) Trajectron++: dynamically-feasible trajectory forecasting with heterogeneous data. In: European Conference on Computer Vision, pp 683–700. Springer

  14. Shafiee N, Padir T, Elhamifar E (2021) Introvert: human trajectory prediction via conditional 3d attention. In: Proceedings of the IEEE/cvf Conference on Computer Vision and Pattern recognition, pp 16815–16825

  15. Xu Y, Ren D, Li M, Chen Y, Fan M, Xia H (2021) Tra2tra: trajectory-to-trajectory prediction with a global social spatial-temporal attentive neural network. IEEE Robot Autom Lett 6(2):1574–1581

    Article  Google Scholar 

  16. Wu H, Nie J, He Z, Zhu Z, Gao M (2022) One-shot multiple object tracking in uav videos using task-specific fine-grained features. Remote Sens. https://doi.org/10.3390/rs14163853

    Article  Google Scholar 

  17. Wu H, He Z, Gao M (2023) Gcevt: learning global context embedding for vehicle tracking in unmanned aerial vehicle videos. IEEE Geosci Remote Sens Lett 20:1–5. https://doi.org/10.1109/LGRS.2022.3228527

    Article  Google Scholar 

  18. Xu Y, Wang L, Wang Y, Fu Y (2022) Adaptive trajectory prediction via transferable gnn. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 6520–6531

  19. Ma Q, Olshevsky A (2020) Adversarial crowdsourcing through robust rank-one matrix completion. Adv Neural Inf Process Syst 33:21841–21852

    Google Scholar 

  20. Ellis D, Sommerlade E, Reid I (2009) Modelling pedestrian trajectory patterns with gaussian processes. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pp 1229–1234. IEEE

  21. Tay MKC, Laugier C (2008) Modelling smooth paths using gaussian processes. In: Field and service robotics, pp 381–390. Springer

  22. Kitani KM, Ziebart BD, Bagnell JA, Hebert M (2012) Activity forecasting. In: European Conference on Computer Vision, pp 201–214. Springer

  23. Bisagno N, Saltori C, Zhang B, De Natale FG, Conci N (2021) Embedding group and obstacle information in lstm networks for human trajectory prediction in crowded scenes. Comput Vis Image Underst 203:103126

    Article  Google Scholar 

  24. Yu C, Ma X, Ren J, Zhao H, Yi S (2020) Spatio-temporal graph transformer networks for pedestrian trajectory prediction. In: European Conference on Computer Vision, pp 507–523. Springer

  25. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inf Process Syst, 30

  26. Liu Y, Zhang J, Fang L, Jiang Q, Zhou B (2021) Multimodal motion prediction with stacked transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7577–7586

  27. Bertugli A, Calderara S, Coscia P, Ballan L, Cucchiara R (2021) Ac-vrnn: attentive conditional-vrnn for multi-future trajectory prediction. Comput Vis Image Underst 210:103245

    Article  Google Scholar 

  28. Ni J, Qiu Q, Chellappa R (2013) Subspace interpolation via dictionary learning for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 692–699

  29. Zhuo J, Wang S, Zhang W, Huang Q (2017) Deep unsupervised convolutional domain adaptation. In: Proceedings of the 25th ACM International Conference on Multimedia, pp 261–269

  30. Wu M, Pan S, Zhou C, Chang X, Zhu X (2020) Unsupervised domain adaptive graph convolutional networks. In: Proceedings of The Web Conference 2020, pp 1457–1467

  31. De Lange M, Aljundi R, Masana M, Parisot S, Jia X, Leonardis A, Slabaugh G, Tuytelaars T (2019) Continual learning: a comparative study on how to defy forgetting in classification tasks. arXiv preprint arXiv:1909.08383

  32. Parisi GI, Kemker R, Part JL, Kanan C, Wermter S (2019) Continual lifelong learning with neural networks: a review. Neural Netw 113:54–71

    Article  Google Scholar 

  33. Lesort T, Lomonaco V, Stoian A, Maltoni D, Filliat D, Díaz-Rodríguez N (2020) Continual learning for robotics: definition, framework, learning strategies, opportunities and challenges. Inf fusion 58:52–68

    Article  Google Scholar 

  34. Michael McCloskey (1989) Catastrophic interference in connectionist networks: the sequential learning problem. Psychol Learn Motiv 24:109–165

    Article  Google Scholar 

  35. Huszár F (2017) On quadratic penalties in elastic weight consolidation. arXiv preprint arXiv:1712.03847

  36. Pellegrini S, Ess A, Schindler K, Van Gool L (2009) You’ll never walk alone: modeling social behavior for multi-target tracking. In: 2009 IEEE 12th International Conference on Computer Vision, pp 261–268. IEEE

  37. Lerner A, Chrysanthou Y, Lischinski D (2007) Crowds by example. In: Computer graphics forum, vol 26, pp 655–664. Wiley Online Library

  38. Mohamed A, Qian K, Elhoseiny M, Claudel C (2020) Social-stgcnn: a social spatio-temporal graph convolutional neural network for human trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 14424–14432

  39. Pang B, Zhao T, Xie X, Wu YN (2021) Trajectory prediction with latent belief energy-based model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 11814–11824

  40. Shi L, Wang L, Long C, Zhou S, Zhou M, Niu Z, Hua G (2021) Sgcn: sparse graph convolution network for pedestrian trajectory prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 8994–9003

  41. Xu P, Hayet J-B, Karamouzas I (2022) Socialvae: human trajectory prediction using timewise latents. Computer Vision - ECCV 2022. Springer, Cham, pp 511–528

    Chapter  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (NO.62176125, 61772272).

Author information

Author notes

  1. HuaiJiang Sun and Tian Xu contributed equally to this work.

Authors and Affiliations

  1. School of Computer Science and Engineering, Nanjing University of Science and Technology, XiaoLinWei, Nanjing, 20094, JiangSu, China

    ChunYu Zhi, HuaiJiang Sun & Tian Xu

Authors
  1. ChunYu Zhi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. HuaiJiang Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tian Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors have made contributions to the final draft of this article.

Corresponding author

Correspondence to ChunYu Zhi.

Ethics declarations

Conflict of interest

All work was completed in Nanjing University of Science and Technology, and there was no other object of interest competition.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhi, C., Sun, H. & Xu, T. Adaptive trajectory prediction without catastrophic forgetting. J Supercomput 79, 15579–15596 (2023). https://doi.org/10.1007/s11227-023-05241-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-023-05241-z

Keywords

Search

Navigation